Can Osiris properly detect an endpoint on a related Ingress for my app ?

[cdmmultimedia]

I'm new to Go, and very new to Kubernetes.

As I can gather from the source code, once the activator detects a request on a known hostname, it :

1. Keeps the connection alive until the pod has been successfully started and is reachable
2. Retrieves the app from the list of apps on which Osiris is enabled
3. Sets the minReplica for the deployment back to 1
4. Waits for the app to be reachable, using k8s library's SharedIndexInformer that watches a list of endpoints that match a specific selector
5. Once reachable, syncs the hijacker and the newly started app endpoints

I have a problem with my config that throws the following error messages:

E0625 19:28:51.850358       1 deployment_activation.go:71] Activation of deployment hello-osiris in namespace default timed out
E0625 19:28:51.850452       1 proxy.go:97] Error executing start proxy callback for host "apps.contoso.io": Timed out waiting for activation of deployment hello-osiris in namespace default: %!s(<nil>)

My app's ingress name was previously suffixed with -customer-ingress. I removed that, thinking that it would help (if my service and my ingress had the same metadata.name) but no luck with that, the scale from zero still times out.

Here is the config of my app, with the added Ingress from the example in this repo:

apiVersion: v1
kind: Service
metadata:
  name: hello-osiris
  labels:
    app: hello-osiris
  annotations:
    osiris.deislabs.io/enabled: "true"
    osiris.deislabs.io/deployment: hello-osiris
    osiris.deislabs.io/ingressHostname: apps.contoso.io
spec:
  type: ClusterIP
  ports:
  - name: http1
    port: 8080
    targetPort: 80
  - name: http2
    port: 8080
    targetPort: 8080
  selector:
    app: hello-osiris
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-osiris
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: apps.contoso.io
    http:
      paths:
      - path: /hello-osiris
        backend:
          serviceName: hello-osiris
          servicePort: http1

---
apiVersion: v1
kind: Secret
metadata:
  name: hello-osiris-cert
  labels:
    app: hello-osiris
data:
  server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQyRENDQXNBQ0NRQy9HeC92dExTTlZEQU5CZ2txaGtpRzl3MEJBUXNGQURDQnJURUxNQWtHQTFVRUJoTUMKVlZNeEV6QVJCZ05WQkFnTUNsZGhjMmhwYm1kMGIyNHhFREFPQmdOVkJBY01CMUpsWkcxdmJtUXhFakFRQmdOVgpCQW9NQ1VSbGFYTWdUR0ZpY3pFVU1CSUdBMVVFQ3d3TFJXNW5hVzVsWlhKcGJtY3hJVEFmQmdOVkJBTU1HR2hsCmJHeHZMVzl6YVhKcGN5NWtaV2x6YkdGaWN5NXBiekVxTUNnR0NTcUdTSWIzRFFFSkFSWWJhMlZ1ZEM1eVlXNWoKYjNWeWRFQnRhV055YjNOdlpuUXVZMjl0TUI0WERURTVNREl5TXpBd01UazBNVm9YRFRJd01ESXlNekF3TVRrMApNVm93Z2EweEN6QUpCZ05WQkFZVEFsVlRNUk13RVFZRFZRUUlEQXBYWVhOb2FXNW5kRzl1TVJBd0RnWURWUVFICkRBZFNaV1J0YjI1a01SSXdFQVlEVlFRS0RBbEVaV2x6SUV4aFluTXhGREFTQmdOVkJBc01DMFZ1WjJsdVpXVnkKYVc1bk1TRXdId1lEVlFRRERCaG9aV3hzYnkxdmMybHlhWE11WkdWcGMyeGhZbk11YVc4eEtqQW9CZ2txaGtpRwo5dzBCQ1FFV0cydGxiblF1Y21GdVkyOTFjblJBYldsamNtOXpiMlowTG1OdmJUQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFMZEJZTFlwT2NaSUdEWWs2Qzl5T1hreE8wcUZQbVkzNGVWanBrc0oKZjZqaUVEUDFWZlBoWXV6TnRwMUY0ZWhlR1h3WEU0cmt4ZjQ5bExtcmU4L3g4NUh6RHNKK1NNdnlaZ09XZjZMcgpoNE53aVBKcmNjcDhGTVlXMmtJenJiVWZFS0wxYUZ1VCtkRXc3NkgxRlhPUmsvS0Y0V3JMYXhkRlBPbDhLMWVPCm1NazFtSkU3NTNYZzVYd2FVVUVHZ2tGbUZkZHJhQ2N3Y1U0QmtnbXRObTdFTExJQ2Nnb3MzNHVmR21ndmN2ZkwKSWhuenZxNmxCNDM4a0hyaG16OG11WFVwYjhQa1k2NmtxRGpxTk53YlBsLzJrMjYvVTg1RUVlazI0YnowMzlBZApsUnpKUTFndUhacmxKOTBQckl0aFJzM3NNZmdzWGtIaGZDR0J5dlVXYWZubUZPY0NBd0VBQVRBTkJna3Foa2lHCjl3MEJBUXNGQUFPQ0FRRUFzckltVURVK0E2YWNHVGloK3N5c2lpQWVWYWtsL0FNcytvWXJIa1NPK2NrOXVNcFUKMUw3dUtrNDZBSGZ0dkplbXJqaFBObHJpSmZOSEh5bEZ0YlpkTjRqM2RmL3p1L1ExbVRMa2dLUXJWZkl6ZFF2eApVUlEyOXB2ZWFBdFJyL0x6VXZINllWRE5lTk9wWXk3ZEJJT1ZqcGpjZFJ5amRHZE1xejBLbEhvUGlnbEdiUEFWCldzdXBmbWI3Nzd1Q3ZtUGRHc1lwb2wvTE9jOU44ZUE0VUdPNk9sWmtWU0NGOTJjSk9oaCtyd1c0cktTOXZFTTcKRzJmaXZVVDZJWCtFamdGQzB0ZytLMkNSSjRoTElnNTFSc0lmdEllNk01MFBpOCsvc3d6andYV2ZuQmpkUWkyegpudC9XYmR3THA2Q2pMR01UdVNrZmVGam00Z2QzM2cyMmRSY09RZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  server.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdDBGZ3RpazV4a2dZTmlUb0wzSTVlVEU3U29VK1pqZmg1V09tU3dsL3FPSVFNL1ZWCjgrRmk3TTIyblVYaDZGNFpmQmNUaXVURi9qMlV1YXQ3ei9IemtmTU93bjVJeS9KbUE1Wi9vdXVIZzNDSThtdHgKeW53VXhoYmFRak90dFI4UW92Vm9XNVA1MFREdm9mVVZjNUdUOG9YaGFzdHJGMFU4Nlh3clY0Nll5VFdZa1R2bgpkZURsZkJwUlFRYUNRV1lWMTJ0b0p6QnhUZ0dTQ2EwMmJzUXNzZ0p5Q2l6Zmk1OGFhQzl5OThzaUdmTytycVVICmpmeVFldUdiUHlhNWRTbHZ3K1JqcnFTb09PbzAzQnMrWC9hVGJyOVR6a1FSNlRiaHZQVGYwQjJWSE1sRFdDNGQKbXVVbjNRK3NpMkZHemV3eCtDeGVRZUY4SVlISzlSWnArZVlVNXdJREFRQUJBb0lCQUdVS2lDK0lSWkc5V0pRcAovMWVCekl5MUIzTU1TcDZEdTJzR2FiOC82b0tNdXRCYk9sd3c3cUdRdjFxeUdHQk4yaEZnaStidVF2anVyVjArClh4TUYzZjJnSFloQnB4UEVnRmtFRnpZV1ZXNjBrdDNQUGp1ZDlMcFFDV0d0S3Q4TjFOZDFKbWd3Qy9NNjN6WFcKYzFCNGVUR2tmZWlyWmsyN1lGMkFtRWs3bDZTQWw4SXVSbHNvUnpSVjBjTGxDQ1ZqeGZrWFFaTko3d0tnQzk2RAovSXppTVhzZ2h3MEd5NFg2L1FadjZnSGFIdHN0d1lITkFyeUF4eExYMVNQOTJhc1BEdWFXYUFMbEJzdGRSc2RtCkpIdU43dWtuMGFHZEMyeVAwQWlDS2prM1NjUnpVdHhyMlBqb3d4WGYwd3BDYUFYN2xVenk2RWpId29VTmg1VEcKenlhRTRoa0NnWUVBN2o2T21lSzdqN1E4SlMwR2NtTDVxbmMzdkRkUkdSdFlhclk2UzR1Qko2UmVnMkJGSFZ4MApPU2lrV1E2Z2dTTkJvMnEwR3pvcVpYZ3FSdGVtYjh1WUhUR25ad2NlRGd4dVhMcXg2cHRKWTlvajJJcDNXQXFaCjdWei95WGl6aWpIWmJZTW5IQmUyODFEMmFUQzlJRXpwdGg2NmlTZ2VYa1pWOFhNL1BCQjk1ZVVDZ1lFQXhPbXcKM04wY2J6QkxPeUN5Zkdhc2phRnAxNml3djFMYVB2b3ZSaG9ubDhkczVmVy9GdC9kMTY1YVorOCtnTU1rN1kxTwpFdkxmVjFkZXVUS0Y0VDV4dC90ZTVYcU5ocXJRbWt5RmZUY0tFem8rODZnUnpxTWRqSXh1eDhjN3FRWjg3Y2xxCitSNTBUbzZraGt1YXlpc1hWT05CV2VxekFSWk9QWmh2L2xSaUl0c0NnWUVBb0xZZ1dkeGg2OW1JTFFmSGJvZ24KcFA5UTRLMXNEb1NzeXlkc0FhUDBsdnBCSzF4WW95ckgxL3I3aW52Y2QrQ0JtYXdVSEwzSzliSHV5dVVVQ0J3TgoyN3V3RWtieDFrWTZlR0VVUFk5TkhZZDhZTWxmSWt2Y2RBc2xIUkpJQXJRSDJPRDlFKzFIWTdFODE4Nmg5ZFVNClh1Y3hxKzRkTmprNkptczR2OXJjSXFVQ2dZQUJrTDRJTTNYTGFIM2duWFR0eWo4cTdSS1RWVkw2WW1VN3hPOWwKUmtYMFRmQ09yM0p5Y3hzbllNcDFNeEN6STFvQ3pYSEdjc25WdnVzUTI5YjJvSEYwL2ZtV0ozQkNsczhMdXZvQQpzZFJSck0vZFRnTytPY3U5VjB4MktCNVFUSzNua2dkWXJhWk5EWk0vUWhDYjlOVzlwZ1RaK3lTcktJczhzQjZMCnpnM3Rxd0tCZ1FDTzhrNkVZRVVoZW5DMWNldSs0ejdEWmZrL01CTWlJci9ob1NySllaSmVOWldBSDJOd2p5M0cKUlhYWTZzdVRZRFRXVUJYWTZZMDl2STdOQzhmRk11ZmhyM28zaThMMVNWMlNCQ0VyMlV4T3RHWnN4TEVMMnhUQwpCbFIrMEF2MnUzSFBKRTBiV3ptVGh3U1RlQ2h0Z3pZQ2tIUlZlNlJMZVhET0w3SkFnQWNyM1E9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-osiris
  labels:
    app: hello-osiris
  annotations:
    osiris.deislabs.io/enabled: "true"
    osiris.deislabs.io/minReplicas: "1"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-osiris
  template:
    metadata:
      labels:
        app: hello-osiris
    spec:
      containers:
      - name: hello-osiris
        image: krancour/hello-osiris:v0.1.0
        args:
        - --https-cert
        - /hello-osiris/cert/server.crt
        - --https-key
        - /hello-osiris/cert/server.key
        ports:
        - containerPort: 8080
        - containerPort: 8081
        - containerPort: 8082
        - containerPort: 4430
        volumeMounts:
        - name: cert
          mountPath: /hello-osiris/cert
          readOnly: true
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
      volumes:
      - name: cert
        secret:
          secretName: hello-osiris-cert

I also foresee a problem with the path in my Ingress spec. As far as I can tell, Osiris has no understanding of the Ingress as it is. It thus does not use the path of my app.

I have several apps, under the same domain name, with each instance accessible under a specific path. Is it something that Osiris supports or plan to support in the near future?

Thank you so much for you work. We like the simplicity of Osiris a lot. If our use case needs some work on the source code, we would be more than happy to contribute to Osiris.

[cdmmultimedia]

After some more research, it appears (and again, I might be wrong, because of my limited knowledge of how Osiris and Kubernetes work):

1. When a pod is hijacked by Osiris, Osiris adds it into a map, where the IP address of the endpoint is the key.
2. As I'm using a dynamic ClusterIP, this IP will change after a pod has been terminated and is recreated.
3. Osiris does not see the new endpoint coming up, because it is looking for endpoints on specific IP, but this IP no longer exists, so it times out while waiting for the pod to be accessible.

If this is the case, then I guess it would be useful to allow dynamic IPs to be used, with for example the proxy-hijacker, notifying the activator about the IP when the pod is up.

I'd very much like to work on this, if it is of any use to the community, as it would allow Osiris to conform to our use case.

Please let me know if my understanding is right, and if I can help in anyway to extend this project, or if by changing my config, I could resolve this problem I have.

[krancour]

Sorry for the delay in response. I'm currently out on paternity leave.

As I can gather from the source code, once the activator detects a request on a known hostname, it :

1. Keeps the connection alive until the pod has been successfully started and is reachable
2. Retrieves the app from the list of apps on which Osiris is enabled
3. Sets the minReplica for the deployment back to 1
4. Waits for the app to be reachable, using k8s library's SharedIndexInformer that watches a list of endpoints that match a specific selector

Pretty accurate.

Not as accurate:

5. Once reachable, syncs the hijacker and the newly started app endpoints

The endpoints hijacker has a very specific and very limited role. When Osiris-enabled services are added/updated, the endpoints hijacker (which is a mutating webhook) mutates the service into a selector-less service, which allows our endpoints controller to take over management of endpoints for the service-- a function that is otherwise accounted for by Kubernetes' own endpoints controller. The hijacker doesn't do anything else.

With this in mind, there are a few other statements you made about the hijacker or hijacking that might not be accurate. For instance:

1. When a pod is hijacked by Osiris, Osiris adds it into a map, where the IP address of the endpoint is the key.

Osiris doesn't hijack pods. By and large, Osiris doesn't interfere with the normal function of any Kubernetes constructs you're familiar with (service endpoint hijacking being the only real exception.)

Re: paths in your ingress-- you are right that that is something that isn't accounted for. The activator doesn't include anything other than hostnames and/or IPs as indexes in its map deployments that may require reactivation.

I'll look at the rest of this issue in greater detail as soon as I am able to. In the meantime, maybe some of the information above will help you dig deeper.

[cdmmultimedia]

Hi Kent,

No need to apologize here! Thank you so much for taking the time to give me some information about my question.
I'll look more into this on the basis of the information you provided.

Thanks again!