Container exits with exception on openshift

[philipp1992]

using cic 1.23.10 installed with helm on openshift 4.9 with openshift OVN

--set nsIP=10.0.38.6,license.accept=yes,adcCredentialSecret=nslogin,openshift=true,exporter.required=true,nsPort=80,nsProtocol=HTTP,clusterName=c4,nodeWatch=true,ipam=true,disableOpenshiftRoutes=true,crds.install=true -n citrix-system,image=quay.io/citrix/citrix-k8s-ingress-controller:1.23.10

2022-04-06 11:56:44,363  - INFO - [config_dispatcher.py:__dispatch_config_pack:352] (Dispatcher) Processing of ConfigPack 'NetScaler Configuration_diff_delete+__synchronize_config___diff_add' is successful
2022-04-06 11:56:44,363  - INFO - [config_dispatcher.py:_synchronize_config:221] (Dispatcher) Config Synchronization ended
2022-04-06 11:56:44,730  - ERROR - [kubernetes.py:main_thread:721] (MainThread) Main thread exits on exception Traceback:
Traceback (most recent call last):
  File "/usr/src/triton/kubernetes/kubernetes.py", line 710, in main_thread
    self.event_handler(event)
  File "/usr/src/triton/kubernetes/kubernetes.py", line 1078, in event_handler
    elif (event['object']['kind'] == 'Node' or event['object']['kind'].upper() in [self.cni.cni_crd.upper(), self.cni.cni_crd.upper()[:-1]]):
AttributeError: 'NoneType' object has no attribute 'upper'
2022-04-06 11:56:44,732  - CRITICAL - [kubernetes.py:main_thread:722] (MainThread)
Exception Occured exiting the CIC

[apoorvak-citrix]

@philipp1992

1. what was the exact set command used? I think the above command shared would have resulted in some error due to the -n namespace in between the value list ?
2. Was the helm charts deployed directly or were there any modifications done say in RBAC?
3. Also can you share the complete CIC logs?

[philipp1992]

yeah that command was wrong but i corrected it.
with openshift SDN its working, openshift OVN failing

[apoorvak-citrix]

@philipp1992
will you be able to share the following details:

1. complete CIC logs.
2. The clusterrole created by the helm for this deployment, it should be prefixed by the name provided during helm install?

[philipp1992]

nsIP: 10.0.38.5
license:
  accept: yes
adcCredentialSecret: nslogin
openshift: true
exporter:
  required: true
nsPort: 80
nsProtocol: HTTP
clusterName: c5
nodeWatch: true
ipam: true
nsSNIPS: 10.0.38.8
disableOpenshiftRoutes: true
crds:
[cic.txt](https://github.com/citrix/citrix-k8s-ingress-controller/files/8441433/cic.txt)

  install: true

logs attached

kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2022-04-06T12:30:49Z"
  name: cic-citrix-ingress-controller-config-networks
  resourceVersion: "83802955"
  uid: c86c6db0-fe4d-4ccc-813d-bb50cd89f99b
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cic-citrix-ingress-controller-config-networks
subjects:
- kind: ServiceAccount
  name: citrix-ingress-controller
  namespace: citrix-system

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-04-06T11:26:54Z"
  name: cic-citrix-ingress-controller-config-networks
  resourceVersion: "83715692"
  uid: a1048f77-0f02-4000-9014-3e44b94e5bc7
rules:
- apiGroups:
  - config.openshift.io
  resources:
  - networks
  verbs:
  - get
  - list

[apoorvak-citrix]

@philipp1992 can you share the complete ClusterRole ?

[philipp1992]

kind: ClusterRole
metadata:
  annotations:
    meta.helm.sh/release-name: citrix-ingress-controller
    meta.helm.sh/release-namespace: citrix-system
  creationTimestamp: "2022-04-06T12:23:56Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: citrix-ingress-controller
  resourceVersion: "83793613"
  uid: 9304779d-1556-4eb1-898d-395d68957b4b
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - pods
  - secrets
  - routes
  - tokenreviews
  - subjectaccessreviews
  - nodes
  - namespaces
  - configmaps
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - services/status
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - citrix.com
  resources:
  - rewritepolicies
  - continuousdeployments
  - authpolicies
  - ratelimits
  - listeners
  - httproutes
  - wafs
  - apigatewaypolicies
  - bots
  - corspolicies
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - patch
- apiGroups:
  - citrix.com
  resources:
  - rewritepolicies/status
  - continuousdeployments/status
  - authpolicies/status
  - ratelimits/status
  - listeners/status
  - httproutes/status
  - wafs/status
  - apigatewaypolicies/status
  - bots/status
  - corspolicies/status
  verbs:
  - patch
- apiGroups:
  - citrix.com
  resources:
  - vips
  verbs:
  - get
  - list
  - watch
  - create
  - delete
- apiGroups:
  - crd.projectcalico.org
  resources:
  - ipamblocks
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - route.openshift.io
  resources:
  - routes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - network.openshift.io
  resources:
  - hostsubnets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - config.openshift.io
  resources:
  - networks
  verbs:
  - get
  - list```

[apoorvak-citrix]

@philipp1992
For OVN CNI we rely on the following two annotations on the nodes to fetch the required podCIDR and gatewayIP to add the routes on the ADC.

podcidr Annotation : k8s.ovn.org/node-subnets
gateway Annotation: "k8s.ovn.org/node-primary-ifaddr

This is failing for the following two-nodes 10.x.x.42 and 10.x.x.12. can you confirm that it's present on these nodes?

[burkhat]

@apoorva-05 I'm a colleague from philipp and this nodes are Windows Nodes and they doens't have this annotations.
Does the citrix-ingress-controller supports windows nodes?

annotations:
csi.volume.kubernetes.io/nodeid: >-
{"csi.vsphere.vmware.com":"422467f7-5d34-78c4-fd35-44e239e1ee06","smb.csi.k8s.io":"chmuw-default-windows-62q8n"}
k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 00-15-5D-87-C3-B7
k8s.ovn.org/hybrid-overlay-node-subnet: 100.124.5.0/24
machine.openshift.io/machine: openshift-machine-api/chmuw-default-windows-62q8n
volumes.kubernetes.io/controller-managed-attach-detach: 'true'
windowsmachineconfig.openshift.io/pub-key-hash: 5436e7a8bcc02d332f30075cfa499abae2711bce4cf5e7765ab62d1f9c104efc
windowsmachineconfig.openshift.io/version: 4.0.1+f66f0980

[philipp1992]

we have added the annotations to all nodes but still get the same error.
cic2.txt

[mayurmohanpatil]

@philipp1992 we need to validate Citrix Ingress controller support on windows based OpenShift 4.9 cluster. Can we engage further over slack channel to know more about your use case and assist you further.
Here is the email id [email protected] where you can share your email id to create slack channel.

[philipp1992]

seems like the email is incorrect [[email protected]](mailto:[email protected])

[mayurmohanpatil]

@philipp1992 in case you are not able to send us an email, please use https://podio.com/webforms/22979270/1633242 invite to share your details where I can create slack channel for you.