Operator deletes resources when it hits an error.

[b13n1u]

Describe the bug
Operator deleted resources when it hit an error like:

ERROR - [config_dispatcher.py:__dispatch_config_pack:347] (Dispatcher) Failed to execute config ADD_sslvserver_sslcertkey_binding_k8Dus1_crd_Listener_xx.xxx.xxx.xx_443_ssl_cs.corp_xx_2024_true_{certkeyname:cs.corp_xx_2024 ca:false snicert:true vservername:k8Dus1_crd_Listener_xx.xxx.xxx.xx_443_ssl } (Status: 104, ErrorCode: 1674, Reason: Nitro Exception: Duplicate domain cert cannot be added.)
...
2024-04-16 11:35:35,566  - INFO - [config_dispatcher.py:__dispatch_config_pack:355] (Dispatcher) Processing of ConfigPack 'Listener.clusterinfra.customersupportpublic.HTTPRoute_delete_spec' is successful
2024-04-16 11:35:35,891  - INFO - [config_dispatcher.py:__dispatch_config_pack:355] (Dispatcher) Processing of ConfigPack 'Listener.clusterinfra.customersupportpublic.delete_spec' is successful
...
DELETE_sslvserver_sslcertkey_binding_k8Ams2ek_crd_Listener_xx.xxx.xx.xx_443_ssl_extcs_2022_true DELETE_csvserver_responderpolicy_binding_k8Ams2ek_crd_Listener_xx.xxx.x.xx_80_http_k8Ams2_crd_Listener_xx.xxx.xx.xxx_80_http_redirect DELETE_responderpolicy_k8Ams2ek_crd_Listener_xx.xxx.xx.xxx_80_http_redirect DELETE_responderaction_k8Ams2ek_crd_Listener_xx.xxx.xx.xx_80_http_redirect DELETE_csvserver_k8Ams2ek_crd_Listener_xx.xxx.xx.xxx_80_http DELETE_csvserver_k8Ams2ek_crd_Listener_xx.xxx.xx.xxx_443_ssl DELETE_sslprofile_k8Ams2_crd_Listener_
....

after the deletion the operator seemed to not do anything. Only after a restart it figured out that the resources are missing and it added them back again.

It looks like there was a SSL certificate added manually into the netscaler instance and the operator could not handle it.

To Reproduce

1. Replace a certificate manually in netscaler and when operator hits (Status: 104, ErrorCode: 1674, Reason: Nitro Exception: Duplicate domain cert cannot be added.) it will delete listener resources.
2. netscalr Controler version: 1.39.6
3. MPX version: NetScaler NS13.1: Build 42.47.nc
4. --ingress-classes citrix --feature-node-watch false --enable-cnc-pbr false --update-ingress-status yes

Expected behavior
The operator should fail with an exception, stop executing or continue working ignoring the change but NOT delete resources.

Logs

kubectl logs
I can provide more logs if needed but I would need to obfuscate some confidential details.

[kaband]

I've run into the same issue using Listener and httproute CRDs. It can occur after an httproute is applied when our netscaler is under a heavy load. If the ingress controller experiences a timeout waiting for a response from the netscaler, it will mark the listener as failed in kubernetes and remove it from the netscaler. This means any other httproutes that were already applied suddenly stop working.

To fix it, I have to recreate the listener and wait for all the httproutes to get applied to it again.

[subashd]

Hi @kaband
Could you please provide the debug logs captured from the NSIC and also you can mail us the details over an email [email protected]?
For the certificate already present in NetScaler, when NSIC tries to create new cert key, it receives the ERROR from the NetScaler. To avoid the certKey already present in NetScaler, you can use the argument preconfigured in the listener. Please refer this.