Failed to create canary patset, Canary deployment will result in patset error when containing "-" symbol the namespace

[progS1m]

Describe the bug
Using canary annotations not working when the namespace name is containing a "-" symbol.

Annotations:
ingress.citrix.com/canary: true
ingress.citrix.com/canary-by-header: X-Canary
ingress.citrix.com/canary-by-header-value: preview
ingress.citrix.com/canary-weight: 0

Namspace:
namespace: my-namespace

Unfortunately we cannot easily change all our namespaces name. Since we have to many of them! And most of them containing "-" symbols!

To Reproduce
Deploy the sample like described here, but in a namespace containing a "-" symbol in its name. https://docs.netscaler.com/en-us/netscaler-k8s-ingress-controller/canary/canary.html#simplified-canary-deployment-using-ingress-annotations

Version of the NetScaler Ingress Controller
1.39.6
Version of MPX/VPX/CPX
NS 13.1 51.15

Expected behavior

Either the Netscaler should accept "-" symbols in patset or the CIC should convert dashes to underlines. The second option would be probably the faster workaround!

Logs

kubectl logs

2024-03-26 08:37:15,674 - ERROR - [nitrointerface.py:create_canary_patset:7887] (MainThread) Failed to create canary patset k710p_my-namespace_myingressname_Canary_Header_Values_0. Reason (3856, 'Invalid policy entity name: names must begin with an ASCII alphabetic character or underscore and must contain only ASCII alphanumerics or underscores; words reserved for policy use may not be used', 'ERROR')

Probably relates to: #547

[ankits123]

@progS1m since '-' is used as a separator in the controller naming convention it is giving above mentioned error.
We will discuss your requirement and get back with a update.

[progS1m]

@ankits123 noticed the canary is not always creating a patset. In which constellation are they being crated and in which not?

Using multiple values like described here is not working, respectively doesn't result into a patset.

ingress.citrix.com/canary-by-header-value: '["value1","value2","value3","value4"]'

results in:

2024-03-27 17:19:10,370  - DEBUG - [nitrointerface.py:is_alternate_backend_default_policy:2106] (MainThread) Checking for default cspolicy for alternate backend : policyname:k915p-showhttpvars-r744_80_csp_l6456jwuknnqbiddtdqxcav46sd74o7h and rule: (((HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/")) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/test"))) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/foo"))) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/bar")) && HTTP.REQ.HEADER("X-Canary").EQ("["value1","value2","value3","value4"]")

https://docs.netscaler.com/en-us/netscaler-k8s-ingress-controller/canary/canary.html#canary-deployment-based-on-the-http-request-header-value

[progS1m]

@ankits123 noticed the canary is not always creating a patset. In which constellation are they being crated and in which not?

Using multiple values like described here is not working, respectively doesn't result into a patset.

ingress.citrix.com/canary-by-header-value: '["value1","value2","value3","value4"]'

results in:

2024-03-27 17:19:10,370  - DEBUG - [nitrointerface.py:is_alternate_backend_default_policy:2106] (MainThread) Checking for default cspolicy for alternate backend : policyname:k915p-showhttpvars-r744_80_csp_l6456jwuknnqbiddtdqxcav46sd74o7h and rule: (((HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/")) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/test"))) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/foo"))) || (HTTP.REQ.HOSTNAME.SERVER.EQ("showhttpvars-web-k915.domain.net") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH("/bar")) && HTTP.REQ.HEADER("X-Canary").EQ("["value1","value2","value3","value4"]")

https://docs.netscaler.com/en-us/netscaler-k8s-ingress-controller/canary/canary.html#canary-deployment-based-on-the-http-request-header-value

Figured out, the patset was introducing in newer CIC builds, where in older builds no patset has been created.

[progS1m]

@ankits123 /usr/src/triton/kubernetes/canary/ingress_canary.py is the file that has the buggy function
get_canary_patset_name

Please fix it by adding an unsupported character substitution something like this:
namespace = re.sub('[^A-Za-z0-9]+', '',namespace)

I think it's an easy fix. When can we expect a fix? Unfortunately It is important for us to get this addressed very soon.

[pfyod]

@ankits123: could you please share timeline for this fix? We are unable to use canary features because of this bug - blocked for two months already!

[ankits123]

hi @pfyod we would like to discussion some points over mail .
Can you please drop us a email at < [email protected] > for the same .

[pfyod]

@ankits123: I would prefer to keep it here. Tried the latest CIC controller and it's still not fixed.

This method:

    @staticmethod
    def get_canary_patset_name(cic_prefix, namespace, ingress_name):
        ingress_name = re.sub('[^A-Za-z0-9]+', '', ingress_name)
        cic_prefix = re.sub('[^A-Za-z0-9]+', '', cic_prefix)
        return CanaryConstants.CANARY_PATSET_NAME.format(cic_prefix, namespace, ingress_name)

should be changed to:

    @staticmethod
    def get_canary_patset_name(cic_prefix, namespace, ingress_name):
        ingress_name = re.sub('[^A-Za-z0-9]+', '', ingress_name)
        namespace = re.sub('[^A-Za-z0-9]+', '', namespace)
        cic_prefix = re.sub('[^A-Za-z0-9]+', '', cic_prefix)
        return CanaryConstants.CANARY_PATSET_NAME.format(cic_prefix, namespace, ingress_name)

This will not break existing deployments, unless there are namespace collisions after escaping, e.g. na-me-spa-ce and namespace.

[subashd]

hi @pfyod
We are working on the fix.

[subashd]

@pfyod
This issue has been fixed in latest release.
https://github.com/netscaler/netscaler-k8s-ingress-controller/releases/tag/2.2.10