Could not decrypt value after Loading Secret Value

[imdevopx]

tried to load secret value. and got this error, any suggestions how to fix it? didn't set SECRET_SALT.
Please see attached screenshot of error after Loading Secret Value.

[rubenfiszel]

Hi,

As the error suggest, it means the workspace key is different than the one used to encrypt that value. The workspace key is unique to each workspace and randomly generated. It's available to set in the workspace settings.

[imdevopx]

Hi,

As the error suggest, it means the workspace key is different than the one used to encrypt that value. The workspace key is unique to each workspace and randomly generated. It's available to set in the workspace settings.

hmmm but we didn't touch/change the workspace key. it's working fine before. any suggestions how we can fix this issue?

[rubenfiszel]

No recommendation as we are not aware of an existing bug. If you can send us a full reproduction where creating a variable would end up in that state with the precise version of Windmill used we can investigate. Otherwise, we only provide in-depth custom support to EE customers.

[imdevopx]

what if we will try to change this encryption key? would it possibly fix the issue? or any concerns when we change it?

[rubenfiszel]

No it won't, if it can't decrypt the value prior then it won't be able to decrypt and re-encrypt the value as needed by the workspace key change.

[imdevopx]

No recommendation as we are not aware of an existing bug. If you can send us a full reproduction where creating a variable would end up in that state with the precise version of Windmill used we can investigate. Otherwise, we only provide in-depth custom support to EE customers.

hi @rubenfiszel variables was created days before the issue appears. it is working fine before, then now all variables are not able to retrieved out of nowhere.
version: CE v1.450.1

[imdevopx]

or any way so we can retrieve the old encryption key?

[rubenfiszel]

No we don't store the history of the encryption keys.

We have no reports of anyone else having similar issue so far.

[imdevopx]

meaning only fix to resolve this issue is to recreate the secrets?

[imdevopx]

any suggestions how to prevent this issue so it won't happen again?

[rubenfiszel]

We do not know the cause of this. Without any reproduction and without anyone else facing this, human error is the most likely possibility.

If you really believe it's a software bug, then use the image of an older release with your current database and check if the value then get decrypted correctly.

[imdevopx]

still not able to fix the issue with the older version. any recommendations so we back to working condition again? we have a lot of workflow/scripts, secrets on this windmill project as we mentioned earlier the issue just came out of nowhere. we didn't change encryption key.

just to add. we have two active workspaces on this windmill. 1 with many secrets/variables and 1 with just 2 secrets..
the 1 with just 2 secrets works properly in loading the secrets value.

[rubenfiszel]

Included in the enterprise edition, there is some support for cases like this where we can help you troubleshoot the cause of the issue. There are more than 10k instances being used daily, it's the first time we hear an issue like this so we do believe it's a human error until proven otherwise.
To get the enterprise edition, you can contact [email protected]

[imdevopx]

can we recreate new encryption key as SECRET_SALT and store it somewhere, then recreate all new variables/secrets in windmill so that the new encryption key in SECRET_SALT will be used on those recreated variables/secrets? will this prevent the said issue to re-occur?