Skip to content
This repository has been archived by the owner on Jul 14, 2021. It is now read-only.

Windows: cert already in hash table #492

Open
cshowers opened this issue Aug 31, 2015 · 22 comments
Open

Windows: cert already in hash table #492

cshowers opened this issue Aug 31, 2015 · 22 comments

Comments

@cshowers
Copy link

My env
Windows 7
chefdk 0.7
PowerShell 4
chef-sever-core 12.1.2


I've been testing the new chefdk policyfile by following the policyfile guided tour https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/#more-16518 Every time I run chef provision it failes with SSL_read errors

I created the special provision cookbook as per example and ran following
chef provision staging --sync -n aar-staging-01 -o port=8000
Uploading policy to policy group staging
[2015-08-30T22:00:25-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_read: cert already in hash table
[2015-08-30T22:00:27-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
Error: Failed to upload policy to policy group staging
Reason: (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table

I setup chef server with ssl. I ran knife ssl fetch and knife ssl check is good

PS C:\Users\cshowers\git\chef-cookbooks\aar> knife ssl check
Connecting to host server-centos-65:443
Successfully verified certificates from `server-centos-65'

even if I run 'chef push staging' I get the same ssl errors. Am I missing something? It's almost like chef provision command doesn't use the same ssl as knife. I am pretty new to chef and I am trying to get my head around policfiles.


My knife.rb
current_dir = File.dirname(FILE)
log_level :info
log_location STDOUT
node_name "cshowers"
client_key "#{current_dir}/cshowers.pem"
chef_server_url "https://server-centos-65/organizations/test"
cookbook_path ["#{current_dir}/../cookbooks"]

@danielsdeleo
Copy link
Contributor

Does it work if you give the -c CONFIG_FILE option? Trying to figure out if it's skipping the config file load if you don't pass the option explicitly.

@cshowers
Copy link
Author

cshowers commented Sep 2, 2015

no it gives the same error. I tried a few tricks with config file. like relocate it to a different dir and ref it using -c. I did not add client.pem on purpose to see if it complains. And yes it complained about not having proper ssl. So I added pem and it resumes with cert already in hash.

@cshowers
Copy link
Author

cshowers commented Sep 2, 2015

Output of using -c

PS C:\Users\cshowers\git\chef-cookbooks\aar> chef provision staging --sync -n aar-staging-01 -o port=8000 -c c:\chef\client.rb
Uploading policy to policy group staging
[2015-09-02T14:01:15-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_read: cert already in hash table
[2015-09-02T14:01:16-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
[2015-09-02T14:01:16-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
Error: Failed to upload policy to policy group staging
Reason: (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table

PS C:\Users\cshowers\git\chef-cookbooks\aar> knife ssl check -c c:\chef\client.rb
Connecting to host server-centos-65:443
Successfully verified certificates from `server-centos-65'

@danielsdeleo
Copy link
Contributor

Thanks for that, I wanted to make sure this wasn't the same thing as a different bug where the config file simply wasn't getting loaded.

This seems to be an issue with Ruby itself, where when you have active threads and are using the SSL_CERT_FILE environment variable to configure OpenSSL (which we have to do on Windows), you can get the (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table error. We are looking into ways we could fix this in the build.

The answer on this SO post might help, but ChefDK patches ruby to set the SSL_CERT_FILE variable (otherwise SSL cert verification wouldn't work in any circumstance), which might break the workaround. http://stackoverflow.com/questions/23263121/opensslx509storeerror-cert-already-in-hash-table

@cshowers
Copy link
Author

cshowers commented Sep 2, 2015

ok I see. So since this problem only shows up on Windows I could test this out on linux until there is a chefDK release that fixes this.

@danielsdeleo
Copy link
Contributor

Yes, we are looking at whether Ruby 2.2 would fix it, or if we could build Ruby on Windows ourselves rather than relying on RubyInstaller/DevKit builds (this would allow us to control the build-time configuration of OpenSSL so we wouldn't need to set SSL_CERT_FILE via a patch).

@SympaHannuPiki
Copy link

Any change on this yet? When trying to upload even a bit more complex cookbook into server, I will get the same error. Simple cookbooks are fine, but "windows" cookbook from supermarket isn't something it will digest very well. We are using valid certificate from Thawte and everything else work just fine, including "knife ssl check" is returning successful results.

Currently using ChefDK on Windows 8.1

Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2

@danielsdeleo
Copy link
Contributor

If you make a policyfile and upload it with chef push does that fail in the same way? Can you knife cookbook upload the same cookbook?

@danielsdeleo
Copy link
Contributor

Also, what chef server are you using, and have you modified its SSL configuration?

@gitmlynch
Copy link

I'm having the same problem if I do a cookbook upload with --include-dependencies and the windows cookbook is one of the dependencies. This wasn't a problem until I upgraded from Chefdk 0.4.0 and knife-windows 0.8.5.

I was able to upload the windows cookbook by itself, so that got me past my immediate problem, but for automated deployments, that's not a useful work around.

Currently using ChefDK on Windows Server 2008 R2

Chef Development Kit Version: 0.9.0
chef-client version: 12.5.1
knife-windows: 1.1.1

@danielsdeleo
Copy link
Contributor

Are you specifically seeing SSL_read: cert already in hash table ? If not, it may be a different issue.

@gitmlynch
Copy link

I am seeing SSL_read: cert already in hash table

@gitmlynch
Copy link

FWIW, I have found that using the --concurrency option, set to 1, in my 'knife cookbook upload' commands to be 100% successful at avoiding the SSL error problem described above.

@SympaHannuPiki
Copy link

@danielsdeleo, tested "knife cookbook upload" and problem seems to be the same. Chef server is 12.3.1. Regarding SSL configuration, I let Chef to generate self-signed certificate and then replaced (crt/key) files with new (valid) certificate... so no actual modifications are made for it. Most of the modifications for any configs are for Analytics.

When uploading depended cookbooks one by one, I finally managed to upload original "windows" cookbook.

PS D:\xxx\sandbox\cookbooks> knife cookbook upload windows
Uploading windows        [1.38.4]
ERROR: SSL Validation failure connecting to host: chef.rs-sympa.com - SSL_read: cert already in hash table
ERROR: SSL Validation failure connecting to host: chef.rs-sympa.com - SSL_read: cert already in hash table
ERROR: Could not establish a secure connection to the server.
Use `knife ssl check` to troubleshoot your SSL configuration.
If your Chef Server uses a self-signed certificate, you can use
`knife ssl fetch` to make knife trust the server's certificates.

Original Exception: OpenSSL::SSL::SSLError: SSL_read: cert already in hash table

@bradwilliamson
Copy link

I am also getting this error consistently . I try to the knife ssl check and fetch. Everything looks good. I get this randomly when using chef-client -o 'cookbook::recipe' -c C:\chef-repo.chef\knife.rb for provisioning virtual machines with the chef-provisioning-vsphere gem.

@bradwilliamson
Copy link

This issue is still present in the latest rub gems provided with chefdk 0.11.2

@bradwilliamson
Copy link

bradwilliamson commented Apr 22, 2016

This issue is still persistent in chefdk 0.12.0. It seems to always occur if there is a syntax error in your provisioning recipe & you try to run it again after. If I reboot that sometimes clears it up. Doing a knife ssl fetch does not fix the issue. This occurs with machine_batch paralegal provisioning of servers.

@bradwilliamson
Copy link

bradwilliamson commented Apr 26, 2016

I have started to use a Linux Chef Workstation as a work around when I provision more than one server at a time. The bug doesn't appear in the ruby gems provided with chefdk on linux.

@jdivine
Copy link

jdivine commented Jul 20, 2016

I am also seeing this when my Windows nodes converge - usually when I first provision them and they do the initial converge. Not on my workstation where I trigger the provisioning (although I have seen that too!) but on the nodes themselves when they converge with chef-client, as they synchronize the cookbooks from the server. It seems to happen about half of the time - frequently I just manually kick off the Chef Provisioning run again and the nodes will successfully converge.

Same issue with both chef-client 12.5.1 and 12.12.5

[2016-07-19T19:53:29-05:00] INFO: Storing updated cookbooks/java/recipes/ibm_tar.rb in the cache. [2016-07-19T19:53:29-05:00] ERROR: SSL Validation failure connecting to host: chef.spokvdev.com - SSL_read: cert already in hash table [2016-07-19T19:53:29-05:00] INFO: Storing updated cookbooks/java/recipes/oracle.rb in the cache.

                   [2016-07-19T19:53:31-05:00] ERROR: SSL Validation failure connecting to host: chef.spokvdev.com - SSL_read: cert already in hash table

                   ================================================================================
                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/01proxy.erb in the cache.
                   Error Syncing Cookbooks:
                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/10recommends.erb in the cache.
                   ================================================================================

                   Unexpected Error:
                   -----------------
                   OpenSSL::SSL::SSLError: SSL_read: cert already in hash table

                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/20auto-upgrades.erb in the cache.
                   [2016-07-19T19:53:31-05:00] ERROR: Running exception handlers
                   [2016-07-19T19:53:31-05:00] ERROR: Exception handlers complete```

@glennsarti
Copy link

Could this be a Chef manifestation of the similar issue in Puppet?

puppetlabs/puppet@35572a9

  • The solution is to update the OpenSSL::X509::Store monkey path to
    pass a predicate to Rubys uniq function, to ensure that certs are
    compared by their DER byte representation. Without this change, the
    default uniq implementation uses the .hash function on the
    X509::Certificate object, which varies by object instance, not actual
    cert contents. Although the PEM representation of a cert could be
    used for comparison, DER generates smaller objects for comparison.

...

  • Duplicate detection is necessary to perform in Puppet, because Rubys
    OpenSSL wrapper provides no ability to enumerate the store, or query
    if a cert has been added already. The code simply fails if dupes are
    added:

https://github.com/ruby/ruby/blob/ruby_2_1/ext/openssl/ossl_x509store.c#L303-L316

That code delegates to the OpenSSL function X509_STORE_add_crl, which
ultimately uses X509_OBJECT_retrieve_match to determine if a given
cert matches any in the store already:

https://github.com/openssl/openssl/blob/OpenSSL_1_0_2g/crypto/x509/x509_lu.c#L572-L597

@bradwilliamson
Copy link

This error still occurs as of latest Chef DK release. I get this error mainly when using machine_batch on a Windows Chef Workstation provisioning multiple Virtual Machines.

@cheeseplus cheeseplus changed the title chefdk 0.7.0 chef provision fails with SSL_read error Windows: cert already in hash table Jul 14, 2017
@irvingpop
Copy link

If I'm reading this Ruby issue correctly, it should be fixed in Ruby 2.4 which is used by ChefDK 2.x - can those impacted confirm?

cc: @thommay @trevorghess

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

10 participants