preserve-environments using r10k deployed environments not working

[walkleyn]

Description of problem

• What did you do?
  I am trying to run octocatalog-diff on a puppetserver running puppet 4.10.12 with puppet environments managed by r10k in the following structure:
  `/etc/puppetlabs/puppet
  • environments
    • production
      • environment.conf
      • manifests
        • site.pp
      • modules
      • site
    • staging
      • environment.conf
      • manifests
        • site.pp
      • modules
      • site
    • testing_branch
      • environment.conf
      • manifests
        • site.pp
      • modules
      • site`
        I am trying to run octocatalog-diff against the checked out code using the --preserve-environments option as follows:
• What happened?
  octocatalog-diff exits with an error code 1, expecting a basedir configuration specified, which isn't required as per documentation on advanced environments
• What did you expect to happen?
  I expected octocatalog-diff to create and compare catalogues created from the two r10k deployed environments.
  Puppet agent runs against both environments work as expected.
• How can someone reproduce the problem?
  Create two different working puppet environments and run octocatalog-diff as explained above

Command used and debugging output

octocatalog-diff -n [nodename] --bootstrapped-from-dir /etc/puppetlabs/puppet --bootstrapped-to-dir /etc/puppetlabs/puppet --from-environment production --to-environment testing_branch --preserve-environments
D, [2018-06-13T17:07:41.649263 #24328] DEBUG -- : Running octocatalog-diff 1.5.3 with ruby 2.1.5 D, [2018-06-13T17:07:41.649465 #24328] DEBUG -- : Command line arguments: ["-n", "node_name", "--bootstrapped-from-dir", "/etc/puppetlabs/puppet", "--bootstrapped-to-dir", "/etc/puppetlabs/puppet", "--from-environment", "production", "--to-environment", "testing_branch", "--preserve-environments", "-d"] D, [2018-06-13T17:07:41.649552 #24328] DEBUG -- : Running on host puppetserver_name (x86_64-linux-gnu) /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/git.rb:63:in fetch': key not found: :basedir (KeyError)
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/git.rb:63:in branch_sha' from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/cached_master_directory.rb:112:in git_repo_checkout_bootstrap'
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/cached_master_directory.rb:49:in run' from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/cli.rb:106:in cli'
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/bin/octocatalog-diff:34:in <top (required)>' from /usr/local/bin/octocatalog-diff:23:in load'
from /usr/local/bin/octocatalog-diff:23:in <main>'

Platform and version information

• Your OS: Debian Linux 8 (Jessie)
• Your Ruby version: ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]
• Your version of Puppet: puppet 4.10.12 OSE
• Your version of octocatalog-diff: octocatalog-diff 1.5.3

Do the tests pass from a clean checkout?
yes

Anything else to add that you think will be helpful?
When I add the -f and -t options, the catalog compiles, but then I get errors about custom providers not found:
command:
octocatalog-diff -n node_name --bootstrapped-from-dir /etc/puppetlabs/puppet/environments/production --bootstrapped-to-dir /etc/puppetlabs/puppet/environments/testing_branch --from-environment production --to-environment testing_branch -f production -t testing_branch -d
catalog error:
Error: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper Error: Could not autoload puppet/type/network_config: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/network_config: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper at /tmp/ocd-ipc-20180613-28056-1iximb4/ocd-builddir-20180613-28061-30pfww/environments/testing_branch/modules/monolith/quotaman/manifests/config.pp:80:3 on node quotaman-staging3.cpt3.host-h.net Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/network_config: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper at /tmp/ocd-ipc-20180613-28056-1iximb4/ocd-builddir-20180613-28061-30pfww/environments/testing_branch/modules/monolith/quotaman/manifests/config.pp:80:3 on node quotaman-staging3.cpt3.host-h.net Error: Failed to compile catalog for node quotaman-staging3.cpt3.host-h.net: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/network_config: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper at /tmp/ocd-ipc-20180613-28056-1iximb4/ocd-builddir-20180613-28061-30pfww/environments/testing_branch/modules/monolith/quotaman/manifests/config.pp:80:3 on node node_name

[vinzent]

@walkleyn can you paste the complete commandline you ouse and the complete output of octocatalog-diff?

[walkleyn]

@vinzent here you go:
octocatalog-diff -n [nodename] --bootstrapped-from-dir /etc/puppetlabs/puppet --bootstrapped-to-dir /etc/puppetlabs/puppet --from-environment production --to-environment testing_branch --preserve-environments
D, [2018-06-13T17:07:41.649263 #24328] DEBUG -- : Running octocatalog-diff 1.5.3 with ruby 2.1.5 D, [2018-06-13T17:07:41.649465 #24328] DEBUG -- : Command line arguments: ["-n", "node_name", "--bootstrapped-from-dir", "/etc/puppetlabs/puppet", "--bootstrapped-to-dir", "/etc/puppetlabs/puppet", "--from-environment", "production", "--to-environment", "testing_branch", "--preserve-environments", "-d"] D, [2018-06-13T17:07:41.649552 #24328] DEBUG -- : Running on host puppetserver_name (x86_64-linux-gnu) /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/git.rb:63:in fetch': key not found: :basedir (KeyError)
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/git.rb:63:in branch_sha' from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/cached_master_directory.rb:112:in git_repo_checkout_bootstrap'
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/catalog-util/cached_master_directory.rb:49:in run' from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/cli.rb:106:in cli'
from /var/lib/gems/2.1.0/gems/octocatalog-diff-1.5.3/bin/octocatalog-diff:34:in <top (required)>' from /usr/local/bin/octocatalog-diff:23:in load'
from /usr/local/bin/octocatalog-diff:23:in <main>'

[xakraz]

Hello !

We are facing the same issue with the elastic_search module (https://github.com/elastic/puppet-elasticsearch) v6.3.0 ..

Error: Failed to compile catalog for node xxx9-8.domain.io: 
Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/elasticsearch_user_file: 
Could not autoload puppet/provider/elasticsearch_user_file/xpack: 
cannot load such file -- puppet/provider/elastic_parsedfile at /tmp/ocd-ipc-20180910-186-kdxjnr/ocd-builddir-20180910-191-g6wb0/environments/production/modules/elasticsearch/manifests/init.pp:545:6 on node xxx9-8.domain.io

CMD

cmd: octocatalog-diff \
  --retry-failed-catalog 1 -o /dev/null \
  --hostnamexxx9-8.domain.io \
  --command_line --certname\=xxxx \
  --command_line --confdir\=/home/xxxx/.puppetlabs/etc/puppet \
  --catalog-only --debug

Platform and version information

Your OS: Debian Linux 8 (Jessie)
Your Ruby version: ruby 2.4.4
Your version of Puppet: puppet 4.10.12 OSE
Your version of octocatalog-diff: octocatalog-diff 1.5.3

I had to do ugly things to work around:

$ git diff modules/elasticsearch/
diff --git a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/elasticsearch_users.rb b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/elasticsearch_users.rb
index 2acf877369..af6af98375 100644
--- a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/elasticsearch_users.rb
+++ b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/elasticsearch_users.rb
@@ -1,4 +1,5 @@
-require 'puppet/provider/elastic_user_command'
+#require 'puppet/provider/elastic_user_command'
+require File.join(File.dirname(__FILE__), '..','..','..', '..', 'lib/puppet/provider/elastic_user_command')
 
 Puppet::Type.type(:elasticsearch_user).provide(
   :elasticsearch_users,
diff --git a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/esusers.rb b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/esusers.rb
index f900f51cfd..983629f402 100644
--- a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/esusers.rb
+++ b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/esusers.rb
@@ -1,4 +1,5 @@
-require 'puppet/provider/elastic_user_command'
+#require 'puppet/provider/elastic_user_command'
+require File.join(File.dirname(__FILE__), '..','..','..', '..', 'lib/puppet/provider/elastic_user_command')
 
 Puppet::Type.type(:elasticsearch_user).provide(
   :esusers,
diff --git a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/users.rb b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/users.rb
index d078a48cee..e6122eae76 100644
--- a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/users.rb
+++ b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/users.rb
@@ -1,4 +1,5 @@
-require 'puppet/provider/elastic_user_command'
+#require 'puppet/provider/elastic_user_command'
+require File.join(File.dirname(__FILE__), '..','..','..', '..', 'lib/puppet/provider/elastic_user_command')
 
 Puppet::Type.type(:elasticsearch_user).provide(
   :users,
diff --git a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user_file/oss_xpack.rb b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user_file/oss_xpack.rb
index a160bea7de..23fb0f30f7 100644
--- a/modules/elasticsearch/lib/puppet/provider/elasticsearch_user_file/oss_xpack.rb
+++ b/modules/elasticsearch/lib/puppet/provider/elasticsearch_user_file/oss_xpack.rb
@@ -1,4 +1,5 @@
-require 'puppet/provider/elastic_parsedfile'
+#require 'puppet/provider/elastic_parsedfile'
+require File.join(File.dirname(__FILE__), '..','..','..', '..', 'lib/puppet/provider/elastic_parsedfile')
...