Add a Rake task class that just adds the CI::Reporter formatter

[abaird]

I have the following rake task (see gist w/ output) that incorporates ci-reporter-rspec. Whenever I run this with bundle exec rake spec I do not see my html documentation. Even if I change the t.rspec_opts line to just --format documentation (which should print progress to STDOUT) I do not see that output. When I remove the ci-reporter require and rake task dependency, I see the formatted output just fine. Is there something else that I need to do to make ci-reporter-rpsec not clobber my rspec output?

There is a similar issue reported here on SO. I am using Ruby 2.1.5, ci-reporter 2.0, ci-reporter-rspec 1.0, rspec 3.1.0 and rake 10.4.2.

[shepmaster]

I am unable to reproduce; you may need to provide a more comprehensive set of steps. Here's what I did:

I set up a brand-new Ubuntu docker container:

$ docker run -it ubuntu

# apt-get update
# apt-get install curl
# gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
# \curl -sSL https://get.rvm.io | bash -s stable
# source /etc/profile.d/rvm.sh
# rvm install 2.1.5

I copied over your files:

# cat Gemfile
source "https://rubygems.org"

gem 'ci_reporter', '~>2.0'
gem 'ci_reporter_rspec', '~>1.0'
gem 'rspec', '~>3.1.0'
gem 'rake', '~>10.4.2'

# cat Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    builder (3.2.2)
    ci_reporter (2.0.0)
      builder (>= 2.1.2)
    ci_reporter_rspec (1.0.0)
      ci_reporter (~> 2.0)
      rspec (>= 2.14, < 4)
    diff-lcs (1.2.5)
    rake (10.4.2)
    rspec (3.1.0)
      rspec-core (~> 3.1.0)
      rspec-expectations (~> 3.1.0)
      rspec-mocks (~> 3.1.0)
    rspec-core (3.1.7)
      rspec-support (~> 3.1.0)
    rspec-expectations (3.1.2)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.1.0)
    rspec-mocks (3.1.3)
      rspec-support (~> 3.1.0)
    rspec-support (3.1.2)

PLATFORMS
  ruby

DEPENDENCIES
  ci_reporter (~> 2.0)
  ci_reporter_rspec (~> 1.0)
  rake (~> 10.4.2)
  rspec (~> 3.1.0)

# cat spec/heartbeat/heartbeat_spec.rb
describe "an example" do
  it "fails" do
    expect(true).to eql(false)
  end
end

I ran Rake:

# ls spec/reports
ls: cannot access spec/reports: No such file or directory

# bundle exec rake
rm -rf spec/reports
rm -rf failure_screenshots
rm -rf spec/reports
/usr/local/rvm/rubies/ruby-2.1.5/bin/ruby -I/usr/local/rvm/gems/ruby-2.1.5/gems/rspec-core-3.1.7/lib:/usr/local/rvm/gems/ruby-2.1.5/gems/rspec-support-3.1.2/lib /usr/local/rvm/gems/ruby-2.1.5/gems/rspec-core-3.1.7/exe/rspec spec/heartbeat/heartbeat_spec.rb --format html --out spec/reports/results.html
F

Failures:

  1) an example fails
     Failure/Error: expect(true).to eql(false)

       expected: false
            got: true

       (compared using eql?)
     # ./spec/heartbeat/heartbeat_spec.rb:3:in `block (2 levels) in <top (required)>'

Finished in 0.00087 seconds (files took 0.09041 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/heartbeat/heartbeat_spec.rb:2 # an example fails
/usr/local/rvm/rubies/ruby-2.1.5/bin/ruby -I/usr/local/rvm/gems/ruby-2.1.5/gems/rspec-core-3.1.7/lib:/usr/local/rvm/gems/ruby-2.1.5/gems/rspec-support-3.1.2/lib /usr/local/rvm/gems/ruby-2.1.5/gems/rspec-core-3.1.7/exe/rspec spec/heartbeat/heartbeat_spec.rb --format html --out spec/reports/results.html failed

I saw results:

# cat spec/reports/SPEC-an-example.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="an example" tests="1" time="0.00069968" failures="1" errors="0" skipped="0" timestamp="2015-03-28T15:11:56+00:00">
  <testcase name="an example fails" time="0.000468792">
    <failure type="RSpec::Expectations::ExpectationNotMetError" message="...">

expected: false
     got: true

(compared using eql?)
 (RSpec::Expectations::ExpectationNotMetError)

expected: false
     got: true

(compared using eql?)

./spec/heartbeat/heartbeat_spec.rb:3:in `block (2 levels) in &lt;top (required)&gt;'    </failure>
  </testcase>
</testsuite>

[shepmaster]

Ugh. I just realized that the issue is that the HTML documentation isn't output, not the CI::Reporter output. I'm still looking.

[shepmaster]

OK, so the problem is that we are using the SPEC_OPTS environment variable to pass options to RSpec. RSpec prefers environment options over CLI options.

As a workaround, you could use something like:

ENV['SPEC_OPTS'] ||= ''
ENV['SPEC_OPTS'] += ' --format html --out spec/reports/results.html'

RSpec::Core::RakeTask.new(:spec) do |t|
  t.pattern = 'spec/heartbeat/heartbeat_spec.rb'
end

You could also set those environment variables in another dependent task, in case you only want them set sometimes.

[shepmaster]

As a longer term fix, it might be nice to create something like a CI::Reporter::RSpec::RakeTask. This could wrap / delegate to the upstream RSpec RakeTask, but just ensure that the proper options are present.

I guess that also provides another alternative for you, instead of setting the environment variables. You could just add the appropriate arguments into your RakeTask.

[abaird]

@shepmaster - this worked for me, thanks. I suppose I could provide an argument for rake, but since I'm already using CI, the solution you provided is an appropriate workaround.

[shepmaster]

@abaird sounds good. I'm going to repurpose this ticket to maybe make a special Rake task that would make this less complicated. (In all my free time!)

[shepmaster]

Suggestions for a future code writer:

class CI::Reporter::RSpec::RakeTask < RSpec::Core::RakeTask

Override the value of rspec_opts to ensure the proper options are present.