Introduce a SimpleCov.collate entry point

CHANGELOG.md

@@ -3,6 +3,7 @@ Master (unreleased)
 
 ## Enhancements
 * Allow early running exit tasks and avoid the `at_exit` hook through the `SimpleCov.run_exit_tasks!` method. (thanks [@macumber]: https://github.com/macumber))
+* Allow manual collation of result sets through the `SimpleCov.collate` entrypoint. See the README for more details (thanks [@ticky](https://github.com/ticky))
 
 0.18.0.beta2 (2020-01-05)
 ===================

README.md

@@ -472,12 +472,11 @@ end
 
 You normally want to have your coverage analyzed across ALL of your test suites, right?
 
-Simplecov automatically caches coverage results in your (coverage_path)/.resultset.json. Those results will then
-be automatically merged when generating the result, so when coverage is set up properly for Cucumber and your
-unit / functional / integration tests, all of those test suites will be taken into account when building the
-coverage report.
-
-There are two things to note here though:
+Simplecov automatically caches coverage results in your
+(coverage_path)/.resultset.json, and will merge or override those with
+subsequent runs, depending on whether simplecov considers those subsequent runs
+as different test suites or as the same test suite as the cached results. To
+make this distinction, simplecov has the concept of "test suite names".
 
 ### Test suite names
 
@@ -531,14 +530,84 @@ SimpleCov.command_name "features" + (ENV['TEST_ENV_NUMBER'] || '')
 
 [simplecov-html] prints the used test suites in the footer of the generated coverage report.
 
-### Timeout for merge
 
-Of course, your cached coverage data is likely to become invalid at some point. Thus, result sets that are older than
-`SimpleCov.merge_timeout` will not be used any more. By default, the timeout is 600 seconds (10 minutes), and you can
-raise (or lower) it by specifying `SimpleCov.merge_timeout 3600` (1 hour), or, inside a configure/start block, with
-just `merge_timeout 3600`.
+### Merging test runs under the same execution environment
+
+Test results are automatically merged with previous runs in the same execution
+environment when generating the result, so when coverage is set up properly for
+Cucumber and your unit / functional / integration tests, all of those test
+suites will be taken into account when building the coverage report.
+
+#### Timeout for merge
+
+Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically merging
+subsequent test runs, result sets that are older than `SimpleCov.merge_timeout` will not be used any more. By default,
+the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifying `SimpleCov.merge_timeout 3600`
+(1 hour), or, inside a configure/start block, with just `merge_timeout 3600`.
+
+You can deactivate this automatic merging altogether with `SimpleCov.use_merging false`.
+
+### Merging test runs under different execution environments
+
+If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a single
+result set using the `SimpleCov.collate` method. This can be added to a Rakefile or script file, having downloaded a set of
+`.resultset.json` files from each parallel test run.
+
+```ruby
+# lib/tasks/coverage_report.rake
+namespace :coverage do
+  desc "Collates all result sets generated by the different test runners"
+  task :report do
+    require 'simplecov'
+
+    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
+  end
+end
+```
+
+`SimpleCov.collate` also takes an optional simplecov profile and an optional
+block for configuration, just the same as `SimpleCov.start` or
+`SimpleCov.configure`.  This means you can configure a separate formatter for
+the collated output. For instance, you can make the formatter in
+`SimpleCov.start` the `SimpleCov::Formatter::SimpleFormatter`, and only use more
+complex formatters in the final `SimpleCov.collate` run.
+
+```ruby
+# spec/spec_helper.rb
+require 'simplecov'
+
+SimpleCov.start 'rails' do
+  # Disambiguates individual test runs
+  command_name "Job #{ENV["TEST_ENV_NUMBER"]}" if ENV["TEST_ENV_NUMBER"]
+
+  if ENV['CI']
+    formatter SimpleCov::Formatter::SimpleFormatter
+  else
+    formatter SimpleCov::Formatter::MultiFormatter.new([
+      SimpleCov::Formatter::SimpleFormatter,
+      SimpleCov::Formatter::HTMLFormatter
+    ])
+  end
+
+  track_files "**/*.rb"
+end
+```
+
+```ruby
+# lib/tasks/coverage_report.rake
+namespace :coverage do
+  task :report do
+    require 'simplecov'
 
-You can deactivate merging altogether with `SimpleCov.use_merging false`.
+    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
+      formatter SimpleCov::Formatter::MultiFormatter.new([
+        SimpleCov::Formatter::SimpleFormatter,
+        SimpleCov::Formatter::HTMLFormatter
+      ])
+    end
+  end
+end
+```
 
 ## Running coverage only on demand
 

features/test_unit_collate.feature

@@ -0,0 +1,43 @@
+@test_unit
+Feature:
+
+  Using SimpleCov.collate should get the user a coverage report
+
+  Scenario:
+    Given SimpleCov for Test/Unit is configured with:
+      """
+      require 'simplecov'
+      SimpleCov.start
+      """
+
+    When I successfully run `bundle exec rake part1`
+    Then a coverage report should have been generated
+    When I successfully run `mv coverage/.resultset.json coverage/resultset1.json`
+    And I successfully run `rm coverage/index.html`
+
+    When I successfully run `bundle exec rake part2`
+    Then a coverage report should have been generated
+    When I successfully run `mv coverage/.resultset.json coverage/resultset2.json`
+    And I successfully run `rm coverage/index.html`
+
+    When I open the coverage report generated with `bundle exec rake collate`
+    Then I should see the groups:
+      | name      | coverage | files |
+      | All Files | 91.38%   | 6     |
+
+    And I should see the source files:
+      | name                                    | coverage |
+      | lib/faked_project.rb                    | 100.0 %  |
+      | lib/faked_project/some_class.rb         | 80.0 %   |
+      | lib/faked_project/framework_specific.rb | 75.0 %   |
+      | lib/faked_project/meta_magic.rb         | 100.0 %  |
+      | test/meta_magic_test.rb                 | 100.0 %  |
+      | test/some_class_test.rb                 | 100.0 %  |
+
+      # Note: faked_test.rb is not appearing here since that's the first unit test file
+      # loaded by Rake, and only there test_helper is required, which then loads simplecov
+      # and triggers tracking of all other loaded files! Solution for this would be to
+      # configure simplecov in this first test instead of test_helper.
+
+    And the report should be based upon:
+      | Unit Tests |

lib/simplecov.rb

@@ -46,15 +46,50 @@ class << self
     #
     def start(profile = nil, &block)
       require "coverage"
-      load_profile(profile) if profile
-      configure(&block) if block_given?
+      initial_setup(profile, &block)
       @result = nil
-      self.running = true
       self.pid = Process.pid
 
       start_coverage_measurement
     end
 
+    #
+    # Collate a series of SimpleCov result files into a single SimpleCov output.
+    # You can optionally specify configuration with a block:
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' # using rails profile
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"] do
+    #     add_filter 'test'
+    #   end
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
+    #     add_filter 'test'
+    #   end
+    #
+    # Please check out the RDoc for SimpleCov::Configuration to find about
+    # available config options, or checkout the README for more in-depth
+    # information about coverage collation
+    #
+    def collate(result_filenames, profile = nil, &block)
+      raise "There's no reports to be merged" if result_filenames.empty?
+
+      initial_setup(profile, &block)
+
+      results = result_filenames.flat_map do |filename|
+        # Re-create each included instance of SimpleCov::Result from the stored run data.
+        (JSON.parse(File.read(filename), symbolize_names: true) || {}).map do |command_name, coverage|
+          SimpleCov::Result.from_hash(command_name => coverage)
+        end
+      end
+
+      # Use the ResultMerger to produce a single, merged result, ready to use.
+      @result = SimpleCov::ResultMerger.merge_and_store(*results)
+
+      run_exit_tasks!
+    end
+
     #
     # Returns the result for the current coverage run, merging it across test suites
     # from cache using SimpleCov::ResultMerger if use_merging is activated (default)
@@ -255,6 +290,12 @@ def write_last_run(covered_percent)
 
   private
 
+    def initial_setup(profile, &block)
+      load_profile(profile) if profile
+      configure(&block) if block_given?
+      self.running = true
+    end
+
     #
     # Trigger Coverage.start depends on given config coverage_criterion
     #

lib/simplecov/result_merger.rb

@@ -62,6 +62,12 @@ def results
         results
       end
 
+      def merge_and_store(*results)
+        result = merge_results(*results)
+        store_result(result) if result
+        result
+      end
+
       # Merge two or more SimpleCov::Results into a new one with merged
       # coverage data and the command_name for the result consisting of a join
       # on all source result's names

spec/simplecov_spec.rb

@@ -230,10 +230,95 @@
     end
   end
 
+  describe ".collate" do
+    let(:resultset1) do
+      {source_fixture("sample.rb") => {lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil]}}
+    end
+
+    let(:resultset2) do
+      {source_fixture("sample.rb") => {lines: [1, nil, 1, 1, nil, nil, 1, 1, nil, nil]}}
+    end
+
+    let(:resultset_path) { SimpleCov::ResultMerger.resultset_path }
+
+    let(:resultset_folder) { File.dirname(resultset_path) }
+
+    context "when no files to be merged" do
+      it "shows an error message" do
+        expect do
+          glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH)
+          SimpleCov.collate glob
+        end.to raise_error("There's no reports to be merged")
+      end
+    end
+
+    context "when files to be merged" do
+      before do
+        expect(SimpleCov).to receive(:run_exit_tasks!)
+      end
+
+      context "and a single report to be merged" do
+        before do
+          create_mergeable_report("result1", resultset1)
+        end
+
+        after do
+          clear_mergeable_reports("result1")
+        end
+
+        it "creates a merged report identical to the original" do
+          glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH)
+          SimpleCov.collate glob
+
+          expected = {"result1": {coverage: {source_fixture("sample.rb") => {lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil]}}}}
+          collated = JSON.parse(File.read(resultset_path), symbolize_names: true).transform_values { |v| v.reject { |k| k == :timestamp } }
+          expect(collated).to eq(expected)
+        end
+      end
+
+      context "and multiple reports to be merged" do
+        before do
+          create_mergeable_report("result1", resultset1)
+          create_mergeable_report("result2", resultset2)
+        end
+
+        after do
+          clear_mergeable_reports("result1", "result2")
+        end
+
+        it "creates a merged report" do
+          glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH)
+          SimpleCov.collate glob
+
+          expected = {"result1, result2": {coverage: {source_fixture("sample.rb") => {lines: [1, 1, 2, 2, nil, nil, 2, 2, nil, nil]}}}}
+          collated = JSON.parse(File.read(resultset_path), symbolize_names: true).transform_values { |v| v.reject { |k| k == :timestamp } }
+          expect(collated).to eq(expected)
+        end
+      end
+
+    private
+
+      def create_mergeable_report(name, resultset)
+        result = SimpleCov::Result.new(resultset)
+        result.command_name = name
+        SimpleCov::ResultMerger.store_result(result)
+        FileUtils.mv resultset_path, "#{resultset_path}#{name}.final"
+      end
+
+      def clear_mergeable_reports(*names)
+        SimpleCov.clear_result
+        SimpleCov::ResultMerger.clear_resultset
+        FileUtils.rm resultset_path
+        FileUtils.rm "#{resultset_path}.lock"
+        names.each { |name| FileUtils.rm "#{resultset_path}#{name}.final" }
+      end
+    end
+  end
+
   # Normally wouldn't test private methods but just start has side effects that
   # cause errors so for time this is pragmatic (tm)
   describe ".start_coverage_measurement", if: SimpleCov.coverage_start_arguments_supported? do
-    before :each do
+    after :each do
       # SimpleCov is a Singleton/global object so once any test enables
       # any kind of coverage data it stays there.
       # Hence, we use clear_coverage_data to create a "clean slate" for these tests

test_projects/faked_project/Rakefile

@@ -8,3 +8,20 @@ Rake::TestTask.new(:test) do |test|
   test.test_files = FileList["test/**/*_test.rb"].sort
   test.verbose = true
 end
+
+Rake::TestTask.new(:part1) do |test|
+  test.libs << "lib"
+  test.test_files = FileList["test/**/*_test.rb"].sort
+  test.verbose = true
+end
+
+Rake::TestTask.new(:part2) do |test|
+  test.libs << "test"
+  test.test_files = FileList["test/**/*_test.rb"].sort
+  test.verbose = true
+end
+
+task :collate do
+  require "simplecov"
+  SimpleCov.collate Dir["coverage/resultset*.json"]
+end