test: improve code coverage reporting (#4849)

* Speed up the generation of coverage reports by using multiple cores.

* Add codecov step to coverage workflow.

Author: Bronek

.codecov.yml

@@ -1,5 +1,6 @@
-
-codecov:
-  ci:
-    - !appveyor
-    - travis
+coverage:
+  status:
+    project:
+      default:
+        target: 60%
+        threshold: 2%

.github/actions/build/action.yml

@@ -6,6 +6,8 @@ inputs:
     required: true
   cmake-args:
     default: null
+  cmake-target:
+    default: all
 # An implicit input is the environment variable `build_dir`.
 runs:
   using: composite
@@ -26,4 +28,5 @@ runs:
         cmake \
           --build ${build_dir} \
           --config ${{ inputs.configuration }} \
-          --parallel ${NUM_PROCESSORS:-$(nproc)}
+          --parallel ${NUM_PROCESSORS:-$(nproc)} \
+          --target ${{ inputs.cmake-target }}

.github/workflows/nix.yml

@@ -49,7 +49,7 @@ jobs:
               cc: /usr/bin/clang-14
               cxx: /usr/bin/clang++-14
     runs-on: [self-hosted, heavy]
-    container: thejohnfreeman/rippled-build-ubuntu:12e19cd9034b
+    container: rippleci/rippled-build-ubuntu:aaf5e3e
     env:
       build_dir: .build
     steps:
@@ -125,7 +125,7 @@ jobs:
           - "-Dunity=ON"
     needs: dependencies
     runs-on: [self-hosted, heavy]
-    container: thejohnfreeman/rippled-build-ubuntu:12e19cd9034b
+    container: rippleci/rippled-build-ubuntu:aaf5e3e
     env:
       build_dir: .build
     steps:
@@ -159,3 +159,72 @@ jobs:
       - name: test
         run: |
           ${build_dir}/rippled --unittest --unittest-jobs $(nproc)
+
+
+  coverage:
+    strategy:
+      fail-fast: false
+      matrix:
+        platform:
+          - linux
+        compiler:
+          - gcc
+        configuration:
+          - Debug
+    needs: dependencies
+    runs-on: [self-hosted, heavy]
+    container: rippleci/rippled-build-ubuntu:aaf5e3e
+    env:
+      build_dir: .build
+    steps:
+      - name: download cache
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
+      - name: extract cache
+        run: |
+          mkdir -p ~/.conan
+          tar -xzf conan.tar -C ~/.conan
+      - name: check environment
+        run: |
+          echo ${PATH} | tr ':' '\n'
+          conan --version
+          cmake --version
+          gcovr --version
+          env | sort
+          ls ~/.conan
+      - name: checkout
+        uses: actions/checkout@v3
+      - name: dependencies
+        uses: ./.github/actions/dependencies
+        with:
+          configuration: ${{ matrix.configuration }}
+      - name: build
+        uses: ./.github/actions/build
+        with:
+          generator: Ninja
+          configuration: ${{ matrix.configuration }}
+          cmake-args: >-
+            -Dcoverage=ON
+            -Dcoverage_format=xml
+            -DCODE_COVERAGE_VERBOSE=ON
+            -DCMAKE_CXX_FLAGS="-O0"
+            -DCMAKE_C_FLAGS="-O0"
+          cmake-target: coverage
+      - name: build
+        shell: bash
+        run: |
+          mv "${build_dir}/coverage.xml" ./
+      - name: archive coverage report
+        uses: actions/upload-artifact@v3
+        with:
+          name: coverage.xml
+          path: coverage.xml
+          retention-days: 30
+      - name: upload coverage report
+        uses: codecov/codecov-action@v3
+        with:
+          files: coverage.xml
+          fail_ci_if_error: true
+          verbose: true
+          token: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/windows.yml

@@ -90,6 +90,7 @@ jobs:
           configuration: ${{ matrix.configuration }}
           # Hard code for now. Move to the matrix if varied options are needed
           cmake-args: '-Dassert=ON -Dreporting=OFF -Dunity=ON'
+          cmake-target: install
       - name: test (permitted to silently fail)
         shell: bash
         # Github runners are resource limited, which causes unit tests to fail

BUILD.md

@@ -262,12 +262,72 @@ It patches their CMake to correctly import its dependencies.
    generator. Pass `--help` to see the rest of the command line options.
 
 
+## Coverage report
+
+The coverage report is intended for developers using compilers GCC
+or Clang (including Apple Clang). It is generated by the build target `coverage`,
+which is only enabled when the `coverage` option is set, e.g. with
+`--options coverage=True` in `conan` or `-Dcoverage=ON` variable in `cmake`
+
+Prerequisites for the coverage report:
+
+- [gcovr tool][gcovr] (can be installed e.g. with [pip][python-pip])
+- `gcov` for GCC (installed with the compiler by default) or
+- `llvm-cov` for Clang (installed with the compiler by default)
+- `Debug` build type
+
+A coverage report is created when the following steps are completed, in order:
+
+1. `rippled` binary built with instrumentation data, enabled by the `coverage`
+   option mentioned above
+2. completed run of unit tests, which populates coverage capture data
+3. completed run of the `gcovr` tool (which internally invokes either `gcov` or `llvm-cov`)
+   to assemble both instrumentation data and the coverage capture data into a coverage report
+
+The above steps are automated into a single target `coverage`. The instrumented
+`rippled` binary can also be used for regular development or testing work, at
+the cost of extra disk space utilization and a small performance hit
+(to store coverage capture). In case of a spurious failure of unit tests, it is
+possible to re-run the `coverage` target without rebuilding the `rippled` binary
+(since it is simply a dependency of the coverage report target). It is also possible
+to select only specific tests for the purpose of the coverage report, by setting
+the `coverage_test` variable in `cmake`
+
+The default coverage report format is `html-details`, but the user
+can override it to any of the formats listed in `Builds/CMake/CodeCoverage.cmake`
+by setting the `coverage_format` variable in `cmake`. It is also possible
+to generate more than one format at a time by setting the `coverage_extra_args`
+variable in `cmake`. The specific command line used to run the `gcovr` tool will be
+displayed if the `CODE_COVERAGE_VERBOSE` variable is set.
+
+By default, the code coverage tool runs parallel unit tests with `--unittest-jobs`
+ set to the number of available CPU cores. This may cause spurious test
+errors on Apple. Developers can override the number of unit test jobs with
+the `coverage_test_parallelism` variable in `cmake`.
+
+Example use with some cmake variables set:
+
+```
+cd .build
+conan install .. --output-folder . --build missing --settings build_type=Debug
+cmake -DCMAKE_BUILD_TYPE=Debug -Dcoverage=ON -Dcoverage_test_parallelism=2 -Dcoverage_format=html-details -Dcoverage_extra_args="--json coverage.json" -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake ..
+cmake --build . --target coverage
+```
+
+After the `coverage` target is completed, the generated coverage report will be
+stored inside the build directory, as either of:
+
+- file named `coverage.`_extension_ , with a suitable extension for the report format, or
+- directory named `coverage`, with the `index.html` and other files inside, for the `html-details` or `html-nested` report formats.
+
+
 ## Options
 
 | Option | Default Value | Description |
 | --- | ---| ---|
 | `assert` | OFF | Enable assertions.
 | `reporting` | OFF | Build the reporting mode feature. |
+| `coverage` | OFF | Prepare the coverage report. |
 | `tests` | ON | Build tests. |
 | `unity` | ON | Configure a unity build. |
 | `san` | N/A | Enable a sanitizer with Clang. Choices are `thread` and `address`. |
@@ -362,6 +422,8 @@ If you want to experiment with a new package, follow these steps:
 [5]: https://en.wikipedia.org/wiki/Unity_build
 [6]: https://github.com/boostorg/beast/issues/2648
 [7]: https://github.com/boostorg/beast/issues/2661
+[gcovr]: https://gcovr.com/en/stable/getting-started.html
+[python-pip]: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
 [build_type]: https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
 [runtime]: https://cmake.org/cmake/help/latest/variable/CMAKE_MSVC_RUNTIME_LIBRARY.html
 [toolchain]: https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html