For Wordle GS, I’ve been trying to maintain decent unit test coverage of the core game logic. My approach has been to leverage Cpputest, gcov, & lcov both locally as well as on the Ubuntu GitHub Runner based CI builds. To simplify checking coverage on PRs, I added lcov-reporter-action to the mix. There was just one problem – the summary numbers in the comments left were clearly bogus. With overall coverage showing as NaN% (with a +/- change of NaN% to boot) and the totals for all the other metrics showing as 100% (despite the clear presence of uncovered lines), this wasn’t as usable as I’d hoped.

After a fair bit of working through the stack to narrow the problem down, it became clear that something was going on with the way lcov was producing its output lcov.info file. Locally, the file produced looked like the following:

TN:
SF:/Users/.../wgs_dictionary.c
FN:54,Dictionary_Create
...
FN:145,Dictionary_GetWord
FNDA:13,Dictionary_Create
...
FNDA:13,Dictionary_GetWord
FNF:10
FNH:10
BRDA:71,1,0,12
...
BRDA:149,1,1,13
BRF:16
BRH:14
DA:54,13
...
DA:153,13
LF:63
LH:63
end_of_record

However, when run on the GitHub Ubuntu Runner, the lcov.info file ended up looking like so:

TN:
SF:/home/.../wgs_dictionary.c
FN:54,Dictionary_Create
FNDA:13,Dictionary_Create
...
FN:145,Dictionary_GetWord
FNDA:13,Dictionary_GetWord
DA:54,13
...
BRDA:71,0,0,12
...
DA:153,13
end_of_record

While the ordering of lines differs between the two, the truly concerning thing is the complete absence of some key fields that would line up well with the problem:

Field Name	Purpose
FNF	Number of functions found
FNH	Number of functions hit
BRF	Number of branches found
BRH	Number of branches hit
LF	Number of lines found
LH	Number of lines hit

This led me to check versions, turns out I was running 1.15 locally and 1.14-2 on CI. I found a number of reports of versions prior to 1.14-2 having trouble with current versions of compilers; but, none which exactly lined up with the symptoms I’m seeing. However, switching from using the apt package of 1.14-2 to building 1.15 from source did turn out to be the solution to my problem. After doing that, the comments are better aligned with reality.

I still need to set up caching; however, this is my current approach on my actions yaml file:

jobs:
  run_unit_tests:
    name: Run Unit Tests
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:

...

    - name: Install lcov
      run: |
        wget https://github.com/linux-test-project/lcov/releases/download/v1.15/lcov-1.15.tar.gz
        tar -zxf lcov-1.15.tar.gz
        cd lcov-1.15
        sudo make install
...