Running lcov on a GitHub Linux Runner

For Wordle GS, I’ve been trying to maintain decent unit test coverage of the core game logic. My approach has been to leverage Cpputest, gcov, & lcov both locally as well as on the Ubuntu GitHub Runner based CI builds. To simplify checking coverage on PRs, I added lcov-reporter-action to the mix. There was just one problem – the summary numbers in the comments left were clearly bogus. With overall coverage showing as NaN% (with a +/- change of NaN% to boot) and the totals for all the other metrics showing as 100% (despite the clear presence of uncovered lines), this wasn’t as usable as I’d hoped.

After a fair bit of working through the stack to narrow the problem down, it became clear that something was going on with the way lcov was producing its output lcov.info file. Locally, the file produced looked like the following:

TN:
SF:/Users/.../wgs_dictionary.c
FN:54,Dictionary_Create
...
FN:145,Dictionary_GetWord
FNDA:13,Dictionary_Create
...
FNDA:13,Dictionary_GetWord
FNF:10
FNH:10
BRDA:71,1,0,12
...
BRDA:149,1,1,13
BRF:16
BRH:14
DA:54,13
...
DA:153,13
LF:63
LH:63
end_of_record

However, when run on the GitHub Ubuntu Runner, the lcov.info file ended up looking like so:

TN:
SF:/home/.../wgs_dictionary.c
FN:54,Dictionary_Create
FNDA:13,Dictionary_Create
...
FN:145,Dictionary_GetWord
FNDA:13,Dictionary_GetWord
DA:54,13
...
BRDA:71,0,0,12
...
DA:153,13
end_of_record

While the ordering of lines differs between the two, the truly concerning thing is the complete absence of some key fields that would line up well with the problem:

Field NamePurpose
FNFNumber of functions found
FNHNumber of functions hit
BRFNumber of branches found
BRHNumber of branches hit
LFNumber of lines found
LHNumber of lines hit

This led me to check versions, turns out I was running 1.15 locally and 1.14-2 on CI. I found a number of reports of versions prior to 1.14-2 having trouble with current versions of compilers; but, none which exactly lined up with the symptoms I’m seeing. However, switching from using the apt package of 1.14-2 to building 1.15 from source did turn out to be the solution to my problem. After doing that, the comments are better aligned with reality.

I still need to set up caching; however, this is my current approach on my actions yaml file:

jobs:
  run_unit_tests:
    name: Run Unit Tests
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:

...

    - name: Install lcov
      run: |
        wget https://github.com/linux-test-project/lcov/releases/download/v1.15/lcov-1.15.tar.gz
        tar -zxf lcov-1.15.tar.gz
        cd lcov-1.15
        sudo make install
...

Standard

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s