For Wordle GS, I’ve been trying to maintain decent unit test coverage of the core game logic. My approach has been to leverage Cpputest, gcov, & lcov both locally as well as on the Ubuntu GitHub Runner based CI builds. To simplify checking coverage on PRs, I added lcov-reporter-action to the mix. There was just one problem – the summary numbers in the comments left were clearly bogus. With overall coverage showing as NaN% (with a +/- change of NaN% to boot) and the totals for all the other metrics showing as 100% (despite the clear presence of uncovered lines), this wasn’t as usable as I’d hoped.
After a fair bit of working through the stack to narrow the problem down, it became clear that something was going on with the way
lcov was producing its output
lcov.info file. Locally, the file produced looked like the following:
TN: SF:/Users/.../wgs_dictionary.c FN:54,Dictionary_Create ... FN:145,Dictionary_GetWord FNDA:13,Dictionary_Create ... FNDA:13,Dictionary_GetWord FNF:10 FNH:10 BRDA:71,1,0,12 ... BRDA:149,1,1,13 BRF:16 BRH:14 DA:54,13 ... DA:153,13 LF:63 LH:63 end_of_record
However, when run on the GitHub Ubuntu Runner, the lcov.info file ended up looking like so:
TN: SF:/home/.../wgs_dictionary.c FN:54,Dictionary_Create FNDA:13,Dictionary_Create ... FN:145,Dictionary_GetWord FNDA:13,Dictionary_GetWord DA:54,13 ... BRDA:71,0,0,12 ... DA:153,13 end_of_record
While the ordering of lines differs between the two, the truly concerning thing is the complete absence of some key fields that would line up well with the problem:
|FNF||Number of functions found|
|FNH||Number of functions hit|
|BRF||Number of branches found|
|BRH||Number of branches hit|
|LF||Number of lines found|
|LH||Number of lines hit|
This led me to check versions, turns out I was running
1.15 locally and
1.14-2 on CI. I found a number of reports of versions prior to
1.14-2 having trouble with current versions of compilers; but, none which exactly lined up with the symptoms I’m seeing. However, switching from using the
apt package of
1.14-2 to building
1.15 from source did turn out to be the solution to my problem. After doing that, the comments are better aligned with reality.
I still need to set up caching; however, this is my current approach on my actions
jobs: run_unit_tests: name: Run Unit Tests runs-on: ubuntu-latest timeout-minutes: 10 steps: ... - name: Install lcov run: | wget https://github.com/linux-test-project/lcov/releases/download/v1.15/lcov-1.15.tar.gz tar -zxf lcov-1.15.tar.gz cd lcov-1.15 sudo make install ...