That's why I created tools to convert the output from different tools into a common CSV format that can be databased and used to compare output from different tools, or from different versions of the code (e.g., after fixing errors reported by the tools).
These tools currently work with cppcheck, clang and PVS-Studio and can be found here: http://btorpey.github.io/blog/categories/static-analysis/
Personally, I'm happier with plain old text files that can be manipulated with awk, grep, etc., can be databased if needed (since they're csv files) -- and can also be compared using my all-time favorite software, Beyond Compare. (http://btorpey.github.io/blog/2013/01/29/beyond-compare/).
"Overall, the error trace found by Infer has 61 steps, and the source of null, the call to X509 _ gmtime _ adj () goes five procedures deep and it eventually encounters a return of null at call-depth 4. "
I think the example Amazon gave for TLA+ was thirty-something steps. Most people's minds simply can't track 61 steps into software. Tests always have a coverage issue.
>. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of both mobile (Android and Objective C) code and backend C++ code
> All codebases see thousands of code modifications each day and our tools run on each code change. For Zoncolan, this can amount to analyzing one trillion lines of code (LOC) per day.
11 "missed bugs" on the 100 mm server-side lines of code per run, or ever?
I think this is where languages with stronger inbuilt analysis (e.g. Rust) win: The results are better, and since the analysis is always running as part of a compiler pass there are no huge jumps in indicated bugs at once (like what would happen if one would run Coverity on a legacy C++ codebase).
> We also use the traditional security programs to measure missed bugs (that is, the vulnerabilities for which there is a Zoncolan category), but the tool failed to report them. To date, we have had about 11 missed bugs, some of them caused by a bug in the tool or incomplete modeling.
A missed bug is presumably one that the tool is designed to spot, but which it didn't during the period in which it has been running.
Hopefully Software Heritage (https://www.softwareheritage.org) will help with that.
Edit: It worked again right after I posted this comment.
