Discussion about this post

User's avatar
Amit Spitzer's avatar

The number worth sitting with in the AWS item is the 37.8 percent, not the 89. Passing the benchmark's own test suite only proves the patch didn't break the build, not that it closed the specific hole the exercise was built around. I ran into the same split a few weeks back looking at Anthropic's own vulnerability count, the headline discovery number and what actually got fixed turned out to be two different claims wearing one number. Anyone buying Continuum or anything like it should be asking for that stage two metric by default, not the end to end pass rate vendors lead with.

richardstevenhack's avatar

"decades of existing security technical debt"

Actually, decades of "software engineering" debt, as I've been saying for twenty five years or more.

Ready for more?