They contained the impact. Root causing or “understanding” should come after impact mitigation. If needed find a safe way to reproduce the bug without customer impact.
We reverted the refactoring, deployed, un-banned the CEO, and set about analysis.
It’s one backed by a lot of data. One example is from the Android project.
The percent of vulnerabilities caused by memory safety issues continues to correlate closely with the development language that’s used for new code. Memory safety issues, which accounted for 76% of Android vulnerabilities in 2019, and are currently 24% in 2024, well below the 70% industry norm, and continuing to drop.
https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html
There’s an argument that critical infrastructure software vendors are already meeting standards for basic, non-memory related items. Yes, there are other categories, but memory safety is one that’s harder to verify. Moving to memory safe languages is an ensure a category of correctness. This excludes usage of unsafe escape hatches.
I’m going to whoosh the joke for a moment.
Avoid allocating memory in sig segv handlers. I worked at a company once that had written their own handlers that tried to print a stack trace. I was fortunate enough to find a machine in a test environment that segfaulted while allocating memory. The handler then caused a deadlock in a call to malloc (through new). I ended up rewriting the handler to fix two sources of memory allocation. First I preallocate enough space to do string manipulations in the handler. Second I made eager calls to a few libc functions used in the handler to avoid memory allocation from lazy binding from ld.so.