Core dump epidemiology: fixing an 18-year-old bug

OpenAI discovered crashes in Rockset, a C++ service that helps ChatGPT search for relevant data, where programs would mysteriously stop after functions returned to invalid memory addresses. After investigating like epidemiologists and analyzing crash data patterns, the team discovered two unrelated bugs: silent hardware corruption on an Azure server where the CPU made math errors, and an 18-year-old race condition (a bug where two processes interfere with each other) in GNU libunwind, a widely-used open source library for tracking function calls.