Debugging Under Fire: Keeping Your Head when Systems Have Lost Their Mind.

As software is increasingly developed to be deployed as part of a service, the manifestations of defects have changed: the effects of broken software are increasingly unlikely to be felt by merely one user, but many (or even all) — with concomitant commercial consequences. Debugging service outages put everyone in an uncomfortable spot: operators must learn how to deal with the uncertainties of broken snowflakes, while developers must adapt their debugging techniques to the constraints of a production environment. And in all but the most immature systems, service outage denotes cascading failure: there is not one bug but several — often in different components that interact in unforeseen ways.
These technical complexities, coupled with the high visibility of a downed service, can lead to stress, confusion, and (in the worst cases) panic.

In this talk, we will address debugging during an outage, looking at specific technical challenges (and techniques to address them) and the psychology, team dynamics, and organizational challenges of debugging under fire.

Bryan Cantrill – Chief Technology Officer at Joyent