DTrace
Or: How I Learned to Stop Worrying and Love Debugging in Production
DTrace is a tool for querying both the kernel and userspace applications about what they are doing. Rather than waiting for a program to crash, or scrambling to sift through incomplete logs, DTrace allows you to ask nearly-arbitrary questions of running programs.
In 2019, I gave a talk at ETSA about the fundamentals of DTrace. It is available on YouTube: https://www.youtube.com/watch?v=9E6ov16tly8