Personal Notes

Random thoughts, mostly about systems work.

On read latency budgets

When the read SLO is p99 < 50ms, the network round-trip alone often spends a third of that. I’ve been measuring our internal cache hits and the long tail is dominated by GC pauses, not the network.

Reverse proxy timeouts that bite

An upstream that closes its side first leaves a half-open socket sitting in FIN_WAIT_2 until the kernel’s timeout. Set tcp_fin_timeout sanely or your conntrack table fills up.

Boring tech beats novel tech

The best technical decisions I’ve made this year were choosing the older, less interesting option. The novel one always promises more and bills more.