Monitoring
We test the service, not the health page. A monitor that only asks a system whether it feels well will report green while every real request fails — we have seen it happen, and we build against it.
- Real RPC calls against every endpoint we serve
- Block-height lag compared to the network, not to ourselves
- Email to the on-call address on the first confirmed failure, once per incident and once when it clears
- A separate watchdog checks the monitor itself every ten minutes, restarts it, and escalates if that does not help
- No pager or SMS channel yet: if nobody reads email at 03:00, nobody is woken. We would rather write that here than imply otherwise