Production Monitoring

Key idea: An offline test set only checks what you thought of. Watch real traffic, add what fails to the test set, and fix it so both scores agree.

Offline score vs. live score

v1v1 + failures in test setv2v2 + failures in test set
0%25%50%75%100%offline = live0%25%50%75%100%Offline test set scoreRun v1 to plot it here
  1. Run v1. The offline test set says 8/8. What does the live week say?
  2. Look at the live misses in the results. What do they have in common?
  3. Turn on Add this week's failures to the test set and run v1 again. Does the offline score finally show the problem?
  4. Switch to v2 and run. Do the offline and live scores agree now?
  5. Which day looked worst on the dashboard? How long would you have shipped broken routing without it?
Test set8 tickets written offlineLive traffica week, 28 ticketsRouterpicks a teammistral-smallTeamwhere it wentReviewerlabels the right teamDashboardoffline vs. live score
Press Run to watch it run

Results

Run it to see the offline and live results.

Behind the scenes

Run it and every ticket, offline and live, will show up here with where it was routed.