undefined | Better HN

0 pointsikeboy9y ago0 comments

You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.

0 comments

maxander9y ago

You may be able to build an elaborate system of dummy network operations to test with, but this system may wind up with bugs that mask what would be errors in the real system. And how to you test against that? A dummy network to test the dummy network operations on? What if the dummy network contains bugs that make it behave significantly different from the real network, in error cases? How do you test for that?

Its turtles all the way down!

ikeboyOP9y ago

You can test it against the actual network; if something goes wrong, you'll have downtime, but you'll be prepared to get it all back up.

Or, to test whether the "prevent errors from going to new places" works, temporarily configure the new places to ignore new configs; if the system works, no messages will be sent there; if the system doesn't work, they ignore the message and you learn about a bug.

dward9y ago

You mean an ICMP request? The IPs were anycast and did not become unreachable until all edge routers had stopped announcing BGP routes. At that point the failure was global. Check out the postmortem, it's a good read.

ikeboyOP9y ago

The servers became unreachable, but the IPs weren't unreachable until all the servers were reconfigured.

My test should have caught this bug:

> In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

j / k navigate · click thread line to collapse

0 comments

maxander9y ago

Its turtles all the way down!

ikeboyOP9y ago

You can test it against the actual network; if something goes wrong, you'll have downtime, but you'll be prepared to get it all back up.

dward9y ago

ikeboyOP9y ago

The servers became unreachable, but the IPs weren't unreachable until all the servers were reconfigured.

My test should have caught this bug:

j / k navigate · click thread line to collapse