undefined | Better HN

0 pointsdotancohen3mo ago0 comments

Yes.

When my database logging fails, I write a file that logs the database fail (but not the original log file).

When my file logging fails, depending on application, I'll try another way of getting the information (the fact that for file logging failed) out - be that an http request or an email or something else.

Databases fail, file systems fill up. Logging logging failures is extremely important.

0 comments

amluto3mo ago

And when that last way fails, what do you do?

I like to have a separate monitoring process that monitors my process and a separate machine in a different datacenter monitoring that. But at the end of the day, the first process is still going to do try to log, detect that it failed, try the final backup log and then signal to its monitor that it’s in a bad state. It won’t make any decisions that depend on whether the final backup logging succeeds or fails.

dotancohenOP3mo ago

I'm not working on anything life-critical, one additional layer of redundancy is all I budget for. If both my database is down and my local filesystem is full simultaneously, things have gone bad and I've likely got lots of other symptoms I'm going to find to direct me.

j / k navigate · click thread line to collapse