The four pillars of data observability: metrics, metadata, lineage, and logs (opens in new tab)

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs?

For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I've seen understand anything beyond text. So I then have to build my own UI or open the images in turn.

Is there a pattern to handle this?

Seems like the the key pillars are: freshness, volume, schema, distribution, and lineage.

Makes more sense this way, I think...

If you think about metrics, traces, and logs (software observability pillars) as three distinct things, it's hard to view metadata separate from metadata, lineage, or logs. Metadata is kind of the glue that holds everything together.

This article has more relevant sources, IMO, even if it is from a SaaS vendor.

https://www.montecarlodata.com/blog-what-is-data-observabili...

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments.

I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me.

YMMV.

There's not enough information shared about what metrics people monitor and why. Accurately measuring complex system performance and utilization over time is hard work! Observability platforms such as DataDog and New Relic are very expensive. Engineering teams repeat mistakes, recording too much information, getting the bill, and rushing high-priority pruning exercises through engineering to stop the financial bleeding. I encourage everyone to share their monitoring setups!

nobody needs this new saas stuff. I prefer the traditional pillars of: emails from users, live chat feature in product where users shame you publicly if something is wrong, twitter search 'is X down', and 4) having laptop open in passenger seat on commute, tethered to blackberry, and periodically hitting F5 on the page which hits the most APIs