undefined | Better HN

0 pointsspinax4y ago0 comments

> You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs.

It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs.

So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon).

It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick.

> (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)

I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")

0 comments

_d7dt4y ago

If you want to use journald for those programs you could just configure them to pipe the logs there, or you could just disable journald logging and have it pipe its logs to the syslog. If your distro didn't configure all those programs to log to the same place, that's more of a distro configuration problem than a problem with any specific syslogger. I personally dislike having a bunch of services that try to implement their own log rotation, I would rather have that handled by the system.

spinaxOP4y ago

System Ops (at scale, large company, lots of teams) in a nutshell: you did not build the system or choose what kind, it was most likely installed using vendor defaults by the systems owner and you were called in because it's misbehaving. Most likely 50 other people have touched various parts of it, some with skill some without.

"Give me an IP, username and password - what OS is it?" are about all you start with and go from there. It's probably a critical system to someone, and everyone swears on a stack of bibles that nobody did anything, touched anything or made a change. You have very specific domain knowledge (kernel, grub, SAN/storage, systemd, dbus, etc.) and typically ask a lot of questions to the systems owner as your fingers are flying ruling out reasons (low hanging fruit common issues).

zxzax4y ago

Ok so complain to the ops department that they need to unify their logs. That's their problem, not yours. If the company is big I would expect them to be doing that anyway, either they coalesce around a journald-type thing that aggregates the logs locally, or they'll use another centralized service like datadog, splunk, etc. Edit: If you are ops then this is your entire wheelhouse, you should be able to solve it at scale without messing everything up.

1 more reply