undefined | Better HN

0 pointsdijit5y ago0 comments

Not sure if joking (since, sysvinit is very old and was in need of replacement) so I'll answer in good faith.

> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?

"oh my process failed; let's add `-x` to the invocation."

> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?

as above.

> One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?

`dmesg` or /var/log/messages (or /var/log/dmesg)

0 comments

theamk5y ago

Puzzle level 1) ini file:

you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?

I would not call this "easy" or "solved". Possible, yes. Pleasant, no.

Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.

Puzzle level 3)

Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

The sysvinit recommended way is to call things like "log_failure_msg" and "log_warning_msg", and those only print to stdout, which is the console. That's it. Gentoo's OpenRC does claim to have boot console logger, but it was broken every time I tried to use it.

Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.

dijitOP5y ago

> you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?

I mean, yes? Granted I'm not a developer but printing out exactly what's happening line by line with all variables expanded and all logical conditions printed is awesome, it's something we try to reverse engineer into programs with tools like bpftrace and strace.

Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.

> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

Actually, that's the opposite. A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.

> Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.

Console scrollback is "nice to have", personally I'm very thankful that it does not use memory keeping things there because there is the odd regression when you configure something (like when you tell IPtables to log deny's: that also gets printed to console.)

re----gardless. I think we're talking passed each other. I had no real issue as a sysadmin with a fully fledged bash, because I can debug it. I get that there's issues with that and most people do not like it- I'm not advocating for it, just saying that what you presume to be "impossible" was somehow solved for decades, and quite competently. It may not be pleasant to you but the alternative is not pleasant for me.

The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.

With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).

Anyway, I'm getting ahead of myself.

If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)

theamk5y ago

Are you perhaps confusing something?

> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.

The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)

To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:

                if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then
                    #   Announce the local time.
                    verbose_log_action_msg "System Clock set to: `date $UTC`"
                else
                    log_warning_msg "Unable to set System Clock to: `date $UTC`"
                fi

This version has been around since at least 1998 [0]. Old school enough for you?

So here is a simple question: let's say my hwclock chip/driver is unreliable and it did not work on startup for some reason, so that "hwclock" call is failing. My kernel timezone is not set right, the clock is off, and the system does log_warning_msg to warn you about it. How do I find out about this warning?

To make your job simpler, here is how the how this function is defined on my system:

    if log_use_fancy_output; then
        YELLOW=`$TPUT setaf 3`
        NORMAL=`$TPUT op`
        echo " $YELLOW*$NORMAL $@" || true
    else
        echo " * $@" || true
    fi

Note no dmesg here, no syslog, no other loggers. You can say "bootlog", but it has been broken every time I tried to use it.

Now, nothing is impossible! I can hack this script so it writes to a temporary file. I can re-execute it again, and hope it fails again. But this gets old, you know? I can spend all day debugging early boot shell, but I did it for a long time already, so now I just want to type "systemctl show --failed", see all the failed services, fix it quickly, and move on to more interesting stuff.

[0] https://www.mail-archive.com/debian-bugs-dist@lists.debian.o...

1 more reply

j / k navigate · click thread line to collapse

0 comments

theamk5y ago

Puzzle level 1) ini file:

I would not call this "easy" or "solved". Possible, yes. Pleasant, no.

Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.

Puzzle level 3)

Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

dijitOP5y ago

Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.

> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.

With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).

Anyway, I'm getting ahead of myself.

If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)

theamk5y ago

Are you perhaps confusing something?

> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.

The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)

To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:

                if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then
                    #   Announce the local time.
                    verbose_log_action_msg "System Clock set to: `date $UTC`"
                else
                    log_warning_msg "Unable to set System Clock to: `date $UTC`"
                fi

This version has been around since at least 1998 [0]. Old school enough for you?

To make your job simpler, here is how the how this function is defined on my system:

    if log_use_fancy_output; then
        YELLOW=`$TPUT setaf 3`
        NORMAL=`$TPUT op`
        echo " $YELLOW*$NORMAL $@" || true
    else
        echo " * $@" || true
    fi

Note no dmesg here, no syslog, no other loggers. You can say "bootlog", but it has been broken every time I tried to use it.

[0] https://www.mail-archive.com/debian-bugs-dist@lists.debian.o...

1 more reply

j / k navigate · click thread line to collapse