undefined | Better HN

0 pointsscottlamb2y ago0 comments

The idea of structured logs is that every place in the code where you define a log message, you can throw in extra attributes (key/value) pairs.

As far as I can tell, the lnav feature you describing allows you to define a JSON log format with predefined fields. But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output or filter on it in the UI and no ability to see it in the SQLite virtual table. That's why I say lnav doesn't appear to support structured logs.

edit: oh, I missed `"hide-extra": false`! That significantly improves things! Still, I don't see a way to access it from the SQLite virtual table. I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. (I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.)

0 comments

tstack2y ago

> But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output

Properties in the log message that are not in the "line-format" are displayed below the message as key-value pairs. As an example, the bunyan[1] log format[2] has a few standard properties that are used in the "line-format" to form the main message. Then, as shown in this test[3] for this log[4], the remaining properties (even ones not mentioned in the format file) are shown underneath.

> or filter on it in the UI

Since they are part of the message as mentioned above, they can be filtered on.

> and no ability to see it in the SQLite virtual table.

The "log_raw_text" column in the table can be used to access the original log message from the file. So, you can use the JSON functions in SQLite to retrieve the value:

    ;SELECT log_raw_text ->> '$.repository' from bunyan

[1] - https://github.com/trentm/node-bunyan

[2] - https://github.com/tstack/lnav/blob/master/src/formats/bunya...

[3] - https://github.com/tstack/lnav/blob/master/test/expected/tes...

[4] - https://github.com/tstack/lnav/blob/master/test/logfile_buny...

scottlambOP2y ago

Thanks for this. When I switched my project to structured logging, I deleted its lnav log format file and was sad about it. I'll add one back now!

tstack2y ago

> I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format.

Have a look at the following comment on an issue that might be similar to what you're thinking of:

https://github.com/tstack/lnav/issues/1065#issuecomment-1602...

> I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.

Unfortunately, the "module-field" does not work for JSON logs at the moment. It's something I should really fix.

Ultimately, lnav has existed for almost two decades now and I use it every day. So, it's always seeing improvements. If you're having a problem with it, file an issue on github. I don't always get around quickly to fixing other folks feature requests / issues, but it tends to happen eventually.

Thanks.

j / k navigate · click thread line to collapse

0 pointsscottlamb2y ago0 comments

The idea of structured logs is that every place in the code where you define a log message, you can throw in extra attributes (key/value) pairs.

0 comments

tstack2y ago

> But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output

> or filter on it in the UI

Since they are part of the message as mentioned above, they can be filtered on.

> and no ability to see it in the SQLite virtual table.

The "log_raw_text" column in the table can be used to access the original log message from the file. So, you can use the JSON functions in SQLite to retrieve the value:

    ;SELECT log_raw_text ->> '$.repository' from bunyan

[1] - https://github.com/trentm/node-bunyan

[2] - https://github.com/tstack/lnav/blob/master/src/formats/bunya...

[3] - https://github.com/tstack/lnav/blob/master/test/expected/tes...

[4] - https://github.com/tstack/lnav/blob/master/test/logfile_buny...

scottlambOP2y ago

Thanks for this. When I switched my project to structured logging, I deleted its lnav log format file and was sad about it. I'll add one back now!

tstack2y ago

Have a look at the following comment on an issue that might be similar to what you're thinking of:

https://github.com/tstack/lnav/issues/1065#issuecomment-1602...

> I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.

Unfortunately, the "module-field" does not work for JSON logs at the moment. It's something I should really fix.

Thanks.

j / k navigate · click thread line to collapse