Show HN: Parsing horse racing charts with Apache PDFBox (opens in new tab)

PDFBox committer here, if you want even lower-level access to the page content stream, without anything 'clever' at all, check out the PDFGraphicsStreamEngine class, which is a superclass of the text extraction and rendering classes. Gives you access to the raw glyphs. You can override PageRenderer too, for visual debugging, e.g. render glyph bounding boxes. We have an interactive Swing PDFDebugger which does just that.

https://github.com/apache/pdfbox/blob/6f18d7c4bef4d23a22d/ex...

Yes I encountered similar issues but many of them were able to be solved.

With PDFBox I was able to deal with the content at a very low level (on a per-character basis), so that when for instance building a String, I would insert a pipe character when the distance between adjacent characters was greater than the width of the space character and then detect that when translating to a certain field.

See the convertToText() method for an example: https://github.com/robinhowlett/chart-parser/blob/master/src...

and https://github.com/robinhowlett/chart-parser/blob/f8d651e9a1... for when I used this technique

Huh, interesting. I was looking around for PDF libs previously and PDFBox didn't show up in google results. pdftk was the only one that showed up in Google results anywhere useful.

Edit: Looks like it's on the second page of results and I never made it that far, heh. Goes to show how biasing the first page of results is.

The keyword is "publishing" --- as in, producing human-readable physical copies, not electronic ones. It just so happens that the format was relatively suitable for the latter too (because it actually looks like a printed document rendered on the screen --- unlike HTML or other formats around at the time), which is why that use-case became popular. PDF is basically a descendant of PostScript, which was designed to control printers.

(Its PostScript origins may also explain the bizarre mix of text and binary that constitute the file format. For example, page contents are in a relatively free-form PostScript-ish RPN-like textual language, but are found in "content streams" which may be compressed or encoded into a binary format. Data "object" structures include things like '<<'-delimited dictionaries, '[' arrays ']', textual "/Names", and even provisions for comments(!?).

Then there are things like the cross-reference table of all objects in the file, which is an array of fixed-width textual numbers representing file offsets, e.g. "0000001056 00000 n" refers to something 1056 bytes from the start of the file. Reactions of WTF!? from those working with the format for the first time are not uncommon.)

PDF has a feature called Tagged PDF, which allows the document to be annotated with a semantic structure. Almost nobody bothers to generate such PDFs, but the support is there!

Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...

I came across https://github.com/pdfminer/pdfminer.six recently and was impressed with what it could get done. The documentation can be challenging to parse, so I relied on a code sample from a StackOverflow answer. Have you had a chance to try it out? Curious about how/if it works well across platforms.

Author here; well, PDFBox is good for simple text stripping. If I wanted to print all the text on the PDF, that would be very straightforward and not much code. However, the PDF chart here is in essence a representation of structured data. I wanted to get the content in that format so that I could both serialize to JSON plus have an SDK to boot.

Most PDF viewers I am aware of (including my own, PostView) have 2 up modes, are those not sufficient?