A Scholarly Markdown (opens in new tab)

Thanks, this is exactly what I was looking for. A simple syntax for documentation that is extensible with macros. I am glad AsciiDoc includes them, they are very useful.

I love AsciiBook. Everything but the name. If we ever get an amazing collaborative development tool like Github that has built-in AsciiBook rendering, this project would have been called "Scholarly AsciiBook (but we got Unicode support too I swear really)".

Edit: I was wrong about Github not supporting AsciiDoc

AsciiDoc is a nice project, but I think that pandoc's variant of Markdown has a lot of advantages over AsciiDoc for academic writing. Let's just compare support for math and citations, for example. Pandoc's citation support is output-format independent: you can write the citations, specify a CSL stylesheet and bibliography, and you'll get the same output in every format pandoc supports. In AsciiDoc, as I understand it, if you want automatic citation support you need to use LaTeX, and then you're limited to output in LaTeX and PDF. Similarly for math: pandoc actually converts your LaTeX math to MathML (for formats that like that) or to native Word equations (for docx); in AsciiDoc, as I understand it, your LaTeX math will work if you target LaTeX, but if you want MathML you need to put MathML in the source. If you want to target both, then maybe you need two different source documents?

I don't have much experience with AsciiDoc, but I've encountered other limitations in writing pandoc's AsciiDoc renderer. For example, I didn't see any way to include multiple paragraphs (or other block content) in an AsciiDoc footnote. That's a deal-breaker for many academics.

I also want to emphasize something that is often not mentioned in comparisons of Markdown and AsciiDoc. As John Gruber emphasizes in his Markdown documentation, Markdown emphasizes ease of reading in source format. AsciiDoc has different priorities, and it sacrifices the readability of the source document to get them. Here's an example involving nested lists, from the AsciiDoc manual:

    1. List item one.
    +
    List item one continued with a second paragraph followed by an
    Indented block.
    +
    .................
    $ ls *.sh
    $ mv *.sh ~/tmp
    .................
    +
    List item continued with a third paragraph.
    
    2. List item two continued with an open block.
    +
    --
    This paragraph is part of the preceding list item.
    
    a. This list is nested and does not require explicit item continuation.
    +
    This paragraph is part of the preceding list item.
    
    b. List item b.
    
    This paragraph belongs to item two of the outer list.
    --

Markdown equivalent:

    1.  List item one.
    
        List item one continued with a second paragraph followed by an
        Indented block.
    
            $ ls *.sh
            $ mv *.sh ~/tmp
    
        List item continued with a third paragraph.
    
    2.  List item two continued with an open block.
    
        This paragraph is part of the preceding list item.
    
        1.  This list is nested and does not require explicit item continuation.
    
            This paragraph is part of the preceding list item.
    
        2. List item b.
    
        This paragraph belongs to item two of the outer list.
    

Note that the AsciiDoc is at least as easy to write, perhaps easier, because you don't need to worry about indentation. But the Markdown source is more readable; the indentation makes clear the structure of the list in a way that mirrors how it would be displayed in a browser or on the page.

I have yet to see a good stylesheet for AsciiDoc's DocBook output for creating a decent PDF. Do you have recommendations? I love AsciiDoc, but am only using it to generate HTML right now.

Are there any good wiki softwares that use Asciidoc?

Although I'm involved in both these enterprises, let's not confuse their goals.

CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out.

Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).

CommonMark should be mentioned on the ScholarlyMarkdown frontpage. I thought it was some competing project.

Don't they already use LaTex?

Note: I'm the maintainer of this project

Basically this is forked because I've been wanting to see how much change to the AST is needed to include most of the academic-specific features. Since Pandoc's AST definition is a separate package from Pandoc itself and could possibly be a dependency of other projects, I thought it would be best to figure most of it out first and end up with just one proposal. Scholdoc has a much more limited number of input/output syntax, so it has much more flexibility when it comes to adding new document element types.

Consider this a self-motivated skunkworks project for Pandoc.

This is a fork because making these changes to pandoc itself needs a lot of consideration.

Internal referencing and attributes on figures are two things that are currently being discussed for pandoc. The discussion has been going on for quite a while though - hence people making forks.

Discussion on internal referencing: https://github.com/jgm/pandoc/issues/813

Discussion on image attributes: https://github.com/jgm/pandoc/issues/261

This was my first thought as well. Pandoc has a rather excellent infrastructure in place to let you extend and build on it, so if you instead choose to fork, there better be a good reason...

I'm in fundamental disagreement with the principle behind CommonMark. They prioritize standardization over practical usability, and for that reason people will keep splintering markdown.

For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown. Then, because a lot of people actually need this feature, they still have to patch or extend or plug-in the functionality into any implementation of CommonMark, leading to fragmentation again. And frankly, the reason that I start writing more markdown is precisely because of the syntax highlighting ability. Oh, and to write scientific articles, math formulas are a deal-breaker.

They want to have a standard, unambiguous syntax specification, a suite of comprehensive tests and a cleanly implemented parser. They want to unify the community of markdown users and developers. All of those are commendable goals. But at the end of the day, it doesn't satisfy my needs, so I'd rather use a messy, poorly specified markdown flavor or even just render the markdown with GitHub's service.

Wow, I've never seen CommonMark -- hope that comes into common use over everything else!

The way I see it, the main feature is being able to plug math formulas in, rather than formatting documents. It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics.

Alternatively you can use org-mode and extend the document with latex where appropriate.

I read your link and, per my main comment, I'm pretty sure AsciiDoc answers 99% of your needs.

http://www.methods.co.nz/asciidoc/#_overview_and_examples

I've just updated the website to include a link to a repo where I am putting all the examples I have permission to share.

https://github.com/timtylin/scholdoc-examples

More will be added as I tweak the syntax. My hidden goal is to add enough to ScholarlyMarkdown to be able to reproduce around 70% of ArXiv.

Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.

Another thing I'm curious about is how the ScholarlyMarkdown authors would go about trying to get papers formatted in this way to go into a database like PubMed but that's pretty far down the track, and not addressed by any of the other solutions either.

I've actually been wondering if this is possible from a theoretical standpoint. I'm thinking you can use Pandoc's LaTeX to MD conversion mode, save changes to a copy, wdiff to get the total change set, then somehow convert that into a LaTeX patch. Whether you can guarantee that this won't break anything is going to be a really challenging problem, as my head is already swimming with edge-cases. I guess we'll know the day that some genius comes up with automatically generated non format-breaking Critic Markup diffs. At least the problem should be easier than LaTeX --- look how long it took for the latexdiff tool to be what it is today, and it still breaks often when you have anything slightly exotic.

I'm currently using a workflow in my thesis where I use ScholarlyMarkdown to write individual chapters for final inclusion into an existing LaTeX book document. I find that ScholarlyMarkdown works quite well this way, and it potentially allows collaborators, since individual parts are isolated.

That's probably never going to happen (for any Markdown-based project).

As you probably know, TeX is a full programming language (albeit an odd one), which means that to compile (La)TeX to Markdown, Markdown needs to support all the features of TeX, which means that it has to be Turing-complete. Not to mention the fact that parsing TeX is kind of nuts too.

My guess is that he is building the PDFs through latex/xetex.

I built something similar to that, but instead of forking Pandoc (and investing the time to learn Haskell), I just added filters through Python. It's not at fast, but you can intercept mostly anything.

Thus, the flow ends up as:

paper.md -> Pandoc -> paper.json + metadata -> paper.tex -> paper.pdf

Yep, that's the idea. Published work today should be able to take advantage of modern screens and its ability to resize, reflow, and animate.

MediaWiki syntax is incredibly grody ('''bold''' / ''italic'', tables, template logic, etc), and has exactly one implementation in common usage (MediaWiki itself -- written in PHP, and horribly convoluted), and has no formal specification. The diff feature you're referencing as a "syntax-highlighted diff" is a simple diff of the source -- it's entirely ignorant of formatting, and the XML export is similarly just an export of the raw source.

Nobody likes MediaWiki syntax, not even its own users. It's awful. The only reason it's still used at Wikipedia is because there's too much content to reasonably convert.