It still leaves the "I have to do a bunch of work beyond simply scrolling down to read this discussion" problem. I don't like solutions that put excessive burden on the largest audience of anonymous readers. Write once, read many, and all that.
I don't see how collapsing them all to a linear discussion would help this: you simply get one massive topic overtaking the entire conversation, making it impossible to even have the second discussion at all... if the content doesn't get posted because it is off-topic in the linear model, no amount of scrolling will find it.
Again, though, this is why almost no sites are actually a linear model; the closest you seem to get to a linear community currently is in the medium of real-time chat, where you still have servers divided into channels, but the channels are relatively fixed and difficult to browse.
The result is that you have communities existing within a single server/channel; as an example, there is a community which I started on an IRC server (not iPhone hacking, btw: a bunch of random friends and acquaintances that has grown and shrunk at various points over time, mostly concentrated with people form the small college that I attended) that has existed for years with a single linear thread.
Real-time chat, though, is a drastically different kind of medium: handling multiple real-time conversations is simply hard in a way that having multiple asynchronous conversations is not. I maintain, then, that this is why almost no discussion community has ever had a truly linear model: content tends to fall into 4-5 levels of hierarchy, with at least two of them being under direct user control.
Once you expand the scope of where you look for the tree in this way, you then note that StackOverflow is actually more threaded than classic discussion forums: of the five levels it has (sites, tags--which I will again explicitly point out is just another hierarchy level where the entries in lower levels might be seen in multiple places--, questions, answers, comments), four of them are under immediate user control, and the fifth is decided by a largely democratic process.
In practice, Hacker News threads sometimes get really deep (and I again agree that they can get confusing down there), but the very top-level is sufficiently shallow that you need some of that depth below to have any interesting conversations: if there were only one linear conversation per posted link, things would rapidly get really boring when only one of the politics/economics/etc. angles was able to be covered at a time (by which point the others would have missed their window).
Hacker News, does, though, actually have a form of thread limiting; I didn't go into it in detail in my earlier post, but I'll go ahead and do so now: as you get deeper into a thread, the "reply" button takes longer and longer to appear. This mechanism is actually defeat-able (which is funny, and some might argue a bug: you click "link", and on the comment's own page you can always reply immediately), but it still does help slow the rate at which a thread can get very deep.
Pretty much every forum on the web is a flat discussion. There are a lot of forums out there. It is quite rare to find a forum that has any kind of threading at all. If you can point me to any web discussion communities that are 10+ years old which use threading, I'd love to see them.
Now, you could argue that lots of topics on a forum is a "kind" of threading, and I guess if I squint really hard I could see it, but that's a far cry from here or Reddit, where it's indentation city all the time.
> it still does help slow the rate at which a thread can get very deep.
I think that's too clever by half, but more than that, nowhere near as effective as an explicit cap on reply depth. Hacker News would be a better discussion system for everyone if the reply depth was capped at, say, 3.
(And of course if it had the ability to collapse branches, which is sorely needed. Nothing is more frustrating to me than being forced to read through 50+ replies about some dumb tangent I don't care about, to possibly get to something I do care about.)
And now that I've written the above, I realize that there might still be some awesomely insightful reply in that 50+ branch on politics -- but I'd never, ever see it because it is forever married to its place in the tree and the iron-clad parent child reply hierarchy... and cannot be sorted independently on its own merits and votes. That's sad.
Yes, not only could I, but I have in all of my posts here so far, very explicitly. ;P A "classic web forum" has four levels: categories, forums, threads, and posts. It is critical to look at a web forum as having all of these levels if you want to compare it to a site like HN, as otherwise it is impossible to discuss how end users are able to navigate the site.
As an example, let's say that we were going to talk about your article on a normal web forum. First of all, that forum may already be divided into "on-topic" and "meta" forum categories, and there may then be multiple forums that are interested in your talk: the conversations don't conflict.
However, even within the same forum on the site (which will often then have the same general participant pool), your article will spawn multiple unrelated threads, each of which will consist of linear posts. Each of these threads will be discussing different areas of the topic, and you can ignore the threads you aren't interested in.
Hacker News can't do that: your article can only create a single top-level link, and all conversation has to exist underneath that. You get the same levels near the top (although managed in a distributed collaborative fashion by the users, as opposed to directed by the site operators), but additionally as you drill down it might get very deep.
... and, once again, I will agree with you that HN has a problem managing the discussion near the leaves of the tree. (FWIW, I actually was not allowed to reply to your post, but decided "screw that" and did it anyway: it could be nice to see something linear once you get this deep instead of just turning off "reply" altogether.)
> If you can point me to any web discussion communities that are 10+ years old which use threading, I'd love to see them.
In addition to classic web forums, which I maintain are the typical 4-5 level hierarchical tree, you will find websites like Slashdot, which is definitely a 10+ year old web discussion community, and is insanely threaded with tons of awesome mechanisms to handle that complexity (such as the feature of "bubbling up" good comments I mentioned elsewhere in this thread).
> but I'd never, ever see it because it is forever married to its place in the tree and the iron-clad parent child reply hierarchy
The solution reddit has for this--both in the commonly-used Reddit Enhancement Suite and as a feature you can buy as part of reddit Gold--is to mark new sections of the tree in an obvious way (such as with a blue box). I believe DISQUS does this as well now, and I have a similar feature in the HN script I sometimes run. In a system like USENET, you would have the option of resorting the messages in your client by date.
However, I maintain that a purely linear discussion model doesn't solve these problems: it just makes the content not exist in the first place. If you were to build even just a two-level system--links and posts; to contrast with a class web forum's four levels--which on HN would look like each article discussion being a linear chat, you could only have one of these conversations, as the interleaving would be crazy.
There are virtually no such communities because 10+ years ago people were still having serious conversations on mailing lists and USENET. Those were threaded and had specialized clients allowing to easily find new posts, replies to own posts, filter content, etc.
The great improvement web brought wasn't flatness but post-moderation, editing and later karma/voting/sorting. You simply couldn't moderate articles after they were posted on USENET. And editing ("superseding"?) mostly caused confusion over what your readers are seeing depending on the server and software they're using.
