A Social Filesystem (opens in new tab)

There was never any such incentive. MS Office formats were undocumented for years because MS had no incentive to document them. Merely using files did not help at all. Actually the vast majority of all file formats have never been open. Think about all the custom file formats used by video games, for instance.

I think it's on open social apps to show that they're actually meaningfully better products, and that is possible because they're open. With luck, this may lead to an ecosystem where it's worth staying compatible and interoperable, and where users scoff if someone is trying to break it, and where users have an easy way to walk away. I know this sounds super idealistic but this did essentially happen with open source over a long time. At some point, people were just as skeptical of open source as we might be about open social.

It’s a long road but the plan is to standardize AT. See https://docs.bsky.app/blog/taking-at-to-ietf for ongoing first steps.

>I’m somewhat concerned that the “file system” or the storage where all of our things are supposed to be stored is now suddenly in the cloud. We actually have a real file system … it backs itself up on iPhone even.

I don’t know how this is related to what my article talks about. The data we’re taking about is already public and in the cloud, it’s just locked into specific app’s databases.

The point isn’t about some single specific app, but about creating an ecosystem where the things you create (StackOverflow posts, Reddit discussions, Instagram follows, Letterboxd recipes) are not tied to the fate of that particular app and the whims of its creators. Then, if a service shuts down or turns to shit, developers can compete without solving the cold start problem. You just need to make a better app for same data, not convince everyone to recreate everything on your app.

Thanks! I saved a link to pdsls. I think there's room for improvement in making the UI user-friendly; maybe I'll try it someday.

This was a great article. Have you got any thoughts on Farcaster and its Snapchain?

fwiw, I don't agree. These detailed articles are excellent.

AFAIK nothing special for it at this point, so whatever you would do with a public cleartext broadcasting platform.

What’s the plan for this? Sharing with a specific group (DM, group chat, feeds from people you follow) are critical to social media.

Sounds interesting, but I can't find any links to an actual live B2C storefront..

This is a nice introduction:

https://openship.org/about

Yes, I should have been clearer. Reddit and Instagram do not operate this way today, but open social alternatives to them could. The idea is that people create personal websites where posting, commenting, and other social actions live, and that becomes the filesystem they own.

Open social networks would simply index or pull from those sites using agreed-upon lexicons and protocols. Existing platforms could either adopt the open social model, or, more realistically in the short term, be treated as syndication targets where posts are pushed via their APIs when someone publishes on their own site.

Except that a lot of the app builders in ATProto seem to think the protocol was designed to make their lives easier when bootstrapping their network from Bluesky userbase.

(Imo, that is a perverse interpretation, it's about user choice, which they are effectively taking away from me by auto importing and writing to my Bsky graph)

re: the debates on reusing follows from Bluesky in other apps instead of their own

For me, part of it is that we have no power collectively against products turning their back on users because coordination to "export data all at once and then import it into specific other place" is near-impossible. So this creates a perverse cycle where once you capture enough of the market, competition has very little chance unless they change the category entirely.

What AT enables is forking products with their data and users. So, if some product is going down a bad road, a motivated team can fork it with existing content, and you can just start using the new thing while staying interoperable with the old thing. I think this makes the landscape a lot more competitive. I wrote about this in detail in https://overreacted.io/open-social/#closed-social which is another longread but specifically gets into this problem.

I hear you re: not wanting "weird aggregation", that just be a matter of taste. I kind of feel like if I'm posting something on the internet, I might as well have it on the open web as aggregatable by other apps.

But what if your entire social graph didn't choose to transfer over as well? What if they don't want to be on that app? What if someone that was very indecent made a compatible app? Would you want your entire Twitter history represented on there?

For better or worse, I don't think it makes sense to decentralize social. The network of each platform is inherently imbued with the characteristics and culture of that platform.

And I feel like Twitter is the anomalous poster child for this entire line of thinking. Pour one out, let it go, move on, but I don't think creating generalized standards for social media data is the answer. I don't want 7 competing Twitter-like clones for different political ideologies that all replicate each others' data with different opt-in/opt-out semantics. That sounds like hell.

But.... why ? Is this assuming a huge portion of the people you interact with on Twitter are also all moving identities to the new platform ?

I can see for something like artistic expression it being useful to export and and move over to a new platform (like moving a whole photography portfolio, short story writing samples, or similar). But who cares about your "likes" on a totally new and separate social media platform ?

I would not call the people who ran Twitter before Elon Musk bought the company and took it private "decent people". I think that Musk purchasing the company and running it in a way that a lot of previous userbase objected to was ultimately a huge boon for software freedom - because without that, the large number of people who stopped using Twitter and went to the ATProto ecosystem instead would have been happy to continue using completely-proprietary Twitter. A lot of people were suddenly and viscerally faced with the downsides of building a digital "home" on someone else's platform.

There's also a risk of adversarial cross-site syndication: your stuff can and probably will show up on websites you don't control.

That's just how it works and I accept the risk.

People concerned about that probably shouldn't publish on Bluesky. Private chat makes more sense for a lot of things.

Trust is subjective! Let's establish trust in each other rather than rely on one-size-fits-all solutions.

Personally, I trust my friends, family, and some public figures and institutions to varying degrees. I want to see social experiences that reflect that.

I trust person A, person A trusts B

When I browse to random site, I can see in my browser that A's first level contact trusts this site.

Now I can make a decision based on the amount of trust I have on A. Maybe after exploring the site I can mark specific pages or the whole domain as trusted, so people in my network can see the same.

On a larger level I might trust the Country of Finland, who will only mark their official sites as Trusted. This way I instantly know if I'm on an official site or something pretending to be one.

But you don’t need the AT protocol and lexicons to describe semantics. Ontologies have been a thing for decades, and search engines (the “rich results”) and ActivityPub use them.

Why use the lexicons you described, instead of the more supported https://www.w3.org/ns/activitystreams ontology that ActivityPub uses, for example?

If you allow me to, I would be interested in re-writing your article but by using ActivityPub concepts instead. Maybe I will learn about a fatal flaw in one of the protocols during that process.

> the expressed preference

Marketing and culture can do wonders to one's preference.

> synonym for capability Yes and no. An "app" is usually a set of capabilities, not a single capability. Many of the core *nix utilities for example are specifically designed to have a single capability each, and these are then composed for complex requirements. But for the most part today's apps contain multiple, usually overlapping capabilities. For example Microsoft Word offers not only text editing and formatting, but also has a dictionary+thesaurus, spelling+grammar check, mail merge and lots more. Of course, you can abstract this all away as "word processor capability", but there are other word processing apps out there with varying capabilities, and so things become problematic fast especially in terms of interoperability.

> OpenDoc and OLE These "never worked" due to people/policy, not due to tech ability. OpenDoc essentially stagnated at Apple due to conflicts and lack of clarity, and then was outright killed by Steve Jobs[0]. OLE(tm) hasn't failed per se, but it's proprietary to Microsoft[1] and thus doesn't have much adoption outside Windows.

> Brand names matter All they really do is put a label on a bundle - or bundles - of capabilities to create handy associations. YouTube is associated primarily with accessing videos that can be made and uploaded by anyone, secondarily allows commenting on and liking said videos, etc. Again, marketing is a significant factor.

> a built-in feature You seem to be missing the forest for a tree here. That's a single example, of infinite permutations. Also, even given that example you're still missing parts of the flow.

[0] https://en.wikipedia.org/wiki/OpenDoc [1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding

I'm saying it would be a lot simpler to just provide an API to download the data. I don't think it's worth the complexity and network of PDS' to preserve this data in perpetuity, but even if I did, nothing would stop me from using the download API and storing tapes in Iron Mountain or whatever.

I think there's also weird cases where you don't really want things replicated: revenge porn, snuff, CSAM, etc. Like, if someone we're like "yes it's true we lose all data forever if we shut down, but we're not indiscriminately pushing some of the worst content to everyone replicating" I'd take that tradeoff immediately.

RSS is signed because it's served over HTTPS which involves an SSL certificate exchange. It is also real time. You fetch the feed and get the most up to date content on it. You can host any XML document you want on your website and there are W3C recommended standards that allow the integration of rich data structures within web pages.

The sole interesting part of ATProto is the aggregation part, which can also be solved in the same way with existing web technologies, that is a web indexer. Google allows you to set up notifications whenever a link is created to your website, which is essentially the unique offering of AT. None of the ideas it presents are new, but they are all incompatible with existing Web technologies. That is why I dislike it. I already have a website. I already have a set of RSS feeds that I follow. I do not want another website hosted on a strange and convoluted service in a bajillion different JSON records when there is a folder of XML on my computer called “my-website” which I can copy to any server and have all I need. Yet, in the name of a federated internet, the BSky folks have herded people away from the real distributed internet and into a crappy Twitter knockoff. Perhaps there are some business-unaware programmers very passionate about the idea of a distributed web and I commend them for that but it already exists and, in every real aspect, they are harming it. If this ATProto thing is successful in its goals, it will cause much inconvenience to me and greatly diminish my ability to publish my works on the internet.

Yes Solid is another idea that's clearly the right thing, especially now. It might mlnit be easy to convince everyone they need it though, and the economics will be uphill given the trillions in entrenched, incumbent opposition.

I also don't think the average user gets the value of the protocol yet. Most of those users were looking for a new, more politically palatable home but with the same features as Twitter. The new generation of apps on the protocol will be vital in showing users what's possible. IMO the two most valuable features at a practical level are: - social graph portability, which might look like having an onboarding experience that bootstraps your community on that app - lexicon cross compatibility, i.e. your data from app A shows up in a contextually relevant spot in app B. Or app B writes records that show up in app A. This is pretty key to get right because it might confuse or anger users if they aren't condition to expect it. Once the average user groks these features though, I'd be surprised if they voluntarily switch back to the standard corpo apps that eventually exit to some company who tries to monetize the shit out of every feature.

We want more control over data that we've created, and more control over data that's about us. I'm not sure either of these concepts align well with "ownership" though. Property and data are concepts that don't mix.

Language nitpicking aside... you subvert the walls of their gardens and aggregate the walled-off data without the walls, so users face a choice not between:

- facebook

- everything else

but instead between

- facebook and everything else

- just facebook

But that approach only works if we can solve the "data I created" problems in a way that doesn't also require us to acknowledges facebook's walls.

Share and you can't control what everyone else does with the data. Hence, untether is a non concept. In the sense that that data has now lost it's uniqueness. You can untether all you want. Copies of it will be subjected to any kind of tethering and transformation. So, if your point was about control, forget it.

Sure, you'd have something like:

    {:record   {:person-key **public-key** 
                :type :twitter-post
                :message "My friend {:person-key **danabramov-public-key**} suggested I make this on this HN post {:link **record-of-hn-post-hash**}. Look at his latest post {:link **danabramov-newtwitter-post-hash** :person-key **danabramov-public-key**} it's very cool!"}
    :hash      **hash-of-the-record**
    :signature **signature-by-author**}

So everything is self-contained. The other features you'd build on top of this basic primitive

- Getting the @danabramov username would be done by having some lookup service that does person-key->username. You could have several. Usernames can be changed with the service.. But you can have your own map if you want, or infer it from github commits :)) There are some interesting ideas about usernames about. How this is done isn't specified by the Record

- Lexicon is also done separately. This is some validation step that's either done by a consumer app/editor of the record or by a server which distributes records (could be based on the :type or something else). Such a server can check if you have less than 300 graphemes and reject the record if it fails. How this is done isn't specified by the Record

- Collection.. This I think is just organizational? How this is done isn't specified by the Record. It's just aggregating all records of the same type from the same author I guess?

- Hashes.. they can point at anything. You can point at a webpage or an image or another record (where you can indicate the author). For dynamic content you'd need to point at webpage that points at a static URL which has the dynamic content. You'd also need to have a hash->content mapping. How this is done isn't specified by the Record

This kind of setup makes the Record completely decoupled from rest of the "stack". It becomes much more of independent moveable "file" (in the original sense that you have at the top) than the interconnected setup you end up with at the end.

That's true if you define the problem as "does my parser crash" and not whether the app is perceived as working correctly. If some platform adds support for video posts, then the next thing that happens is people start making posts that are only video. Meaning that in every other client, users see what appears to be an entirely empty post. Which will be considered a bug.

This is the core argument of Moxie's seminal essay, The Ecosystem Is Moving:

https://signal.org/blog/the-ecosystem-is-moving/

One of the controversial things we did with Signal early on was to build it as an unfederated service. Nothing about any of the protocols we’ve developed requires centralization; it’s entirely possible to build a federated Signal Protocol-based messenger, but I no longer believe that it is possible to build a competitive federated messenger at all.

That was written in 2016 but it was true then and continues to be true today. Users reject federated open platforms because the coordination costs mean they don't move as fast as proprietary centralized platforms, and they often appear broken even if technically working as designed.

Nothing about that analysis is unique to social media. It is also true of file formats. OpenOffice never took off because new features got added to Office first, so files that used those features would open in semi-corrupted ways in OpenOffice. The fact that OO represented things internally using open unions didn't matter at all.

Are these just guidelines or is this enforced in some way? I guess readers could validate and skip anything that doesn't match their schema.

There's also software called Constellation[0] from microcosm that exclusively collects "backlinks", which are items which link to a certain thing. e.g. follow records for an account, reposts/replies/likes on a post. If you click on the Backlinks tab when examining something on pdsls.dev you can see a list of them.

It can be self hosted and is much lighter than running a full appview. The author has a public endpoint for it that's keeping up with the whole network running on a single raspberry pi on his home network.

0: https://www.microcosm.blue/

Yes. I describe this in this part of the article: https://overreacted.io/a-social-filesystem/#:~:text=One%20ch...

It's basically event sourcing. You listen to the data you care about from the network and update the local index (DB). There are also tools like Tap (https://docs.bsky.app/blog/introducing-tap) that do the plumbing work and let you backfill automatically.

For the commenters who have made similar comments, I'd be curious to hear what they think could be cut. I suspect different readers will have different opinions on this, which means it's probably a good thing you didn't make cuts.

Exactly. Anything that's ever been public on the internet is never really gone anyways, and it's unsafe to assume so. This is similar to publishing a website or a blog post. Plus, from a practical (non-opsec) point of view, you can delete items (posts, likes, reposts, etc.) on ATProto, and those items will disappear from whatever ATProto app you are using - usually even live. You need to dive into the protocol layer to still see deleted items.

Your last point is one that I used to be very strongly favor of, and today?

Nooooooooooo. No. No. No.

It's not going to happen and we shouldn't even consider it. Seriously. This thing we are doing here, which is "connecting people to each other," those forces for MANY will be far more powerful than "let me stop and think about the fact that this is forever." I just don't think we are wired for it, we're wired for a world in which we can just talk?

I think it's better to try to engineer some specific counter to "everything is recorded all the time" (or, as in here, not try to usher it into existence even more) than to try to say "welp, everything is recorded all the time, better get used to it."

It's not about the access, it's about the completeness. Imagine this paradigm takes off (I hope it does!), everyone has their own PDS and finally owns their data. Social apps link into their PDS to publish and share data exactly as they're supposed to.

Well now someone's PDS is a truly complete record of their social activity neatly organized for anyone that's interested. It's not a security issue, after all the data was still public before, but the barrier to entry is now zero. It's so low that you can just go to stalker.io, put in their handle, and it will analyze their profile and will print out a scary accurate timeline of their activity and location leveraging AI's geoguesser skill.

Maybe some could want to publish something publicly but anonymously?

Microblogging is also the least interesting part of the ATProto ecosystem. I've switched all my git hosting over to https://tangled.org and am loving it, not least of which is that my git server (a 'knot' in Tangled parlance) is under my control as a PDS and has no storage limits!

yeah, I was one of them. developers are not the endgame, though. true social media needs people who are not going to do anything more complicated than "go to website, sign up". there's no world where setting up your own pds is that simple without an organized piece of software to do that kind of thing.

personally, I could probably get behind recommending something like umbrel[0], if it included something like a "include a pds" option during config. but even that is asking for a lot of mind-share for a non-tech user. it would take a super smooth setup process for that to be realistic. point is, though, I'm not saying it can't be done; I'm saying no one is doing it and what people are doing is not getting the job done for wider adoption.

[0] https://umbrel.com/ *and, naturally, at this point, I'd prefer they include something that isn't based on AT proto for social publication. I wouldn't mind if they had both, but just an AT proto implementation wouldn't attract me.

What's a PDS?

come on, man, let's be real. you're talking modern, practical application; I'm talking reasonable user buy in at big boy social media levels. the video hosting IS what I'm talking about being expensive. you think bsky is going to be successfully while ignoring the instagram crowd forever? what are we doing here?

bsky saves the video processing and bandwidth by not hosting that content on bsky. it's a smaller problem, but in a large enough pool, images become heavy, too. and, either way, the egress of that content is expensive if you're doing it for the entire world, instead of letting each individual's computer (their pds) do it.

I'm happy to admit that text is cheap and bsky isn't looking to offload their data as it stands now. but let's be honest about the long term, which is what my original comment takes aim at.

> Do you have a more pointed criticism?

Sure.

After taking the time to re-read the article since I initial posted my (admittedly shallow) dismissal, I realized this article is really a primer/explainer for the AT protocol, which I don't really have enough background in to criticize.

My criticism is more about the usefulness of saying "what if we treated social networking as a filesystem": which is that this doesn't actually solve any problems or add any value. The idea of modeling a useful thing (social media)[0] as a filesystem is generalizing the not-useful parts of it (ie. the minutia of how you actually read/write to it) and not actually addressing any of the interesting or difficult parts of it (how you come up with relevant things to look at, whether a "feed" should be a list of people you follow or suggestions from an algorithm, how you deal with bad actors, sock puppets, the list goes on forever.)

This is relevant to Joel's blog because of the point he makes about Napster: It was never about the "peer to peer" or "sharing", that was the least interesting part. The useful thing about Napster was that you could type in a song and download it. It would have been popular if it wasn't peer to peer, so long as you could still get any music you wanted for free.

Modeling social media as a filesystem, or constructing a data model about how to link things together, and hypergeneralizing all the way to "here's how to model any graph of data on the filesystem!" is basically a "huh, that's neat" little tech demo but doesn't actually solve anything. Yes, you can take any graph-like structured data and treat it as files and folders. I can write a FUSE filesystem to browse HN. I can spend the 20 minutes noodling on how the schema should work, what a "symlink" should represent, etc... but at the end of the day, you've just taken data and changed how it's presented.

There's no reason for the filesystem to be the "blessed" metaphor here. Why not a SQL database? You can `SELECT * FROM posts WHERE like_count > 100`, how neat! Or how about a git repo? You can represent posts as commits, and each person's timeline as a branch, and ooh then you could cherry-pick to retweet!

These kind of exercises basically just turn into nerd-sniping: You think of a clever "what if we treated X as Y" abstraction, then before you really stop to think "what problem does that actually solve", you get sucked into thinking about various implementation details and how it to model things.

The AT protocol may be well-designed, it may not be, but my point is more that it's not protocols that we're lacking. It's a lack of trust, lack of protection from bad actors, financial incentives that actively harm the experience for users, and the negative effects on what social media does to people. Nobody's really solved any of this: Not ActivityPub, not Mastadon, not BlueSky, not anyone. Creating a protocol that generalizes all of social media so that you can now treat it all homogeneously is "neat", but it doesn't solve anything that you couldn't solve via a simple (for example) web browser extension that aggregated the data in the same way for you. Or bespoke data transformations between social media sites to allow for federation/replication. You can just write some code to read from site A and represent it in site B (assuming sites A and B are willing.) Creating a protocol for this? Meh, it's not a terrible idea but it's also not interesting.

- [0] You could argue whether social media is "useful", let's just stipulate that it is.