From a very quick skim of the repositories, this appears to be quite limited transparency. The documentation gives a decent high-level overview of how Tweet recommendation works—no surprises—and the code tracks that roadmap. Those are meaningful positive steps. But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
I am assuming that open sourcing the code aims to increase transparency about the business logic of the ranking decisions. At the same time you don't want spammers to be able to easily run experiments against a cloned version of your system.
Haven't gone through yet, but yeah, if that's the case, all this is, is a glorified framework to plug your own in.. Not exactly what was promised.
From a quick clone and line-count, it has:
235 kLOC .scala
136 kLOC .java
22 kLOC .py
7 kLOC .rs
I also haven't skimmed this code except very superficially, but perhaps you should since you're out there making statements with your Princeton credentials.
(I posted this comment with the heads-up a few minutes after your comment above and then expanded it as you didn't respond.)
Ignoring the global nature of Twitter for a moment.
And neither can spammers find and test the cracks and edge cases that would allow them to break the system, that does sound reasonable to me. If they were public there would be an arms race between spammers/those wishing to game the system and Twitter engineers.
This is the problem with most of social media today. It is a very well known problem in ML [1], but nobody is willing to do anything about it because it's a fundamental UX change. Facebook, Twitter, YouTube, TikTok, they have defined themselves by their recommendation engines.
[1] https://towardsdatascience.com/dangerous-feedback-loops-in-m...
Worth exploring.
So wonder if some value is wrong in one of those constants. Anyways, the blocking feature is broken..
If Twitter wants to put a stop to the user exodus and save lots of money in the process, here's what they could do:
1) Add an off switch to the for you feed. I'll click it right away and never turn it on again. Stop wasting minutes of CPU time on my behalf. I never asked for it. It doesn't do anything for me that I need or want.
2) Sort by time, filter by hashtag. Twitter used to be about real time information. I don't care about things that happened days or weeks ago. I don't need to see all of it. This is the core feature that made Twitter popular. Mastodon has it and it is absorbing users from Twitter by the millions. It still works. Restore this feature and make it the default.
3) Join the fediverse. That's where a lot of the former hard core users went. They still exist. They still post messages. They still engage with each other. They just don't use Twitter anymore. Allow people to follow mastodon users. Allow mastodon users to follow Twitter users. Not that hard to implement and probably would do wonders for user engagement.
That's probably one of the biggest reasons that I have remained on Twitter even after setting up a Mastodon persona.
// we only keep unfollows in the past 90 days due to the huge size of this dataset,
// and to prevent permanent "shadow-banning" in the event of accidental unfollows.
// we treat unfollows as less critical than above 4 negative signals, since it deals more with
// interest than health typically, which might change over time.
val unfollows: SCollection[InteractionGraphRawInput] =
GraphUtil
.getSocialGraphFeatures(
readSnapshot(SocialgraphUnfollowsScalaDataset, sc),
FeatureName.NumUnfollows,
endTs)
.filter(_.age < 90)
(
"author_is_elon",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
(
"author_is_power_user",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
(
"author_is_democrat",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
(
"author_is_republican",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
) /**
* These author ID lists are used purely for metrics collection. We track how often we are
* serving Tweets from these authors and how often their tweets are being impressed by users.
* This helps us validate in our A/B experimentation platform that we do not ship changes
* that negatively impacts one group over others.
*/
Additionally, from another Twitter engineer, the Democrat/Republican flags are apparently 10 years old and not important and do not have high feature importance.
[1] https://twitter.com/elonmusk/status/1641880448061120513?s=20
https://twitter.com/elonmusk/status/1641908130274525187?t=5t...
I am surprised at the number of inherently redundant and colinear features, though. (e.g. has_1_image, has_2_images, has_3_images, has_4_images)
https://twitter.com/ZoeSchiffer/status/1641902570921943044?s...
Or did they leave this in just so they could hold its removal up as an example of listening to the community?
If !user.follows_author(author) then don't show tweet on timeline Else if tweet.timestamp is later than all other tweets show tweet first
This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tweets seen by their followers. When Twitter moved away from this, it rendered my timeline useless so I started just pulling up people's profiles to read their tweets in order and eventually deleted my (pseudonymous) account that had several thousand followers. Almost nobody was seeing my tweets anyway thanks to this algorithm and deleting the account did not prevent me from browsing accounts I'm interested in.
All Elon needed to do to fix Twitter was to reverse all of the bad changes they've made since 2015 or so and restore the platform to what it was in the late 00s/early 10s.
As with all of the media outlets that elevate these two private clubs into the arbiters of truth, votes for Community Notes have to be relatively balanced between the two parties. Bipartisanship is a trash metric for determining truth, but absolutely none of the people raging at Musk in this thread would disagree with it.
So, for technical / performance reasons, changes to the algos might want to be benchmarked against this account in particular, because it’s the account most likely to be at the centre of capacity- / load-related issues.
private val DarkRequestAnnotation = "clnt/has_dark_request"
private val Democrats = "democrats"
private val Republicans = "republicans"
private val Elon = "elon"
private val Vits = "vits"> This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Is this new? Perhaps Twitter already removed the code from their main branch? Or was this just a joke from the beginning?
Isn’t this top of the page disclaimer relevant for you? It seems not to be part of the main branch.
https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh
Permalink: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
It's not an absolute rule, I've certainly inherited projects in a consulting capacity that were written by small teams and were atrocious. But more often than not, a small team working for a small company has fewer of the internal "forces" that incur "technical debt."
Those forces are things like
- Silo'd teams working on a common code base in parallel but never talking to each other, thus duplicating code and having wildly different conventions
- Layers of middle management each with different management styles, leading to inconsistency and product-wide short-cuts
- Dealing with sudden success-induced scalability disasters that result in bandaid solutions
- More employee churn which means that the way we did things yesterday is not the way we're doing things today because someone new is in charge ... more inconsistency in code and software decisions
- More "old code." Companies very rarely do rewrites and when they do they're often failures. So the bigger the company, the more "legacy" spaghetti code typically because you don't fix what isn't broken (especially when the entire system is broken because it's one big giant mess that no one understands and yet somehow it actually works ... as long as we don't breathe on it or get a sudden surge of new account sign-ups).
And thank you to everyone at Twitter who helped organize this release. Open sourcing something like this is no small effort.
[1]: https://github.com/twitter/the-algorithm/blob/main/home-mixe...
https://i.imgur.com/F8GSeyH.png
And, no, this wasn't in a merge-request, it was in the "main" branch of HomeTweetTypePredicates.scala.
\*
\* These author ID lists are used purely for metrics collection. We track how often we are
\* serving Tweets from these authors and how often their tweets are being impressed by users.
\* This helps us validate in our A/B experimentation platform that we do not ship changes
\* that negatively impacts one group over others.
\*
I have spent significant effort creating a network and there you go choosing to ignore my efforts by putting in 50% of crap-I-don't-want-to-see.
That is why I despise your algorithm.
This is just one feed (the "For You" recommendations feed), they also have the "following" feed tab next to it that is 100% your network (want you want), and it remembers your selection when you change between them (they fixed that a few months ago), so really this is kind of a pointless thing to despise for that reason. It's just an option you can 100% avoid if you don't want to see it.
In fact, Twitter is probably one of the only few left in the large social media space that actually gives you an 100% following network feed (minus maybe ads) in chronological order that REMEMBERS your selection (Facebook, Instagram, and TikTok don't). Which makes this even more silly to say. Facebook, Instagram, and TikTok do all have in-network exclusive chronological order feeds, BUT they are extremely hard to find, or don't remember your selection to them.
Hate of Twitter is easy to spoon out, but at least complain about things that aren't already solved for you.
There's no way to actually use this limited release to understand how or why any tweet is boosted, so we're in exactly the same boat we were in yesterday.
This is a great thing.
But really, if people respond to Twitter's actions politically, that response exists within a context that was certainly influenced by Twitter's prior actions.
> Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user.
It would’ve been interesting to see what changes were made since Musk’s takeover. As someone who followed 5,000+ users, I know I never saw a tweet that wasn’t either from nor retweeted by someone I followed — e.g. I never saw those “[user you follow] liked [someone you don’t follow] tweet”
50%/50% in FYP seems to reflect my experience today — which is much worse, to the point that I’ll regularly switch to viewing by List b/c I miss seeing people who I want to read.
I wonder how much testing and analysis went into deciding on the 50/50 ratio — e.g. how does it impact user engagement and behavior. Because it sounds like an easy round value that you’d land on when thinking “users should be pushed out of their bubbles”
For ranking the candidates these predictions are combined into a score by
weighting them:
"recap.engagement.is_favorited": 0.5
"recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": 11* (the
maximum prediction from these two "good click" features is used and weighted by
11, the other prediction is ignored).
"recap.engagement.is_good_clicked_convo_desc_v2": 11*
"recap.engagement.is_negative_feedback_v2": -74
"recap.engagement.is_profile_clicked_and_profile_engaged": 12
"recap.engagement.is_replied": 27
"recap.engagement.is_replied_reply_engaged_by_author": 75
"recap.engagement.is_report_tweet_clicked": -369
"recap.engagement.is_retweeted": 1 "recap.engagement.is_video_playback_50": 0.005
I wonder if this is why threads rank so obnoxiously high. They get artificially boosted by the author replying to their own tweet
Code: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
Should be: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),
> the main neural network part of @Twitter recsys algo is based on 2021 work of #SinaWeibo - Chinese clone of Twitter
https://twitter.com/Ben_Cary_/status/1641893540614623258
> Twitter use to rank posts higher for those who had more followers/less people they follow
> They are removing that as of today but kinda interesting that someone with 10k/10k followers would get less reach than if they had 10k followers and only followed 6khttps://twitter.com/modern_mindset/status/164207843202770534...
> Twitter algo is finally opensource.
> • Twitter Blue 2x boosts
> • Likes have 30x comment value
> • Links/mentions/names deboosts
> • Retweets have 20x comment value
> • Restrictions/suspensions deboost
> • Images/videos/trending topics 2x boost
> Will write a thread about it later. GM
> Twitter algo 101
> Boosts
> - Likes 30x
> - Retweets 20x
> - Twitter Blue 2-4x
> - Trusted circle 3x
> - Images/videos 2x
> - Replies 1x
> Negatives
> - URL only
> - No text
> - Mute
> - Block
> - Unfollow
> - Report > Part of twitter's algo Jack Dorsey, Katy Perry, Stephen Curry and Barack Obama as “testing accounts” for getting random Tweets for testingThis does a lot of heavy lifting here.
https://github.com/twitter/the-algorithm
Not sure if it has what you were looking for (and maybe you already checked this repo, too!), but it's more relevant than the linked repo imo
Let's not focus criticism on an attempt to do something.
I know that open source code around commenting online directly impacted the direction my current team went building our community tooling.
I’ll take even a glimpse into the machinations of any social media giant. It’s better than nothing!
https://github.com/twitter/the-algorithm-ml/blob/main/projec...
In realgraph you can see some of the things they keep track of, which include what you have in your address book, total time spent "dwelling" and a few other interesting nuggets.
The minority of people who understood what this was already worked for platform companies and wanted to again, and the few who didn't but also knew how invasive this was could always be discredited as conspiracy theorists.
Ever wonder who else gets those graphs from platform companies? Today this is all interesting, but a couple of weeks from now when this all sinks in, I wouldn't be surprised if I were mad as hell.
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
which is a shame.
[1]: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
This is standard practice when it comes to open-sourcing such repos that were closed-source for years.
Well written article, from an engineer's perspective.
Ranking is achieved with a ~48M parameter neural network that is continuously trained on Tweet interactions to optimize for positive engagement (e.g. Likes, Retweets, and Replies). This ranking mechanism takes into account thousands of features and outputs ten labels to give each Tweet a score, where each label represents the probability of an engagement. We rank the Tweets from these scores.
"Use 4 spaces per indentation level."
https://www.quora.com/Why-does-Google-use-2-spaces-for-Pytho...
Search for Elon gives this: https://github.com/twitter/the-algorithm/search?q=Elon&type=
Of course it could also be that they change their mind when spammers abuse the openness.
> The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.
Edit: hi friend
Perhaps this calls for an HN poll...
I want to see a chronological list of things sources I follow have posted.
Yes, I understand you can do this on Twitter still, but I would guess most people are more influenced by "the algorithm".
Maybe this is helpful to anyone for navigating what's in there!
Like a dig at the code quality.
I found it interesting that there is no attribution. Most other companies list the authors on engineering blogs (eg. Facebook, Uber, etc.)
This topic seems to draw the attention of unhinged people, so I suppose I wouldn't want my name on it either.
https://thehill.com/policy/technology/3928219-musk-was-denie...
for Google Ads, you couldn't easily know what ads would be shown for a given query, without a whole lot of data that's not contained in any code: the experiment settings in the server, for one thing. And the user who's doing the query, for another.
An "experiment" could apply to 100% of the traffic, so it's not really an experiment anymore. And even if you think X has been put into production, there is still a "holdback" experiment, where some part of the traffic does not get X applied to it.
And i don't even live in the US.
It would explain why they are tracking it, to increase visibility.
https://github.com/twitter/the-algorithm/tree/main/src/java/...
https://www.databreachtoday.com/twitter-says-source-code-lea...
Maybe we’ll all get lucky and Elon will cause Twitter to go away forever.
s.SpaceSafetyLabelType.MedicalMisinfo -> MedicalMisinfo,
s.SpaceSafetyLabelType.GenericMisinfo -> GenericMisinfo,
s.SpaceSafetyLabelType.DmcaWithheld -> DmcaWithheld,
s.SpaceSafetyLabelType.HatefulHighRecall -> HatefulHighRecall,
...
s.SpaceSafetyLabelType.UkraineCrisisTopic -> UkraineCrisisTopic,
Where's the beef?
The algorithm really needs to recognise when tweets are time-sensitive and not recommend them just because they got a lot of engagement the previous day!
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
This should be seen as a possible snapshot of some code, that might have run, might run in the future, or is possibly running in some parts of the production infrastructure at Twitter.
I get way to much random crap now, promoted tweets, "thing that might interest me", users that seem to never get on my feed etc.
Twitter seems to go in the direction of all other social media, feeds that are 100% digital crack with no way to control your media diet.
There is code that favor Elon's tweets so I'd yes that's probably what they use
They made my morning
I don't see any Typelevel stuff. This probably lets them hire and train engineers faster while still gaining most of the benefits
I hope this will encourage more companies to pick Scala.
the majority of users didn't ask for the this so not sure what's the exact motive behind thier efforts. it could be a PR stunt.
Maybe an UML graph or even a presentation or written guide on how they measure and apply each weigh or group policy would make it easier to have some solid take on how it works
author_is_uwu
[1] https://en.wikipedia.org/wiki/It%27s_the_economy,_stupid
It looks like once again these lot predicting that he won't open source the algorithm and are going to start eating their words again [0], just like they did around incorrectly predicting Twitter's immediate collapse [1] and will look at the source code anyway and continue to talk about "Twitter" again.
If Twitter can open-source their algorithm, Why not TikTok? Either way, the bots are now going to have a very expensive time on Twitter.
I especially like the suggestions to rewrite the algorithm in Rust [1] and this pull request which simplifies the algorithm to a single c file [2].
[1] https://github.com/twitter/the-algorithm/issues?q=is%3Aissue... [2] https://github.com/twitter/the-algorithm/pull/712
What does the commit history say? There are 3 commits, like a very very real programming project. The issues and pull requests show how much people are fooled by this very transparent move.
So this is an obvious attempt at a digital potemkin village, that like the real one, poorly succeeds in hiding the truth. Elon does not not want to upset the apple cart (political economical or ideological) but make his followers believe in it, and so we get this. Great spectacle, if that's what you're interested in.
The default feed view has grown increasingly useless over the past ~6 months.
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
Reminds me of the Sirius Cybernetics Nutri-matic drinks machine.
#!/bin/sh
exit 0Wow, we're getting some collaboration going!
Twitter hmu if you need help trying Pytorch 2.0 ;)
private def getLinearRankingParams: ThriftRankingParams = { ThriftRankingParams( `type` = Some(ThriftScoringFunctionType.Linear), minScore = -1.0e100, retweetCountParams = Some(ThriftLinearFeatureRankingParams(weight = 20.0)), replyCountParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)), reputationParams = Some(ThriftLinearFeatureRankingParams(weight = 0.2)), luceneScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)), textScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 0.18)), urlParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)), isReplyParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)), favCountParams = Some(ThriftLinearFeatureRankingParams(weight = 30.0)), langEnglishUIBoost = 0.5, langEnglishTweetBoost = 0.2, langDefaultBoost = 0.02, unknownLanguageBoost = 0.05, offensiveBoost = 0.1, inTrustedCircleBoost = 3.0, multipleHashtagsOrTrendsBoost = 0.6, inDirectFollowBoost = 4.0, tweetHasTrendBoost = 1.1, selfTweetBoost = 2.0, tweetHasImageUrlBoost = 2.0, tweetHasVideoUrlBoost = 2.0, useUserLanguageInfo = true, ageDecayParams = Some(ThriftAgeDecayRankingParams(slope = 0.005, base = 1.0)) ) }
https://twitter.com/alexblechman/status/1641905502043926530?...
My conclusion is that it's basically entertainment, with very little of what I'd call high-quality useful information that deserves further examination (unlike a lot of HN posts, in contrast). I also notice something of a Tik-Tok approach to video being implemented, which is not surprising given Tik-Tok's success (and makes one wonder who exactly it is lobbying so hard for a Tik-Tok ban, and whether it's just a commercial competition issue more than anything else).
As far as the recommendation algorithm, it appears to be a siloing setup - look at content of one particular flavor, it gives you more of that flavor. A 'flush settings' or 'forget browsing history' or 'reset to defaults' button would be useful, if probably not what advertisers want in terms of delivering to target audiences. I suppose setting up multiple accounts is something of a solution, although too much effort to be that interesting.
In terms of news reports, it's broader in scope than traditional corporate media outlets, so that's a plus in its favor. Reliability is perhaps similar (i.e. low).