Twitter reveals part of its source code, including its recommendation algorithm

As repeatedly promised by Twitter CEO Elon Musk, Twitter has done it open part of its source code for public viewing, including the algorithm it uses to recommend tweets in users’ timelines.

On GitHub, Twitter has published two repositories containing code for many parts that make the social network work, including the mechanism Twitter uses to control the tweets users see on the For You timeline. In a blog post, Twitter called the move “a first step into the future[ing] more transparent” and at the same time “[preventing] risk” for Twitter itself and people on the platform.

In a Twitter Spaces session today, Musk clarified:

“Our first release of the so-called algorithm will be quite embarrassing and people will find a lot of bugs, but we will fix them very quickly,” Musk said. “Even if you don’t agree with something, at least you know why it’s there and that you’re not being surreptitiously manipulated… The analog here that we’re aiming for is the great example of Linux as an open-source operating system… You can.” theoretically discover many exploits for Linux. In reality, the community identifies and fixes these exploits.”

As for the second point in the risk avoidance blog post, the open-source releases do not contain the code that powers Twitter’s ad recommendations, nor the data used to train Twitter’s recommendation algorithm. Additionally, they include some instructions for reviewing or actually using the code, reinforcing the idea that the releases are strictly developer-focused.

“[We excluded] Any code that would compromise user security and privacy or the ability to protect our platform from bad actors, including undermining our efforts to combat child sexual exploitation and manipulation,” Twitter wrote. “We [also took] Steps to ensure user security and privacy are protected.”

Twitter says it’s working on tools to manage code suggestions from the community and sync changes to its internal repository. Presumably these will be available at a later date – currently there are no signs of it.

“We’ll be looking for suggestions, not just about bugs, but about how the algorithm should work,” Musk said at the Spaces session. “It will be an evolving process. I wouldn’t expect it to be a straight up move…but we’re very open to things that would improve the user experience.”

At first glance, the algorithm is quite complex – but not exactly surprising from a technical point of view. It consists of several models, including a model for detecting “unsafe for the job” or abusive content, determining the likelihood that a Twitter user will interact with another user, and calculating a Twitter user’s “reputation”. . (It’s unclear exactly what “reputation” refers to; the high-level documentation isn’t clear.) Multiple neural networks are responsible for ranking tweets and recommending accounts to follow, while a filtering component hides tweets , um – pardon the jargon – “Aid regulatory compliance, improve product quality, increase user trust, protect sales by using hard filtering, visible product treatment, and coarse-grained downranking.”

In an engineering blog entryTwitter reveals more about the referral pipeline, which is said to run about five billion times a day:

“We’re trying to extract the top 1,500 tweets from a pool of hundreds of millions… Today, the For You timeline is 50% full [tweets from people you don’t follow] and 50% [tweets from people you follow] on average, although this may vary from user to user,” Twitter wrote. “Ranking [tweets] is achieved using a neural network with ~48 million parameters continuously trained on tweet interactions to optimize for positive engagement (e.g. likes, retweets and replies).”

The release of the source code comes after several controversies over the past few months involving tweaks to Twitter’s recommendation algorithm. According to Platformer, in February Musk asked Twitter’s engineers to reconfigure the algorithm so his tweets would attract more attention. (Twitter later rolled back that change — at least somewhat.) In November, Twitter began showing users more tweets from people they don’t follow — a move the platform attempted before acquiring Musk, but later after user backlash undone.