A Thousand Newlines: Matrix Spam, #3 — We are so back!

I managed to write scripts that were good enough to mitigate spam as it was happening but it was all still pretty janky and worst of all, still reactionary. I needed something to close the door before they got in.

Something with unexpected consequences happened

On the second of September 2025, matrix.org had a huge 24h outage. You can read the blog post with the post mortem here, it is pretty interesting. At first this seems completely irrelevant to the topic of this blog post series but trust me, this will make a lot of sense in a bit.

How notaries work

To understand what will follow now, we first have to take a quick detour to how certain aspects of matrix work.
In Matrix, every event is signed by the originating servers signing-key. In order to actually verify the events that you get sent over federation, your server WILL need to have the other servers signing key.

Now, what can we do if we get an historical event from a server that has been offline for years? This is something that regularly happens as rooms age, a server participated in there at some point and eventually shut down for whatever reason. Now you join it and suddenly see events from someone that technically doesn't exist anymore. You can't just contact it directly and get it's signing key.

This is where the so called "notary" servers come into play. A notary is a server that you trust with providing you the public signing-keys of another server. Notaries also tend to cache/store all keys they ever see in their database, so as long as the notary you selected has federated with that server that you want the keys from now, the chances are high that the notary has stored that key back then and still has it today!
Notaries have some more valuable use cases though the above described one is one of the most common ones, the other use cases mainly have performance reasons or can help in cases of network segmentation.

For synapse servers, the default notary is matrix.org and during that 24h outage many server admins saw a ton of errors in their logs over failed request to their notary.

One of those admins was ~creme, the admin of envs.net, a cool tilde community that also ran one of, if not the largest public matrix server directly after matrix.org at the time and who also happened to be a member of my "Synapse admins" room that got attacked so regularly.

Becoming a notary by accident

The conclusion of the admins in my room: Add a few additional trustworthy servers as notary.

I didnt't notice it right away but it turned out that ~creme decided to add codestorm.net as a trusted notary for envs.net. Funnily this was pretty pointless for him since envs.net was much much older than my server, so the chances of me being able to return keys for a server that envs.net has never seen before were basically zero, envs was also well connected and had IPv4 and IPv6 connectivity, so I couldnt even assist in bridging gaps between a IPv4-only and a IPv6-only servers either.

Now, what happens when a notary gets a request for a key that it does not yet have?

Pretty simple, it tries to connect to the server in question to get those keys too, if that works it stores them and also returns them in the response to whoever asked for them.
So a notary can basically also just be your proxy for getting keys from other servers.

Now comes the really interesting part. For every event that Synapse sees, it first checks if it has a fitting/matching key already in it's own database, if that is not the case it then asks ALL of it's configured notaries in parallel and only if none of them can provide the keys does it try to directly contact the origin server.

Let that sink in.... every new event that envs.net now sees will potentially trigger my server to connect to another matrix server and retrieve it's signing keys. Signing keys do also expire, so even for servers that envs already had a key it will at some point need to refresh that too. Guess who will get asked for that now? Yes, my server.

The server_signature_keys jackpot

Remember when I said that envs.net was probably the second largest matrix server at the time? Yeah, so this also means it was in millions of rooms and as a result federating with everyone and their mom. This resulted in me suddenly having insights into a massive portion of the open federation, not only do I obviously know about all the servers I directly federate with, but by being a notary for envs I now also know about every single server that envs.net is federating with.

Lets revisit the approach of preemptively scanning a list of servers.
In the second blog post I checked the destinations table with rather disappointing results. I got a grand total of ~13k servers

But now that I became a notary for one of the biggest servers out there, lets see what new options we have.

I present to you:
The server_signature_keys table

synapse=# select count (distinct server_name) from server_signature_keys;
 count 
-------
 26824
(1 row)

Jippie! now this is a much much higher number than before!

Automating discovery

I start working on a simple bash script, it pulls the list of distinct server names from the server_signature_keys table, pipes that list into the regchecker binary from the Meowlnir repo and checks the output for dangerously open servers, then stores the list of already scanned servers in a little helper textfile to deduplicate future runs. it's set up to run every 10 seconds on a systemd timer. It really is nothing more than a bit of glue code and extremely bare bones. I also add a simple notification for new servers like that to my Reportingroom:

At the same time I create a new private ban list called "CS-Globs" where I automatically add user glob policies for every domain that gets detected as open reg (example: @*:due.ren).
The reason to do it this way is that glob bans, or user bans in general are invisible until they get executed. Unlike a ACL where the entire list of denied servers will be visible in the ACL event it'self that I have to send into the room I want to protect in order for it to be effective.

For example, a ACL event in a room might look like this:

{
    "allow": [
        "*"
    ],
    "allow_ip_literals": false,
    "deny": [
        "bad-server-1:example.org",
        "bad-server-3:example.org",
        "bad-server-4:example.org"
    ]
}

My biggest fear at the time is to accidentally help the spammers by assisting them in finding vulnerable servers.

From that time on every user from a server with dangerously open registration gets automatically banned in my rooms.

I also add a new weekly job to rescan the full list of servers to account for servers changing their registration requirements over time, especially those that might have fixed it and no longer pose a risk and who would otherwise be unfairly banned.

The time is now mid October 2025 and after Tulir asked me to create a version of my ban list with actual ACL policies so he can use it for his rooms with the join-gate I created the #cs-auto-open_reg:codestorm.net policy list. The list is invite only at that point and the only condition under which I allow people to use it is if they aren't leaking all the servers in one way or another.

Over the next few weeks it stays somewhat quiet in terms of new spam. A few buddies and I are regularly commenting on the new servers that get discovered because some of them have quite funny domains and I work on improving the scripts and also get some additional lists of servers from others. Some of the people in my close circles start configuring me as one of their trusted notary servers to passively help with discovery of new unsafe servers. The system discovers new dangerous servers every single day from then on, sometimes only one but often times multiple a day.

I feel pretty confident that the system is now solid enough to protect me from new spam.

Nov 10: wildcard bans aren't enough

Time passes and eventually on November the 10'th we get another large spam attack:

But hey, for all the servers that were used in this spam attack, I already had wildcard bans in place, so it should have been easily mitigated, right???

No :/

Out of all the 56 accounts that participated in the raid, about 43 managed to send messages before getting banned, some only 1 or 2, but a lot of them managed to send messages in the double digits range, which resulted in a total of ~700 spam messages getting sent despite my bot and server working overtime and banning as fast as they can.

Here is how that looked:

My server also wasnt overloaded cpu or memory wise, there was enough capacity to deal with this. What broke the defense was simply a matter of network and processing latency in a federated network. When the spamming account has already sent 20 messages before the event of them joining even arrived at my server then no amount of tuning and optimizing can help here.

What also didnt't help is when certain chains of events get stuck in the so called "federation staging area".
This is a spot where synapse parks events because certain previous events that are necessary to verify the legitimacy of the current event are missing for example.
The problem is that for as long as the events are parked there, they aren't processed and all of my tooling is blind to their existence. This has the side effect that countless spam messages might accumulate in that staging area without any of my bots having a clue of what is about to happen only for all of these events to then appear all at once as soon as whatever was blocking them got resolved.

See the following screenshot for a example of how events can accumulate in that staging area:

At this point im pissed, I tried to play nicely, I tried reacting as fast as possible and it didnt't work, I tried to mitigate this spam without leaking the list of abusable servers to the public and it didnt't work.

And before we continue, I really really want to highlight just how problematic I find it to leak that server list and explain why I was so hesitant with doing it.

There is a lot of stuff that this list could be abused for once it's public. For example there is a disgusting amount of people seeking illicit and horrific content that might take advantage of those often clueless admins that have accidentally misconfigured their server and who aren't aware of it.
Someone can make a account on it, join rooms dedicated for sharing illegal content, which will also download the media to that homeservers disk, meaning that some random nerd could now potentially host CSAM on their VPS and might be accidentally supporting a network of pedophiles by providing the server infrastructure for their user accounts.
This can and does of course also happen with servers that are actually meant to be open to the general public but the difference is that the admins of these servers are usually reasonably aware of the possible risks and made that decision conciously. Whereas the admins of those misconfigured personal servers have no reason to check for abuse originating from their server because why would they?
I also wouldn't knock on the bathroom door before stepping in when I live alone and have no reason to believe that someone else can even be there.

Going public

The reasons for hiding what is vulnerable are pretty similar to CVE disclosures in general. By making this public I'm increasing the risk for everyone while at the same time providing the solution to properly protect yourself against it. For the initial time after disclosure it shifts from a luck based game to a time race between those trying to quickly abuse it all as soon as the cat is out of the bag and those applying the mitigation.

I considered the decision for a while but eventually had enough and saw no other way to put an end to this than to go all in.

The next morning after the November 10 spam attack I make the following announcement in my Synapse room as well as in a few other communities with relevant people that are potentionally interested in it.

The list goes public and it's a massive success, after the November 10 attack there has been no furter spam attack in this fashion in any community that use this list now.

Furthermore, many admins were very positive about it and I managed to collaborate with a few bigger players to automatically share lists of server names with me by either using me as a notary or by sharing the list as a txt file on a schedule.
This is a great example of how the community works together and everyone helps even if it's just a little bit.

A centralized list in a decentralized network?

Yeah... thats one of the issues with the list right from the start.

Its a system that fully depends on me, one me and my infrastructure running, on me not abusing the trust that people put in to me by making their bots subscribe to the list.
If my server is down the list wont be updated anymore.
I want to distribute responsibility and free myself of the burden of providing this now essential service to the community all on my own. Thats why in June of 2026 I started developing a Maubot plugin that will hold all the functionality that my current collection of janky bash glue code and the Meowlnir regchecker binary has so that it will be as easy as possible for others to also run a scanner like me. Many people already have a maubot set up and drag&drop uploading a plugin to a webinterface is pretty simple.

This plugin comes with all the fancy stuff like metrics and an accompanying grafana dashboard which allows operators to track important metrics of their scanner. But most importantly I now have the ability to properly decentralize the list and with the awesome help of people in the community there are now multiple vantage points in different countries around the world.

And you know what? If you don't want to trust me and subscribe to my ban list you can now use that bot plugin to run your own version fully independent of me and do what you want with it.
I happily invite you to do so actually because I believe that no one should have to put that much trust in others.
The bot code is open source and mirror'd to Github.

I will continue to work on that scanner and add improvements as needed. At the same time I will keep my version of that ban list up and running as long as possible, there are other trusted people that can take over if I should get hit by a bus or randomly decide to disappear tomorrow.

Closing words

At the end of this journey over 6 thousand different accounts from several hundred servers were used for these spam attacks during that time.
The ban list currently has a bit over 900 known dangerous servers listed (offline servers get automatically removed after 60 days).
There are currently over 200 accounts in the ban list room and from looking at the usernames, most of them are moderation bots, each protecting who knows how many rooms.
To this day its unclear who was behind this spam, we don't even know if its one person or a group, maybe multiple individuals operating independently?

I wrote this blog post series because I believe that this type of spam is now pretty much solved and we already have too many negative posts about spam on matrix with no happy end, so I wanted to yap about something good. I know that there is still a lot to do with other types of spam and abuse and time will tell how that turns out, maybe I'll write some more blog posts once I have something nice to tell on those fronts too ;)

Until then I hope you were entertained and had fun reading it

~Sky