A Thousand Newlines: Matrix Spam, #2 — The heat is on

When you're walking into unknown territory and face new opponents you sometimes have to throw stuff at the wall and see what sticks.

In part one of the series I explained in detail how the spam looks, what issues it causes and why it's so hard to defend (If you are unfamiliar with scripted open reg spam and haven't seen that one I highly suggest to read Part 1 before this one).
This post will focus on how we reacted to the spam and how we tried to develop countermeasures.

Phase 1: I need more visibility

You see, as contrary as it might first seem, it can actually be quite hard to quickly spot where exactly the spam is happening. You rely on your clients room list preview and you catching a bunch of quickly sent empty messages.
The cool homelabbers that actually host their homeserver at home can even hear the spam because of the ramp up in fan rpm.
Though even when you finally spot it, what do you do next? You need to get a list of user ID's and their servers so that you can add them to a ban list.
Easier said than done especially when it takes you ~40 full page scrolls to even get past a single message to then click on the sender of the next message in order to copy/paste their user id. Doing this via Element Web or any other client is extremely tedious and unfeasible.

So, thinking about it for a while, the first idea was to write a bunch of scripts to fetch the relevant data via the Client-Server API via cURL, but thats usually not pretty and at least for me not great from a UI/UX pov.

This would also rely on Synapse as the application layer, the thing that is already under heavy load during these attacks.
I needed something better, something that is highly customizeable and better suited for the task. Grafana!
Who would have guessed it, the tool basically made for gaining visibility in computer systems. What played into my cards here was also that Synapse runs on Postgres and Grafana having native support to connect to Postgres as a datasource.

So, what things do we need to know?

• User ID's of joined users
• The rooms they join
• General activity in rooms

Building a dashboard

---

The first panel I made is an aggregated counter across all member events in all rooms that my server can see. That allows me to very very quickly see a sudden increase in joins (which is something you have to do first before you can start to spam and thus a strong first indicator).

---

The second panel I made is the result of a absolute mess of a postgres query with way too many left joins.
It shows me exactly what i need, every member event that happens in any of the rooms I'm in, the room where it happened, all kinds of details AND also if this account is already banned on the cme ban list or any of my other personal ban lists.
Why? you might ask. Well, with something like 5 accounts you can probably relatively easy work through it and ban them, even if they show up multiple times because of them having joined many different rooms. But with how generic these usernames are you will very quickly get to the point where you completely lose track of which accounts you already banned and which not.
Having this row in the table allows me to filter for only those accounts that arent banned yet.

---

And now the third and final panel in the dashboard.
What you see here are tiles that are sized by the total count of events, grouped per room and in itself split into the different kind of events.

This is probably by far the most difficult to visualize data.

The end result was jokingly named "Matrix Moderationclient from Wish" and this dashboard has kept this name internally to this day.

---

Hunting for the server list

So, after I finally had more visibility, I got a lot better in the reactionary defense. The cleanup was much easier now. The problem however was the same and no matter how shiny the dashboards get, this doesn't help in preventing it.

At the time, Tulir wrote a registration scanner in GO that made it very easy to verify if a server was actually open reg before we added it to the various ban lists after they were used in attacks.

What this also meant was that I can now in theory easily scan a bunch of servers..... if only I had a list to actually scan.

I knew they must be finding these servers somewhere, so I started brainstorming. What if they just scrape servers from random public rooms? I joined a bunch of really large rooms, like Matrix HQ and the old synapse admins rooms and some more that I could find on matrixrooms.info and pulled lists of servers in the room, ran them all through the scanner aaaaaaand nothing :(.
Sure, I found some open reg servers here and there but I never managed to find even a single new one that hasnt been abused yet and thus been placed on a ban list alredy, meanwhile they kept showing up with completely new, never seen before servers, every day.

Then I remembered, my server should have already seen a bunch of other servers, so at least somewhere in the database there must be a place with a large list and after a while I found the destinations table. It looks like this:

         destination         | retry_last_ts | retry_interval |  failure_ts   | last_successful_stream_ordering 
-----------------------------+---------------+----------------+---------------+---------------------------------
 ipfs.io                     |             0 |              0 |               |                                
 matrix.serveursdupeuple.net | 1706196937967 |    17896085947 | 1702548261365 |                                
 fx46.in                     | 1718657379556 |    31536000000 | 1702502061105 |                                
 inter.rocks                 | 1715081096739 |    31536000000 | 1702375844959 |                                
 ds01.de                     | 1701972621262 |    31536000000 | 1694723552297 |                                
 darknight-coffee.org        | 1734944939791 |    31536000000 | 1718958861765 |                                
 daisuke.osrx.de             | 1733788093741 |    31536000000 | 1722023692555 |                                
 kesuek.de                   | 1730517059981 |    31536000000 | 1720859902098 |                                
 andersson.im                | 1716915143938 |    31536000000 | 1694723569345 |                                
 matrix.vmnets.de            | 1722416101166 |    31536000000 | 1703935643384 |                       

Ok, lets see how many servers that gives us:

synapse=# select count (distinct destination) from destinations;
 count 
-------
 13521
(1 row)

I run a scan for all of them too but many are just offline and many of the ones that have super duper dangerous open registration here are already known too but I do actually find 47 new ones that haven't been used for spam yet. Nice!

I store that list of newly found servers for now, no idea what to do with it since none of the lists I'm working with at the time would do preemptive bans and only banning them in my own room via the ACL would potentially leak them to the attackers too.

A bit of time passes by and eventually some new spam attacks happen.

Lets see, did they use the servers I already found in the past?

I assumed that they might either get them from something like shodan by looking for results to public API endpoints of matrix servers, like the version probe for example.
I signed up for shodan and did some searches, I don't remember the exact specifics anymore but I hardly found actual matrix servers at all.

Trying to outrun them

Ok, so preventative measures are not within reach, at least not yet. Lets try to be quicker than them. Maybe maybe maybe..... if we stop spam quick enough AND consistently enough, they may get frustrated and stop?

I set up my "Reportingroom" and a script on my server that I run every 10 seconds on a systemd timer.

#!/bin/bash

DB_NAME="synapse"
DB_USER="grafana"
DB_PASSWORD="09F911029D74E35BD84156C5635688C0"
DB_HOST="localhost"
DB_PORT="5432"

# Alert configuration
TIME_WINDOW_MINUTES=5
JOIN_THRESHOLD=10

CURRENT_TS=$(date +%s%3N)
TIME_WINDOW_TS=$((CURRENT_TS - (TIME_WINDOW_MINUTES * 60 * 1000)))

echo "Checking for rooms with >$JOIN_THRESHOLD joins in last $TIME_WINDOW_MINUTES minutes..."


QUERY=$(cat <<EOF
SELECT 
    rsc.room_id,
    COALESCE(rsc.name, 'Unnamed Room') AS room_name,
    COUNT(rm.event_id) AS join_count
FROM 
    room_stats_state rsc
JOIN 
    room_memberships rm ON rsc.room_id = rm.room_id
JOIN
    events e ON rm.event_id = e.event_id
WHERE 
    rm.membership = 'join'
    AND e.origin_server_ts > $TIME_WINDOW_TS
GROUP BY 
    rsc.room_id, rsc.name
HAVING
    COUNT(rm.event_id) > $JOIN_THRESHOLD
ORDER BY 
    join_count DESC;
EOF
)

# Execute query and process results
PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -t -A -F "|" -c "$QUERY" | \
while IFS='|' read -r room_id room_name join_count; do
    
    room_id=$(echo "$room_id" | xargs)
    room_name=$(echo "$room_name" | xargs | sed "s/'/''/g")
    join_count=$(echo "$join_count" | xargs)

    echo "ALERT: Room '$room_name' ($room_id) has $join_count recent joins"
    
    # Send alert
    payload=$(jq -n \
      --arg room_id "$room_id" \
      --arg room_name "$room_name" \
      --arg join_count "$join_count" \
      --arg TIME_WINDOW_MINUTES "$TIME_WINDOW_MINUTES" \
      '{
        "msgtype": "m.text",
        "body": "Alert Sky room: https://matrix.to/#/\($room_id)?via=codestorm.net | Room Name: \($room_name) Is getting raided!",
        "formatted_body": "Alert <a href=\"https://matrix.to/#/%40sky%3Acodestorm.net\">Sky</a> room: https://matrix.to/#/\($room_id)?via=codestorm.net | Room Name: \($room_name) Is getting raided! There were \($join_count) joins within the last \($TIME_WINDOW_MINUTES) minutes",
        "m.mentions": {
          "user_ids": [
            "@sky:codestorm.net"
          ]
        },
        "format": "org.matrix.custom.html"
      }')

    curl -sS -X PUT \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer syt_09F911029D74E35BD84156C5635688C0" \
        -d "$payload" "https://matrix.codestorm.net/_matrix/client/v3/rooms/%21naAedOOzARDkWVFRXt%3Acodestorm.net/send/m.room.message/$(date +%s)" && sleep 0.2s
done

echo "Monitoring complete at $(date)"

I hope you are impressed by this beautiful jank. All it does is to run a query against my postgres DB to get a list of rooms in which there were more joins than what I configured in the threshold within the configured timespan and then sends me a notification with a ping into my Reportingroom.

In case you're already copied the tokens from the script to your clipboard, stop wasting your time ;) The original creds got replaced with the AACS key.
In case you actually catched that detail.... you're such a nerd <3

Now we just need to wait for another spam attack! Aaaaaaaand it didnt take too long, just 4 days after setting up the alerts, the first raid happened!
Here is how it looked:

Yeah... There is definitely room for improvement. So, first of all, the spam happened at ~5:30 in the morning for me and I tend to not read messages when I sleep.
Secondly, we need some cooldowns.

The real test however is still pending: Can I stop a spam attack by reacting super quickly?
I didnt had to wait long for the next spam attack, only a few days later I got hit again, this time I was not sleeping and able to quickly react to it.

This time I was awake AND had the ability to react to it basically instantly but did I manage to mitigate it? Not really. Yes I say "this worked out quite well" in that screenshot, but the reality is that in this spam attack, the spammers managed to send ~800 messages in 3 minutes. 800 messages and mitigation within 3 minutes is still a massive win over what it took previously, but the thing I learned in that moment was that no matter what you think how good you're prepared, when "it" actually happens you suddenly need to shift your mind from:
"Damn, I wonder who will be the murder in this movie" to:
"oh wtf, where is the spam?, where are my scripts?!?, where was my access token again?????, where is the dashboard bookmark?, uuuuh what was the plan again for how I wanted to deal with this??!?! aaaaaaaaaaa I need to do something NOW QUICK!".

It's like going to a first aid course and the person tells you that you need to follow these simple steps when you come across a car accident. Only for you to then actually encounter a car accident and throw everything overboard again and be a headless chicken for the first few minutes until you magically find your sanity again.

Phase 2: I need to automate things

The obvious next step was to let scripts fight against scripts!
My script has now shown that it can detect the raids and I know that my rooms arent that popular to naturally attract 30 new people in 5 minutes, so I got back to scripting again.
There was a second defense now in addition to the alerts. The so called "stormbreaker" script, which has pretty much the same logic as the raid detector though this one will just put all the participants of these raids on a private ban list that my moderation bots watch.

The reality is that no matter how good I get at reacting to the notifications, I will never get into the range of sending the first bans within something like 10 seconds, meanwhile for a script this is a piece of cake.

......

Fast forward to the 20'th of August and we got another round of spam!

This time it also started with a ton of accounts joining...... but! would you look at this! They are starting to get banned before even a single message was sent!

Here is another screenshot of all the join events, quickly followed by the rising line of bans:

This spam attack also went on for about 2-3 minutes, but the scripts actually worked and not a single spam message got through!
All that happened was a bunch of member events, which is still not ideal but much much much better than before.

Ok cool, so with all of what I got now I felt like I was pretty much holding up against it equally, not getting ahead of it yet, but also no longer being caught off guard all the time.

However, this was only true for my own rooms and many other rooms were still constantly seeing spam, which also still affected my server performance-wise, so the battle was not yet over!

Before we head into part 3 and focus on the final breakthrough, here is the quick recap of this post:

• I now have a dashboard with great visibility
• We have tooling to scan for insecure servers
• However, still no "good enough" source for a list of servers to scan
• Using shodan to find servers also doesn't work
• Counter-spam scripts are deployed and working for now
• Reaction times got massively reduced but it's still all reactionary
• My scripts are only suitable for my own rooms, others are still vulnerable