A Thousand Newlines: Matrix Spam, #1 — Man, I just want to chat

I made some public chat rooms to chat about servers and networking, but after a while we got attacked by spam from servers all around the world.
I had to choose between giving up on public rooms and taking the L or fighting against the spam to demonstrate that we can in fact have nice things.

This post will mainly explain the spam itself, if you're already familiar with scripted open reg spam then you can skip this one and jump directly to the second post.

Introduction: how did we get here?

Back then I joined Matrix because I wanted to bridge Whatsapp and Signal so that I no longer have to use two different apps to stay in touch with all my contacts, but after doing so I discovered many nerdy public Matrix rooms and found myself very welcomed by the community. After a while I made a few public rooms myself, one of them being a "Community edition" of a Synapse support room (Synapse is a software for hosting Matrix servers).

At the start it was really nice, we were able to help each other with our setups, shared yaml files and generally had a good time.
But then strange things started happening, people started to attack the room with spam, not only my room but all kinds of public rooms that had more than just a hand full of users.

At the time I was luckily already using a moderation bot and subscribed to a few ban lists, most notably the CME ban list. This list is highly trusted in the community and has volunteer operators all around the world in many different timezones which means that without much effort there are people online pretty much at all times that have the ability to add spam accounts to the list. The Moderationbot can then read those bans and will ban the users from my room automatically.

Even better, I was also already part of the CME community through some volunteer moderation work for envs.net.

This works really well for "regular" spam where one or a hand full of accounts join rooms from time to time and drop a scam link or maybe even a shock image but this spam was very different and neither me, nor anyone else was prepared for it.

This first blog post of the series will focus mainly on explaining and visualizing this spam and it's effects as best as possible. I will use lots of images with shiny colors so you don't get tired and I invite you to click on them and take your time to look at some of the data in a bit more detail.
Note that not all of the screenshots are related to the exact same incident but are instead selected based on how good they convey the information that I aim to display in that moment, so don't be confused if timestamps on graphs are not consistent across this article.

What kind of spam is this?

When talking about spam, many people probably think of random crypto or romance scams but in this case the spam was actually meant to annoy and disrupt the chat itself as much as possible.

It did this by sending messages containing purely newline characters and mentions of random users from the member list of the room so that everyone gets pinged by thousands, sometimes tens of thousands of empty messages with thousands of blank lines.
An actual real world example of such a raw message body looks like this:

{
    "body": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
    "m.mentions": {
      "user_ids": [
        "@example_user_0:example.org",
        "@example_user_1:example.org"
      ]
    },
    "msgtype": "m.text"
}

This message contains a staggering 2 thousand newlines.

To put this into perspective, here is a screenshot of me fitting ~50 newlines on a 1249px screen with no zoom factor.
The original spam messages were 40 times larger than what you can fit on a regular sized chat window. You can imagine how many times one would need to scroll just to get past a single one of these messages.

But wait, there is more!

If this was only one account, then sure enough you can easily copy the user id, paste it in your room with your moderation bot and let it handle the rest, ban the user and clean up the mess....

This is another example from my room. Here you can see a wave of new accounts with randomly generated usernames all join at the same time.

As you can see, it's not just a single or even just a hand full of accounts, the whole screen is filled.

The largest amount of accounts that we saw being used in a single attack were 935.

The keen observer might have even noticed that these accounts are not only seemingly random in their names, which makes it basically impossible to write functioning glob or regex patterns to catch them, they are also all coming from several different homeservers. This means a simple wildcard ban rule for all users of one particular server also won't work.

On top of this, once a spam attack like this one started, clients like element web become practically unusable due to how much they start lagging under the rate of incoming messages (insert advertisement for Gomuks Web here).
Servers of the Conduit family further worsen this by NOT having any functionality for rate limiting user actions; meaning that a spam account on a conduit based server will send new messages/events as fast as the server possibly can.

Now, what is wrong with these servers and why are they sending so much spam you might ask. The reality as it turns out, is that whoever is behind these spam attacks is searching for and finding servers that (usually) accidentally were configured with extremely insecure registration requirements. This means no captcha, no email verification, no nothing to stop or slow down bots/scripts, all you need is a username and a password and you have your new account.
This is what the spammers are abusing, they run their script and have hundreds of accounts within a few seconds.
In most cases these servers were set up by some individual as a personal server and they may have configured it like that during setup so they can register their own account or a friends one and then simply forgot about it.

This means that it's not the server operators themselves launching some attacks here but much more them also falling victim to the same abusers but in a slightly different way than those on the receiving end of it.

The numbers!!! How does this look in reality?

See the below screenshot for a visualization in grafana that shows a count of message events (y axis), per individual user over time (x axis).

Here a single account managed to send over 3k messages during the 15 minutes spam wave. That's quite a bit more than what normal activity looks like, so this unusual load is also affecting the homeserver.

For those that come from Discord: In Matrix when we speak of servers we mean the actual thing with the blinking lights, "Synapse" for example.

More graphs!

Let's look at what this means for the actual server itself.

As you can see, my server is keeping up quite well, the CPU doesn't even go over 30% on any of the processes. However, compared to how the lines look on the left and the right side, you quickly realize how big the difference is.

What I haven't told you yet either is that despite codestorm.net being a private homeserver for just myself, it not only runs on stupidly strong hardware, but is also highly optimized.

What you're looking at is a Matrix server that runs as a lxc container on my Proxmox host, with access to 10 cores of a overclocked Ryzen 9 7900 and 30GiB memory, a postgres db that was configured and tuned specifically for Matrix and runs in the same container to ensure the lowest possible latency.
All components talk to each other over unix sockets and the database runs on a pcie-gen4 nvme ssd with read/write speeds of up to 5GB/s (and trust me, during some attacks synapse has made postgres push gigabytes/s in disk I/O).

So while my server has yet to face a wave of spam that would actually manage to shake it down, people hosting their server on weaker hardware stand absolutely no chance against something like this, especially the cheap 4C/4GB VPS that many like to choose.
I saw servers of friends needing several hours in some cases to catch up with processing the backlog of events with their CPU pinned to 100% and being practically unusable during that phase.

Well, sounds like something a rate limit could solve?

Almost, but there are multiple problems with this. The first one is that due to how matrix works, lots of events might have already been sent "to the room" before my server has even seen a single one of them. This combined with that in order for my moderationbot to even realize this, my server first has to receive and process these events. By the time that happened (we are talking about milliseconds to seconds), countless messages might have already been sent.
Well, that would still be better than nothing but in reality it doesn't matter too much if you have 3k or 20k spam messages in your room. The timeline and history will still very likely get messy and clients like Element web and others already start struggling after a few hundred redactions in a row that they have to process while you try to scroll up.

On top of that, the only available option at the time was the BasicFloodingProtection in Draupnir which comes with a warning that it contains bugs. Bugs like for example not realizing that certain events were sent by privileged users. So when I try to ban a bunch of accounts very quickly with a script, my own bot tries to ban me for sending too many events in a too short timespan.

So, how often did this even happen then?

I'm glad you asked!
See the below screenshot for a overview of how many accounts got added per day to the previously mentioned CME ban-list in all of 2025.

Basically every day where the bar went above ~20 and the graph turned yellow or even red was a day with a spam wave like this (the first horizontal line starts at 50 btw).
The accounts and servers involved in each spam attack got of course all added to the ban list but as we would eventually find out, there seems to be plenty of easy to abuse servers.

As you can surely imagine, being affected by spam attacks this frequently is not nice. The cherry on top is that due to how matrix works, by being beautifully decentralized, a spam wave doesn't even have to happen in my room in order to cause load on my server.
Any room that a user of my server (me) is in is a potential source of load, caused by spam attacks.

So, to give a short summary and wrap it up before the next part:

• It is not the server operators doing the spam
• Spam comes from random servers all around the world
• Ban one server and a new one shows up
• You can't preemptively add filters to catch the user IDs
• Spam causes so much load on servers that they become unresponsive
• Existing tools are insufficient
• Many spam accounts get created on servers with no rate limits
• All tooling is purely reactionary

Alright, That's it for the introduction, you should now have a rough understanding of the nature of the spam itself, the effects it has and it's challenges.
The story will continue in part 2 of the series, you can use the navigation box below.