Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Benefit of using Cloudflare CDN:
Commenting and editing took about 0.5 seconds!
Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):
64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms 64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms 64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms 64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms 64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms 64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms 64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms 64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms 64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms 64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms 64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms 64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms 64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms 64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms 64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 msThe bugs in Lemmy are such that you don’t even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today’s crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.