On January 20th and 22th, one of the servers of la Quadrature du Net, the Paris-based digital freedom association I co-founded and that my company is freely hosting, got hit by some distributed denial of service attacks.
Here is a quick analysis of what we were able to understand about those attacks…
Introduction : the DNS
When you surf the web for a webpage (eg http://fr.rsf.org) your browser starts by asking a server called the DNS server, a service your Internet Access Provider usually provide you with. This service tells you where this domain name and website are on the Internet, replying usually with an IP address (here 188.8.131.52 for rsf)
Then your browser calls this IP address and ask for the page (here http://fr.rsf.org). The web server at this IP address knows it is hosting this domain name, it take the web page from its harddrive and return it to you.
Story of the attack against LQDN
So, on January 20th, we saw a big spike in the load of a server, named PI, owned by La Quadrature du Net. We logged into this machine and saw a few interesting things happening:
– Many requests on the DNS server of La Quadrature, but for domain names that we don’t host here, those requests are therefore refused, and a line is written into our log files to remember this.
– many many unterminated requests to the HTTPS web server, blocking it, waiting for answers from the Internet user, creating a saturation of this service.
– many requests to both HTTP and HTTPS servers to web pages we are not hosting here, such as tracker.thepiratebay.org or www.facebok.com!
Those 3 problems were found in the log files of the server, as you can see in the examples below:
Example of a denied DNS request:
Jan 20 08:03:59 pi named: client 184.108.40.206#33795: query (cache) 'graph.facebook.com/AAAA/IN' denied Jan 20 08:09:19 pi named: client 220.127.116.11#10144: query (cache) 'cs20.wpc.edgecastcdn.net/AAAA/IN' denied
Example of a non-terminated web request:
18.104.22.168 - - [20/Jan/2015:09:31:29 +0100] "-" 408 - "-" "-" 0 - 22.214.171.124 - - [20/Jan/2015:09:31:29 +0100] "-" 408 - "-" "-" 0 -
Example of requests for domain names we are not hosting here: (access then error logs are provided)
126.96.36.199 - - [20/Jan/2015:09:32:30 +0100] "GET /announce?info_hash=&peer_id=&ip=188.8.131.52&port=11628&uploaded=703804438&downloaded=703804438&left=314346&numwant=200&key=27066&compact=1 HTTP/1.0" 404 418 "-" "Bittorrent" 0 tracker.thepiratebay.org 184.108.40.206 - - [20/Jan/2015:09:31:58 +0100] "POST /restserver.php HTTP/1.1" 404 416 "-" "-" 0 api.facebook.com [Tue Jan 20 07:46:34 2015] [error] [client 220.127.116.11] File does not exist: /var/alternc/bureau/admin/announce [Tue Jan 20 08:33:29 2015] [error] [client 18.104.22.168] File does not exist: /var/alternc/bureau/admin/graph.facebook.com, referer: http://quetwo.com/2011/12/01/arduinoconnector-ane/
How did we block that?
We have 3 patterns that we should block.
The first, the denied DNS requests, is not easy to block, since the DNS does not consist of exchanging information with the source of the request. Any Internet user could send a DNS request with the source address of somebody else. The answer (or the answer denial) will come to this somebody else!
So, if we block the IP addresses doing those bizarre queries, we will likely block anybody else than the attacker, therefore denying access to our websites to legitimate users…
The second one, the SYN flood attack, was not anonymous. Indeed, when we connect to a web server, we send and receive a few packets of information between the server and the browser, in a way described in the TCP protocol. In a simplified way, that’s:
- Browser sends a TCP/SYN packet to Server
- Server replies with a TCP/SYN+ACK packet to Browser
- Browser replies with a TCP/ACK to Server
- The discussion can now starts, the 2 parties recognized properly eachother.
This protocol implies that the source IP address (the Browser) must be who it pretends: If I emit a TCP/SYN packet with the source IP address of somebody else, the TCP/SYN+ACK packet will not come back to me (it will arrive at the counterfeited IP address) and therefore, I’ll not be able to answer with the proper TCP/ACK.
In our case, with those unterminated (timeout) pages, we had a proper SYN, then SYN+ACK, then ACK exchange. The source IP address is the one currently attacking us, we can block it.
The third case is even easier, we received properly formatted HTTP requests from the browsers asking those random pages that we don’t host, and we passed some time answering “Page not Found”, the famous 404 Error!
So we used the IPSET filter we usually use at Octopuce (my hosting company, article in French), precisely used to block this kind of attacks, usually not dangerous for the network, but exhausting for web servers.
Thanks to the 2 rules below, we were able to block those malicious Internet users:
pi# tail -F access.log|grep --line-buffered '"-" 408 -' |sed -ue 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1.\2.\3.\4/'|blacklist add pi# tail -F error.log|grep --line-buffered "File does not exist: /var/alternc/bureau/admin/" | sed -ue 's/^.*client \([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*$/\1.\2.\3.\4/'| blacklist add
Within a few minutes, we were able to block 3000 then up to 17400 IP addresses.
A few minutes later, the attack was completely blocked, the firewall refusing any connection from those IP addresses.
Second attack, enlarge your blocking
The number of invalid queries went down to almost nothing, and the IP addresses progressively disappeared from the blocklist (since they didn’t try again). Everything was calm the morning after.
On January 22th, 48 hours after the first attack, we had the same issue. Adrienne, who coordinate the campaigns of la Quadrature du Net, told me she was not able to read her email: she detected the attack just before our monitoring software!
Immediately, I restart the 2 automated blocking requests from above, but, quickly, the blacklist tool told me that it was not able to add more IP addresses! We went up to 1 million blocked IP addresses, filling up the IPSET blacklist table!
As a result, we went to the second step of the blocking system: instead of blocking individual IP addresses, we started to block /24 prefixes (the name we give in the network world to 256 successive IP addresses, whose 3 first numbers are identical)
The filtering rules became:
pi# tail -F access.log|grep --line-buffered '"-" 408 -' |sed -ue 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1.\2.\3.0\/24/'|blacklist add pi# tail -F error.log|grep --line-buffered "File does not exist: /var/alternc/bureau/admin/" | sed -ue 's/^.*client \([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*$/\1.\2.\3.0\/24/'| blacklist add
We quickly went up to 8000 blocs of 256 IP addresses blocked, and soon everything went calm again.
We were able to see, after the operation (this graph was captured the next sunday) the spikes in the Apache web server of la Quadrature (please note that this is not the main web server of LQDN, this one only serve some satellite websites and the webmail. Tau, our main server for www.laquadrature.net, was not attacked)
Where do those attacks come from? and why?
From the size of this second attack (we are talking about a million of IP addresses and millions of DNS / HTTP / HTTPS incorrect requests), an inquiry, even a quick one, was required, to better know, and maybe find a reason for this attack, and maybe be able to stop it faster next time.
Of course, we immediately checked the country of those IP addresses, and the result is quite expected:
- 7980 blocs from China
- 4 from Japan
- 6 from Honk Kong
- 8 from Taiwan
The attack is coming mostly from Chinese computers.
On the Chinese Internet, and on its censorship system
Since then, with a nice help from Stéphane Bortzmeyer summing up (in French) those events, we finally understood what was going on:
- China is censoring its Internet, that’s well known
- to do this, this country censors (among others) DNS queries in its network (and also censoring as a side effect, the rare Japanese, Korean or Taiwanese queries going through China)
- when it answers a DNS query to a censored website, it answers with “any incorrect IP address” instead.
we thought for a long time that those incorrect IP addresses were randomly chosen, but nothing proves it either …
But, since a few days, we see spikes of requests to websites censored in China coming to IP addresses such as those of La Quadrature du Net. Other people had this same issue : http://furbo.org/2015/01/22/fear-china/
So, the end story is that we just saw censored websites requests coming to La Quadrature du Net’s IP address from China, due to how the Chinese Internet censorship is working!
We felt it like the side effect of the great firewall of China against freedom of speech and free flow of information… Quite unexpected to see it pointing to La Quadrature !
Whether it is done on purpose or not, this technical behavior of the great firewall of China can come from different scenarios:
- The censorship system in China chose to answer to DNS requests on censored websites with IP addresses less random than before, (creating a concentration of traffic to those IP addresses)
- Maybe one of the system administrator of the great firewall of China is gaining some small and quick money selling DDOS, selling Internet attacks to the highest bidder (in bitcoin? 😉 ) and using that censorship system as a weapon
- Maybe China chose a precise list of targets to send censored traffic to, adding to this technical “useful” process (the censorship) a “nice” one (putting down foreign opponents’ websites)… La Quadrature du Net, as a digital freedom association, seems to be too nice a target (among others of course).
- Maybe nothing of this is true, we were just randomly chosen, and that’s no luck
Anyway, this attack shows something obvious on the way the Internet works today: a big guy (here a very big, namely China), can easily shoot down a small or a very small one, saturating his network, his servers, his infrastructures, with dummy requests or unusually high volumes of data.
Internet starts to become hard to handle when you are not Orange, Cloudflare or Google, and when you are hosting contents that people doesn’t really like.
What can we do against that? Putting some anti-ddos system at the core of the network as some Illiad, OVH or Cloudflare do? With Arbor brand or equivalent hardware? This is sad, since this kind of hardware is clearly doing DPI (Deep Packet Inspection) seeing and learning on what packets they route, which is clearly a violation of the network neutrality principle… And more, those anti-DDOS system requires big pipes anyway, pipes capable of filtering those big traffic volumes, and therefore they require you be a big network operator.
Is the Internet starting to condemn smaller actors?
… think about it.