- Thread starter
- Staff
- #1
This is a story about the internet, and how it doesn’t work like it should, when it works at all, except when it doesn’t. It’s a story about how the internet stopped working for our customers, and how we fixed it.
The internet is more of a wild frontier than we give it credit for being. Stray from the safe confines of your two-factor identification and trusted cookies, and it can be a bit of a wilderness out there, full of random trolls with the maliciousness—and, increasingly, the ability—to make your life pretty miserable for no other reason than because they can.
Back in 2010, the method those trolls were using was distributed denial-of-service (DDoS) attacks. “Distributed” refers to a large number of internet hosts maliciously flooding a particular target IP with traffic in an effort to overwhelm the network stack. This is called a volumetric attack, and the intent isn’t to try to get into the network. It’s just trying to deny service. A distributed denial of service means that legitimate people who want legitimate service are crowded out by the bad traffic.
We tried several solutions to deal with DDoS attacks before we arrived at one that worked. Initially, we attempted to filter the traffic with a powerful network switch. Unfortunately, this type of filtering is inherently difficult to do with game traffic. It is the nature of game servers to receive unsolicited TCP traffic from arbitrary IP addresses. Imagine you had a post office that weeded out unwanted junk mail for you. But now imagine your job is as an advice columnist, and you receive tons of legitimate mail from random strangers all the time. For you, the post office doesn't know what's junk mail and what isn't. That's how traffic to game servers tends to look. Furthermore, the source IP in TCP packets is not secured, and can be easily spoofed. Our post office cannot even look at the return address on the envelope for clues, because the senders of junk mail forge that.
To prevent attackers from using our own network to attack our servers, we needed to control all the entrances and secure them. We accomplished this by creating proxies for game traffic, routing every single packet of data transmitted across the network through relays. Now when a client wanted to talk to a game server, it had to do so through a relay that both authenticated it and proxied that traffic to the game server. This meant the IP address of the server was always hidden—the attacker simply had no idea where to attack.
To re-use our antiquated post office metaphor from earlier, our spammer no longer had an address to send junk mail to. They could send it to every post office in the area and ask them to mail it, but without authorization, that post office isn’t going to. (Moreover, that post office would find it a little suspicious that someone was trying to send a single person 100,000 letters.)
But couldn’t you just attack the relay? Technically, you could. But we have an essentially limitless number of them, and we built them to be attacked. A “relay” is just another word for a computer running software. You can attack it or take it offline, but the protocol was designed with that assumption in mind. If a client is trying to play a game and loses contact with a relay, it just switches to another. Relays are like hundreds of pawns scattered around the world with the singular purpose of guarding the game server. (Incidentally, taking out a relay is harder than it sounds. They’re engineered pretty well and positioned in a specific part of the network, so although they were built to be taken offline, we haven’t lost one yet.)
The solution was straightforward but effective. Before, if someone wanted to disrupt a game, they could just overpower a single game server (a very low bar to clear). Now they had to overpower essentially the entire data center—a much, much, much higher bar. Are there attacks that could still accomplish this? Of course. Are there attacks that can do this that anyone online could buy for five dollars? No. An attack this sophisticated was officially out of the price range of most people.
With this new system up and running, we had an epiphany: If we controlled our own private network, we wouldn’t be beholden to how the normal internet works. We could use it to make the customer experience even better. With the normal internet, when you send a packet from one IP address to another, the route you use is determined by Border Gateway Protocol (BGP). This is a routing algorithm that decides how your packet will travel across a network, and you have no choice in the route it picks.