HCGH: The Gang Learns about DOS Attacks
How Crypto Gets Hacked: The Solana Shutdown
Note: This post is too long for your email. Open it in a new window to read the whole thing!
Disclaimer: I’ll be talking about individual crypto projects in this series but this is for informational purposes only and not a solicitation to buy or sell any cryptoassets.
Do your own due diligence.
For those who of you who don’t know about Solana or what it does, people have called it an “ETH-killer” because it does many of the same things as the rival blockchain. You can create decentralized applications on it, you can create DeFi protocols to trade tokens and cryptocurrencies, you can create and exchange NFTs, etc. You can read more about it on Packy’s Not Boring newsletter if you haven’t already (don’t worry, it’s long but not boring).
It operates in a slightly different way than Ethereum does, using something they call “proof-of-history” to get faster transaction speeds, pay lower transactions fees, and handle more transactions per second (TPS). Unfortunately for them, even though it can handle more TPS they hadn’t experienced anything like what happened next. According to Solana, it’s currently built to handle around 65,000 TPS, but on Tuesday, Sept 14th, it experienced a massive increase in transaction volume to 400,000 TPS, which was freezing up the network and started to fork the blockchain.
After 17 hours, they successfully restarted “Mainnet Beta” (yes, this project is skyrocketing while still in the beta testing phase) and getting the blockchain back on track. Something similar happened the same day to Arbitrum, (a layer 2 protocol that supports the Ethereum blockchain) where large transaction volumes caused an outage that lasted for 45 minutes. When this happens it is called a “denial of service,” or when a system is flooded with so much traffic that it makes it inaccessible to any of the users of that system.
However, this is not the same thing as the dreaded DDOS (decentralized denial of service) attacks which are a huge threat to centralized server systems. A DOS comes from a single location, in this case a single smart contract, which overloads the system with packets of data. DDOS attacks come from multiple locations, making them faster, flooding the system with more data, harder to locate and harder to defend against.
In theory, DDOS attacks are harder to perform on blockchain protocols because of their decentralized nature, and may even make them obsolete. Because there are many validators or miners all over the world processing transactions, even if some were overrun, the others would continue to support the network.
If that’s the case, then how did a single denial-of-service cause the network to freeze up? According to one Solana developer’s theory, it was probably caused by a packet-forwarding system used by the blockchain’s validators. Here’s the thread:
Basically, his/her theory is when one validator (like a miner in a proof-of-work blockchain but for proof-of-stake blockchains) starts running behind on the volume of transactions they get, they can forward the next incoming transaction orders to the next validator, then the next one, then the next one, until the que is so backed up that the network becomes inoperable.
I should also mention that although large transaction volumes caused the DOS, I don’t think it’s right to call this a DOS “attack,” which to me would involve malicious intent. The most likely explanation for what happened was a flood of excitement for a new token issuance called GRAPE, as explained by the Solana Labs CEO:
Fun fact: an “IDO” (initial decentralized exchange offering) is like an “ICO” (initial coin offering) that were all the rage back in 2017, except they are only sold through a DEX (decentralized exchange).
I also thought this was a good explanation when someone on Twitter asked how a DDOS attack could happen to a blockchain:
So this was a DOS that caused an outage, not a DDOS attack, because it mostly came from a single source (the IDO smart contract) and there was no way to run them in parallel through the validators.
Scalability is still an issue for crypto networks, and Solana is still a work in progress.
What made it more tragic for Solana was the fact they were probably days away from a patch that would have prevented the outage and system reboot:
Poor Solana.
NOTE:
As I was writing this, on September 20th, Solana released their official statement of the events:
Turns out the crowd was correct:
The cause of the network stall was, in effect, a denial of service attack. At 12:00 UTC, Grape Protocol launched their IDO on Raydium, and bots generated transactions which flooded the network. These transactions created a memory overflow, which caused many validators to crash forcing the network to slow down and eventually stall. The network went offline when the validator network could not come to agreement on the current state of the blockchain, which prevented the network from confirming new blocks.
Like I said, I don’t know if I’d call it an “attack” per se. But they definitely DOS’d themselves.
Poor Solana.
Many were quick to jump on Solana for it’s missteps, but in reality every new blockchain has its moments.
For example, something more nefarious happened to Ethereum back in 2016:
Shots were first fired at ethereum's big developer conference, Devcon2, with a mysterious message written in German and delivered via transaction method payload. The message said "Go home", but to those who have been following the network's contentious changes this summer, the full meaning was clear.
Since then, block creation and transactions have continued to be impacted, with nodes syncing up to the network more slowly. But while various fixes have since been implemented, the attacker continues to find vulnerabilities to exploit and, in turn, create new ways to launch denial-of-service (DoS) attacks.
The result: the network is being flooded with transaction spam.
Unlike Solana, this seemed to be a true “attack,” where someone seemed to be trying to bring down the network on purpose. As explained in a Medium post by Thomas Rush:
The attacker created millions of useless accounts across thousands of transactions prior to the forks. Because the hard forks do not actually remove the dead accounts directly, an off-chain process was initiated to ‘touch’ these accounts so they would be removed. This worked, but it created a second huge amount of ‘cleanup’ transactions each with its own large set of traces. Needless to say — this entire section of the blockchain — from the start of the hack to the end of the cleanup — is ugly and very bloated (which translates in our world to “slow” which we hate!).
This was more like a true DDOS attack, where hundreds of dead accounts were wreaking havoc on Ethereum.
And in another instance, something VERY similar to what just happened to Solana, happened to Ethereum when an ICO for the Bancor token was issued:
On Monday, a much-hyped startup called Bancor, which wants to do away with cryptocurrency exchanges using smart tokens, managed to raise $153 million in Ether in just three hours. This was the largest Initial Coin Offering—a way of crowdfunding a blockchain startup—in the network's history. However the huge number of transactions being carried out as people rushed to buy Bancor tokens caused the Ethereum network to grind to a halt.
In other words, the popularity of the Bancor ICO had caused the Ethereum network to essentially launch a distributed denial of service (DDoS) attack on itself.
My only gripe is if the Solana situation was a DOS, then this was also a DOS and not a DDOS, but I’ll let others smarter than me decide that.
And finally, even the mighty, legendary Bitcoin was trying to be DOS’d at one point:
The issue is that the digital "blocks" that contain every Bitcoin transaction are being filled up. Transactions are put into blocks by Bitcoin users, who are incentivized in part by fees that people attach to their transactions. Transactions aren't complete until they're put into blocks, or "confirmed." But with near-full blocks, there's more competition for space inside them, and so miners logically seek out the transactions with a decent reward attached before those that don't—meaning those who are willing to pay more in fees will get their transactions confirmed first.
Transactions without a reward are thus left to languish in what's known as the "memory pool," which is stored on every computer running Bitcoin software. When that memory pool fills up, the whole system slows down, and that's exactly what's happening right now. The question now is why.
According to longtime Bitcoin developer Jeff Garzik, someone might actually be taking advantage of fuller blocks with an attack that artificially pushes the system over the edge.
"There is an uptick in correlated transactions in the lower fee strata," Garzik wrote me in an email. "This would seem to imply that a single wallet or set of wallets is potentially sending the same coins over and over, to themselves. This may be innocent transaction activity, coin 'tumbling,' or an unknown actor paying transaction fees to load the bitcoin network with traffic."
Eventually though, like Solana, these issues were eventually resolved with new updates and improvements that would make the networks safer.
Thoughts:
The Solana network was not “attacked” as much as it was a victim of its own success. A flood of excited users trying to get in on a new and untested token ended up creating a self-inflicted DOS on Solana.
What was most impressive to me, in this instance and in all the previous instances of the past with other blockchains, was the quick turnaround time to get consensus from miners/validators to come to a solution. From Solana:
One of the biggest benefits of blockchains is that, even in complete liveness failure for any reason, the validators are individually responsible for recovering the state and continuing the chain without relying on a trusted third party. On a decentralized network, each validator works to bring it back and has their work guaranteed and verified by everyone else. This was a coordinated effort by the community, not only in creating a patch, but in getting 80% of the network to come to consensus. There’s a big difference between an outage like this happening on a centralized network (like Amazon Web Services) and a decentralized network like Solana. If AWS crashes, users have to trust Amazon to bring it back to the right state. The credit and obligation for restoring network operations on any blockchain is in the hands of the community. Operators across the world worked together to reach a solution and restored functionality.
Blockchains are only as strong as the community that holds them up.
I had no idea until I started doing this that Bitcoin and Ethereum have also basically handled their own versions of DOS attacks in the past, and look at them now! At the time of the above articles, bitcoin was trading around $432, and ETH was trading around $12 during DevCon and then around $357 during the Bancor ICO. They were able to come together, solve the issues at the time, and continue building.
As I write this (September 2021), crypto markets are crashing.
Guess we can all have fun staying poor together now! YAY!
Thanks for reading.