4/30/2022: Post-mortem of Mainnet sUSDv2 metapool exploit

UPDATE: Figures for exploit amount rectified to reflect total amount hacked.

On 4/30/2022, the sUSD metapool on Mainnet was exploited for $11.9m due to a vulnerability resulting from reusing an incorrect library deployment.

Because of the vulnerability, a malicious blackhat hacker was able to use a flash loan attack to drain $11.9m in funds from the sUSDv2 metapool.

The total amount of funds drained would have been greater. However, $3.97m was secured by BlockSec, a whitehat security firm. By using an internal bot that detects and tracks hacking activities on the blockchain, BlockSec was able to frontrun the theft of an additional $3.97m by the attacker. In addition, due to rapid response to the incident, the attack affected only one of three pools that were vulnerable to this exploit.

Upon learning of the exploit, Saddle immediately paused all pools. On 5/2/2022, Saddle pushed out a fix for the vulnerability. Today, Saddle is resuming metapool operations for unaffected pools, as detailed below:

• Arbitrum USDs (Sperax) metapool - unaffected by vulnerability - Same pool is resumed
• Evmos tBTC metapool - unaffected by vulnerability - Same pool is resumed

We’re pushing new contracts with the fix for the affected pools, which will be deployed later this week as new pools.

The team is also continuing work on remuneration plans for affected LPs, a bounty of ~$400K to BlockSec (pending governance vote), and implementing additional security and monitoring measures. Read on to learn more.

The response strategy

All metapools on Saddle were paused within approximately ~1.5 hours of identifying the attack – both those affected by the vulnerability, as well as those unaffected, were paused. While individual asset withdrawals were prohibited, balanced withdrawals – meaning any withdrawals that did not rebalance the pool – were still possible.

The fix

In addition to the exploited sUSD metapool, two other pools that still contained the previous, incorrect MetaSwapUtils library were identified. Testing was done to ensure that the hardhat-deploy plugin was deploying the correct, new library on these pools. All pools and metapools were audited to ensure they were using the correct libraries.

Timeline of events

• 12/11/2021 - “wCUSD metapool updated” deployed with incorrect metaswaputils contract [link]
• 12/15/2021 - “sUSD metapool updated” deployed with incorrect metaswaputils contract [link]
• 4/30/2022 12:40 AM PDT - sUSD metapool is exploited for $10.2m [link] by unknown attacker
• 01:01 AM PST - Community member @expenbik shares an etherscan link to the exploit transaction in the “Saddle Community” Telegram group
• 01:04 AM - BlockSec Whitehat Ops frontruns another attack and removes $3.97m from the sUSDv2 pool [link]
• 01:20 AM - Blockchain Security company Peckshield tweets at Saddle with a link to the BlockSec transaction
• 01:24 AM - sUSD metapool is exploited for an additional $1.6m [link] by same attacker
• 01:29 AM - Multisig member @Aurelius pings the multisig chat with the same tweet and tags the Saddle engineering team
• 01:35 AM - @Aurelius starts creating transactions to pause pools
• 01:40 AM - Tweet is viewed by Saddle core contributors and engineers
• 01:42 AM - Pool paused: tBTCv2 Metapool MAINNET [link]
• 01:53 AM - Pool paused: USDs Metapool ARBITRUM [link]
• 02:13 AM - Pool paused: sUSD Metapool MAINNET [link]
• 02:19 AM - Pool paused: wCUSD Metapool MAINNET [link]. This is the last metapool with significant TVL, thus completing the mitigation
• 5/02/2022 - Fix for the vulnerability is pushed out
• 11:00AM - Team completes a post-mortem and assigns action items
• 5/03/2022 - Two metapools that were confirmed to be unaffected by the vulnerability are unpaused

What is a metapool

There are 2 kinds of StableSwap pool implementations:

• Standard StableSwap pools - A standard StableSwap pool is composed of two or more assets which share a value peg, e.g. WETH and sETH.
• Metapools - A metapool also pairs several pegged assets together. However, one of the assets is an LP token belonging to another Saddle base pool. Through this mechanism, it is possible to create many pools of different assets, while still sharing a price peg and not fragmenting liquidity.

Due to these features, metapools have a higher degree of complexity compared to standard pools, resulting in more potential attack surface.

Why the vulnerability affected the metapools

A vulnerability in a MetaSwapUtils library for metapools had been previously identified. After that, a new library had been created and deployed which patched this vulnerability.

However, the old libraries, with the vulnerability in them, were reused incorrectly. This incorrect MetaSwapUtils library was used for three pool deployments: sUSD Meta V2, tBTC Meta V2, and wCUSD Meta V2.

The reason the old libraries persisted was because deployments are controlled by the “hardhat-deploy” plugin. This plugin has the behavior of defaulting to using previously-deployed instances of libraries, instead of redeploying new libraries if code is updated.

By deploying using the previous vulnerable library – instead of the new library which patched that vulnerability – the vulnerability was able to persist. Further, had we fully operationalized internal QA practices and routinely carried out post-deploy verification steps, the vulnerability would have been likely to have been identified earlier.

What comes next

The underlying vulnerability is fixed. All pools will be unpaused or redeployed. As of today, we’ve unpaused the pools that were found to be unaffected by the vulnerability. Next, for pools that were affected, we’ll be re-deploying the pools with instructions for LPs on how to migrate to the redeployed pools.

LPs who lost value in the attack will be remunerated. Saddle expects to return the $3.97m to affected users pro-rata and award a bounty of ~$400k to BlockSec, pending governance vote. The Saddle community is also currently discussing a remuneration plan for LPs who lost value from the attack, which will also be put to governance vote. We’ll communicate further details through Twitter, Discord, and Telegram.

Security will remain the #1 priority for Saddle going forward. In addition to having pushed out the fix described above, we’re also taking the following steps to maximize security and safety of metapools, contracts, and deployment practices:

• Implementing OpenZeppelin Defender to monitor Saddle metapools’ virtual price and to monitor for large outflows of capital
• Engaging with Certora to conduct formal verification of all smart contracts
• Soliciting auditors for a follow-up formal audit for the metapool contracts
• Strengthening internal QA and post-deployment verification processes

With the above steps, we’re confident that the vulnerability would not have been missed and that our ability to detect security issues in the future is greatly enhanced. Saddle is as secure as peer protocols.

Saddle is resilient

We agree with our Bankless friends: crypto is resilient. Not only Saddle, but several DeFi protocols were targeted, reflecting the growing pains of an ecosystem that more and more people see value in. How we respond to come back stronger and build an even more secure, safe, and anti-fragile ecosystem of decentralized financial building blocks is how we will all make it.

Special thanks goes out to the BlockSec team for their quick response; to @Aurelius of Synapse (on the Saddle multisig) for initiating pausing of all metapools; and to many Saddle community members and core contributors who looked out for each other by educating, spreading accurate information, and responding proactively and positively.

As a community and an ecosystem, we’re stronger when we come together.