In what stands as one of the most catastrophic failures in both cryptocurrency and cloud infrastructure history, FTX's collapse serves as a stark reminder of how poor cloud practices can accelerate a company's downfall.
The Infrastructure Nightmare
On March 28th, 2024, FTX's previous CEO, Sam Bankman-Fried, was finally sentenced to 25 years in prison. Ordered to forfeit $11 billion, he had been convicted on multiple counts of fraud related to his management of the FTX companies.
So, what had happened behind the scenes? When John J. Ray III took over as CEO during FTX's bankruptcy proceedings, what he discovered was beyond concerning. The company, valued at tens of billions billion at its peak, had no documentation of their cloud architecture. Security protocols were virtually non-existent, and access controls were managed through shared credentials stored in an unsecured group chat.
Customer funds, corporate assets, and critical data were all managed through a tangled web of poorly configured cloud services. The lack of proper segregation between development and production environments meant that anyone with access could potentially modify live trading systems. Aside from this, the John J. Ray's First Interim Report revealed a plethora of other cloud security flaws.
Retributions from the Report
The following cloud security oversights come primarily from John J. Ray III’s First Interim Report, and are cited by page number and/or footnotes:
1.) Lack of Proper Asset Segregation- FTX.com, FTX.US, and Alameda shared a single AWS account, violating fundamental security principles of segmentation (p. 33). The shared infrastructure meant a single compromise could expose all entities' assets.
- Failure to maintain separate collaboration platforms and password vaults for different entities increased security risks (p. 33, footnote 34).
- The company failed to implement appropriate network segmentation to contain potential breaches and limit the impact of unauthorized access (p. 33, footnote 34).
- Over a dozen people had access to central omnibus wallets containing billions in crypto assets (p. 30).
- Failed to implement "least privilege" principle for system access (p. 30).
- Did not enforce multi-factor authentication for critical services like Google Workspace and 1Password (p. 31). This lack of MFA was particularly ironic given that FTX recommended customers use MFA on their own accounts.
- FTX Group generally did not use SSO, an authentication scheme used to manage user access centrally. Without SSO, the company couldn't effectively manage or revoke user access, enforce MFA, or prevent users from having multiple accounts with separate passwords (p. 32).
- The company did not institute any basic mechanism to be alerted to "root" login to its AWS account (p. 34).
- Amazon GuardDuty, an AWS feature that supports threat detection, was not enabled at all on FTX.com (p. 34, footnote 36).
- VPC flow logs that can capture IP traffic information were only enabled to log rejected traffic, and only in some networks (p. 34, footnote 36).
- To manage inbound internet traffic on a key server, the FTX Group used a version of software that was nearly four years out of date, leaving the server exposed to known vulnerabilities (p. 35).
- As a result from many of the above, the FTX Group did not learn of the November 2022 Breach until the Debtors' restructuring advisor alerted employees after observing suspicious transfers via Twitter and other public sources (p. 34).
- Absence of comprehensive organizational charts prior to late 2021 (pp. 8-9).
- No written plans, processes, or procedures explaining the architecture or operation of computing environment or storage of crypto assets (p. 38).
- Lacked tracking of employee workstations, software application servers, and business data. Failed to maintain inventory of third-party cloud service. (p. 35)
- IT professional's attempts to create device inventory were undermined by management's non-cooperation. Senior management failed to identify electronic devices they were using despite requests (p. 35).
- Had to analyze financial records and search through employee communications just to understand the scope of services used (p. 35).
- The FTX Group had no comprehensive record from which it could even identify critical assets and services, including employee workstations, software application servers, business data, and third-party cloud services (p. 35).
- Stored sensitive data including private keys and API keys in unencrypted files (p. 36).
- No effective process for securely introducing, updating, or patching software (p. 37).
- Minimal code review and testing procedures (p. 37).
- Hundreds of secrets, including passwords for crypto wallet nodes and API keys for exchanges, were kept in source code repositories with wide accessibility (p. 36).
- Absence of procedures like scanning to continually ensure the integrity of code running on FTX Group servers (p. 37).
- Inadequate review, testing, and deployment practices failed to ensure code was functioning as expected and free of vulnerabilities (p. 37).
- A senior developer deleted a file containing secrets from a repository but failed to remove it from the code history. This practice went against GitHub's recommended practices, leaving the file exposed to anyone accessing the code repository (p. 36, footnote 41).
- Absence of a structured approach to incorporate security requirements into the system development lifecycle (p. 37, footnote 42).
Lessons from the Ashes
The culmination of all these endless cloud security malpractices are best represented by John J. Ray's lamentation during the filing with the District of Delaware Bankruptcy Court:
"I have over 40 years of legal and restructuring experience... Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy financial information." - John Ray III, CEO.
As of this moment, a number of other senior-level employees of the company, including Bankman-Fried’s co-founders, are continuing to cooperate with the Department of Justice. Most of the FTX funds have not been recovered, and forensic analysis of FTX is ongoing.
The FTX fiasco teaches us that even in the fast-paced world of cryptocurrency, fundamental cloud practices cannot be ignored. Proper infrastructure management isn't just about keeping systems running—it's about protecting billions in customer assets and maintaining the trust that's essential to any financial platform.
As the dust continues to settle, one thing becomes crystal clear: in the world of cloud computing, there are no shortcuts to success. The price of ignoring basic cloud practices isn't just technical debt—it's potentially criminal liability.