Railway Says Stolen Customer AI Keys Led to $740,000 in Estimated Unauthorized Usage Charges

Railway, a platform-as-a-service provider, announced that a breach in October resulted in the theft of AI provider keys stored by customers, leading to unauthorized spending estimated at $740,000. The company has committed to covering these charges when AI providers do not offer refunds. This incident highlights the risks associated with storing sensitive credentials online, as attackers decrypted environment variables in up to 1.85% of workspaces, targeting AI provider keys specifically.

Railway's platform is designed to build and run applications for its clients, storing sensitive information like API keys and database credentials as environment variables. During the breach, the attacker decrypted variables from approximately 100,000 workspaces, focusing on AI provider keys. The company initially informed customers that it could not refund unauthorized AI spending and advised them to contact their AI providers directly. However, it has now pledged to cover costs not refunded by these providers.

The company's incident report revealed that the breach also exposed linked-account tokens, including GitHub OAuth tokens and BYOK agent keys, for 1,852 users. These tokens were reset, and users were notified, though no unauthorized logins were detected. Additionally, the attacker accessed credentials for cloud, database, and payment services, but forensics found no evidence of customer databases being accessed or code repositories being downloaded.

The breach began on October 20, when the attacker exploited a previously unknown flaw in Railway's build system, allowing partial escape of filesystem isolation and access to host virtual machine files. This access provided a credential used by build servers, which led to further exploitation of internal systems. The attacker eventually obtained a credential capable of decrypting customer variables, leading to the unauthorized access of AI provider keys.

Railway has advised customers to revoke and replace any sensitive variables stored on the platform before October 21. The company has also taken steps to improve security, including running each build in its own short-lived virtual machine and restricting credentials to specific tasks. Despite these measures, the exact method of estimating the $740,000 in unauthorized charges remains unclear, as does the period covered by this estimate.