RMM Alerts at 2 AM: An MSP and SOC Incident Response Story
A managed service provider faced significant downtime after a cyberattack compromised their remote monitoring tools.
In early September, a 2 AM alert from the Huntress Security Operations Center (SOC) revealed suspicious activity involving remote monitoring and management (RMM) tools within the network of managed service provider TruePath IT, putting nearly 1,000 endpoints at risk. The incident resulted in an estimated $3 million in downtime and ongoing stress for the client.
The initial alert prompted immediate action as Huntress identified compromised accounts with extensive administrative privileges. Attackers had deployed rogue instances of ScreenConnect, a remote access tool, allowing them to control numerous endpoints within the client's network. Just over a week later, the alert system indicated renewed suspicious activity, confirming that the attackers had maintained access to the environment, leading to further complications.
In the first incident, the Huntress SOC took swift action by isolating the affected endpoints and issuing a critical incident report. This report detailed the nature of the attack and the steps needed for containment. Tom Warren, Vice President of Operations at TruePath IT, engaged with the SOC to discuss the incident, ensuring that the client's leadership had accurate information as they navigated the crisis. The SOC's continuous support allowed the IT director to relay verified facts to the executive team during the incident response.
Remote monitoring and management (RMM) tools like ScreenConnect enable technicians to access and manage devices remotely, facilitating support and maintenance. While beneficial for legitimate technical assistance, these tools can be exploited by attackers to gain unauthorized access to systems. In this case, the presence of rogue ScreenConnect instances created multiple entry points for the cybercriminals, heightening the risk for TruePath IT's clients.
The impact of the attacks was significant, with the client experiencing two days of downtime following the first incident. The urgency to restore operations led to decisions that ultimately compromised security, as some endpoints were put back into service before being fully cleaned. This misstep allowed the attackers to deploy a second rogue ScreenConnect instance, which appeared to deliver ransomware. The persistence of the threat highlighted the dangers of rushing recovery efforts without thorough verification.
Following the second incident, TruePath IT took decisive action to re-secure the environment. The client's IT team was instructed not to return any systems to production until remediation was confirmed and approved by an executive. This cautious approach ensured that they would not face another ransomware threat and allowed for a more gradual and secure recovery process.
Key questions included who has the authority to isolate systems during an active threat and how recovery efforts can be expedited without compromising security. The need for a well-defined recovery strategy was underscored by the challenges faced during the incident.
While the immediate crisis was addressed, several unknowns remain. It is unclear how the attackers initially gained access to the compromised accounts or what specific measures could have prevented the incidents. Organizations are encouraged to review their remote access policies and ensure that only necessary tools are permitted within their networks.