InfoSec-Specific Executive Development for
CISOs and Aspiring Security Leaders.
Live Faculty-led instruction and interactive
labs to build you and your team's InfoSec skills
The SolarWinds breach underscores just how complex supply chain attacks can be. Because it targeted network management software, the impacts were extremely damaging. Attackers gained immediate access to an extremely sensitive system, centrally positioned
in the network, with credentials that, in most cases, could be used immediately to move laterally to other systems.
If you were compromised by this SolarWinds software update, know that regardless of the procedures you followed for validating software updates, you were likely to have been impacted. There is very little organizations can do to prevent a software supply
chain attack like this, but threat modeling will provide the best possible outcome should one occur. This report outlines best practices for responding to breached SolarWinds Orion servers, including threat modeling network management systems in general.
On Dec. 13, 2020 Reuters reported the U.S. Treasury Department had been breached. The Washington Post quickly followed with news the same attackers who had breached FireEye (reported the week of Dec. 7) were also responsible for the Treasury breach. The
Post also reported sources close to the investigation claimed a supply chain attack on SolarWinds’ Orion network management software was responsible for both intrusions, and that both intrusions were the work of Russia’s SVR intelligence
service (aka APT29/Cozy Bear).
Organizations using any version of Orion (not just the one confirmed by SolarWinds as compromised) should assume they have been breached and begin threat hunting for intruders. Unfortunately, this attack group was sophisticated, both in its development and operational activities.
Due to the way the Orion user interface is built, it can be difficult to understand the full set of credentials configured for use by a SolarWinds Orion server. Unfortunately, this is critical in the context of evaluating the impact of a breach. Fortunately,
Rob Fuller (@mubix) has been researching Orion servers for years in the context of red team operations. He published a blog post compiling information he's learned from targeting SolarWinds Orion servers over the years. In it, he explains how credentials are stored across different versions of Orion and how they can be decrypted. He also provides links to some of his conference
talks discussing how attackers can use a compromised Orion server. Watching these videos is a must for security and operations teams seeking to threat model the impact from a compromised network management system.
Fuller also released a tool, called SolarFlare, that decrypts passwords and credentials configured in the SolarWinds Orion database. He says credentials previously configured in Orion may persist even after being removed in the Orion console. This is
likely the result of normal activity in the database engine, because deleted entries are frequently retrieved from database tables in forensic operations. SolarFlare performs a similar operation, so it stands to reason it can recover inactive credentials
operations that personnel believe were previously deleted.
Teams should also rotate any credentials used on a compromised Orion server. However, the credentials stored on the Orion server do not tell the whole story. Most Orion servers can retrieve data from other systems, often including network device configurations.
If attackers have a stolen network device configuration, they can recover a variety of sensitive data (as applicable to the specific device), including:
The private keys should be considered especially important. While many attackers would find it difficult to operationalize a private key, the attackers associated with the SolarWinds breach are extremely capable and could make use of it in the context
of an intelligence operation if they have somehow compromised an ISP.
If the Orion server is configured with passwords providing access to web servers, SSH servers or any other device where keying material must be kept secret, those keys should also be rotated.
When in doubt, bring in red team members to evaluate the impact of compromised material. If dissent still exists on the necessity to rotate the passwords or authentication material, we recommend erring on the side of caution. Under no circumstances should
you think "the attackers won't figure out how to use this." While that might be a viable approach in some circumstances, it is not the case here.
Several best practices can help protect against attacks that target network management software, including:
Limiting its communications: Limiting access to the internet is considered a best practice for network management systems in every case. Since the systems can provide a high degree of access to an attacker, their network communications
scope should be limited. Ensure network ACLs limit the system to only communicate with managed devices and only using the specific protocols required.
Reviewing its configurations: Most network management systems are configured by operations teams without consulting security. Security should review existing configurations.
Threat modeling: Security should also threat model the access attackers might gain by a successful compromise. Historically, organizations that threat model what a compromise of a critical asset would give an attacker fare the best in
extremely complex situations like a supply chain attack. In general, InfoSec should be involved in threat modeling the configuration and use of all orchestration and automation platforms. Security should not become a blocker to the use of these platforms,
but information security teams brings valuable perspective because it understands attacker tradecraft in a way operations teams may not.
It is not enough to simply install a new version of Orion. The Windows server on which Orion is installed must also be reloaded. And avoid restoring from backup. SolarWinds has not yet completed its internal investigation and we don't know for sure how
far back this intrusion goes. There are also questions about whether the new version of Orion that SolarWinds released is clean. While we have scanned this version with the Yara signatures released by FireEye and found nothing, this would be expected
even if the attackers are compromising the SolarWinds build process. There are countless researchers digging further into the latest Orion release. However, knowing how well the original backdoor was camouflaged and how large the installation is,
it could be days or even weeks before another backdoor is discovered if present.
The compromised Orion installation calls out to a command-and-control server to download secondary malware payloads (FireEye calls one such payload "BEACON," which is a reference to a loader for Cobalt Strike). This domain is used to bootstrap additional
operations, but it is now under the control of Microsoft, likely mitigating any direct threat from a freshly installed version of Orion. Microsoft controlling the domain is sufficient to prevent new breaches as a result of the backdoor, but it is
important to remember the attacker may have already downloaded other malware to an Orion server. All samples of secondary malware detected so far use other domains for command and control.
The group responsible for this attack is known to take countermeasures to avoid detection and practice active counterintelligence, adapting its operations based on what it believes is publicly known about them. This means any publicly available indicators
of compromise (IoCs) are unlikely to be useful in detecting the intrusion. By the time you start using the publicly available indicators in your environment, the attacker will have likely changed those indicators to prevent detection. Don't interpret
this as a requirement to more quickly operationalize indicators. Instead, interpret it as a need to think about logging.
By having logging available, defenders can execute the discovery course of action (CoA). The discovery CoA looks backward for newly discovered indicators, while the detection CoA looks forward. The sophistication of these attackers' actions are likely
to render detection CoA difficult. We recommend organizations consider what indicators they can execute the discovery CoA against and make logging changes to close gaps going forward.
As we continue to learn about more instances of confirmed compromise, we have yet to see overlap in network indicators, beyond the initial backdoor callback domain. While we predicted the attackers would change IoCs rapidly on disclosure, it now appears
they were also ready for the inevitable investigations. We should not anticipate network-level IoCs will be useful in this case.
A researcher named Ean Meyer is compiling information from threat researchers, including IoCs, into this GitHub repository. It is an excellent resource that is being updated frequently.
The group behind this supply chain attack is extremely capable. Even without the privileged access provided by an Orion server compromise, it would likely elevate to domain administrator in most networks. If IoCs or anomalous logs are discovered that
lead the security team to believe a domain admin-level compromise occurred, be ready to take the appropriate actions to secure the do.
Once an attacker takes over the domain admin account, it is extremely difficult to track what they did that may allow them to regain access quickly once they rejoin the network. Consult with Active Directory (AD) security professionals to ensure malicious
modifications have not been made, including:
Additionally, if the attackers obtained domain admin permissions, changes will need to be made to the krbtgt account to prevent them from immediately regaining these permissions using a Golden Ticket Attack (see this Microsoft AD documentation for information
on resetting the krbtgt password).
The following recommendations apply to all systems that hold credentials, including:
For all these systems, InfoSec teams should consider:
Supply chain attacks are amazingly complex. Even when properly investigated, attackers appear to come out of nowhere. Defending against these attacks is difficult as well. If your organization was impacted by this attack, know that you could have done
everything right and still have been impacted. This may not have been a failure of your information security
Organizations that move forward assuming their Orion server and managed devices have been breached will be in a better position to respond and repel the attackers. In addition, threat modeling and applying robust security controls around your network
management software should help reduce the risk of similar supply chain attacks in the future.
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in connection with such information, opinions, or advice.
May 19, 2022
By IANS Faculty
Understand potential security risks for executives on social media. Find information on attack trends and guidelines to help identify potential attacks and keep both social media accounts and the organization secure.
May 17, 2022
Learn how to make progress with zero trust, including common zero trust use cases, success stories, tooling guidance and tips for effectiveness.
May 12, 2022
Gain an understanding of the role executives play in incident response (IR). Find guidance on key actions to take before, during and after a security incident.