Build a Cloud Forensics Practice

The Takeaway

When building a cloud forensics practice for a shared services organization, first build forensics playbooks for common types of incident investigations and then translate each playbook to a runbook with specific steps for each supported cloud platform. This writeup covers training recommendations for staff responsible for forensics and incident response on cloud technologies, as there are simply too many variations in cloud platform deployment models to cover everything in playbooks and runbooks.

Build Playbooks and Runbooks

Forensic playbooks and runbooks are significant enablers for forensics practitioners. Playbooks describe a set of high-level steps for an investigation, and runbooks describe the specific steps to be taken. For example, a playbook might say “collect the blob container logs,” and the corresponding runbook would then describe the specific steps to obtain those logs in a specific cloud platform. Endpoint and mobile forensics generally do not require these guides because investigations don’t differ significantly. However, playbooks and runbooks are a necessity for cloud forensics where investigations differ considerably.

Faculty recommends creating playbooks for each of the following types of investigations:

• Blob/file storage
• Database PaaS
• IaaS
• Container environments
• Control/management plane compromise

These five playbooks will cover most incidents the organization is likely to respond to, especially in a shared services environment. Analyze your own historical incident data to determine whether other playbook types should be prioritized instead. The shared services model complicates things a bit due to the variety of cloud environments and logging postures in place at different subordinate organizations.

After each playbook is created, build a corresponding runbook detailing the specific steps to take for each cloud platform. The rationale for building both playbooks and runbooks is twofold:

1. Playbooks are higher-level documents, so they require far less ongoing maintenance to account for platform changes.
2. Runbooks require much more work to create in the first place.

Organizations can quickly establish high-level guidance across all relevant types of investigations and later devote resources to adding detailed steps for specific cloud platforms. This approach ensures forensic analysts have something, even if it is high level, to refer to for their investigations. Faculty has observed several organizations delay the hard work of creating runbooks and then experience an incident before any relevant documentation has been created.

Note the following additional guidance on cloud forensics playbooks:

1. Kubernetes is intentionally not listed as an incident type, as it is usually covered by either container or IaaS playbooks, depending on how the organization being investigated uses Kubernetes in the cloud.
2. IaaS runbooks must account for logs created by platform agents that execute in a privileged context in a VM to ensure such logs are investigated.
3. Playbooks for control/management plane compromise should be targeted in investigation recommendations. Playbooks that say “investigate everything” are of little value. Faculty generally recommends focusing on IAM changes.
4. Database PaaS playbooks should recommend communicating to stakeholders early that little can be done with available logging to validate the integrity of data or confidentiality (usually unauthorized disclosure) of specific records. Consider writing database PaaS playbooks with a set of steps analysts should perform—after which they can comfortably express that no additional investigation will provide additional information.
5. To discuss recommendations for specific playbooks and runbooks, schedule an AAE with the author.

Cloud-Specific Forensics Tooling

There is no cloud-specific forensics tooling that Faculty finds useful across varying cloud environments. Most endpoint forensics software offers collection tools for major cloud and SaaS environments. However, in Faculty’s experience, these collections are often incomplete. Given that analysts must be familiar with retrieving configurations and data using platform-native tools, this is recommended as the default approach.

Some teams make use of tools like F-Response for remote collections of IaaS. However, given that cloud platform tools can be used obtain volume snapshots, tools like F-Response are rarely needed. As a practical matter, container monitoring tools, cloud security posture management and cloud-native application protection platforms are some of the richest sources of forensic data outside of platform logs.

One tooling consideration is whether to run forensics workstations in a dedicated cloud tenant or in the tenant being investigated. Unfortunately, there’s no best practice here. While there is a clear integrity/assurance benefit to running a forensic workstation in a known-clean environment, this may present challenges in obtaining the data needed for forensics. There are cost implications of moving data between regions for a given cloud provider (and even between availability zones, in many cases).

Examine the specific circumstances of each investigation before deciding where to deploy an analysis workstation. Many organizations want to preposition analysis workstations as an incident response preparation step. However, this introduces new month-to-month carry costs, even if there is nothing to investigate. Instead of prepositioning forensics infrastructure, build infrastructure-as-code templates that allow analysis workstations to be rapidly deployed in any environment, minimizing analysis delay and eliminating infrastructure carry costs.

Provide Governance Recommendations

Forensic analysts can only work with the data available to investigate a given incident. The data available will largely depend on configuration settings and sometimes on deployed platform tools. For example, in Amazon S3, bucket-level events are logged by default but object-level events are not. Enabling object-level events without a cost analysis is not recommended, as this can increase spending significantly for a deployed application. Similarly, in Azure many logs are “generated automatically” (at least per Microsoft documentation) but must be routed to Azure Monitor logs to save or query them. In other words, if you don’t configure a target for the logs, they will disappear into the ether as “automatically” as they were generated.

In Faculty’s experience, most organizations are not aware of limitations in cloud platform logging until an incident occurs. One recommendation is to provide governance advice to constituent organizations detailing the types of investigations that will be impossible unless logging is enabled. Dictating specific logging practices is not advised, as specific architectures may render these cost prohibitive. Given the client’s operating model, Faculty recommends establishing minimum expected security and the logging posture it will support for each organization.

Train Staff on Cloud Platform Architecture

Cloud platforms differ significantly from traditional computing technology deployments both in their available components and the access available to forensic analysts. Separately, no two AWS deployments are the same (AWS is used here as a straw man, the same applies to other cloud platforms). Before beginning forensics work, an analyst must first understand the application components and how they work together (the application architecture), and inventory available logging data for each platform component.

The skills required to complete this work are usually outside the specialties of forensic analysts. For this reason, Faculty recommends providing these forensic analysts with cloud technology training, expanding to each platform they are expected to support. For best results, train forensic analysts using architecture/engineer study materials rather than security. Cloud security materials focus on securing specific platform components, not understanding how components work together to deliver an outcome. The latter is the focus of cloud architecture/engineer courses and is the job analysts will be asked to perform before they can begin forensics.

All analysts should receive training for at least the “basics” certification (AZ-900, CLF-C02, etc.) for any cloud platform they are expected to investigate. This will ensure they are at least conversant in platform technologies.

Team leaders should be sure to budget disproportionately for continuing education for these analysts. Cloud platforms are a moving target, and each one changes significantly every year, introducing new services, changing logging, etc. The same phenomenon affects DevOps teams as well, but with one key difference: DevOps teams can wait to receive training or governance before adopting a new platform technology, while forensics analysts are expected to hit the ground running in any investigation.

Get Started Building a Cloud Forensics Practice

Faculty recommends the following:

• Educate forensics and incident response staff on cloud technologies.
• Build playbooks for the recommended investigation types.
• Translate these playbooks into runbooks for each specific cloud platform in which the constituent organizations. operate in, focusing on those with the largest footprint across those organizations.
• Provide governance recommendations for each subordinate organization you will support.