Preventing credential leakage through compromised build pipeline

Context

Recently a sophisticated attack injected malicious behaviour to some trivy image scanning software, that resulted in pipelines running malicious code that extracted sensitive content and injected malicious beahaviour into artifacts.

I'd like to believe that the organisations that I have worked for have had robust safeguards in place that would have prevented them from being vulnerable to this type of attack, but to keep myself busy and "never let a good crisis go to waste" I am jotting down some thoughts around how such a compromise could be detected and prevented from inflicting damage.

I will base my evaluation on the assumption that the artifacts involved here are a scanning tool running within a build pipeline, and a Docker image that has been generated as part of an earlier stage of the pipeline process. 

Clean separation of concerns

Do not include any sensitive data in the container image

If there is no sensitive data such as API keys, tokens, credentials, secrets as part of the artifact being scanned then a compromised tool cannot find them to expose them.

The build environment should be completely isolated from the environment(s) that the container will ultimately be running in, so even if the mechanism for obtaining credentials was discovered it should not be possible to leak secrets.

Don't allow arbitrary network egress

By locking down the build environment to only have access to the resources that are relevant to the processes involved we can reduce the scope of what a compromised system could do.

If the malicious software cannot phone home then it cannot expose what it has found, or update itself with further actions, such as triggering as part of a botnet.

Alongside prevention, we would ideally also want detection for picking up on if and when an unexpected attempt was made to call out to the network.

Read-only access

Given that the responsibility of the scanner should be limited to reading of the content of the container image, it is reasonable to not permit write access to the files involved.

As an additional safeguard, the image generation stage should include writing of a checksum for additional verification that the integrity of the container image is in tact between build and deploy.

Observability

The sophistication and frequency of software supply chain attacks is only going to continue to increase, so it is more vital than ever that we treat our build and deploy environments with the same level of scrutiny and attention to detail as our top tier production services.

Logs, metrics and alerts must be in place in the build pipeline to enable teams to stay ahead of attackers and continuously improve the resilience and integrity of the release processes.