CI/CD: Necessary for modern software development, yet it carries a lot of risk

CI/CD over the past decade has become the cornerstone of modern software development. The term – for continuous integration and continuous delivery (sometimes the "D" also can mean "deployment") – emerged in the late 2000s with the rise of DevOps, defining a way to more quickly create and update applications by leaning heavily …

[that one in the corner]

> "Attackers within an organization can add configuration to build phases...

so these injections would be 'invisible' compared to a nefarious configuration or source commit."

So these build phases have configuration that *isn't* being committed into version control? What? Any vaguely competent CI/CD build starts with getting a verified copy of *everything*, build configuration and all, put of version control, surely?

So the only non-VCed config is literally just the top-level command "pull label xxxx" - and if you don't manage to log[1] *that* then something is drastically wrong!

[1] log? Heck, drop it into everything that gets built so it'll show up in every --version or About box.

[SunnyS]

I’m looking at you Azure DevOps pipelines and your classic editor! Now that didn’t use your repo and so was/is a way for bad guys to do bad things. I believe the default editor is now YAML so is more secure but you need to specifically disable the helpful hint to use the classic editor.

Another reason I use GitHub Actions…

[Michael Wojcik]

Exactly. Any CI/CD system that doesn't at a minimum support version control and individual authentication for all of its configuration components is rubbish. Any organization that uses CI/CD and doesn't at a minimum keep all of that under version control with individual authentication is direly broken and, frankly, deserves to get pwned.

Hell, moving more stuff into the version-control domain is one of the biggest advantages of IoC, along with the usual things like repeatability, auditing, recoverability...

It'd be better if CI/CD configuration was signed, and those signatures verified regularly, but version control and strong authentication are at least a start.

[claimed]

IaC - infrastructure as code

Typo (i think) confused me

[runt row raggy]

isn't the argument here "ci/cd scary so in conclusion, buy our product"?

[klh]

Pretty much. Though I'm not surprised some of the scare comes from MS, as someone mentioned Azure DevOps CI was an insecure GUI mess by default the last time I had to use it.

[Anonymous Coward]

CI/CD doesn’t log?

“ CI/CD pipelines tend to get less security attention and have little if any logging for what developers execute as part of the build, package, or deploy phases.”

That’s not been my experience at all, the logs are so voluminous it’s often a slog to find what you’re looking for.

[Michael Wojcik]

Re: CI/CD doesn’t log?

Right, which is why they should be feeding into a SIEM, and probably into a UIBA system to flag unusual behavior.

Logs aren't good for much other than post-exploit forensics – if that – if they're not being continuously analyzed, and there's no way humans can do that. Even if you could find people to try, human beings are not good at constant vigilance.

[captain veg]

"The term – for continuous integration and continuous delivery (sometimes the "D" also can mean "deployment") – emerged in the late 2000s with the rise of DevOps, defining a way to more quickly create and update applications by leaning heavily on automation for everything from building to testing to deploying systems, pulling together contributions from myriad contributors into a pipeline, and speeding up release cycles."

Or, as we used to say, if it compiles, ship it.

-A.

[abend0c4]

The problem is not the concept, it's the implementation

Source control, unit- and integration-testing and regular builds have been cornerstones of software development for decades. The rate at which you turn the handle doesn't really change the fundamentals.

The thing that has fundamentally changed is that previously it would all be done in house using locally-developed code. These days your own code is hosted by a third-party. The code on which it depends and which you likely haven't tested or inspected more than cursorily is controlled by unknown hordes of third parties. The build pipeline may be wholly or partly hosted by a third party and is likely to have poorly-understood and wrongly-configured access to the swathe of disparate resources it needs to run its course.

If you deliver software using a process you don't own, don't control, don't fully understand and share with the great unwashed then risk is not something that should take you by surprise.