The cloud can be extremely hard to operationalize at scale. In our previous posts in this series, we discussed the complexity of Service Sprawl and Inter-Dependencies as well as More Dynamic Applications. Now comes perhaps the toughest dimension of all: People.
Complexity #3: Growing Cloud Teams
Recently, Gartner calculated that 1.7 million decisions are required for an engineer to simply deploy an EC2 instance. One point seven million decisions! To spin up a server! With 70 instance types, multiple purchase options, multiple operating systems, dozens of regions and availability zones, you begin to see how the small choices compound into death by a thousand paper cuts.
And that’s just for one AWS service. Now add in your other standard cloud services—IAM, S3, RDS, and more. Then consider your applications with access to customer data or that connect to other systems in your company and you can watch the level of risk multiply with each eager engineer you add to your team. And as your organization becomes more cloud mature, you need to add more and more engineers into the fray.
People Don’t Know What They Don’t Know
One of the early learning experiences people have working in cloud is the superhero revelation that with great power comes great responsibility—and this is often learned by mistakes that can have pretty bad results.
You don’t have to look far to find examples of these type of mishaps. In the last 45 days alone there have been three major data leaks caused by preventable misconfigurations that left critical information like voter registration, passwords to US government systems, and customers’ names, numbers, and security pins codes accessible.
- 200 Million Voters Exposed Due to Open AWS Repository
- Top Defense Contractor Left Sensitive Pentagon Files on Amazon Server With No Password
- Verizon data of 6 million users leaked online
Fundamentally, the cloud is still a new paradigm that’s continuously creating new rules for building great applications. Historically, different individuals owned networking, security, and the systems and application space, but with cloud infrastructure, every individual now has the power to create what they need. As a result, it’s more and more common for application developers to do work that falls in the security and networking realm. But due to the historical siloing of these responsibilities, very few people have the experience to confidently own all of these areas, so there are suddenly many people who are learning primarily through trial by fire.
To understand and adapt to this new level of responsibility, DevOps has emerged as a methodology to align ownership and control for both development and operations of an application. A key component of the DevOps methodology is to create automated solutions that allow the organization to iterate and release quickly. And automation is great, but…
Automation Will Not Protect You
Several of our customers and prospects have told us that they have a goal to turn off AWS console access in an attempt to avoid inevitable misconfiguration errors that come with people clicking around on their own. This is how worried many are about our team making mistakes in the cloud—there are just too many knobs and configuration nuances to keep track of.
Great, so in their perfect world, everything is automated and there are no more point and click interventions. But what happens when one of your engineers runs up against the limitations of your tool of choice? They work around it by going to the console as an escape hatch. One such team member admitted, “It was late, my wife was calling me and wanted me home for dinner…so I did it. I was having trouble making the change in Terraform so I popped up the AWS console made the change real quick and I went home. Then I forgot to go back and fix it in Terraform. Oops.”
Additionally, many standardized automation tools make it too easy to grab a template that doesn’t fit your context and unknowingly introduce a vulnerability. Recently, we heard a story about a developer who accidentally used a CloudFormation template that gave all AWS users access to the data for his new application. He later realized his mistake—those permissions would be perfectly fine for a file sharing site, but not for an application that needed to hold customer and location data.
Context for the application and how it’s intended to be used is critical. Automation gets it done faster, but it doesn’t mean the templates you are using always get the job done correctly. And as your cloud and DevOps team broadens, it gets significantly harder to tell what you actually have deployed, much less detect when one of thousands of configuration elements poses a new security risk. True visibility into your cloud—and its vulnerabilities—has to start from the ground up, with the very team members who are launching and managing cloud configuration.
To enable true, continuous cloud security, security must be integrated into your existing deployment and CI/CD processes to highlight violations and vulnerabilities in each and every batch of code.
Giving Cloud Teams the Visibility they Need
These days, even leading cloud companies have trouble hiring great DevOps engineers, due to the limited supply of ‘cloud know how.’ Talent and experience in cloud may be in short supply, but every company still needs to figure out how to enable the broader team without either creating roadblocks or giant security holes.
There are many options for cloud vulnerability scanning tools that will go out and look for dangerous elements within your public cloud configuration. This can be considered an over the top scan: occasional, point in time checking of your cloud accounts to see if there are any vulnerabilities. This type of vulnerability scan for cloud infrastructure should be a part of every single company that is operating in AWS, Azure or Google Cloud.
While over the top scanning is a must, it’s often managed by a central security team who is fighting to keep up with all the alerts, much less understand the context for each application and team that they serve. As a result, these type of tools do little to help integrate Security into Cloud Engineering teams and processes. Instead, the noise of the alerts overwhelms Security, and still leaves them on the outside looking in, trying to get engineering owners to fix issues well after the fact.
To meet the challenges of cloud, vulnerability checking must be integrated at the true source of cloud: enabling DevOps owners to check their cloud while they are building and deploying infrastructure. That’s why we’ve created Dev Time Checks.
In addition to over the top scanning of your cloud accounts for best practice violations and vulnerabilities, CloudCoreo can provide feedback to directly to developers while they’re using the AWS CLI. Run a command or a template that launches or modifies AWS infrastructure as usual, and receive real-time feedback, inline, that immediately let’s you know if there’s an issue with your deployment. Give your team the tools to understand the impact their changes are having on the cloud, and best of all, gain the expertise to stop vulnerabilities at the source.
Is your cloud team expanding? Drop us a line if you’re interested in learning more about how a one time install can make every deployment better.