Leading Kubernetes Mistakes (And How to Avoid Them)

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services. The system has a large and rapidly growing following. Unfortunately, as with any application tool, users can make mistakes that quickly negate its benefits.

“There are many common mistakes,” says Mike Loukides, vice president of emerging tech content at IT learning services provider O’Reilly Media. They fall into a few classes: misunderstanding how to use pods effectively, using Kubernetes in a way that makes observability difficult or impossible, getting security wrong, and using Kubernetes without automating the deployment. There are many other common mistakes and, to some extent, these all overlap and interact.

Security Blunders

It’s possible, although not advisable, to limp along with many Kubernetes blunders, but security mistakes can take down an entire enterprise, Loukides warns. “Unfortunately, many of the people responsible for Kubernetes aren’t fully aware of security problems and don’t have the skills to lock it down properly,” he notes.

Security mistakes can go unnoticed for a long time — until there’s a successful attack. By then, the entire enterprise may be compromised, leading to potentially dire consequences. “Private data may end up for sale on the dark web, or you’re offline because a ransomware guy has encrypted all your systems,” Loukides cautions. “Security is difficult; it’s a pain, it’s an expense for which there’s little return — until somebody breaks in.”

Architecture Misconceptions

It’s easy to forget that Kubernetes is a distinct architecture, not simply another virtual management environment that teams may already be familiar with. “Instead of building or rebuilding applications toward decoupled, transient microservices, [teams] continue to focus on optimizing code for a legacy environment,” explains Tim Serewicz, training and certification program director at the Linux Foundation. “Developers who don’t understand the new architecture will build non-optimized applications.”

Education is the best way to avoid future mistakes, Serewicz says. “A lack of understanding architectural differences exists from entry level to senior leadership, causing tactical and strategic mistakes that undo the advantages of Kubernetes and the cloud,” he notes.

Underestimating Complexity and Cost

A common mistake development teams make is underestimating the complexity and costs associated with setting up and operating Kubernetes. At least part of the complexity is due to the system’s distributed nature, its foreign concepts for teams new to containers and orchestration, and the variety of potential implementation approaches, says Anay Nawathe, principal consultant of the digital strategy and solutions group at global technology research and advisory firm ISG.

Kubernetes adds layers of abstraction between infrastructure and applications, and performs background tasks that make cost allocation and subsequent cost optimization a challenge without proper standards, governance, and tooling, Nawathe explains. “Some development teams tend to use Kubernetes to solve too many of their problems with an infrastructure solution when they should be considering alternative architectures, such as serverless and edge deployments to better meet those needs.”

Kubernetes developers should avoid accidentally misconfiguring deployments. “This includes mistakes like not setting resource limits, using default configurations, and leaving unneeded privileges on service accounts,” Nawathe says.

Developer Distraction

A mistake many delivery teams make is attempting to push out functionality at a high frequency, then becoming distracted by high-risk behaviors related to the new technology, says Jeff Cratty, senior director of advanced technologies at systems integrator and cloud services company Blue Mantis.

A new technology is always attractive to developers, providing an opportunity to learn new things and expand skillsets. Yet it’s easy to get lost in the nuances of a fresh technology and lose sight of the bigger picture. “The goal of any organization adopting a new technology like Kubernetes always was, and still remains, to solve business problems and deliver value,” Cratty says. “Development teams need to remain focused on creating value for their customers and build, buy, and partner for components that fall outside of their core competencies.”

Accidental Misconfigurations

Accidental misconfigurations are one of the biggest problems in Kubernetes environments. This is largely attributable to the fact that security has not always been prioritized by developers, says James Maskelony, senior detection and response engineer for security operations provider Expel. “For years, the prevailing mindset among many developers was that security is the responsibility of the security team, rather than something they needed to consider,” he explains. “Unfortunately, this historical lack of focus on security often provided opportunities for attackers to exploit.”

To some degree, misconfigurations are unavoidable, Maskelony observes. “They can occur any time a user does something that unintentionally introduces risk into the environment.” Such mistakes range from adding a workload that grants excess permissions to accidentally allowing a stranger from the Internet to access the system. “These mistakes are easy to make, particularly since Kubernetes is still relatively new to most users,” he says. “Addressing misconfigurations and limiting them in the future is essential.”