Kubernetes is growing in popularity, and its use within organizations is maturing. According to the latest Cloud Native Computing Foundation (CNCF) annual survey, 93% of organizations are using or planning to use containers in production, and 96% are using or evaluating Kubernetes. According to a previous survey, 28% of organizations have more than 11 Kubernetes production clusters.
At the same time, security concerns around Kubernetes are mounting. A Red Hat survey of Kubernetes adoption and security showed that, of 500 DevOps professionals surveyed:
55% delayed an application release due to security issues.
94% experienced at least one Kubernetes security incident in the past year.
59% said security is their biggest concern with regard to continued use of Kubernetes and containers.
Read on to better understand the security risks facing Kubernetes environments, understand what security mechanisms Kubernetes natively provides, and discover best practices for improving your Kubernetes security posture.
This is part of an extensive series of guides about Kubernetes.
In this article, you will learn:
8 Kubernetes Security Best Practices
Kubernetes Security Solutions
What Are the Main Kubernetes Security Issues?
Kubernetes Security Fundamentals: What You Need to Secure Your Cluster
Kubernetes Security and Observability with Calico
8 Kubernetes Security Best Practices
Below are critical best practices you must implement to secure your Kubernetes clusters, divided into the three main stages of the development lifecycle: build, deploy, and runtime.
Build-Time Security
Here are key best practices that will help you secure containers during the build phase of your software development lifecycle (SDLC).
1. Image Scanning
It is essential to make sure that container images are free of vulnerabilities, because every container created from an image will inherit its vulnerabilities. This is typically done by scanning the base image and all packages against a vulnerable database.
Image-scanning tools verify that:
Applications or libraries included in container images are free from Common Vulnerabilities and Exposures (CVEs) and other vulnerabilities shared by public sources.
The base image, on which the container image was built, is free of vulnerabilities that could affect the operating system (if included) or any other libraries.
Images must be scanned at all stages of the CI/CD pipeline, and access to image registries must be controlled to avoid tampering.
2. Host Operating System Hardening
Ensure that containers only have the minimal required privileges on the host. Use a hardened host operating system with adequate controls to restrict system calls and file system access, and have strong isolation between processes. This can help prevent privilege escalation attacks, where a compromised container gains access to the host operating system, or other containers running on the system.
3. Minimizing the Attack Surface: Base Container Images
When selecting a base image for your containers, use an image with the minimal software packages absolutely necessary for your container to function. It is even better to build a container without a full base image:
Use the Docker FROM scratch directive to create a minimal image and explicitly select the packages you want to add to it.
Start from a slimmed down Linux distribution image (“distroless” image) or an Alpine minimal image.
Learn more in our blog post: What Your Kubernetes Security Checklist Might Be Missing
Deploy-Time Security
When the time comes to deploy your containers to production, you can secure them using the following measures.
4. Harden Your Kubernetes Clusters
Kubernetes cluster configuration is not secure by default. To ensure your clusters are secure, follow these steps:
Perform a review of current Kubernetes cluster configuration and identify gaps with security best practices. You can use automated tools like kube-bench to scan your clusters for compliance with benchmarks like the CIS Benchmark for Kubernetes.
Build a trust model for each component of your cluster by identifying probable threats and specifying how the cluster will respond to and mitigate each threat.
Use label taxonomies and label governance to implement your trust model.
Set up Role Based Access Control (RBAC) to define access to the cluster, aligning it with your threat model and label structure.
Secure etcd, the Kubernetes datastore, and the Kubernetes API server with multi-factor security and Transport Layer Security (TLS) for data in transit, and limit access to public networks.
5. Integrating Security Tools with Kubernetes Clusters
Typically, existing security tools like web access control gateways and Next Generation Firewalls (NGFW) are not aware of Kubernetes resources. You can overcome this by integrating your Kubernetes clusters with the existing security toolset. This allows perimeter security systems to work together with security measures deployed inside your clusters.
An effective way to do this is to feed the IP addresses and TCP/UDP ports used by your workloads into perimeter security tools, on an ongoing basis, to allow them to identify and secure Kubernetes assets.
Another approach is to use cloud provider security groups to limit network connectivity to and from Kubernetes nodes. Security groups are more aligned with Kubernetes architecture than traditional security tools.
Runtime Security
Here are best practices that can help you secure Kubernetes cluster infrastructure and workloads during runtime.
6. Network Security Controls
A critical concern in Kubernetes environments is that security controls in corporate networks, as well as security groups managed by cloud providers, are all focused on the node level. You can control traffic to and from a node, but existing controls do not know which pod or service is running on a node. This makes security controls largely ineffective at runtime, because different services may be running on the same node at different times, each with its own security requirements.
Kubernetes workloads are dynamic, and automated CI/CD processes constantly deploy different services, or new versions of the same services, on Kubernetes cluster nodes. To further complicate matters, the same workloads can transition between on-premises and different cloud environments, each of which has its own network security controls.
To achieve network security in a Kubernetes environment, you must build network security definitions into your workloads, using a declarative model. Security definitions must be an inseparable part of Kubernetes workloads, and must be portable across Kubernetes distributions and data centers. Wherever the workload runs, it must always carry its security definitions with it. This can be achieved in two ways:
Using a Kubernetes-native network policy solution – A few examples are Calico, Weavenet, Kube-router, and Antrea. These tools apply a network policy at network layers 3 and 4 (TCP/IP).
Using a Kubernetes-native proxy – A commonly used proxy is Envoy. This can help define application layer policy at network layer 7 (HTTP/HTTPS) to secure communication between microservices. Defining security policies at the proxy level can help you, for example, define that a specific microservice should only accept HTTP GET requests, and should reject HTTP POST requests.
Related content: Read our guide to Kubernetes network policy
7. Enterprise Security Controls
In addition to network security controls, apply the following enterprise security controls to your Kubernetes environment:
Encrypt data in transit – You can achieve this by enabling TLS encryption for workloads, leveraging a service mesh to enable mutual TLS, or using a VPN approach with tools like open-source WireGuard.
Automate compliance reports – Ensure you collect data in a way that enables automated reports for relevant compliance standards, such as PCI, HIPAA, GDPR, and SOC2.
Aim for continuous compliance – Use Kubernetes-native automation to check and remediate compliance issues. For example, if a pod fails a compliance check, the cluster can automatically update the container image, or shut down the pod until the issue is remediated.
Related content: Read our guide to securing Kubernetes host endpoints
8. Threat Defense
Malicious activity in a Kubernetes cluster can involve threat actors exploiting vulnerabilities in components running in containers, exploiting insecure configurations, or directing malicious traffic at nodes, pods, or the Kubernetes control plane.
To defend a Kubernetes cluster against threats, you need two key capabilities:
Intrusion detection – The ability to analyze data, identify anomalies, and triage security incidents.
Intrusion prevention – The ability to block malicious activity and respond to attacks.
A key challenge when attempting to detect intrusions is that a Kubernetes cluster generates huge amounts of log data. Use the following guidelines to make data more manageable, and derive security insights.
Aggregate by pods – Group together “similar” pods communicating on a certain port, instead of using the traditional 5-tuple technique (Source IP, Source Port, Destination IP, Destination Port, Protocol).
Leverage machine learning – Use tools that can automatically analyze pod traffic metrics to detect anomalies.
Use threat intelligence – Leverage databases of known malicious IPs and domains to identify malicious traffic in your cluster.
Related content: Kubernetes security policy design best practices
Kubernetes Security Solutions
Traditional security tools cannot handle a highly dynamic environment like Kubernetes, which constantly deploys new containers and retires old ones, especially at a large scale. For example, using a single peripheral firewall for the entire application can expose you to greater risk, because once threat actors breach the firewall, they gain access to the entire system.
In addition to the dynamic nature of containers, security standards are also undergoing constant change, often upgraded more quickly than traditional security solutions can keep up. You need advanced security tools to help manage this constantly changing environment.
By default, Kubernetes assigns an IP address to each pod in the cluster to provide IP-based security. However, it includes only basic security measures, which means administrators must handle advanced security monitoring and compliance enforcement, usually using third-party tools designed to secure the Kubernetes stack.
Advanced Kubernetes security tools offer various capabilities, including detecting vulnerabilities and ensuring each container runs as intended. Container security processes must run continuously to protect the container host, its management stack, and its network traffic while monitoring the build pipeline’s integrity and the application’s security.
What Are the Main Kubernetes Security Issues?
The following are some of the main risks facing Kubernetes production deployments.
Compromised Images and Image Registries
To ensure the security of images, organizations should implement strong governance policies that ensure images are securely built and stored in trusted registries. For example, organizations should make sure that container images are built using pre-approved and secure base images, which should be regularly scanned for issues and vulnerabilities.
Organizations should standardize registries by creating a list of image registries that are allowed for use. Images should always be scanned before being used to create containers in a Kubernetes cluster, to avoid tampering.
Compromised Containers or Malicious Traffic
Containers and pods need to communicate with each other as part of normal operations. However, this communication can be exploited by threat actors. A breached container can affect other containers and pods.
To ensure communication is secured, organizations should put in place network policies that limit communication to the minimum necessary for workloads to function. This includes both north-south traffic (ingress/egress traffic) and east-west traffic within the cluster. Network policies should be automatically adjusted to ensure they do not hurt productivity.
Lack of Visibility
Visibility is critical to ensure security is maintained. However, it can be challenging to achieve visibility in complex, distributed, containerized environments.
There can be a large number of containers scheduled, deployed, and terminated—all of which need to be tracked, monitored, and managed.
The distributed and dynamic nature of a containerized workload makes it difficult to collect, track, and understand relevant metrics.
Kubernetes can be deployed in multi-cloud or hybrid cloud environments. Each cloud vendor provides its own monitoring and visibility tools, and it becomes very difficult to allow consistent visibility across environments.
Securing applications without visibility can be difficult or even impossible. Without visibility, it is not possible to detect critical vulnerabilities or discover misconfigurations before threat actors exploit them. Visibility is also critical to monitor containers at runtime to detect attacks, and to keep track of containers that are no longer in use in order to properly retire them before they become a liability.
Unsecure Default Configurations
Kubernetes is built to speed up the deployment of applications and simplify operations and management. While Kubernetes provides a wide range of controls that can help organizations effectively secure clusters and applications, it does not provide secure configurations out of the box.
For example, Kubernetes network policies behave similarly to firewall rules, controlling how pods can communicate with each other as well as other endpoints. Once a pod is assigned a network policy, it is only allowed to communicate with the assets declared in the network policy. However, Kubernetes does not apply network policies to pods by default. This means that after being deployed, all pods can talk to all other pods within a Kubernetes environment. This makes it critical to ensure all cluster resources have appropriate security policies defined.
Another concern is secret management. Secrets define how sensitive information, like keys and credentials, is accessed and stored. When managing secrets, it is critical to make sure that they are not used as environment variables, or hardcoded within images. Secrets should be managed externally to containers, with careful access control to protect secrets from unauthorized parties.
Compliance Challenges
For many organizations, compliance is a critical concern. Achieving compliance in cloud-native environments is a highly challenging endeavor. To achieve compliance, organizations are usually required to implement certain security measures. This often requires enforcing best practices, benchmarks, and industry standards, as well as internal organizational policies.
In addition to maintaining compliance on a regular basis, organizations are also required to provide proof of compliance. This requires visibility, monitoring, and logs—the data needed to perform audits. To ensure auditors have access to the data or the systems, organizations need robust features that are not necessarily provided by plain, open-source Kubernetes.
Learn more in our guides to Kubernetes vulnerabilities and Kubernetes security issues
Kubernetes Security Fundamentals: What You Need to Secure Your Cluster
Kubernetes provides several concepts and mechanisms that can help secure your clusters. Here are the most important ones, as listed in the official Kubernetes project documentation.
Network Policy
Kubernetes uses a flat network model that allows each pod to communicate with any other pod in the cluster by default. This creates major security concerns, because it allows attackers who compromise one pod to freely communicate with all other resources in the cluster. To secure pod-to-pod communications, Kubernetes uses the concept of network policies.
A network policy is like a virtual firewall set up within each pod. When network policies are defined by an administrator, they are automatically updated at the pod level, and the pod accepts or rejects network traffic according to the policy. In Kubernetes, the mechanism for defining allowed or disallowed communication is label selectors, not IP addresses or ranges.
Pod Security Policy
Kubernetes originally used the PodSecurityPolicy (PSP) object to control security-related pod configuration. A PSP defines the minimal conditions a pod must meet in order to run in the cluster. However, PSP has been found to be difficult to use, has important functional limitations, and cannot be enabled by default, and so will soon be deprecated and replaced.
The replacement feature has the temporary name “PSP Replacement Policy”. The new policy will be designed for simplicity of deployment, will have a built-in admission controller, and will be easy to use while providing flexibility for large-scale production deployments. Importantly, unlike the current PSP, you will be able to retrofit it to existing clusters via soft rollout.
Kubernetes Secrets
Secrets are sensitive information like authentication tokens, SSH keys, or passwords. It is dangerous to store secrets in plaintext as part of container images or pod configurations, as these can easily be compromised by attackers who then gain access to any system accessible via those credentials.
The built-in Kubernetes Secrets feature lets you store sensitive information within a secret object in Kubernetes. You must first create a secret, populate it with secret data, update a service account to reference the secret, and only then create pods that use the secret. A pod can access a secret as an environment variable or a file (this is known as a secret volume).
Kubernetes RBAC
Kubernetes uses the concept of ClusterRoles and Roles, which specify what each user can perform in a cluster or an entire Kubernetes namespace. You can use ClusterRoleBinding to apply a role to all resources in the cluster, or RoleBinding to apply the role to every resource in a namespace. Kubernetes offers the following default user-facing roles, and also lets you create additional custom roles:
Cluster-admin – Can perform any action on anything in a cluster.
Admin – Allows unlimited read/write access to resources in a namespace, but does not provide write access to the namespace itself.
Edit – Allows read/write access within a namespace, but does not allow viewing or modifying roles and bindings.
View – Like Edit, but allows read-only access in a namespace.
Authentication
The primary access point for a Kubernetes cluster is the Kubernetes API. Users and service roles access it via the kubectl utility, direct REST API requests, or client SDKs. By default, the API is accessed via port 445 and is protected by Transport Layer Security (TLS).
The Kubernetes API authentication process works as follows:
The client attempts to access the API, presents a TLS client certificate, and a TLS connection is established.
The API server runs one or more Authenticator modules. These allow cluster administrators to define user and service accounts and specify authentication strategies, including client certificates, tokens, JSON Web Tokens, and passwords.
The Authentication modules accept the client’s entire HTTP request and attempt to authenticate it. If there are several authentication modules, they are tried one by one until one succeeds.
If authentication fails, the API returns status code 401. If it succeeds, the client is authenticated as a username, and can use this username in subsequent activities.
The Authentication module might assign the user group membership. If so, the user will have the additional privileges of the group.
TLS For Kubernetes Ingress
Kubernetes Ingress is an object that allows services in a Kubernetes cluster to be accessed from outside the cluster. It facilitates connections via HTTP/S, and specifies routing rules.
To secure an Ingress object, you specify a secret containing two keys—tls.crt and tls.key—which contain the certificate and private key, respectively. TLS encryption works up to the ingress point, and traffic within the cluster is not encrypted by default. If the ingress configuration includes multiple hosts, they are multiplexed on the same port.
Quality of Service
Kubernetes automatically assigns a Quality of Service (QoS) class to pods, which helps optimize scheduling and eviction of pods. The kubelet uses a QoS class to decide the order in which pods are evicted from nodes, and to make smarter decisions when scheduling pods on nodes. DevOps teams can customize QoS classes for pods if necessary.
You can place pods in a Guaranteed QoS class, which means that each container is guaranteed to receive enough resources from the node. This requires that all containers have a CPU and memory limit. Kubernetes assigns exactly enough CPU and memory resources to meet the requirements of containers in the pod.
Kubernetes Security and Observability with Calico
Tigera’s commercial solutions provide Kubernetes security and observability for multi-cluster, multi-cloud, and hybrid-cloud deployments. Both Calico Enterprise and Calico Cloud provide the following features for security and observability:
Security
Zero trust for workloads – Prevent lateral movement of threats and maintain compliance by applying fine-grained security policies to restrict communication between workloads and third-party applications, the internet, and other workloads.
Compliance reporting and alerts – Continuously monitor and enforce compliance controls, easily create custom reports for audit.
Intrusion detection & prevention (IDS/IPS) – Detect and mitigate Advanced Persistent Threats (APTs) using machine learning and a rule-based engine that enables active monitoring.
Microsegmentation across Host/VMs/Containers – Deploy a scalable, unified microsegmentation model for hosts, VMs, containers, pods, and services that works across all your environments.
Data-in-transit encryption – Protect sensitive data and meet compliance requirements with high-performance encryption for data-in-transit.
Observability
Dynamic Service and Threat Graph – Observe upstream and downstream dependencies for microservices, and service-to-service interactions within a Kubernetes cluster, with a dynamic live view that helps identify security gaps and troubleshoot connectivity issues faster.
Application-layer observability – Gain visibility into service-to-service communication within your Kubernetes environment, without the operational complexity and performance overhead of service mesh.
Dynamic Packet Capture – Perform packet capture for a specific pod or collection of pods with this self-service, on-demand tool. The tool integrates with Kubernetes RBAC to limit and secure users’ access to the endpoints and namespaces assigned to them.
DNS Dashboard – Quickly confirm or eliminate DNS as the root cause for microservice and application connectivity issues in Kubernetes.
Flow visualizer – Get a 360-degree view of a namespace or workload, including analytics around how security policies are being evaluated in real time and a volumetric representation of flows.
Next steps:
Read our guide: Kubernetes observability
Read our O’Reilly book: Kubernetes Security and Observability (free download)
See Our Additional Guides on Key
