How to Optimize Kubernetes Pod Performance with Pod-Level Resource Managers (Alpha)

By ● min read

<h2>Introduction</h2><p>Kubernetes v1.36 introduces a powerful alpha feature called <strong>Pod-Level Resource Managers</strong> that transforms how you allocate CPU and memory for performance-sensitive workloads. Instead of forcing every container in a pod to require exclusive, integer-based resources, this enhancement lets you define a <strong>pod-level resource budget</strong> alongside flexible per-container allocations. The result? You can achieve NUMA alignment and Guaranteed QoS for your main application while letting lightweight sidecars share a common pool—saving cores and reducing waste.</p><figure style="margin:20px 0"><img src="https://picsum.photos/seed/3138074739/800/450" alt="How to Optimize Kubernetes Pod Performance with Pod-Level Resource Managers (Alpha)" style="width:100%;height:auto;border-radius:8px" loading="lazy"><figcaption style="font-size:12px;color:#666;margin-top:5px"></figcaption></figure><p>This guide walks you through enabling and using Pod-Level Resource Managers, from prerequisites to real-world configuration examples. By the end, you’ll be able to deploy pods that combine exclusive resources for critical containers with shared resources for helpers, all without losing deterministic performance.</p><h2>What You Need</h2><p>Before you begin, ensure your environment meets these prerequisites:</p><ul><li><strong>Kubernetes cluster running v1.36 or later</strong> (alpha features require this baseline).</li><li><strong><code>kubelet</code> with feature gates enabled</strong>: <code>PodLevelResourceManagers=true</code> and <code>PodLevelResources=true</code>.</li><li><strong>Topology Manager configured</strong> (set scope to <code>pod</code> via kubelet flag <code>--topology-manager-scope=pod</code>).</li><li><strong>Knowledge of NUMA architecture</strong> and how CPU/Memory Managers allocate resources.</li><li><strong>A performance-critical workload</strong> that includes sidecar containers (e.g., ML training, high-frequency trading, low-latency databases).</li><li><strong>Cluster admin privileges</strong> to modify kubelet flags and apply pod manifests.</li></ul><h2>Step-by-Step Guide</h2><h3>Step 1: Determine If Pod-Level Resource Managers Are Right for You</h3><p>Pod-Level Resource Managers are ideal when your pod contains at least one container that needs exclusive, NUMA-aligned resources (e.g., a database engine) and others that can tolerate shared resources (e.g., a logging sidecar). If all containers in your pod demand exclusive resources or you don’t require NUMA alignment, the existing per-container model may suffice.</p><p><em>Tip:</em> This feature is still alpha. Test it in a non-production cluster first.</p><h3>Step 2: Enable Feature Gates on the Kubelet</h3><p>Edit the kubelet configuration (usually <code>/var/lib/kubelet/config.yaml</code> on the node) to activate the two required feature gates:</p><pre><code>featureGates:<br> PodLevelResourceManagers: true<br> PodLevelResources: true</code></pre><p>Restart the kubelet service (<code>systemctl restart kubelet</code>). Verify the gates are active by checking the kubelet logs or using <code>kubectl describe node</code> (look for “PodLevelResourceManagers” in the feature gates list).</p><h3>Step 3: Configure Topology Manager Scope to Pod</h3><p>Set the Topology Manager scope to <code>pod</code> so that NUMA alignment considers the entire pod’s resource budget, not just individual containers. Add this flag to the kubelet:</p><pre><code>--topology-manager-scope=pod</code></pre><p>If you previously used <code>container</code> scope, change it now. Restart the kubelet again. Confirm the scope with <code>kubectl describe node | grep TopologyManager</code>.</p><h3>Step 4: Define Pod-Level Resources in Your Pod Spec</h3><p>Add a <code>resources</code> field at the pod level (<code>spec.resources</code>) to establish the total CPU and memory budget available to all containers. This budget also defines the alignment size for NUMA.</p><pre><code>apiVersion: v1<br>kind: Pod<br>metadata:<br> name: tightly-coupled-database<br>spec:<br> resources:<br> requests:<br> cpu: "8"<br> memory: "16Gi"<br> limits:<br> cpu: "8"<br> memory: "16Gi"</code></pre><p>In this example, the pod requests and limits 8 CPU cores and 16 GiB of memory. The kubelet will attempt to allocate all these resources from a single NUMA node.</p><h3>Step 5: Specify Per-Container Resource Requirements</h3><p>For each container, you can now set container-level resources that are <strong>relative to the pod’s budget</strong>. The main application container gets exclusive, integer-based resources (e.g., 4 CPUs), while sidecars can use fractional or shared allocations.</p><pre><code>containers:<br>- name: database<br> image: postgres:15<br> resources:<br> requests:<br> cpu: "4"<br> memory: "10Gi"<br> limits:<br> cpu: "4"<br> memory: "10Gi"<br>- name: metrics-exporter<br> image: prom/node-exporter:v1<br> resources:<br> requests:<br> cpu: "100m"<br> memory: "128Mi"<br>- name: backup-agent<br> image: my-backup:latest<br> resources:<br> requests:<br> cpu: "200m"<br> memory: "256Mi"</code></pre><p>Notice that the sidecar containers use millicores (<code>m</code>) and small memory amounts. The kubelet will: (1) allocate exclusive 4 CPUs + 10 GiB to the database from the pod budget, (2) create a “pod shared pool” from the remaining budget (~4 CPUs + 6 GiB), and (3) run the sidecars from that shared pool.</p><h3>Step 6: Apply the Pod and Verify Allocation</h3><p>Use <code>kubectl apply -f pod.yaml</code> to create the pod. After the pod runs, inspect its resource allocation:</p><pre><code>kubectl describe pod tightly-coupled-database</code></pre><p>Look for the <code>Conditions</code> section to confirm the pod is <code>PodLevelResourcesAllocated</code>. Also check the <code>Allocated Resources</code> section (kubelet metrics or node status) to see that exclusive and shared pools are formed correctly.</p><p>For deeper verification, run <code>kubectl exec</code> into each container and check <code>/proc/self/status</code> for Cpus_allowed_list and Mems_allowed_list to confirm NUMA affinity.</p><h3>Step 7: Tweak and Iterate</h3><p>Experiment with different pod-to-container resource ratios. If a sidecar becomes a bottleneck, increase its request from the shared pool. If the main container needs more isolation, raise its exclusive allocation. Remember that total container requests cannot exceed the pod-level requests (that would be rejected by the scheduler or kubelet).</p><h2>Tips for Success</h2><ul><li><strong>Start small</strong>: Use a test pod with one heavy container and one light sidecar to understand the behavior before deploying to production.</li><li><strong>Monitor resource contention</strong>: Shared pool containers can compete for resources under high load. Use metrics to ensure sidecars don’t starve each other.</li><li><strong>Combine with CPU Manager policies</strong>: For exclusive containers, set CPU Manager policy to <code>static</code> (default <code>none</code>). This is already handled when you request integer CPUs.</li><li><strong>Check feature gate stability</strong>: Since this is alpha, upgrade to a newer version or wait for beta before relying heavily in production.</li><li><strong>Document your allocation strategy</strong>: Clearly label which containers use exclusive vs. shared resources; it helps with troubleshooting and future migrations.</li><li><strong>Use namespace-level ResourceQuotas</strong>: Ensure the total pod-level requests don’t exceed cluster capacity; set quotas accordingly.</li></ul>