perf inside a Docker container can be tricky because the container’s kernel view is isolated from the host’s.

Let’s see perf in action. Imagine we have a simple Python app that does some busy-waiting:

import time

def busy_wait(duration):
    start_time = time.time()
    while time.time() - start_time < duration:
        pass

if __name__ == "__main__":
    print("Starting busy wait...")
    busy_wait(10)
    print("Done.")

We’ll build a Docker image for this:

FROM python:3.9-slim
WORKDIR /app
COPY busy_wait.py .
CMD ["python", "busy_wait.py"]

Now, let’s build and run it, and try to use perf from inside the container:

docker build -t perf-test .
docker run -it --name my-perf-app perf-test

Inside the running container:

root@container:/app# perf top
Warning:
Cannot initialize perf:/sys/kernel/debug/tracing/trace_marker: No such file or directory.
/bin/perf: Unknown error -1

This is where things get interesting. The error Cannot initialize perf:/sys/kernel/debug/tracing/trace_marker: No such file or directory tells us perf can’t access the necessary kernel tracing infrastructure. By default, Docker containers run with a restricted view of the host’s kernel, and the performance monitoring unit (PMU) events and tracing points that perf relies on are often not exposed or are restricted.

The fundamental problem is that perf needs direct access to the host’s performance counters and kernel tracing mechanisms. When you run a container with default settings, it’s isolated from these resources to ensure security and stability.

Here are the common reasons perf fails in Docker and how to fix them:

  1. Lack of Privileged Access: perf requires access to /proc/kcore and /sys/kernel/debug/tracing, which are often restricted in unprivileged containers.

    • Diagnosis: Check ls -l /proc/kcore and ls -l /sys/kernel/debug/tracing inside the container. You’ll likely see permission denied errors.
    • Fix: Run the container with the --privileged flag. This grants the container almost all the capabilities of the host.
      docker run -it --privileged --name my-perf-app perf-test
      
    • Why it works: --privileged effectively removes the isolation for kernel-level access, allowing perf to see and interact with the host’s performance monitoring hardware and tracing infrastructure.
  2. Missing sys_ptrace Capability: Even with --privileged, some security profiles might still restrict ptrace operations, which perf uses for certain types of analysis.

    • Diagnosis: While --privileged usually covers this, explicitly checking capsh --print inside the container (if capsh is installed) can show capabilities.
    • Fix: If not using --privileged, add the SYS_PTRACE capability.
      docker run -it --cap-add SYS_PTRACE --name my-perf-app perf-test
      
    • Why it works: The SYS_PTRACE capability is specifically required by perf to attach to processes and read their memory or registers, which is essential for sampling and tracing.
  3. Host Kernel Configuration (Less Common): Certain kernel configurations or security modules (like SELinux or AppArmor) on the host might prevent access to performance counters even when the container has elevated privileges.

    • Diagnosis: This is harder to diagnose from within the container. Check host dmesg or /var/log/audit/audit.log for SELinux/AppArmor denials related to Docker or perf access.
    • Fix: Temporarily disable SELinux (setenforce 0) or adjust AppArmor profiles for the Docker daemon. This is a host-level change and requires root access. For SELinux, you might need to allow specific Docker operations.
      # On the host
      sudo setenforce 0
      
    • Why it works: These security modules can enforce granular access controls. Disabling them or creating specific rules allows perf (via the privileged container) to access the necessary hardware.
  4. perf_event_paranoid Setting: The host kernel has a setting, perf_event_paranoid, that controls how much access unprivileged users (and by extension, containers) have to performance events. A high value can block perf.

    • Diagnosis: Inside the container, check the value:
      root@container:/app# cat /proc/sys/kernel/perf_event_paranoid
      
      If it’s 3 or 4, it’s too restrictive.
    • Fix: Change the setting on the host. For profiling all processes (including kernel), set it to 1 or 0.
      # On the host
      echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid
      
      To make it persistent across reboots, edit /etc/sysctl.conf or a file in /etc/sysctl.d/.
    • Why it works: This kernel parameter directly governs the availability of performance monitoring hardware events. Lowering the value enables access for perf.
  5. Container Runtime/Version Issues: Older Docker versions or specific container runtimes might have bugs or limitations in how they expose kernel features.

    • Diagnosis: Check your Docker version (docker version) and the container runtime (containerd --version, crio --version).
    • Fix: Update Docker and the container runtime to the latest stable versions.
      # Example for updating Docker
      sudo apt-get update
      sudo apt-get upgrade docker-ce docker-ce-cli containerd.io
      
    • Why it works: Newer versions often include fixes for kernel interaction and capability handling.
  6. Mounting Debugfs: Sometimes, the specific debugfs mounts required by perf might not be automatically available or correctly set up in the container’s namespace, even with --privileged.

    • Diagnosis: Inside the container, check if /sys/kernel/debug/tracing exists and is accessible.
    • Fix: Explicitly mount debugfs when running the container.
      docker run -it --privileged -v /sys/kernel/debug:/sys/kernel/debug --name my-perf-app perf-test
      
    • Why it works: This ensures that the host’s debugfs filesystem, which contains critical tracing information, is directly mapped into the container’s filesystem, making it visible and accessible to perf.

After applying --privileged (which is the most common and effective solution for most users), you should be able to run perf top inside the container and see your Python application:

root@container:/app# perf top
# Output will show Python processes and their hot functions

If you fix all these and try to run perf record -g -o output.perf -- your_command, the next error you might encounter is a perf: Failed to open /sys/kernel/debug/tracing/events/syscalls/sys_enter_clone/enable: Permission denied if you are not using --privileged or if specific tracepoints are blocked.

Want structured learning?

Take the full Perf course →