perf inside a Docker container can be tricky because the container’s kernel view is isolated from the host’s.
Let’s see perf in action. Imagine we have a simple Python app that does some busy-waiting:
import time
def busy_wait(duration):
start_time = time.time()
while time.time() - start_time < duration:
pass
if __name__ == "__main__":
print("Starting busy wait...")
busy_wait(10)
print("Done.")
We’ll build a Docker image for this:
FROM python:3.9-slim
WORKDIR /app
COPY busy_wait.py .
CMD ["python", "busy_wait.py"]
Now, let’s build and run it, and try to use perf from inside the container:
docker build -t perf-test .
docker run -it --name my-perf-app perf-test
Inside the running container:
root@container:/app# perf top
Warning:
Cannot initialize perf:/sys/kernel/debug/tracing/trace_marker: No such file or directory.
/bin/perf: Unknown error -1
This is where things get interesting. The error Cannot initialize perf:/sys/kernel/debug/tracing/trace_marker: No such file or directory tells us perf can’t access the necessary kernel tracing infrastructure. By default, Docker containers run with a restricted view of the host’s kernel, and the performance monitoring unit (PMU) events and tracing points that perf relies on are often not exposed or are restricted.
The fundamental problem is that perf needs direct access to the host’s performance counters and kernel tracing mechanisms. When you run a container with default settings, it’s isolated from these resources to ensure security and stability.
Here are the common reasons perf fails in Docker and how to fix them:
-
Lack of Privileged Access:
perfrequires access to/proc/kcoreand/sys/kernel/debug/tracing, which are often restricted in unprivileged containers.- Diagnosis: Check
ls -l /proc/kcoreandls -l /sys/kernel/debug/tracinginside the container. You’ll likely see permission denied errors. - Fix: Run the container with the
--privilegedflag. This grants the container almost all the capabilities of the host.docker run -it --privileged --name my-perf-app perf-test - Why it works:
--privilegedeffectively removes the isolation for kernel-level access, allowingperfto see and interact with the host’s performance monitoring hardware and tracing infrastructure.
- Diagnosis: Check
-
Missing
sys_ptraceCapability: Even with--privileged, some security profiles might still restrictptraceoperations, whichperfuses for certain types of analysis.- Diagnosis: While
--privilegedusually covers this, explicitly checkingcapsh --printinside the container (ifcapshis installed) can show capabilities. - Fix: If not using
--privileged, add theSYS_PTRACEcapability.docker run -it --cap-add SYS_PTRACE --name my-perf-app perf-test - Why it works: The
SYS_PTRACEcapability is specifically required byperfto attach to processes and read their memory or registers, which is essential for sampling and tracing.
- Diagnosis: While
-
Host Kernel Configuration (Less Common): Certain kernel configurations or security modules (like SELinux or AppArmor) on the host might prevent access to performance counters even when the container has elevated privileges.
- Diagnosis: This is harder to diagnose from within the container. Check host
dmesgor/var/log/audit/audit.logfor SELinux/AppArmor denials related to Docker orperfaccess. - Fix: Temporarily disable SELinux (
setenforce 0) or adjust AppArmor profiles for the Docker daemon. This is a host-level change and requires root access. For SELinux, you might need to allow specific Docker operations.# On the host sudo setenforce 0 - Why it works: These security modules can enforce granular access controls. Disabling them or creating specific rules allows
perf(via the privileged container) to access the necessary hardware.
- Diagnosis: This is harder to diagnose from within the container. Check host
-
perf_event_paranoidSetting: The host kernel has a setting,perf_event_paranoid, that controls how much access unprivileged users (and by extension, containers) have to performance events. A high value can blockperf.- Diagnosis: Inside the container, check the value:
If it’sroot@container:/app# cat /proc/sys/kernel/perf_event_paranoid3or4, it’s too restrictive. - Fix: Change the setting on the host. For profiling all processes (including kernel), set it to
1or0.
To make it persistent across reboots, edit# On the host echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid/etc/sysctl.confor a file in/etc/sysctl.d/. - Why it works: This kernel parameter directly governs the availability of performance monitoring hardware events. Lowering the value enables access for
perf.
- Diagnosis: Inside the container, check the value:
-
Container Runtime/Version Issues: Older Docker versions or specific container runtimes might have bugs or limitations in how they expose kernel features.
- Diagnosis: Check your Docker version (
docker version) and the container runtime (containerd --version,crio --version). - Fix: Update Docker and the container runtime to the latest stable versions.
# Example for updating Docker sudo apt-get update sudo apt-get upgrade docker-ce docker-ce-cli containerd.io - Why it works: Newer versions often include fixes for kernel interaction and capability handling.
- Diagnosis: Check your Docker version (
-
Mounting Debugfs: Sometimes, the specific debugfs mounts required by
perfmight not be automatically available or correctly set up in the container’s namespace, even with--privileged.- Diagnosis: Inside the container, check if
/sys/kernel/debug/tracingexists and is accessible. - Fix: Explicitly mount debugfs when running the container.
docker run -it --privileged -v /sys/kernel/debug:/sys/kernel/debug --name my-perf-app perf-test - Why it works: This ensures that the host’s debugfs filesystem, which contains critical tracing information, is directly mapped into the container’s filesystem, making it visible and accessible to
perf.
- Diagnosis: Inside the container, check if
After applying --privileged (which is the most common and effective solution for most users), you should be able to run perf top inside the container and see your Python application:
root@container:/app# perf top
# Output will show Python processes and their hot functions
If you fix all these and try to run perf record -g -o output.perf -- your_command, the next error you might encounter is a perf: Failed to open /sys/kernel/debug/tracing/events/syscalls/sys_enter_clone/enable: Permission denied if you are not using --privileged or if specific tracepoints are blocked.