perf is a powerful Linux performance analysis tool, but it often requires elevated privileges to gather detailed system-wide performance data. When you encounter permission errors when trying to use perf, it means the system is preventing your user account from accessing the necessary performance monitoring hardware or kernel interfaces.
Here’s how to diagnose and fix those perf permission issues:
Cause 1: Insufficient perf_event_paranoid Level
The perf_event_paranoid kernel parameter controls which types of performance events users can access. A high value restricts access to prevent unprivileged users from observing sensitive kernel or hardware behavior.
Diagnosis:
Check the current perf_event_paranoid value:
cat /proc/sys/kernel/perf_event_paranoid
Common values are:
-1: All events allowed.0: Disable all CPU events.1: Disable kernel events.2: Disable kernel and user events.3: Disable all events except raw tracepoints.4: Disable all events.
If this value is 2 or higher, you likely can’t access the events perf needs.
Fix:
Temporarily set a less restrictive level (e.g., 1 or 0):
sudo sysctl kernel.perf_event_paranoid=1
This change is temporary and will revert on reboot.
To make it permanent, edit /etc/sysctl.conf (or a file in /etc/sysctl.d/) and add/modify the line:
kernel.perf_event_paranoid = 1
Then apply the changes:
sudo sysctl -p
Why it works: This directly tells the kernel to relax restrictions on what performance events can be sampled, allowing perf to access the necessary data.
Cause 2: Missing CAP_SYS_ADMIN or CAP_PERFMON Capabilities
Even with a low perf_event_paranoid setting, certain perf operations, especially those involving CPU-wide profiling or specific hardware counters, require elevated privileges. These are often managed via Linux capabilities.
Diagnosis:
Run perf with verbose output. You might see errors like "Operation not permitted" or specific mentions of capabilities.
perf top -v
If the above doesn’t give enough detail, check the capabilities of the perf executable itself (though this is less common for user-level perf runs). More relevant is checking if your user has the necessary capabilities, which is usually granted via CAP_SYS_ADMIN for general system access or CAP_PERFMON for specific performance monitoring.
Fix:
-
Add user to
perf_event_usersgroup (if available): Some distributions create a group forperfaccess.sudo usermod -aG perf_event_users $(whoami)You’ll need to log out and log back in for group changes to take effect. Why it works: This grants the user membership in a pre-defined group that has been configured to have specific permissions for
perfoperations, often by setting a broaderperf_event_paranoidor granting capabilities to the group. -
Grant
CAP_SYS_ADMIN(use with caution): This is a broad privilege.sudo setcap cap_sys_admin+ep /usr/bin/perfWhy it works: This directly attaches the
CAP_SYS_ADMINcapability to theperfbinary, allowing it to perform system-level operations without needing to run as root. This is generally discouraged for security reasons; direct user privilege grants are preferred. -
Grant
CAP_PERFMON(more specific):sudo setcap cap_perfmon+ep /usr/bin/perfWhy it works: This grants the more specific
CAP_PERFMONcapability, which is designed for performance monitoring tasks, making it a more secure alternative toCAP_SYS_ADMIN.
Cause 3: SELinux or AppArmor Restrictions
Mandatory Access Control systems like SELinux or AppArmor can impose their own restrictions on what processes can access. They might prevent perf from reading kernel data structures or hardware registers.
Diagnosis:
Check system logs for SELinux or AppArmor denials related to perf or the kernel’s performance event subsystem.
For SELinux:
sudo ausearch -m avc -ts recent
For AppArmor:
sudo dmesg | grep -i apparmor
sudo journalctl -xe | grep -i apparmor
Look for entries mentioning perf, perf_event, or related kernel modules.
Fix:
-
SELinux:
- Temporarily set SELinux to permissive mode:
Ifsudo setenforce 0perfworks now, SELinux is the culprit. - Identify the specific SELinux policy violation and create a new policy module to allow the action. This is distribution-specific but often involves tools like
audit2allow.# After identifying the AVC denial in ausearch output sudo grep perf /var/log/audit/audit.log | audit2allow -M perf_custom sudo semodule -i perf_custom.pp
Why it works: SELinux enforces strict security contexts. Allowing
perfto access specific kernel interfaces by updating its policy modifies the rules to permit the necessary operations. - Temporarily set SELinux to permissive mode:
-
AppArmor:
- Temporarily disable AppArmor profiles for
perf:
(orsudo aa-complain /usr/bin/perfsudo aa-disable /usr/bin/perf) Ifperfworks, AppArmor is the culprit. - Edit the AppArmor profile for
perf(usually in/etc/apparmor.d/) to grant the necessary read access to relevant kernel files or device nodes. Why it works: AppArmor profiles define allowed actions for executables. Modifying the profile allowsperfto interact with the system resources it needs.
- Temporarily disable AppArmor profiles for
Cause 4: Debugfs Not Mounted or Accessible
perf sometimes relies on debugfs for certain information or operations, especially for advanced tracing. If debugfs isn’t mounted or your user can’t access it, perf might fail.
Diagnosis: Check if debugfs is mounted and accessible:
mount | grep debugfs
ls /sys/kernel/debug/
If it’s not mounted, or if you can’t list its contents, this could be an issue.
Fix: Mount debugfs if it’s not already:
sudo mount -t debugfs none /sys/kernel/debug
Ensure your user has read permissions on the debugfs mount point. This is usually handled by system configuration, but if permissions are overly restrictive, you might need to adjust them (though this is rare and generally not recommended due to debugfs’s nature).
Why it works: Debugfs provides a filesystem interface to kernel debugging information, which perf can leverage. Mounting it makes this information available.
Cause 5: Older Kernel or perf Version Bugs
While less common, older kernel versions might have bugs in their performance event handling, or older perf userspace tools might not be compatible with newer kernel features or security models.
Diagnosis: Check your kernel version:
uname -r
Check your perf version:
perf --version
Search online for known issues related to your specific kernel version and perf with the error message you’re encountering.
Fix:
- Upgrade Kernel: If a known bug exists, upgrading to a newer, stable kernel version is the most robust solution.
- Upgrade
perf: Ensure you have the latest version of thelinux-toolspackage (or equivalent for your distribution) that provides theperfuserspace tools.
Why it works: Newer versions often contain bug fixes and improved compatibility with kernel interfaces, resolving the underlying issue.# Example for Debian/Ubuntu sudo apt update && sudo apt install linux-tools-common linux-tools-$(uname -r)
Cause 6: Hardware Issues or Specific CPU Models
In rare cases, the performance monitoring unit (PMU) on your CPU might be faulty, or specific CPU models might have quirks that affect perf’s ability to access certain counters.
Diagnosis:
Try running perf on a different machine with the same OS and perf version.
If perf works there, the issue is likely specific to the hardware.
Check dmesg for any hardware-related errors during boot.
Fix:
- Workaround: If a specific counter or event type is causing issues, try profiling with different events.
- Hardware Replacement: If the PMU is indeed faulty, the hardware itself may need to be replaced.
- Kernel Patch: For specific CPU model quirks, a kernel patch might eventually be released, requiring a kernel upgrade. Why it works: This addresses the problem at its root, either by avoiding problematic hardware features or by fixing flawed hardware behavior through software.
The next common problem you’ll hit after resolving permission issues is perf itself failing to find or access specific hardware performance counters, often manifesting as "No such file or directory" errors for certain event types or "Failed to open /dev/cpu/…/msr" messages.