requirements.txt files are a surprisingly fragile piece of infrastructure that often break in subtle, hard-to-debug ways, especially as projects grow.
The core problem is that pip’s default behavior when installing from a requirements.txt is often too permissive, leading to drift and potential runtime errors. It’s not just about listing dependencies; it’s about pinning them and managing their transitive dependencies effectively.
Here’s a breakdown of how to manage your requirements.txt like a pro, avoiding the common pitfalls:
1. Pin Everything, Always
The single most important practice is to pin every single dependency to an exact version. This means no ranges (like >=1.0) or fuzzy specifiers (like ~=1.0.1).
Diagnosis: Look for lines in your requirements.txt that don’t specify an exact version, such as:
requests
flask>=2.0
django~=3.2.0
Fix: Use pip freeze > requirements.txt to generate an exact list of installed packages. For new projects, install your initial dependencies and then run pip freeze.
# Example requirements.txt
requests==2.28.1
flask==2.2.2
django==4.1.7
Why it works: Pinning ensures that every developer and every deployment environment installs the exact same set of packages. This eliminates "it works on my machine" issues and guarantees reproducibility, preventing unexpected behavior caused by minor version updates in dependencies that might introduce breaking changes or subtle bugs.
2. Generate requirements.txt After Installation, Not Before
Don’t manually create or edit requirements.txt from scratch for a new project. Install your dependencies first, then capture them.
Diagnosis: A requirements.txt file that was hand-written before any packages were installed.
Fix:
- Create a virtual environment:
python -m venv venv - Activate it:
source venv/bin/activate(orvenv\Scripts\activateon Windows) - Install your initial dependencies:
pip install requests flask - Generate the
requirements.txt:pip freeze > requirements.txt
Why it works: This process automatically includes all direct dependencies and their transitive dependencies, ensuring that your requirements.txt accurately reflects what’s actually installed and functional.
3. Use a Virtual Environment Religiously
This is foundational. pip freeze will capture everything in your current Python environment. If you have other projects or global packages installed, they’ll all get included, leading to a bloated and inaccurate requirements.txt.
Diagnosis: requirements.txt contains packages that are not actually used by your current project, or versions that are unexpectedly high.
Fix:
- Always create and activate a virtual environment before installing packages or running
pip freeze.python -m venv .venv source .venv/bin/activate - Install dependencies within the activated environment.
- Generate
requirements.txtfrom within the activated environment.
Why it works: Virtual environments isolate project dependencies. pip freeze then only captures packages installed specifically for that project, ensuring a clean and accurate requirements.txt.
4. Split Requirements for Different Environments
A single requirements.txt for development, testing, and production is a common mistake. Production environments often need fewer, more specific packages than development.
Diagnosis: Your requirements.txt includes development tools like pytest, black, flake8, or ipython, which are not needed at runtime in production.
Fix: Create separate files, e.g., requirements-dev.txt, requirements-prod.txt, requirements-test.txt.
# requirements-prod.txt
flask==2.2.2
gunicorn==20.1.0
# requirements-dev.txt
-r requirements-prod.txt
flask==2.2.2
gunicorn==20.1.0
pytest==7.2.1
flake8==5.0.4
black==22.12.0
Use the -r flag to include base requirements.
Why it works: This allows you to install only what’s necessary for each context. Production deployments are leaner and less prone to conflicts, while development and testing environments have all the tools needed for the job.
5. Regularly Update and Re-pin
Dependencies evolve. Security vulnerabilities are found, and new features are added. Stale dependencies are a major security risk.
Diagnosis: Your requirements.txt hasn’t been updated in months or years.
Fix: Periodically run pip install --upgrade <package_name> for key dependencies and then regenerate requirements.txt with pip freeze. For more automated updates, consider tools like Dependabot or Renovate.
# Example workflow
source .venv/bin/activate
pip install --upgrade requests flask
pip freeze > requirements.txt
Why it works: Keeping dependencies updated mitigates security risks and allows you to benefit from performance improvements and bug fixes. Re-pinning after upgrades ensures that your requirements.txt reflects the latest stable versions you’ve tested.
6. Use a Lock File Generator (Advanced)
While pip freeze is the standard, it can sometimes be difficult to manage transitive dependencies precisely, especially when dealing with complex dependency graphs. Tools like pip-tools offer more control.
Diagnosis: You’re struggling to resolve dependency conflicts or want a more explicit way to manage direct vs. transitive dependencies.
Fix:
- Install
pip-tools:pip install pip-tools - Create a
requirements.infile with your direct dependencies (can use version specifiers).# requirements.in flask>=2.2 requests - Compile it to
requirements.txt:pip-compile requirements.in - Install from the compiled file:
pip install -r requirements.txt
Why it works: pip-compile resolves the entire dependency tree based on requirements.in and creates a fully pinned requirements.txt. This separation makes it clear which dependencies you chose and which were pulled in transitively, and it provides a robust mechanism for updating dependencies incrementally.
The next common issue you’ll encounter after perfecting your requirements.txt is managing Python versions across different environments.