Managing multiple Python packages within a single Git repository, often called a monorepo, presents unique challenges for dependency management and local development. The standard pip tools, designed for single-package projects, can struggle to understand relationships and build processes across these interconnected packages.
Let’s see how pip actually interacts with a monorepo setup, focusing on the pip install -e (editable install) mechanism, which is the cornerstone of local monorepo development.
Imagine this simple monorepo structure:
monorepo/
├── packages/
│ ├── my_package_a/
│ │ ├── pyproject.toml
│ │ └── src/
│ │ └── my_package_a/
│ │ └── __init__.py
│ └── my_package_b/
│ ├── pyproject.toml
│ └── src/
│ └── my_package_b/
│ └── __init__.py
└── pyproject.toml (Root project configuration)
And packages/my_package_a/pyproject.toml:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my_package_a"
version = "0.1.0"
dependencies = [] # For simplicity, no external dependencies
And packages/my_package_b/pyproject.toml:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my_package_b"
version = "0.1.0"
dependencies = [
"my_package_a==0.1.0", # Declares a dependency on package_a
]
If you were to cd into monorepo/packages/my_package_b and run pip install -e ., pip would install my_package_b in editable mode. However, it wouldn’t automatically know about my_package_a unless you explicitly told it to.
This is where the real power (and complexity) of monorepo management comes in. You need a way to tell pip about all the packages in your monorepo and how they relate. Tools like pip-tools, poetry, pdm, or even custom scripts often handle this by generating a consolidated requirements.txt or by using advanced pip features.
Consider a root pyproject.toml that could orchestrate this, though standard pip doesn’t natively support monorepo-wide dependency resolution directly from a single root file. For example, a tool might use this structure:
# monorepo/pyproject.toml (for illustration of a potential monorepo tool)
[tool.monorepo_tool] # Hypothetical tool configuration
packages = ["packages/my_package_a", "packages/my_package_b"]
If you were using a tool like pdm (which has excellent monorepo support), you’d navigate to the root monorepo/ directory and run pdm install. pdm would then discover all sub-packages and manage their dependencies, including the local my_package_a dependency within my_package_b. It would effectively add a line like -e packages/my_package_a and -e packages/my_package_b to an internal requirements list, ensuring both are available and resolvable.
The core problem pip alone faces is that pip install -e . is context-aware only to the current directory. It doesn’t have a global view of your monorepo structure or the dependencies between your local packages.
To make my_package_b aware of my_package_a when installing my_package_b in editable mode, you’d typically need to install my_package_a first, also in editable mode, from the monorepo root:
cd monorepo/packages/my_package_a
pip install -e .
cd ../my_package_b
pip install -e .
This ensures that my_package_a is available in your Python environment when pip tries to resolve the my_package_a==0.1.0 dependency for my_package_b. The pip install -e . command for my_package_b will find the locally installed my_package_a and link it correctly.
The most surprising truth about pip in a monorepo context is that it can work, but it requires an explicit, often manual, orchestration of the installation order or the use of a higher-level monorepo management tool that generates the correct pip commands for you. pip itself is fundamentally a single-package installer; it doesn’t have built-in heuristics for discovering and linking multiple local packages within a single repository.
The true power comes from tools that abstract this. For example, poetry or pdm can be configured to recognize the monorepo structure and manage the editable installs of all internal packages automatically. They often achieve this by creating a virtual environment at the root of the monorepo and installing all sub-packages into it using pip install -e, ensuring all local dependencies are resolvable.
If you’ve installed everything correctly using an appropriate monorepo tool or by carefully ordering your pip install -e commands, you’ll find that my_package_b can now import from my_package_a without issue, and changes in my_package_a are immediately reflected in my_package_b because of the editable installs.
The next hurdle you’ll likely encounter is managing transitive dependencies across your monorepo packages and ensuring consistent versions when you need to build or release individual components.