Developing Python packages often involves a tight loop of writing code, testing, and iterating. The standard pip install <package-name> process, however, requires you to reinstall the package after every change to see it reflected in your environment. This is where pip install -e (or pip install --editable) shines, allowing you to install a package in a way that links directly to your source code.
Here’s a package structure we’ll use for demonstration:
my_package/
├── my_package/
│ ├── __init__.py
│ └── module.py
└── setup.py
my_package/__init__.py:
from .module import greet
__version__ = "0.1.0"
my_package/module.py:
def greet(name="World"):
return f"Hello, {name}!"
setup.py:
from setuptools import setup, find_packages
setup(
name="my_package",
version="0.1.0",
packages=find_packages(),
install_requires=[],
)
Now, let’s install this package in editable mode from the directory containing setup.py:
pip install -e .
This command doesn’t copy your package files into your virtual environment’s site-packages directory. Instead, it creates a .egg-link file in site-packages that points back to your project’s root directory.
Let’s see what this looks like in a typical virtual environment. After running pip install -e . in a virtual environment, you’ll find a file named my_package.egg-link inside the lib/pythonX.Y/site-packages/ directory of your virtual environment. The content of this file will be the absolute path to the directory containing your setup.py file.
Now, if you have a Python interpreter or script running within this environment, you can import and use my_package as usual:
# In a separate file, e.g., main.py
from my_package import greet
print(greet("Alice"))
When you run python main.py, it will output: Hello, Alice!.
The magic happens when you modify my_package/module.py. Let’s change the greet function:
my_package/module.py (modified):
def greet(name="World"):
return f"Greetings, {name}!"
If you were to run python main.py again without reinstalling, you’d see the updated output: Greetings, Alice!. This is because Python’s import mechanism, when encountering an editable install, directly imports from the source files on disk. It doesn’t rely on a static copy within site-packages.
The setup.py file is crucial here. pip uses it to understand how to package your project, even in editable mode. It reads the name, version, and packages (found by find_packages()) to register your project correctly. The install_requires would specify any dependencies your package needs.
When you use pip install -e ., you’re essentially telling pip: "Install this package, but instead of copying the files, just make a note of where the source code lives so I can import it directly." This is incredibly powerful for development because any changes you make to your source files are immediately available in your environment.
Consider a more complex setup where your package has submodules. If my_package/module.py imported another file, say my_package/utils.py, and you were developing utils.py, changes there would also be reflected instantly without a reinstall. The .egg-link file acts as a pointer to the entire package root, enabling this live updating for all modules within it.
The primary benefit is the elimination of the reinstall step during active development. This drastically speeds up the development cycle, especially for larger projects or when making frequent small changes. It also simplifies testing, as you can run your test suite against the latest code without any extra installation steps.
One common pitfall is forgetting that pip install -e . installs based on the state of your project at the time of installation. If you later move the entire project directory, the .egg-link file will become stale, and your package will no longer be importable. You’ll need to uninstall and reinstall it in editable mode from the new location. Similarly, if you modify setup.py (e.g., change the package name or add new discoverable packages), you’ll often need to reinstall.
The underlying mechanism involves setuptools creating an "editable install." This means that instead of copying files into site-packages, setuptools creates a small file (usually .egg-link or a .pth file) in site-packages that tells Python’s import system where to find your actual source code. When you import a module from an editable package, Python follows this link and imports directly from your project directory.
The next concept you’ll likely encounter is managing dependencies for your editable package, especially when those dependencies themselves are also under development.