You can set up pip to look for packages in more than one place, not just the default PyPI.
Let’s see this in action. Imagine you have a private package repository at https://my.private.repo/simple/ and you also want to use the public PyPI.
Here’s a pip.conf (or pip.ini on Windows) file that tells pip to check your private repo first, then fall back to PyPI:
[global]
index-url = https://my.private.repo/simple/
extra-index-url = https://pypi.org/simple/
Now, when you run pip install some-package, pip will first query https://my.private.repo/simple/ for some-package. If it’s not found there, it will then query https://pypi.org/simple/.
This is incredibly useful for a few reasons. Firstly, it lets you use internal, proprietary packages that you don’t want to host on public PyPI. Secondly, it can speed up installations if your private index is geographically closer or has better network connectivity than the public PyPI. Thirdly, it allows for a staged rollout of packages – you can put a new version on your private index, test it internally, and then, once validated, make it available on public PyPI.
The index-url is the primary source pip checks. If a package is found and installed from index-url, pip won’t even bother looking at extra-index-url for that specific installation. The extra-index-url is only consulted if the package isn’t found on the index-url. You can specify multiple extra-index-url lines, and pip will check them in the order they appear.
Here’s how pip resolves dependencies when multiple indexes are configured. Suppose you’re trying to install package-a, and package-a depends on dependency-x.
- Pip looks for
package-aon theindex-url. - If found, it installs
package-a. - Then, it looks for
dependency-xon theindex-url. - If
dependency-xis found onindex-url, it’s installed. - If
dependency-xis not found onindex-url, pip then checks theextra-index-urls in order. - If
dependency-xis found on anextra-index-url, it’s installed. - If
dependency-xis not found on any of the configured indexes, pip will raise an error.
The key is that pip tries to resolve all requirements from the index-url first, and only then resorts to the extra-index-urls if absolutely necessary for any part of the dependency tree. This means if package-a is on your private index but its dependency dependency-x is only on public PyPI, pip will successfully find both. However, if package-a is on PyPI but dependency-x is only on your private index, and your private index is listed as extra-index-url, pip will not find dependency-x because it was looking for package-a on the index-url (PyPI) and didn’t find dependency-x there. It would then check your private index for package-a (which it wouldn’t find if it’s not there) and only then try to find dependency-x on the private index.
This behavior is often a point of confusion. People expect pip to check all indexes for each package, but it prioritizes the index-url. You can override this behavior on the command line, though. For example, pip install --index-url https://pypi.org/simple/ --extra-index-url https://my.private.repo/simple/ some-package will ensure that some-package and all its dependencies are searched for on both URLs. This is generally the more robust approach when you have critical dependencies that might only exist on one of the indexes.
The pip.conf file can be located in several places, depending on your operating system and user context. For global configuration (affecting all users), it might be in /etc/pip.conf (Linux/macOS) or C:\ProgramData\pip\pip.ini (Windows). For user-specific configuration, it’s typically in ~/.config/pip/pip.conf (Linux/macOS) or %APPDATA%\pip\pip.ini (Windows).
A more advanced configuration involves using different index URLs for different package scopes, which is achievable through requirements.txt files themselves. You can specify index-url and extra-index-url directives directly within a requirements.txt file. This is powerful because it allows specific projects or environments to have their own tailored package source configurations without affecting the global pip setup. For instance, a requirements.txt might start with:
--index-url https://my.internal.repo/simple/
--extra-index-url https://pypi.org/simple/
my-internal-package==1.2.3
requests>=2.20.0
When you run pip install -r requirements.txt, these directives override any global or user-specific pip.conf settings for the duration of that installation command. This isolation is critical for reproducible builds, ensuring that a project always uses the exact same set of package sources, regardless of the user’s local machine configuration.
Another subtle but important aspect is how pip handles authentication for private indexes. If your https://my.private.repo/simple/ requires authentication, you can include credentials directly in the URL: https://user:password@my.private.repo/simple/. However, this is generally discouraged for security reasons as it hardcodes credentials. A better approach is to use environment variables or pip’s credential helper mechanisms, although these are more complex to set up.
Understanding how pip resolves package names across multiple indexes is crucial for managing complex dependency graphs and ensuring reliable deployments.
The next thing you’ll likely run into is how to manage different Python versions and their associated package environments.