mongodb/etc/pip

On requirements (*-requirements.txt) files

MongoDB requires multiple pypa projects installed to build and test. To that end, we provide our own *-requirements.txt files for specific domains of use. Inside each requirements file, there are only include statements for component files. These files are the bare requirements for specific components of our python environment. This separation allows us to avoid repetition and conflict in our requirements across components.

For most developers, if you pip-install dev-requirements.txt, you have the python requirements to lint, build, and test MongoDB.

On the constraints (constraints.txt) file

Our requirements files are minimally constrained. For the majority of pypa projects, any given requirements file will install the newest version of the project. For increased stability, there is a constraints.txt file with explicit versions for each required project. This file represents a somewhat portable manifest of the latest acceptable pypa projects version according to a set of local python environments. It is also used to install the site-packages in the MongoDB internal toolchain-builder.

Please note that the verify_pip task in evergreen confirms that constraints.txt satisfies toolchain-requirements.txt. If you modify the pypa project requirements, you should regenerate the constraints file.

How to modify a pypa project requirement in a component

The most common edit of our requirements is likely a change to the constraints on a pypa project that we already use. For example, say that we currently require pymongo >= 3.0, < 3.6.0 in the component core. You would like to use PyMongo 3.7, so you instead modify the line in etc/pip/components/core.req to read pymongo >= 3.0, != 3.6.0. Since this is a modification to an existing component, you do not need to modify any requirements file. However, you do need to regenerate the constraints file. The workflow will usually look like:

$ # Make your changes to the component file
$ $EDITOR etc/pip/components/core.req
$ # Regenerate the constraints file
$ bash buildscripts/generate-pip-constraints.sh -r etc/pip/toolchain-requirements.txt -o etc/pip/constraints.txt

How to add a new component (*.req) file

Occasionally, we will require a set of pypa projects for an entirely new piece of software in our repository. This usually implies adding a new component file. For example, say that we need to add a logging system to both local development and evergreen. This system requires the fictional pypa project FooLog. So we add a file foolog.req and require it from both dev-requirements.txt and evgtest-requirements.txt. Like the majority of our components, we want it in the toolchain, so we also add it to toolchain-requirements.txt. The workflow will usually look like:

$ # Make the component file
$ echo "FooLog" >etc/pip/components/foolog.req
$ # Require the component from the requirements files
$ echo "-r components/foolog.req" >>etc/pip/dev-requirements.txt
$ echo "-r components/foolog.req" >>etc/pip/evgtest-requirements.txt
$ echo "-r components/foolog.req" >>etc/pip/toolchain-requirements.txt
$ # Regenerate the constraints file
$ bash buildscripts/generate-pip-constraints.sh -r etc/pip/toolchain-requirements.txt -o etc/pip/constraints.txt

How to add a new requirements (*-requirements.txt) file

Rarely, we will have an entirely new domain of requirements that is useful. In this case, we need to at least make a new requirements file. For example, say we want to make a requirements file for packaging our code. We would need most of the requirements for dev-requirements.txt but the testing has already been done in our continuous integration. So we create a new file package-requirements.txt and require a smaller subset of components. The new file at etc/pip/package-requirements.txt would look like this:

-r components/platform.req
-r components/core.req

-r components/compile.req
-r components/lint.req

Notice that since we did not change any components files, we do not need to regenerate our constraints.