Skip to content

Draft: Move deps to pyproject.toml. Update Deps. Regen Protos.

Cal Pratt requested to merge cpratt34/requirements-cleanup into master

Description

Use the latest stable configurations for packaging the buildgrid project by leveraging pyproject.toml and setup.cfg to configure package layouts.

As it stands, all of the requirements contents had been duplicated in the main requirements.in file, leaving most of the CI setup redundant by installing the main .in file and the individual extras .in files. This MR leaves the behavior mostly untouched to begin the process of properly defining the extras with pyproject.toml and setup.cfg. It uses pip's recursive dependency feature to construct the "all" extra, removing the need for some of the more complex pip-tools usage.

The protobuf generation utility has been replaced by tox -e protos which now supports creating pyi stub files for the pb2_grpc modules as well. A pb2_grpc_aio module is also created which will allow creating entities using the asyncio fixtures from grpc.aio. The majority of the code changes from this MR are simply from regenerating the protos.

Note: There were a number of unused json and yaml files in the protos packages which I imaging were generated from previous protoc calls. These files have been removed.

The utility for regeneration of the requirements.txt file has also been replaced in this work. You can now use tox -e requirements-append to add newly defined dependencies from the pyproject.toml into requirements.txt (keeping old pins the same) or use tox -e requirements-refresh to upgrade all dependencies to the latest versions (ignoring the previous pins).

Note: Updates to some of the libraries exposed more type errors that I have marked ignored. We should revisit them at some point.

Sadly, pylint does not seem to support the new protobuf generation, failing to resolve imports. I have switched the linting to instead use flake8, which runs much faster and covers a wider number of use-cases. We may also choose to later extend flake8 with plugins, which I have done to continue creating the codeclimate.json file, using the flake8-gl-codeclimate plugin.

Finally, I've updated the docs to point to these new tools for future devs.

Edited by Cal Pratt

Merge request reports