Skip to content

[SE-708] Upgrade all packages to the latest versions. Move to Django 2

Daniel Clemente Laboreo requested to merge clemente/django2-and-other-upgrades into master

This PR upgrades all packages in requirements.txt to their latest versions. In particular it migrates to the latest Django version 2.2. It also updates many core packages like huey, honcho, ipython, … and also the cleanup tests. No functionality is lost. It also introduces pip-tools to manage our depencies file (see below).

This is the work of SE-711 (upgrading packages) plus SE-713 (making tests work again). Also SE-709 (pip-tools), SE-1256 (fix code quality) and SE-1415 (deployment).

Details

OpenStack

  • the parts about OpenStack had to be a bit rewritten, both the connection setup and some parameters. Also the part about num_retries (retrying in case on error). Some more work could help to simplify the code (e.g. we're using 2 OpenStack libraries with a bit different behaviour)

django-angular

  • migrating to the latest django-angular is unbearable for too many reasons: it forces us to rewrite the way we handle files (logo/favicon) and the code that handles it isn't working (render() got an unexpected keyword argument 'renderer') and it requires more packages and even Node. Disabling django-angular is troublesome and requires rewriting many tests. So we're not upgrading django-angular, and instead I fixed the version we're using to make it work in Django 2, it. It's in this branch

quality tests (pylint)

  • upgrading pylint surfaced hundreds of new code quality errors: no documentation, line too long, redefined function with different methods, too many branches, broad exception, unused import, unneeded return, and more. This PR fixes them. It involves refactoring some functions

minor changes

  • some functions were changed to use str instead of bytes, since the new version of some package started passing b"strings like this one" to templates and to DB fields, and they ended up in haproxy config files and in DB fields exactly like that: with a lowercase b, a quotation mark etc. (3 extra bytes)

requirements.txt is now autogenerated by pip-compile

  • earlier: we hand-edit requirements.txt or use pip freeze >requirements.txt. It includes all
  • now: we hand-edit requirements.in to list the software we care about (our direct dependencies). Then we run pip-compile. This creates requirements.txt with the full list. We commit both.

not upgraded to the latest version

  • pylint was a mess; we can't upgrade to the latest versions of packages because there are many dependency conflicts and in addition many open bugs. The status of each package is:
    • pep8-naming: newest version 0.8.2, but latest prospector (1.1.6.4) limits us to 0.4.1
    • pycodestyle: newest version 2.5.0, but latest prospector (1.1.6.4) limits us to 2.4.0. Actually, 2.3.1 due to flake8
    • pyflakes: newest version 2.1.1, but latest prospector (1.1.6.4) limits us to 1.6.0
    • flake8: newest version 3.7.7, but the mentioned pyflakes downgrade limits us to 3.5.0
    • astroid: newest version 2.2.5, but a bug with an old version of pylint/prospector forces us to downgrade to 2.1.0. By the way the maintainer of some of those tools, Carl, worked in OpenCraft (hello if you read this)
    • pylint: newest version 2.3.1 but that same bug affects many components, and downgrading pylint to 2.2.3 avoids it
    • pylint-django: newest version 2.0.9, but it has another bug (open!) that I saw in my Ocim, and I had to downgrade to 2.0.6 to fix it
    • that means that: prospector is holding many of our packages by a few years (we could try to try it with recent versions and upstream it), but in addition some of the pylint-related bugs (including open bugs) would make an upgrade to the latest versions a bit longer (we'd have to upstream some fixes)
  • in addition PyYAML: newest version 5.1.1, but latest jasmine (3.4.0) limits us to 3.13
  • selenium isn't upgraded to version 3 since we found new problems in the test cases that would require a non-trivial rewrite. See SE-1414

Playbooks They don't require any update. .env files don't require updates either.

Testing instructions

  • test recreating vagrant
  • or pip install -r requirements.txt from a current Ocim
  • run Ocim and verify that it runs and can deploy. Tto do real deployments you'll need a good .env. Ask someone else for theirs or take Ocim stage's .env
  • integration tests should pass
  • we'll test this in stage first

Merge request reports