Skip to content
  • Ed Morley's avatar
    Tests: Replace pipenv shunit2 tests with Hatchet tests (#1146) · cfa5c4da
    Ed Morley authored
    The current buildpack-test-runner shunit2 integration tests are not
    fully representative of production, slow, and the test runner in need
    of maintenance.
    
    Using Hatchet for tests ensures parity with production and means we can
    use much greater concurrency (since we can rely on platform capacity).
    
    In the future some tightly scoped bash *unit* tests may also be added
    to the buildpack using a more up to date bash testing framework, once
    the buildpack's bash scripts have been refactored to be more testable.
    However even with these, we'll still want production end to end tests
    using Hatchet.
    
    This is the first of several PRs to convert the existing shunit2 tests
    to Hatchet, and whilst doing so add missing test coverage.
    
    With these new Hatchet tests, I've decided to start validating the build
    log output more strictly (ie: asserting against multiple lines), since:
    * the build log is an important part of a buildpack's UX, and something
      I plan on improving over the coming months
    * it's the only way to ensure various error lines aren't being output,
      without having to report to fragile negative assertions like these:
      https://github.com/heroku/heroku-buildpack-python/blob/90f3eeadfe5ef4aadd7439e9a646868de7180852/spec/hatchet/python_spec.rb#L14
      (longer term after the buildpack is refactored we can reduce the
      chance of these errors being emitted by enforcing exit on error
      everywhere, but even then we likely want to catch non-fatal warnings
      emitted by tools such as pip or pipenv)
    
    The tests were written following the rspec style guide:
    https://rspec.rubystyle.guide/
    
    Note: The newly added Hatchet tests don't yet test post-deployment
    functionality since the existing tests didn't either - this will be
    added in future PRs.
    
    Closes @W-8665943@.
    
    [skip changelog]
    cfa5c4da