Skip to content

[#1074] Create a localization system test prototype

Yunus Emre Çalışkan requested to merge 1074-create-system-tests-for-the-localization-stack into master

Description

Closes #1074 (closed)

Notes for reviewer

This MR implements a simple system test for localization. To use, launch the evaluation launch file, launch the simulator, drive around and then shut down the evaluator, which should print the results:

Processed 91 samples.
Average translation error: 0.173523 meters.
Average rotation error: 0.00298193 radians.

The launch configuration is meant to be a basis for future automated evaluation/system tests.

Details:

The application is initialized at the correct origin and from then on, NDT localizes the vehicle using the previous estimate without the contribution of the odometry data from the simulator, which is the ground truth itself anyway.

The translation error is basically the mean absolute trajectory error where we get the euclidean distance between the estimate and the ground truth. Same goes for the rotation error where it's the mean absolute radial distance between the estimate and ground truth rotations. It should be noted that, the result of the evaluation depends on the kinematic conditions of the test drive. I assume the performance will be better on a slower drive. So when this test is automated, having a fixed trajectory would provide a fixed reference for the evaluation results.

We can extend these metrics by computing their standard deviations etc.

Additional KPIs/metrics that we should add:

  • Average run time of NDT localizer
  • Relative Pose Error: This is computed by comparing the two corresponding relative transformations rather than comparing two absolute poses. It is a useful metric to remove the effect of accumulated drift during the drive which should be the case since we make the NDT initialize itself using the previous pose at each step.

Pre-review checklist for the author before submitting for review

Every developer is encouraged to be familiar with our contributor guidelines.

  1. MR formalities
    1. "WIP" or "Draft" removed from the MR title
    2. MR title and description help a friendly human understand the problem solved
    3. MR has a link to the original issue in the description, if it exists
    4. If the source branch is on a fork, MR is configured to allow commits from developers with access to push to the target branch
    5. Sensible notes for the reviewer added to the section above to facilitate review
    6. Target branch set correctly. Default: master
    7. MR assigned to a capable reviewer. Default: JWhitleyWork
    8. Splitting the MR into smaller, easier-to-review merge requests was considered
  2. Code and tests
    1. Code is properly formatted
    2. Tests affected by new code pass locally
    3. Reasonable coverage with unit tests of 90+%; else create a follow-up ticket
    4. Review any // TODO item added in the MR that can be addressed without the reviewer's help
  3. Documentation
    1. Any new and modified code has accurate doxygen documentation
    2. Any diagrams are committed

Checklist for the reviewer

Only the reviewer is allowed to make changes in this section!

Items not applicable to this MR are crossed out by the reviewer.

  • For new nodes, the checklist is expanded and reviewed
  • For a new package, the checklist is expanded and reviewed
  • Reviewer crossed out non-applicable items

Checklist

If the MR provides an improvement, don't hesitate to add a 👍 emoji for a neat line of code or a "Thanks for implementing this" comment. This will reward the MR author and prevent the review from being only about what still needs to be improved.

Mark all the items that are done.

Checklist for development
  1. Basic checks
    1. The MR title describes what is being done on the ticket
    2. All functional code written in C++14, tooling code may be written in Python 3.5+ or Bash
    3. The first commit has a proper commit message to be used as a basis for the squashed commit created at the very end; e.g. [#928] Fix foo in bar
  2. Code correctness
    1. The problem/feature is solved (reproducibly)
    2. The solution is performant enough for the use case in mind
  3. Open work
    1. Any added source-code comment about future work refers to a follow-up GitLab issue explicitly; e.g., // TODO #551 refactor code below
  4. Documentation
    1. New classes, methods, functions in headers are documented with doxygen-style comments
    2. If implementation (of a function...) is modified, the doxygen documentation is updated accordingly
    3. The design article is updated with the implementation
    4. Drawings are created when needed and committed to git
    5. Modified files have a license that is compatible with AutowareAuto
  5. Testing
    1. Code coverage with unit tests does not decrease. Aim for coverage with unit tests of 90+%; else create a follow-up ticket
    2. Unit tests make sense and cover the business logic and error cases
    3. Integration tests are sensible and not flaky
Checklist for new ROS2 nodes
  1. Every node is registered as a component
  2. The naming conventions are followed
  3. At least the basic launch component test is included
Checklist for new package
  1. Structure
    1. The package name and organization into files is sensible
    2. The files have a license header as per CONTRIBUTING.md
    3. Core functionality is separated from the ROS2-specific part where reasonable
    4. There is a design document that explains the package at a high level
    5. All dependencies are explicitly included in package.xml with the proper <*depend> declaration

When starting from scratch, new packages should be created with the autoware_auto_create_pkg macro and they will automatically satisfy the following criteria.

  1. If an existing package is added to AutowareAuto, it should match the output of autoware_auto_create_pkg regarding
    1. calling autoware_set_compile_options for each compiled target
    2. same set of linters
    3. visibility control
    4. finding build dependencies in cmake with ament_auto_find_build_dependencies()
    5. installing with ament_auto_package()

Post-review checklist for the author

After receiving approval:

  1. Rendered documentation looks as expected
  2. All checkboxes in the MR checklist are checked or crossed out. Syntax example: 1. [ ] ~~this item crossed out~~
  3. Developers were informed about breaking changes on slack in the committers-autoware-auto channel
  4. If there are multiple commits, the MR title has to contain the issue number as it's used for the squashed commit message; e.g. [#928] Fix foo in bar
  5. Assign MR to @merge-bot
Edited by Igor Bogoslavskyi

Merge request reports