This repository was archived by the owner on Jul 26, 2020. It is now read-only.

Implement new vision system#33

Merged

Adimote merged 113 commits intomasterfrom

Feb 25, 2018

Member

Adimote commented Feb 18, 2018 •

edited

Loading

This PR is based off of #29 as it uses some images from the fixed data set in testing. The first commit you should look at is 5bebf1f .

Added a new vision system using the unofficial python3 opencv bindings.
Removed the old ML-based vision system (good riddance)
Use distance_model as the new parameter for the camera model xml file generated by opencv
As the old calibration files are removed too, the default marker size (the size of a marker if its id isn't defined for the game) is now defined in game_specific.py
Update all distance-verifying tests to accept a certain error amount (within 15% of the total distance)

Adimote added 5 commits

February 17, 2018 11:24


          Merge remote-tracking branch 'origin/fixup-tecknet-references'

515d8dc


          add new vision system logic

5bebf1f


          Remove calibration scripts for old ML-based vision

90f80f2


          Rename distance_model to camera_model

85d1aeb

this is because the camera is the calibration subject in the new
model, not the distance of the markers.


          fix tests to use the new vision system

50127cb

Adimote requested review from PeterJCLaw, RealOrangeOne and prophile

February 18, 2018 16:20


          update dependencies

78d85ac

Adimote force-pushed the new-vision branch from 7394840 to 78d85ac Compare

February 18, 2018 16:40

Adimote added 2 commits

February 18, 2018 16:43


          remove scikit learn and pyyaml deps

7223dc8


          add .xml files to deb

b47d2aa

Adimote changed the title ~~Implement new vision system~~ [WIP] Implement new vision system

Adimote added the work-in-progress label

Member Author

Adimote commented Feb 18, 2018

Just trying to get tests to pass before this is ready for review.

Adimote removed request for PeterJCLaw, RealOrangeOne and prophile

February 18, 2018 16:47

RealOrangeOne reviewed

View reviewed changes

sb_vision/tokens.py

+                      marker_id = apriltag_detection.id
                       # *************************************************************************
                       # NOTE: IF YOU CHANGE THIS, THEN CHANGE ROBOT-API camera.py

Member

RealOrangeOne Feb 18, 2018

You have changed this file, however robot-api hasn't been changed (ideally by design). Is this note necessary anymore?

Member Author

Adimote Feb 18, 2018

It happens to be that the changes I made here doesn't affect robot-api's version. If I were to add a new value (i.e. orientation), I'd need to add it in robot-api, so I think we still need have a need for the message

Contributor

PeterJCLaw Feb 18, 2018

I'm not sure that that's strictly true, and certainly nowhere near as dramatic as this comment appears.

Adding a value here makes it possible to also expose that value in robot-api. Removing a value here is (hopefully obviously) a breaking change, so we'd need to modify the consumers of the library anyway.

I'd be in favour of reducing the shoutyness of this comment. However, from what I can see the comment itself doesn't change here, so that would be a separate PR anyway.

RealOrangeOne reviewed

View reviewed changes

sb_vision/vision.py Outdated

-                      distance_model = self.camera.distance_model
+                      camera_model = self.camera.camera_model
+                      detections = list(self.apriltag_detector.detect_tags(img))

Member

RealOrangeOne Feb 18, 2018

Why are you casting this to a list, to then just iterate over it? I think keeping it as-is inside the list comprehension is much nicer

Member Author

Adimote Feb 18, 2018

Good catch, this was because I iterated over it twice at one point when experimenting, fixed in 5239714

Adimote added 5 commits

February 18, 2018 17:05


          add xml files to setup.py

a908e70


          move xml file to manifest

8f23a45


          Merge branch 'master' into new-vision

737d9c4


          possibly fixed CI syntax error?

dc0c52e


          remove redundant list conversion

Adimote mentioned this pull request

Remove our thresholding functionality, as apriltags does it itself #32

Merged


          use Path object for Python 3.5 support

8b8ff4e

PeterJCLaw reviewed

View reviewed changes

Contributor

PeterJCLaw left a comment

I realise this is work-in progress, so some of the internal variances might be things already on your list of TODOs; hopefully this is useful input anyway.

tests/test_coordinates_from_file.py Outdated

    
                          rot_y=math.atan(-0.1),  # or -5.710593137499643 degrees

                          dist=(0.01 + 1)**0.5,  # or 1.004987562112089

                          rot_y=math.atan(-0.1),

                          dist=(0.1 + 1) ** 0.5,

Contributor

PeterJCLaw Feb 18, 2018

If the Cartesian position hasn't changed, then this too should be unchanged.

Member Author

Adimote Feb 18, 2018

Oops that's a miss-calculation, forgot these values are squared. I'll make it more explicit.

tests/test_coordinates_from_file.py

-                          polar_x=1.6704649792860642,
-                          polar_y=1.5707963267948966,
-                          dist=1.004987562112089,
-                      ),

Contributor

PeterJCLaw Feb 18, 2018

Why are we removing this? While it's deprecated, it's still supported. We therefore still need to actually have it so should be validating that it outputs the same values.

Member Author

Adimote Feb 18, 2018

re-added in df9df93

tests/test_coordinates_from_file.py

+              # Rotation tolerance is independent of distance
+              ROTATION_TOLERANCE = 0.075
+              CALIBRATIONS = Path(__file__).parent.parent / 'calibrations' / 'tecknet_25cm'

Contributor

PeterJCLaw Feb 18, 2018

IMO it would be much better if we could have this still use the c270 samples (I realise that means also calibrating a c270) since we have confidence that the c270 model was correct. That would therefore make it much easier to have confidence that the new calibration mechanism is outputting consistent data.

Member Author

Adimote Feb 18, 2018

I don't think we should calibrate a c270 just so we don't need to change these tests. Calibrating has to be done manually using the interactive opencv script, and is thus more effort than it's worth.

tests/test_coordinates_from_file.py Outdated

    
                          rot_y=math.atan(-0.1),  # or -5.710593137499643 degrees

                          dist=(0.01 + 1)**0.5,  # or 1.004987562112089

                          rot_y=math.atan(-0.1),

                          dist=(0.1 + 1) ** 0.5,

Contributor

PeterJCLaw Feb 18, 2018

Why are you removing the comments here? They were present intentionally to help:

humans reason about whether values are correct
make it easier to understand the test failures (since the 'expected' values will be the calculated commented value, rather than the expression)

Member Author

Adimote Feb 18, 2018

re-added them in 5125186

tests/test_coordinates_from_file.py Outdated



		def assertWithinTolerance(val, expected, tolerance, error_msg=""):
		assert (expected - tolerance) < val < (expected + tolerance), error_msg

Contributor

PeterJCLaw Feb 18, 2018

I've not used it, but you might enjoy pytest.approx

Member Author

Adimote Feb 18, 2018

fixed in f1e0154

Contributor

PeterJCLaw Feb 18, 2018

@Adimote Out of curiosity: what are the error messages like from pytest.approx? Are they useful in helping understand the failure?

Member Author

Adimote Feb 24, 2018 •

edited

Loading

Yes, they're actually as detailed as not using approx

sb_vision/distance_finding.py Outdated

		return text


		def get_calibration(file_name: Path) -> Dict:

Contributor

PeterJCLaw Feb 18, 2018

This will (should) be failing type checking -- please can we specify the details of this Dict. If they're too complex, consider using a typing.NamedTuple instead.

Member Author

Adimote Feb 24, 2018

Fixed a while ago (CBA to find commit hash)

sb_vision/distance_finding.py Outdated

+                              values = _get_text(element.find('data'))
+                              if data_type == 'd':  # doubles
+                                  data = np.reshape(np.array([float(v) for v in values]),
+                                                    (rows, cols))

Contributor

PeterJCLaw Feb 18, 2018

What's the benefit of going into numpy here? If there's precision that we're aiming for (numpy's float64 is more precise than float is IIRC), then we should be using float64 to parse the string. Otherwise IMO we should stick to Python types here (not least as it makes the type signatures much easer to reason about).

Member Author

Adimote Feb 18, 2018

fixed in 50d2a45

Member Author

Adimote Feb 18, 2018

sb_vision/distance_finding.py Outdated

		import numpy as np


		def _get_text(element: etree.Element):

Contributor

PeterJCLaw Feb 18, 2018

This doesn't really return "text"; is there a better name for it?

Member Author

Adimote Feb 18, 2018

fixed in 304abe6

sb_vision/distance_finding.py Outdated



		@functools.lru_cache()
		def load_camera_calibrations(file_name: Path) -> Tuple[np.ndarray, np.ndarray]:

Contributor

PeterJCLaw Feb 18, 2018

Note that typing of numpy types doesn't actually work -- they're treated as Any, so if we can avoid using their array types and instead use built-in types that might be better.

sb_vision/vision.py Outdated

                       camera_model = self.camera.camera_model
-                      detections = list(self.apriltag_detector.detect_tags(img))
+                      detections = self.apriltag_detector.detect_tags(img)

Contributor

PeterJCLaw Feb 18, 2018

I think Jake's point was that we could leave this in the comprehension below entirely. This would both minimise the diff and also keep the clarity that the previous spelling had.

Adimote added 2 commits

February 18, 2018 18:53


          fix all linting errors

c1e713b


          fix typechecker errors

d002d3b

PeterJCLaw added 3 commits

February 25, 2018 03:43


          Switch to standard library XML parsing

42f9815


          Introduce helper to resolve type hint issues

713960b


          Ignore type weirdness from Element.text

26c047a

Contributor

PeterJCLaw commented Feb 25, 2018

@Adimote thanks for those changes; I think this branch is now all good except the deployment plan & testing it on a Pi.

My wrapping of solvePnP now works and I've put up #36 with code.

As our usage of lxml was fairly lightweight, I had a quick look at whether we could remove that. It turns out that the etree stuff we were using is present in the standard library. I'm not sure whether it's actually the same code, but it works the same as far as we were using it. I've pushed #37 which moves us to the standard library there.

The combination of those allows us to ship this without any extra packages, which makes this much easier to ship! Once those PRs are merged into this one, this PR can be merged into master.

Contributor

PeterJCLaw commented Feb 25, 2018 •

edited

Loading

@Adimote I've just spotted that this still lists some of the now-redundant dependencies in the debian package control file. We should remove those.

Specifically we can remove python3-sklearn and python3-yaml.

Contributor

PeterJCLaw commented Feb 25, 2018

@Adimote manual testing of this on a Pi mostly works, huzzah!

There are a couple of easily fixable breakages:

robotd will currently always request a calibration for a 'c270' model. We therefore need to not explode if that's asked for. We can do this either by faking the c270 calibration as being the C016 calibration (probably via a symlink) or by actually calibrating a c270. My preference would be for the latter as it's less confusing for anyone reading this repo and will allow developers to use them if that's what they have to hand.
(I worked around this by editing my robotd/camera.py to ask for 'C016')
robotd expects that a Token will have a homography_matrix attribute (with a .tolist() method). We'll need to continue to expose that so as not to break robotd users (I don't see any reason to remove it really).
(I worked around this by commenting out that line in my robotd/camera.py )

Other than that, this now works well and seems to be fairly accurate :)

PeterJCLaw and others added 2 commits

February 25, 2018 18:00


          We also depend on the calib3d parts of OpenCV

bf09b9c


          remove dependencies for sklearn and yaml

70a377f

Adimote force-pushed the new-vision branch from 78937f0 to 8111c8c Compare

February 25, 2018 20:09

Adimote added 2 commits

February 25, 2018 20:12


          re-add Homography Matrices

9db3ae4


          Pretend to support the c270

a398dcb

Adimote force-pushed the new-vision branch from 8111c8c to a398dcb Compare

February 25, 2018 20:12

PeterJCLaw and others added 2 commits

February 25, 2018 21:05


          Clarify where this value comes from

5d32563


          Revert: Go back to original homography handling

0e0ff71

This reverts commit 9db3ae4.

Adimote force-pushed the new-vision branch from 8c069c3 to 0e0ff71 Compare

February 25, 2018 21:40


          Restore original signature shape

28bc729

Adimote mentioned this pull request

Introduce a Homography Coordinate type #39

Open

PeterJCLaw added 9 commits

February 25, 2018 22:19


          Merge branch 'wrap-solve-pnp-directly' into new-vision

92367b0


          Merge branch 'remove-lxml' into new-vision


          Produce an error when loading a mismatching calibration

If we use a calibration for a resolution other than that which it
is intended, we'll get nonsense values. Ensure that that doesn't
happen by requiring that `load_camera_calibrations` is passed the
image resolution which will be used and then validating that
against the value in the calibration file.


          Fix docstring

bd5fd55


          Make this test actually useful

045c7d1


          Fix typo

dada343


          Rename this test now it's more general

2aa1059


          Produce an error when loading a mismatching calibration

090afdb

This merges 'new-vision-check-image-size' into new-vision. If we use
a calibration for a resolution other than that which it is intended,
we'll get nonsense values. Ensure that that doesn't happen by
requiring that `load_camera_calibrations` is passed the image
resolution which will be used and then validating that against the
value in the calibration file.


          Add compatibility note and removal deadline

663a725

PeterJCLaw approved these changes

View reviewed changes

Contributor

PeterJCLaw left a comment

Woo!

Adimote merged commit 59e6819 into master

mxbi commented Feb 28, 2018

Now that this has been merged, is the new vision system in a state where we can load it up on the pi (after installing the libopencv/python headers) and try it out? :)

Contributor

PeterJCLaw commented Mar 1, 2018

@mxbi happily there aren't any new dependencies for this change, so all that's needed is to build the latest package and install it.

Having done that you should find that the existing support in robot-api yields much more accurate results, though you may wish to wait for the next published update as that will include some other niceties too.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet