Passive object tracking using camera

• US 10,719,953 B1
• Filed: 03/27/2018
• Issued: 07/21/2020
• Est. Priority Date: 03/27/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Ye, Yuting, Wang, Robert Y., Twigg, Christopher David, Han, Shangchen, Wu, Po-Chen
• Primary Examiner(s): Yang, Wei Wen
• Application Number:
  US15/937,766
• Time in Patent Office: 847 Days
• Field of Search: None
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A system tracks poses of a passive object using fiducial markers on fiducial surfaces of a polygonal structure of the object using image data captured by a camera. The system includes an object tracking controller that generates an estimated pose for a frame of the image data using an approximate pose estimation (APE), and then updates the estimated pose using a dense pose refinement (DPR) of pixels. The APE may include minimizing reprojection error between projected image points of the fiducial markers and observed image points of the fiducial markers in the frame. The DPR may include minimizing appearance error between image pixels of the fiducial markers in the frame and projected model pixels of the fiducial markers determined from the estimated pose and the object model. In some embodiments, an inter-frame corner tracking (ICT) of the fiducial markers may be used to facilitate the APE.

Claims

1. A system, comprising:

a camera configured to generate image data of an object, the object including a polygonal structure at a first end and a tip at a second end opposite the first end, the polygonal structure including fiducial surfaces, each fiducial surface including a fiducial marker; and

an object tracking controller, configured to;

receive a frame of the image data of the object from the camera;

determine an estimated pose of the of the polygonal structure in the frame based on minimizing reprojection error between a first number of projected image points for each of the fiducial markers in the frame and the first number of observed image points for each of the fiducial markers in the frame, the projected image points determined from spatial locations of the fiducial markers determined from the observed image points;

determine a pose of the polygonal structure by minimizing appearance error between a second number of image pixels for each of the fiducial markers in the frame and the second number of projected model pixels for each of the fiducial markers in the frame, the projected model pixels generated using the estimated pose and an object model of the object, the second number being larger than the first number; and

determine a spatial location of the tip of the object from the pose of the polygonal structure and the object model.

2. The system of claim 1, wherein the object tracking controller is further configured to:

determine spatial locations of the tip of the object over time from multiple frames of the image data captured by the camera; and

determine an image drawn by the tip of the object on a surface based on the spatial locations of the tip over time.

3. The system of claim 1, wherein the object tracking controller is further configured to:

determine the observed image points of the fiducial markers in the frame;

determine the spatial locations of the fiducial markers from the observed image points based on locations of corners of the fiducial markers in the frame;

determine the projected image points of the fiducial markers based on the spatial locations of the fiducial markers;

determine the reprojection error between the projected image points and the observed image points; and

minimize the reprojection error to determine the estimated pose.

4. The system of claim 1, wherein the object tracking controller is configured to:

determine a search region in the frame based on second image pixels of the fiducial markers in a prior frame of the image data; and

search within the search region to determine the observed image points of the fiducial markers in the frame.

5. The system of claim 3, wherein the object tracking controller is further configured to:

determine that less than a predefined number of fiducial markers of the polygonal structure are identified in the frame; and

in response to determining that less than the predefined number of fiducial markers are identified in the frame, apply inter-frame corner tracking of the polygonal structure across multiple frames of the image data to determine observed image points of at least a portion of the fiducial markers.

6. The system of claim 5, wherein the object tracking controller is further configured to:

determine motion samples of the fiducial markers from spatial locations of the fiducial markers across the multiple frames of the image data;

determine an aggregate motion sample of the fiducial markers across the multiple frames of the image data from the motion samples of the fiducial markers;

update the aggregate motion sample by removing spatial locations of fiducial markers that are incompatible with the aggregate motion sample; and

determine observed image points of the at least a portion of the fiducial markers based on the updated aggregate motion sample.

7. The method of claim 1, wherein object tracking controller is further configured to normalize intensity of the image pixels of the fiducial markers in the frame prior to minimizing the appearance error.

8. The system of claim 1, wherein the object tracking controller is further configured to apply a Gauss Newton iteration and a backtracking line search to minimize the appearance error between the second number of image pixels of the fiducial markers in the frame and the second number of projected model pixels of the fiducial markers determined from the estimated pose and the object model.

9. The system of claim 1, wherein the object tracking controller is configured to generate the projected model pixels using mipmaps of the fiducial markers, the projected model pixels of each fiducial marker being generated based on a mipmap from a set of mipmaps corresponding with the fiducial marker, the mipmap selected according to pixel size of the image pixels of the fiducial markers in the frame.

10. The system of claim 9, wherein a portion of the mipmap is masked.

11. The system of claim 1, wherein the object tracking controller is further configured to calibrate the object model using a bundle adjustment to determine relative positions of the fiducial markers and the polygonal structure in the object.

12. The system of claim 1, wherein the object model defines the polygonal structure including a dodecahedron shape, at least a portion of the surfaces of the dodecahedron shape being a fiducial surface including a fiducial marker.

13. A system, comprising:

an object tracking controller, configured to;

receive a frame of image data of an object captured by a camera, the object including a polygonal structure at a first end and a tip at a second end opposite the first end, the polygonal structure including fiducial surfaces, each fiducial surface including a fiducial marker;

determine an estimated pose of the of the polygonal structure in the frame based on minimizing reprojection error between a first number of projected image points for each of the fiducial markers in the frame and the first number of observed image points for each of the fiducial markers in the frame, the projected image points determined from spatial locations of the fiducial markers determined from the observed image points;

determine a pose of the polygonal structure by minimizing appearance error between a second number of image pixels for each of the fiducial markers in the frame and the second number of projected model pixels for each of the fiducial markers in the frame, the projected model pixels generated using the estimated pose and an object model of the object, the second number being larger than the first number; and

determine a spatial location of the tip of the object from the pose of the polygonal structure and the object model.

14. The system of claim 13, wherein the object tracking controller is further configured to:

determine the observed image points of the fiducial markers in the frame;

determine the spatial locations of the fiducial markers from the observed image points based on locations of corners of the fiducial markers in the frame;

determine the projected image points of the fiducial markers based on the spatial locations of the fiducial markers;

determine the reprojection error between the projected image points and the observed image points; and

minimize the reprojection error to determine the estimated pose.

15. The system of claim 13, wherein the object tracking controller is further configured to:

determine that less than a predefined number of fiducial markers of the polygonal structure are identified in the frame; and

in response to determining that less than the predefined number of fiducial markers are identified in the frame;

determine motion samples of the fiducial markers from spatial locations of the fiducial markers across multiple frames of the image data;

determine an aggregate motion sample of the fiducial markers across the multiple frames of the image data from the motion samples of the fiducial markers;

update the aggregate motion sample by removing spatial locations of fiducial markers that are incompatible with the aggregate motion sample; and

determine observed image points of the at least a portion of the fiducial markers based on the updated aggregate motion sample.

16. The system of claim 13, wherein the object tracking controller is further configured to apply a Gauss Newton iteration and a backtracking line search to minimize the appearance error between the second number of image pixels of the fiducial markers in the frame and the second number of projected model pixels of the fiducial markers determined from the estimated pose and the object model.

17. The system of claim 13, wherein the object tracking controller is configured to generate the projected model pixels using mipmaps of the fiducial markers, the projected model pixels of each fiducial marker being generated based on a mipmap from a set of mipmaps corresponding with the fiducial marker, the mipmap selected according to pixel size of the image pixels of the fiducial markers in the frame.

18. A method, comprising, by one or more processors:

receiving a frame of image data of an object from a camera, the object including a polygonal structure at a first end and a tip at a second end opposite the first end, the polygonal structure including fiducial surfaces, each fiducial surface including a fiducial marker;

determining an estimated pose of the of the polygonal structure in the frame based on minimizing reprojection error between a first number of projected image points for each of the fiducial markers in the frame and the first number of observed image points for each of the fiducial markers in the frame, the projected image points determined from spatial locations of the fiducial markers determined from the observed image points;

determining a pose of the polygonal structure by minimizing appearance error between a second number of image pixels for each of the fiducial markers in the frame and the second number of projected model pixels for each of the fiducial markers in the frame, the projected model pixels generated using the estimated pose and an object model of the object, the second number being larger than the first number; and

determine a spatial location of the tip of the object from the pose of the polygonal structure and the object model.

19. The method of claim 18, further comprising, by the one or more processors:

determining the observed image points of the fiducial markers in the frame;

determining the spatial locations of the fiducial markers from the observed image points based on locations of corners of the fiducial markers in the frame;

determining the projected image points of the fiducial markers based on the spatial locations of the fiducial markers;

determining the reprojection error between the projected image points and the observed image points; and

minimizing the reprojection error to determine the estimated pose.

20. The method of claim 18, further comprising, by the one or more processors:

determining a search region in the frame based on second image pixels of the fiducial markers in a prior frame of the image data; and

searching within the search region to determine the observed image points of the fiducial markers in the frame.