SYSTEM AND METHOD FOR MARKER BASED TRACKING

• US 20180046874A1
• Filed: 08/10/2017
• Published: 02/15/2018
• Est. Priority Date: 08/10/2016
• Status: Abandoned Application

• Current Assignee: uSens Inc.
• Original Assignee: uSens Inc.
• Inventors: GUO, Rongwei, MA, Gengyu, FEI, Yue
• Application Number:
  US15/673,568
• Publication Number:
  US 20180046874A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis

Abstract

A tracking method is disclosed. The method may be implementable by a rotation and translation detection system. The method may comprise obtaining a first and a second images of a physical environment, detecting (i) a first set of markers represented in the first image and (ii) a second set of markers represented in the second image, determining a pair of matching markers comprising a first marker from the first set of markers and a second marker from the second set of markers, the pair of matching markers associated with a physical marker disposed within the physical environment, and obtaining a first three-dimensional (3D) position of the physical marker based at least on the pair of matching markers.

Claims

1. A tracking method, implementable by a rotation and translation detection system, the method comprising:

obtaining a first and a second images of a physical environment;

detecting (i) a first set of markers represented in the first image and (ii) a second set of markers represented in the second image;

determining a pair of matching markers comprising a first marker from the first set of markers and a second marker from the second set of markers, the pair of matching markers associated with a physical marker disposed within the physical environment; and

obtaining a first three-dimensional (3D) position of the physical marker based at least on the pair of matching markers.

2. The tracking method of claim 1, wherein:

the physical marker is disposable on an object, associating the object with the first 3D position of the physical marker; and

the first and second images are a left and a right images of a stereo image pair.

3. The tracking method of claim 1, further comprising:

obtaining a position and an orientation of a system capturing the first and the second images relative to the physical environment.

4. The tracking method of claim 1, wherein:

the first and second images comprise infrared images; and

obtaining the first and the second images of the physical environment comprises;

emitting infrared light, at least a portion of the emitted infrared light reflected by the physical marker;

receiving at least a portion of the reflected infrared light; and

obtaining the first and the second images of the physical environment based at least on the received infrared light.

5. The tracking method of claim 1, wherein:

the first and second images comprise infrared images;

the physical marker is configured to emit infrared light; and

obtaining the first and the second images of the physical environment comprises;

receiving at least a portion of the emitted infrared light; and

obtaining the first and the second images of the physical environment based at least on the received infrared light.

6. The tracking method of claim 1, wherein detecting (i) the first set of markers represented in the first image and (ii) the second set of markers represented in the second image comprises:

generating a set of patch segments from the first image;

determining a patch value for each of the set of patch segments;

comparing the each patch value with a patch threshold to obtain one or more patch segments with patch values above the patch threshold;

determining a brightness value for each pixel of the obtained one or more patch segments;

comparing the each brightness value with a brightness threshold to obtain one or more pixels with brightness values above the brightness threshold; and

determining a contour of each of each of the markers based on the obtained one or more pixels.

7. The tracking method of claim 1, wherein determining the pair of matching markers comprises:

generating a set of candidate marker pairs, each candidate marker pair comprising a maker from the first set of markers and another marker from the second set of markers;

comparing coordinates of the markers in the each candidate marker pair with a coordinate threshold value to obtain candidate marker pairs comprising markers having coordinates differing less than the coordinate threshold value;

determining a depth value for each of the obtained candidate marker pairs comprising markers having coordinates differing less than the coordinate threshold value; and

for the each obtained candidate marker pair, comparing the determined depth value with a depth threshold value to obtain the obtained candidate marker pair exceeding the depth threshold value as the pair of matching markers.

8. The tracking method of claim 1, wherein obtaining the first 3D position of the physical marker based at least on the pair of matching markers comprises:

obtaining a projection error associated with capturing the physical marker in the physical environment on the first and second images, wherein the physical environment is 3D and the first and second images are 2D; and

obtaining the first 3D position of the physical marker based at least on the pair of matching markers and the projection error.

9. The tracking method of claim 1, wherein:

the first and the second images are captured at a first time to obtain the first 3D position of the physical marker;

a third and a fourth images are captured at second first time to obtain a second 3D position of the physical marker; and

the method further comprises;

associating inertia measurement unit (IMU) data associated with the first and the second images and IMU data associated with the third and the fourth images to obtain an orientation change of an imaging device, the imaging device captured the first, the second, the third, and the fourth images;

pairing a marker associated with the first and the second image to another marker associated with the third and the fourth image;

obtaining a change in position of the physical marker relative to the imaging device based on the paring;

associating the orientation change of the imaging device and the change in position of the physical marker relative to the imaging device; and

obtaining movement data of the imaging device between the first time and the second time based at least on the orientation change of the imaging device and the associated change in position of the physical marker relative to the imaging device.

10. A tracking system, comprising:

a processor; and

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform a method, the method comprising;

obtaining a first and a second images of a physical environment;

detecting (i) a first set of markers represented in the first image and (ii) a second set of markers represented in the second image;

determining a pair of matching markers comprising a first marker from the first set of markers and a second marker from the second set of markers, the pair of matching markers associated with a physical marker disposed within the physical environment; and

obtaining a first three-dimensional (3D) position of the physical marker based at least on the pair of matching markers.

11. The tracking system of claim 10, wherein:

the physical marker is disposable on an object, associating the object with the first 3D position of the physical marker; and

the first and second images are a left and a right images of a stereo image pair.

12. The tracking system of claim 10, further comprising:

obtaining a position and an orientation of a system capturing the first and the second images relative to the physical environment.

13. The tracking system of claim 10, wherein:

the first and second images comprise infrared images; and

obtaining the first and the second images of the physical environment comprises;

emitting infrared light, at least a portion of the emitted infrared light reflected by the physical marker;

receiving at least a portion of the reflected infrared light; and

obtaining the first and the second images of the physical environment based at least on the received infrared light.

14. The tracking system of claim 10, wherein:

the first and second images comprise infrared images;

the physical marker is configured to emit infrared light; and

obtaining the first and the second images of the physical environment comprises;

receiving at least a portion of the emitted infrared light; and

obtaining the first and the second images of the physical environment based at least on the received infrared light.

15. The tracking system of claim 10, wherein detecting (i) the first set of markers represented in the first image and (ii) the second set of markers represented in the second image comprises:

generating a set of patch segments from the first image;

determining a patch value for each of the set of patch segments;

comparing the each patch value with a patch threshold to obtain one or more patch segments with patch values above the patch threshold;

determining a brightness value for each pixel of the obtained one or more patch segments;

comparing the each brightness value with a brightness threshold to obtain one or more pixels with brightness values above the brightness threshold; and

determining a contour of each of each of the markers based on the obtained one or more pixels.

16. The tracking system of claim 10, wherein determining the pair of matching markers comprises:

generating a set of candidate marker pairs, each candidate marker pair comprising a maker from the first set of markers and another marker from the second set of markers;

comparing coordinates of the markers in the each candidate marker pair with a coordinate threshold value to obtain candidate marker pairs comprising markers having coordinates differing less than the coordinate threshold value;

determining a depth value for each of the obtained candidate marker pairs comprising markers having coordinates differing less than the coordinate threshold value; and

for the each obtained candidate marker pair, comparing the determined depth value with a depth threshold value to obtain the obtained candidate marker pair exceeding the depth threshold value as the pair of matching markers.

17. The tracking system of claim 10, wherein obtaining the first 3D position of the physical marker based at least on the pair of matching markers comprises:

obtaining a projection error associated with capturing the physical marker in the physical environment on the first and second images, wherein the physical environment is 3D and the first and second images are 2D; and

obtaining the first 3D position of the physical marker based at least on the pair of matching markers and the projection error.

18. The tracking system of claim 10, wherein:

the first and the second images are captured at a first time to obtain the first 3D position of the physical marker;

a third and a fourth images are captured at second first time to obtain a second 3D position of the physical marker; and

the method further comprises;

associating inertia measurement unit (IMU) data associated with the first and the second images and IMU data associated with the third and the fourth images to obtain an orientation change of an imaging device, the imaging device captured the first, the second, the third, and the fourth images;

pairing a marker associated with the first and the second image to another marker associated with the third and the fourth image;

obtaining a change in position of the physical marker relative to the imaging device based on the paring;

associating the orientation change of the imaging device and the change in position of the physical marker relative to the imaging device; and

obtaining movement data of the imaging device between the first time and the second time based at least on the orientation change of the imaging device and the associated change in position of the physical marker relative to the imaging device.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a tracking system, cause the processor to perform a method, the method comprising:

obtaining a first and a second images of a physical environment;

detecting (i) a first set of markers represented in the first image and (ii) a second set of markers represented in the second image;

determining a pair of matching markers comprising a first marker from the first set of markers and a second marker from the second set of markers, the pair of matching markers associated with a physical marker disposed within the physical environment; and

obtaining a first three-dimensional (3D) position of the physical marker based at least on the pair of matching markers.

20. The non-transitory computer-readable storage medium of claim 19, wherein:

the first and the second images are captured at a first time to obtain the first 3D position of the physical marker;

a third and a fourth images are captured at second first time to obtain a second 3D position of the physical marker; and

the method further comprises;

associating inertia measurement unit (IMU) data associated with the first and the second images and IMU data associated with the third and the fourth images to obtain an orientation change of an imaging device, the imaging device captured the first, the second, the third, and the fourth images;

pairing a marker associated with the first and the second image to another marker associated with the third and the fourth image;

obtaining a change in position of the physical marker relative to the imaging device based on the paring;

associating the orientation change of the imaging device and the change in position of the physical marker relative to the imaging device; and

obtaining movement data of the imaging device between the first time and the second time based at least on the orientation change of the imaging device and the associated change in position of the physical marker relative to the imaging device.