Digital camera with orientation sensor for optical tracking of objects

• US 9,595,109 B1
• Filed: 01/30/2015
• Issued: 03/14/2017
• Est. Priority Date: 01/30/2014
• Status: Active Grant

• Current Assignee: Inertial Labs, Inc.
• Original Assignee: Inertial Labs, Inc.
• Inventors: Umakhanov, Ievgenii, Struts, Anton, Marraccini, James
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US14/610,627
• Time in Patent Office: 774 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

An object tracking system has an inertial orientation sensor attached to a camera. The sensor uses a rigid body position and orientation (with or without markers) visible to the camera for determining the orientation and the position of the object in the global reference frame, when the camera is not rigidly fixed. Another orientation sensor is attached to the object in order to keep tracking of its orientation when a valid tracking of the object cannot be obtained from the camera. The data from the orientation sensor attached to the object and the data from the orientation sensor attached to the camera is used to increase the accuracy of the optical tracking of the object.

Claims

1. A method for optical object tracking, comprising:

placing an object to be tracked into a field of view of a camera, wherein the camera is not rigidly fixed;

attaching a first orientation sensor to the camera;

attaching a second orientation sensor to the object;

connecting the first and second orientation sensors to a processor for their data to be processed by a sensor fusion algorithm;

acquiring camera orientation data from the first orientation sensor and object orientation data from the second orientation sensor using a fusion filter;

producing a fused object orientation data for tracking the object using the fusion filter based on the camera orientation data and the object orientation data; and

generating an object heading based on the camera orientation data, and object pitch and roll angle from the fusion filter, and marker X and Y coordinates that are identified based on the image sensor of the camera.

2. The method of claim 1, further comprising calculating a global orientation of the object based on a global orientation of the camera.

3. The method of claim 1, further comprising estimating an orientation of the object when the object is out of sight of the camera.

4. The method of claim 1, further comprising placing markers on the object and estimating an orientation of the object using the markers.

5. The method of claim 1, further comprising adjusting a field of view of the camera to compensate for distortion.

6. The method of claim 1, further comprising applying an affine transformation to an image of the object to determine a position and fused object orientation data of the object.

7. The method of claim 1, further comprising selecting one solution out of multiple solutions for the fused object orientation data.

8. The method of claim 1, further comprising using the second orientation sensor to track the object in a global reference frame when the object is not visible to the camera.

9. The method of claim 1, further comprising using the pitch and roll data and the object heading as input for the fusion filter.

10. The method of claim 1, further comprising placing optical markers on the object.

11. The method of claim 1, wherein position and orientation of the image sensor of the camera relative to an inertial sensor of the object orientation sensor remains constant.

12. The method of claim 1, wherein position and orientation of a set of object markers relative to an inertial sensor of the object orientation sensor remains constant.

13. The method of claim 1, wherein an object translation vector is received from the reverse projective transformation processor for calculating object location data.

14. The method of claim 1, wherein the orientation of the object is used together with the orientation obtained from the camera to produce a fused camera orientation.

15. The method of claim 1, wherein fluctuations in magnetic field data obtained from magnetic field sensors included in the second orientation sensor and the fluctuations are used to select whether optical data will be used to correct the object heading or a camera heading.

16. A system for optical object tracking, comprising:

a camera that is not rigidly fixed and is aimed at an object located in a field of view of the camera;

a first orientation sensor attached to the camera;

a second orientation sensor attached to the object;

a fusion filter configured to receive camera orientation data from the first orientation sensor and object orientation data from the second orientation sensor,

wherein the fusion filter produces fused object orientation data for tracking the object based on the camera orientation data and the object orientation data, and wherein a reverse projective transformation processor generates an object heading based on the camera orientation data, and object pitch and roll angle from the fusion filter, and marker X and Y coordinates that are identified based on the image sensor of the camera.

17. The system of claim 16, wherein the first orientation sensor includes three accelerometers for generating acceleration data in three axes, three angular rotation sensor for generation angular rotation data in three axes, and a magnetic field sensor.

18. The system of claim 17, wherein the second orientation sensor includes three accelerometers for generating acceleration data in three axes, three angular rotation sensor for generation angular rotation data in three axes, and a magnetic field sensor.

19. The system of claim 18, further comprising a GPS receiver affixed to or integrated into the camera, the GPS receiver providing location data to the fusion sensor.