Accurate 3D finger tracking with a single camera

• US 9,323,346 B2
• Filed: 12/31/2012
• Issued: 04/26/2016
• Est. Priority Date: 12/31/2012
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Lv, Fengjun, Zhu, Fengqing, Zhou, Liang, Wang, Pengwei
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Nasri, Maryam
• Application Number:
  US13/732,223
• Publication Number:
  US 20140184745A1
• Time in Patent Office: 1,212 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

An object tracking device includes a camera with a field of view oriented in a first direction and a mirror with a field of reflection oriented in a second direction. When an object is in a first region in the field of view of the electronic camera, the camera has a direct view of the object and a reflected view of the object from the mirror. A processor coupled with the camera is configured to receive a first image data set and a second image data set from the camera. The first image data set and the second image data set each include the direct view of the object and the reflected view of the object from the mirror.

Claims

1. An object tracking device, comprising:

a camera with a field of view oriented in a first direction;

a mirror with a field of reflection oriented in a second direction, such that when an object is in a first region in the field of view of the camera, the camera has a direct view of the object and a reflected view of the object from the mirror; and

a processor configured to;

receive a first image data set and a second image data set from the camera, wherein the first image data set includes the direct view of the object and the reflected view of the object from the mirror, and wherein the second image data set includes the direct view of the object and the reflected view of the object from the mirror;

create a three dimensional representation of the object using at least the first image data set;

track motion in three dimensions by tracking allocation on a fingertip of a human hand using at least the second image data set; and

attempt to locate the fingertip using the first image data set by calculating at least one of an intersection and a closest approach of a first ray from the camera and a second ray from the mirror, and if an intersection of the first ray and the second ray exist, use the intersection as the location of the fingertip, and if no intersection between the first ray and the second ray exists, estimate the location of the fingertip to be at the center of the shortest straight line connecting the first ray to the second ray.

2. The object tracking device of claim 1, further comprising an infrared light source which projects infrared light into the first region.

3. The object tracking device of claim 2, wherein the camera produces the first image data set and second image data set using the infrared light.

4. The object tracking device of claim 1, wherein the processor is further configured to track relative motion of a plurality of the fingertips on the human hand.

5. The object tracking device of claim 4, wherein the processor is further configured to interpret a motion of the human hand in three dimensions as an input to a graphical user interface of a computing device.

6. The object tracking device of claim 5, wherein the processor is further configured to interpret discrete gestures of the human hand in three dimensions.

7. An object tracking apparatus, comprising:

a processor configured to;

receive at least a first image data set and a second image data set from a single camera, wherein the first image data set includes a direct view of an object and a reflected view of the object from a mirror, and wherein the second image data set includes the direct view of the object and the reflected view of the object from the mirror;

create a three dimensional representation of the object using the first image data set;

track motion in three dimensions by tracking a location on a fingertip of a human hand using the second image data set;

attempt to locate the fingertip using the first image data set by calculating at least one of an intersection and a closest approach of a first ray from the single camera and a second ray from the mirror, and if an intersection of the first ray and the second ray exist, use the intersection as the location of the fingertip, and if no intersection between the first ray and the second ray exists, estimate the location of the fingertip to be at the center of the shortest straight line connecting the first ray to the second ray; and

control a graphical user interface by inputting data derived from the tracking motion in three dimensions.

8. The apparatus of claim 7, wherein the processor is configured to control a graphical user interface by at least controlling an operating system and a plurality of applications usable with the operating system.

9. The apparatus of claim 8, wherein the processor is configured to control a graphical user interface by providing inputs to perform at least one of opening an application, closing an application, suspending an application, and restarting an application.

10. The apparatus of claim 7, wherein the processor is configured to receive the first image data set and the second data set produced by the single camera configured to produce data from infrared light.

11. The apparatus of claim 7, wherein the processor is configured to track motion in three dimensions including relative motion of a plurality of fingertips on the human hand.

12. The apparatus of claim 11, wherein the processor is configured to interpret the relative motion of the plurality of fingertips on the human hand as an input to a graphical user interface of a computing device.

13. The apparatus of claim 12, wherein the processor is further configured to interpret discrete gestures of the human hand in three dimensions.

14. The apparatus of claim 7, further comprising setting up the single camera and mirror to produce image data of the object by:

orienting a field of view of the single camera in a first direction; and

orienting a field of reflection of the mirror in a second direction, such that when an object is in a first region in the field of view of the single camera, the single camera has two views of the object from two different angles.

15. An object tracking method, comprising:

orienting a field of view of a single camera in a first direction;

orienting a field of reflection of a mirror in a second direction, such that when a human hand is in a first region in the field of view of the single camera, the single camera has two views of the hand from two different angles; and

receiving in a non-transitory memory at least a first image data set and a second image data set from the single camera, wherein the first image data set includes a direct view of the hand and a reflected view of the hand from the mirror, and wherein the second image data set includes the direct view of the hand and the reflected view of the hand from the mirror;

creating a three dimensional representation of the hand using the first image data set by identifying in the first image data set a first location on the hand and a second location on the hand, wherein creating the three dimensional representation of the hand using the first image data set comprises;

calculating a first ray from the single camera to the first location;

calculating a second ray from the single camera to the second location;

calculating a third ray from the mirror to the first location;

calculating a fourth ray from the mirror to the second location;

estimating a first intersection of the first and third ray; and

estimating a second intersection of the second ray and the fourth ray;

tracking motion in three dimensions of at least one location on the object using the second image data set by identifying in the first image data set the first location on the object and the second location on the object; and

controlling a graphical user interface by inputting data derived from tracking motion in the three dimensions.

16. The method according to claim 15, wherein creating the three dimensional representation of the hand using the first image data set includes using a known distance between the single camera and the mirror, and a known angle between the field of view of the single camera and the field of reflection of the mirror, and wherein tracking motion in three dimensions of the at least one location on the object includes using the known distance between the single camera and the mirror, and the known angle between the field of view of the single camera and the field of reflection of the mirror.

17. The method according to claim 16, wherein creating the three dimensional representation of the hand using the first image data set further includes calculating a fifth ray from the single camera to the first location, calculating a sixth ray from the single camera to the second location, calculating a seventh ray from the mirror to the first location, and calculating an eighth ray from the mirror to the second location, estimating a third intersection of the fifth and seventh ray and estimating a fourth intersection of the sixth ray and the eighth ray, calculating a path between the first intersection and the third intersection, calculating a path between the second intersection and the fourth intersection.

18. The method according to claim 15, wherein the first location is a first fingertip and the second location is a second fingertip, and identifying in the first image data set the first fingertip on the hand and the second fingertip on the hand further comprises:

illuminating the hand with an infrared light source;

locating at least two pixel regions with large contrast values;

identifying a contiguous set of boundary points within each of the two pixel regions;

locating boundary points with large curvature values within each of the two pixel regions; and

setting the first fingertip location and the second fingertip location along the boundary points with large curvature values.