Using a combination of 2D and 3D image data to determine hand features information

• US 9,275,277 B2
• Filed: 02/22/2013
• Issued: 03/01/2016
• Est. Priority Date: 02/22/2013
• Status: Active Grant

• Current Assignee: Kaiser Foundation Hospitals (Kaiser Permanente Inc.)
• Original Assignee: Kaiser Foundation Hospitals (Kaiser Permanente Inc.)
• Inventors: Onen, Cengiz, Krishnagiri, Shruti
• Primary Examiner(s): FUJITA, KATRINA R
• Application Number:
  US13/774,326
• Publication Number:
  US 20140241570A1
• Time in Patent Office: 1,103 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method of determining hand features information using both two dimensional (2D) image data and three dimensional (3D) image data is described. In one implementation, a method includes: receiving a 2D image frame; receiving 3D image data corresponding to the 2D image frame; using the 3D image data corresponding to the 2D image frame, transforming the 2D image frame; and using the 3D image data corresponding to the 2D image frame, scaling the 2D image frame, where the transforming and scaling results in a normalized 2D image frame, where the normalized 2D image frame is a scaled and transformed version of the 2D image frame, and where the scaling and transforming is performed using a computer.

Claims

1. A method comprising:

receiving a two dimensional (2D) image frame;

receiving three dimensional (3D) image data corresponding to the 2D image frame;

using the 3D image data corresponding to the 2D image frame, rotating and scaling the 2D image frame,

wherein the rotating and scaling results in a normalized 2D image frame, wherein the normalized 2D image frame is a scaled and rotated version of the 2D image frame,

further wherein the scaling and rotating is performed using a computer;

determining hand features information in two dimensions of a hand in the normalized 2D image frame; and

mapping the hand features information in two dimensions to hand features information in three dimensions.

2. The method of claim 1, wherein the 3D image data corresponding to the 2D image frame is (a) skeletal tracking information corresponding to the 2D image frame or (b) a 3D image frame corresponding to the 2D image frame.

3. The method of claim 1 further comprising:

clipping the 3D image data;

producing a clipped 3D image data, wherein the clipped 3D image data contains a 3D image of the hand in a 3D image frame of a predetermined size;

using the clipped 3D image data, clipping the 2D image frame; and

producing a clipped 2D image frame, wherein the clipped 2D image frame is a 2D image frame of a predetermined size and contains a 2D image of the hand.

4. The method of claim 1, wherein the 2D image frame includes a 2D image of the hand, further wherein the 3D image frame includes a 3D image of the hand, the method further comprising:

determining an aspect ratio of the hand in the 3D image data; and

using the aspect ratio, determining a palm plane for the hand.

5. The method of claim 4, wherein rotating the 2D image frame comprises rotating the palm plane such that it is (a) parallel to a lens of a camera with which the 2D image frame was captured and (b) perpendicular to a floor plane, wherein the floor plane is a plane defined by a floor of a location where the 2D image frame was captured, further wherein the scaling the 2D image frame comprises scaling the 2D image frame such that the 2D image of the hand fits within a frame of a predetermined size.

6. The method of claim 1, wherein, if skeletal tracking information corresponding to the 2D image frame is available, then:

using the skeletal tracking information corresponding to the 2D image frame, rotating and scaling the 2D image frame,

wherein, if skeletal tracking information corresponding to the 2D image frame is not available, but a 3D image frame corresponding to the 2D image frame is available, then;

using the 3D image frame corresponding to the 2D image frame, rotating and scaling the 2D image frame.

7. The method of claim 6, wherein if, within a predetermined time of detecting the 2D image frame, skeletal tracking information is detected, then it is determined that skeletal tracking information corresponding to the 2D image frame is available.

8. The method of claim 1, wherein the 2D image frame is a red green blue (RGB) 2D image frame and is received from a 2D RGB camera, wherein the normalized 2D image frame is a normalized RGB 2D image frame, the method further comprising:

converting the normalized RGB 2D image frame to a binary segmented image frame;

using the binary segmented image frame in a neural gas network to provide hand topology data corresponding to the binary segmented image frame and the normalized RGB 2D image frame;

using the hand topology data, determining hand features information in two dimensions of the hand in the normalized RGB 2D image frame; and

providing hand features information in three dimensions of the hand.

9. The method of claim 1, wherein the 3D image data comprises a 3D image frame, the method further comprising:

ordering data in the 3D image frame in order of increasing depth to produce an ordered set of data;

using the ordered set of data, creating a set of data clusters, wherein each data cluster of the set has a predetermined degree of continuity; and

for each data cluster in the set of data clusters, determining whether such data cluster represents a hand.

10. The method of claim 1, wherein the 2D image frame is a red green blue (RGB) 2D image frame and is captured by a RGB camera, further wherein the 3D image data is captured by an infrared (IR) camera.

11. A method comprising:

capturing a two dimensional (2D) red green blue (RGB) image frame, wherein the 2D RGB image frame includes an image of a hand and is captured using an RGB camera;

capturing three dimensional (3D) image data corresponding to the 2D RGB image frame, wherein the 3D image data corresponding to the 2D image frame is (a) skeletal tracking information corresponding to the 2D RGB image frame or (b) a 3D image frame corresponding to the 2D RGB image frame, wherein the 3D image data is captured using an infrared (IR) camera;

using the 3D image data corresponding to the 2D RGB image frame, producing a normalized RGB 2D image frame, wherein the normalized 2D RGB image frame is a scaled and rotated version of the 2D RGB image frame,

wherein the producing is performed using a computer;

determining hand features information in two dimensions of the hand in the normalized 2D image frame; and

mapping the hand features information in two dimensions to hand features information in three dimensions.

12. The method of claim 11 further comprising:

clipping the 3D image data;

producing a clipped 3D image data, wherein the clipped 3D image data contains a 3D image of the hand in a 3D image frame of a predetermined size;

using the clipped 3D image data, producing a clipped 2D RGB image frame, wherein the clipped 2D RGB image frame is a 2D RGB image frame of a predetermined size and contains a 2D RGB image of the hand.

13. The method of claim 11 further comprising:

determining an aspect ratio of the hand in the 3D image data; and

using the aspect ratio, determining a palm plane for the hand,

wherein producing the normalized 2D RGB image frame comprises;

rotating the palm plane such that it is (a) parallel to a lens of the RGB camera and (b) perpendicular to a floor plane, wherein the floor plane is a plane defined by a floor of a location where the 2D RGB image frame was captured; and

scaling the 2D RGB image frame such that the image of the hand fits within a frame of a predetermined size.

14. The method of claim 11, wherein

if skeletal tracking information corresponding to the 2D RGB image frame is available, using the skeletal tracking information corresponding to the 2D RGB image frame, to produce the normalized 2D RGB image frame;

if skeletal tracking information corresponding to the 2D image frame is not available, but the 3D image frame corresponding to the 2D RGB image frame is available, using the 3D image frame corresponding to the 2D RGB image frame, to produce the normalized 2D RGB image frame.

15. The method of claim 14, wherein if, within a predetermined time of detecting the 2D RGB image frame, skeletal tracking information is detected, then it is determined that the skeletal tracking information corresponding to the 2D RGB image frame is available.

16. A non-transitory computer readable medium including computer executable code, which when executed on a computer, cause the computer to perform a method comprising:

receiving a two dimensional (2D) image frame;

receiving three dimensional (3D) image data corresponding to the 2D image frame;

using the 3D image data corresponding to the 2D image frame, transforming and scaling the 2D image frame, wherein the transforming and scaling results in a normalized 2D image frame, wherein the normalized 2D image frame is a scaled and transformed version of the 2D image frame;

determining hand features information in two dimensions of a hand in the normalized 2D image frame; and

mapping the hand features information in two dimensions to hand features information in three dimensions.

17. The non-transitory computer readable medium of claim 16, wherein the 3D image data corresponding to the 2D image frame is (a) skeletal tracking information corresponding to the 2D image frame or (b) a 3D image frame corresponding to the 2D image frame.

18. The non-transitory computer readable medium of claim 17, wherein the method further comprises:

clipping the 3D image data;

producing a clipped 3D image data, wherein the clipped 3D image data contains a 3D image of the hand in a 3D image frame of a predetermined size;

using the clipped 3D image data, clipping the 2D image frame; and

producing a clipped 2D image frame, wherein the clipped 2D image frame is a 2D image frame of a predetermined size and contains a 2D image of the hand.

19. The non-transitory computer readable medium of claim 16, wherein the 2D image frame includes a 2D image of the hand, further wherein the 3D image frame includes a 3D image of the hand, further wherein the method further comprises:

calculating an aspect ratio of the hand in the 3D image data; and

using the aspect ratio, determining a palm plane for the hand,

wherein transforming the 2D image frame comprises rotating the palm plane such that it is (a) parallel to a lens of a camera with which the 2D image frame was captured and (b) perpendicular to a floor plane, wherein the floor plane is a plane defined by a floor of a location where the 2D image frame was captured, further wherein the scaling the 2D image frame comprises scaling the 2D image frame such that the 2D image of the hand fits within a frame of a predetermined size.

20. The non-transitory computer readable medium of claim 16, wherein, if skeletal tracking information corresponding to the 2D image frame is available, then the method comprises:

using the skeletal tracking information corresponding to the 2D image frame, transforming and scaling the 2D image frame,

wherein, if skeletal tracking information corresponding to the 2D image frame is not available, but a 3D image frame corresponding to the 2D image frame is available, then the method comprises;

using the 3D image frame corresponding to the 2D image frame, transforming and scaling the 2D image frame.

21. The non-transitory computer readable medium of claim 20, wherein, if within a predetermined time of detecting the 2D image frame, skeletal tracking information is detected, then it is determined that the skeletal tracking information corresponding to the 2D image frame is available.

22. The non-transitory computer readable medium of claim 16, wherein the 2D image frame is a red green blue (RGB) 2D image frame and is received from a 2D RGB camera, wherein the normalized 2D image frame is a normalized RGB 2D image frame, the method further comprising:

converting the normalized RGB 2D image frame to a binary segmented image frame;

using the binary segmented image frame in a neural gas network to provide hand topology data corresponding to the binary segmented image frame and the normalized RGB 2D image frame;

wherein determining the hand features information in two dimensions of the hand in the normalized 2D image frame comprises, using the hand topology data, determining hand features information in two dimensions of the hand in the normalized RGB 2D image frame; and

providing hand features information in three dimensions of the hand.