SYSTEM AND METHOD FOR POINT, SELECT AND TRANSFER HAND GESTURE BASED USER INTERFACE

• US 20110219340A1
• Filed: 04/29/2010
• Published: 09/08/2011
• Est. Priority Date: 03/03/2010
• Status: Abandoned Application

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: PATHANGAY, Vinod, Subramanian, Anbumani
• Application Number:
  US12/769,654
• Publication Number:
  US 20110219340A1
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method for a point, select and transfer hand gesture based user interface is disclosed. In one embodiment, a depth image of a hand gesture is captured using an in-front camera substantially on a frame by frame basis within a predefined interaction volume. Also, a nearest point of the hand gesture to a display screen of a display device is found using a substantially nearest depth value in the captured depth image for each frame. Further, an image-to-screen mapping of the captured depth image and the found nearest point to the display screen is performed upon validating the found nearest point as associated with the hand for each frame. Moreover, one of select options displayed on the display screen is pointed and selected when the substantially nearest depth value is within one or more predetermined threshold ranges, and based on the outcome of the image-to-screen mapping.

Claims

1. A computer implemented method for a point, select and transfer hand gesture based user interface, comprising:

capturing a depth image of a hand gesture using an in-front camera substantially on a frame by frame basis within a predefined interaction volume and wherein the in-front camera is substantially disposed around a display device and wherein the display device is designed to display a plurality of select options;

finding a nearest point of the hand gesture to a display screen of the display device using a substantially nearest depth value in the captured depth image for each frame;

performing an image-to-screen mapping of the captured depth image and the found nearest point to the display screen upon validating the found nearest point as associated with the hand for each frame; and

pointing and selecting one of the plurality of displayed select options on the display screen of the display device when the substantially nearest depth value is within one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.

2. The method of claim 1, further comprising:

validating whether the found nearest point is associated with a hand in the captured depth image using a heuristic approach for each frame; and

pointing and selecting one of the plurality of displayed select options on the display screen of the computer when the substantially nearest depth value is within one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping and validating the found nearest point as associated with the hand for each frame.

3. The method of claim 1, wherein the predefined interaction volume substantially extends in front of the display screen by a distance that is approximately in the range of about 0.5 meter to 1 meter.

4. The method of claim 2, wherein finding the nearest point to the display screen using the substantially nearest depth value in the captured depth image for each frame comprises:

finding a pixel associated with the substantially nearest depth value in the captured depth image for each frame.

5. The method of claim 3, wherein validating whether the found nearest point is associated with the hand in the captured depth image using the heuristic approach for each frame comprises:

computing a depth variance using depth values associated with pixels substantially surrounding the pixel associated with the substantially nearest depth value for each frame; and

if the computed depth variance is within a predefined range of variance threshold, then validating the found nearest point as associated with the hand in the captured depth image for each frame.

6. The method of claim 1, wherein performing the image-to-screen mapping of the captured depth image and the found nearest point to the display screen upon validating the found nearest point as associated with the hand for each frame further comprises:

smoothening an estimated pointing location on the display screen by temporal averaging of the estimated pointing location upon performing the image-to-screen mapping of the captured depth image and the found nearest point to the display screen.

7. The method of claim 5, wherein pointing and selecting the one of the plurality of displayed select options on the display screen of the display device when the substantially nearest depth value is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping, comprises:

determining whether the found nearest point associated with the substantially nearest depth value is within a first predetermined threshold range or within a second predetermined threshold range, wherein the found nearest point is a tip of a finger or a part of a hand;

if the found nearest point is within the first predetermined threshold range then declaring the found nearest point as a pointing hand gesture and highlighting the one of the plurality of select options displayed on the display screen based on the outcome of the image-to-screen mapping; and

if the found nearest point is within the second predetermined threshold range then declaring the found nearest point as a selecting hand gesture and selecting the highlighted one of the plurality of select options displayed on the display screen.

8. The method of claim 7, wherein pointing and selecting the one of the plurality of displayed select options on the display screen of the display device when the substantially nearest depth value is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping, further comprising:

if the found nearest point is outside the first predetermined threshold range and the second predetermined threshold range, then doing nothing on the display screen.

9. The method of claim 1, wherein the in-front camera comprises a depth camera disposed in a location selected from the group consisting of above the display device, below the display device, and on side of the display device.

10. The method of claim 1, wherein, in finding a nearest point of the hand gesture, the substantially nearest depth value comprises a highest depth value or a lowest depth value.

11. The method of claim 1, wherein pointing and selecting the one of the plurality of displayed select options on the display screen of the display device, comprises:

pointing and selecting digital content displayed on a display screen of a source display device when the substantially nearest depth value associated with a grabbing hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping;

grabbing the digital content upon pointing and selecting the digital content displayed on the display screen; and

transferring the digital content to a destination display device when the substantially nearest depth value associated with a release hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.

12. A non-transitory computer-readable storage medium for a point, select and transfer hand gesture based user interface having instructions that, when executed by a computing device, cause the computing device to perform a method comprising:

capturing a depth image of a hand gesture using an in-front camera substantially on a frame by frame basis within a predefined interaction volume and wherein the in-front camera is substantially disposed around a display device and wherein the display device is designed to display a plurality of select options;

finding a nearest point of the hand gesture to a display screen of the display device using a substantially nearest depth value in the captured depth image for each frame;

performing an image-to-screen mapping of the captured depth image and the found nearest point to the display screen upon validating the found nearest point as associated with the hand for each frame; and

pointing and selecting one of the plurality of displayed select options on the display screen of the display device when the substantially nearest depth value is within one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.

13. The non-transitory computer-readable storage medium of claim 12, wherein pointing and selecting the one of the plurality of displayed select options on the display screen of the display device, comprises:

pointing and selecting digital content displayed on a display screen of a source display device when the substantially nearest depth value associated with a grabbing hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping;

grabbing the digital content upon pointing and selecting the digital content displayed on the display screen; and

transferring the digital content to a destination display device when the substantially nearest depth value associated with a release hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.

14. A system for a point, select and transfer hand gesture based user interface, comprising:

a processor;

a display device having a display screen and coupled to the processor, wherein the display device is configured to display a plurality of select options;

an in-front camera disposed substantially around the display device; and

memory operatively coupled to the processor, wherein the memory includes a point, select and transfer hand gesture based user interface module having instructions capable of;

capturing a depth image of a hand gesture using the in-front camera substantially on a frame by frame basis within a predefined interaction volume and wherein the in-front camera is substantially disposed around a display device and wherein the display device is designed to display a plurality of select options;

finding a nearest point of the hand gesture to a display screen of the display device using a substantially nearest depth value in the captured depth image for each frame;

performing an image-to-screen mapping of the captured depth image and the found nearest point to the display screen upon validating the found nearest point as associated with the hand for each frame; and

pointing and selecting one of the plurality of displayed select options on the display screen of the display device when the substantially nearest depth value is within one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.

15. The system of claim 14, wherein the point, select and transfer hand gesture based user interface module have further instructions capable of:

pointing and selecting digital content displayed on a display screen of a source display device when the substantially nearest depth value associated with a grabbing hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping;

grabbing the digital content upon pointing and selecting the digital content displayed on the display screen; and

transferring the digital content to a destination display device when the substantially nearest depth value associated with a release hand gesture is within the one or more predetermined threshold values or ranges, and based on the outcome of the image-to-screen mapping.