High resolution tracking and response to hand gestures through three dimensions

• US 10,261,595 B1
• Filed: 05/19/2017
• Issued: 04/16/2019
• Est. Priority Date: 05/19/2017
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Kin, Kenrick Cheng-kuo
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US15/600,689
• Time in Patent Office: 697 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system includes an electronic display configured to display one or more simulated objects in accordance with display instructions, an imaging sensor configured to capture images of a user'"'"'s hands, and a console. The console is configured to receive the captured images from the imaging sensor, extract joint information of the user'"'"'s hands from the captured images, and determine one or more poses based on the extracted joint information. In response to the determined poses indicating the user'"'"'s index finger positioned orthogonally to the user'"'"'s thumb, and the thumb within a minimum distance to the index finger, the console detects a directional pad display gesture, and update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user'"'"'s thumb in a simulated environment that is presented to the user via the electronic display.

Claims

1. A system comprising:

an electronic display configured to display one or more simulated objects in accordance with display instructions;

an imaging sensor configured to capture images of a user'"'"'s hands; and

a console configured to;

receive the captured images from the imaging sensor;

extract joint information of the user'"'"'s hands from the captured images;

determine one or more poses based on the extracted joint information; and

in response to the determination that the poses indicate the user'"'"'s index finger positioned orthogonally to the user'"'"'s thumb, and the thumb is in contact with the index finger, detect a directional pad display gesture; and

in response to the detection of the direction pad display gesture, update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user'"'"'s thumb in a simulated environment that is presented to the user via the electronic display, the simulated directional pad indicating each of one or more selection directions each with a simulated arrow, the selection of the one or more selection directions detected based on a detection of further determined poses indicating a movement of a distal end of the user'"'"'s thumb.

2. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s thumb is moving in a selected direction, the selected direction being one of the selection directions indicated on the simulated direction pad, detect a directional pad selection gesture;

in response to the detection of the directional pad selection gesture, execute instructions associated with the selected direction; and

in response to the detection of the directional pad selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.

3. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger is extended, the user'"'"'s middle finger is positioned orthogonally to the user'"'"'s thumb, and the thumb is within a minimum distance to the index finger, detect a multifunction display gesture;

in response to the detection of the multifunction display gesture, generate a simulated display of multiple options for selection, the multiple options arranged radially and adjacent to the user'"'"'s thumb in the simulated environment; and

in response to the detection of the multifunction display gesture, update the display instructions to cause the electronic display to generate the simulated display of multiple options via the electronic display.

4. The system of claim 3, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger moving in a selected option, the selected option being one of the options indicated on the simulated display of multiple options, detect a multifunction selection gesture;

in response to the detection of the multifunction selection gesture, execute instructions associated with the selected option; and

in response to the detection of the multifunction selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.

5. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s thumb on the user'"'"'s dominant hand is substantially orthogonal to the lateral side of the user'"'"'s index finger, and the distal portion of the thumb being within a minimum distance to the index finger, detect a pen-holding gesture;

in response to the detection of the pen-holding gesture, generate a simulated pen between the user'"'"'s thumb and the users index finger, the simulated pen capable of simulated writing on a simulated pad; and

in response to the detection of the pen-holding gesture, update the display instructions to cause the electronic display to generate the simulated pen.

6. The system of claim 5, wherein the console is further configured to:

in response to the determination that the poses indicate the distal portion of the thumb being exceeding the minimum distance to the index finger, detect a contextual menu gesture;

in response to the detection of the contextual menu gesture, generate a simulated contextual menu adjacent to the simulated pen; and

in response to the detection of the contextual menu gesture, update the display instructions to cause the electronic display to generate the simulated contextual menu.

7. The system of claim 1, wherein the console is further configured to:

in response to a detection that the user is holding a physical pen device, detect a pen-holding gesture;

in response to the detection that the user is holding a physical pen device, generate a simulated pen between the user'"'"'s thumb and the users index finger, the simulated pen capable of writing on a simulated pad and mirroring the position of the physical pen device relative to the user'"'"'s hand; and

in response to the detection that the user is holding a physical pen device, update the display instructions to cause the electronic display to generate the simulated pen.

8. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate that one of the user'"'"'s fingers is elongated and that the distal portion of the one finger is intersecting a simulated pressable element in the simulated environment that is presented to the user via the electronic display, detect a button press gesture;

in response to the detection of the button press gesture, update the display instructions to generate a simulated button press in the pressable element to the user via the electronic display;

in response to the detection of the button press gesture, execute instructions associated with the button press gesture for the pressable element; and

in response to the detection of the button press gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user associated with the pressable element.

9. The system of claim 8, wherein the console is further configured to:

in response to the determination that the poses indicate that the one finger is moving in a direction orthogonal to a previous movement direction of the one finger, detect a sliding gesture;

in response to the detection of the sliding gesture, execute instructions associated with the pressable element for the sliding gesture to cause a translational movement in simulated content; and

in response to the detection of the sliding gesture, update the display instructions to cause the electronic display to generate a translational movement in the simulated content.

10. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the distal portion of the user'"'"'s thumb is within a minimum distance to the distal portion of the user'"'"'s index finger, wherein the user'"'"'s hand is moving along one axis, and wherein the user'"'"'s thumb and the user'"'"'s index finger surround a scrolling element detect a pinch scroll gesture;

in response to the detection of the pinch scroll gesture, execute instructions associated with the scrolling element for a scrolling action to cause a translational movement in simulated content; and

in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the translational movement in the simulated content.

11. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate that both the user'"'"'s hands are performing a pinch gesture, each hand pinching one transform element of a simulated element in the simulated environment, the distance between the two hands changing over time, detect a transform gesture;

in response to the detection of the transform gesture, execute a transform on the simulated element based on the positions of each hand, the transform of the simulated element changing a dimension of the simulated element; and

in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the transform of the simulated element.

12. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate that one of the user'"'"'s hand is performing a pinch gesture on a simulated element in the simulated environment, the user'"'"'s hand moving over time, detect a moving gesture;

in response to the detection of the moving gesture, execute a move on the simulated element based on the position of the hand; and

in response to the detection of the moving gesture, update the display instructions to cause the electronic display to generate the move of the simulated element.

13. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate that one of the user'"'"'s hand is performing a pinch gesture using two fingers and a thumb, detect a selection button gesture, and

in response to the detection of the selection button gesture, execute an action associated with the selection button gesture; and

in response to the detection of the selection button gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.

14. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger and middle finger are extended, the user'"'"'s ring finger positioned orthogonally to the user'"'"'s thumb, and the thumb within a minimum distance to the ring finger, detect a lock gesture; and

in response to the detection of the lock gesture, execute an input lock action to disable changes in the presentation of simulated elements to the user via the electronic display for those simulated elements associated with the input lock action.

15. The system of claim 1, wherein the console is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger and middle finger are extended, the user'"'"'s ring finger positioned orthogonally to the user'"'"'s thumb, and the thumb within a minimum distance to the distal portion of the ring finger, detect an unlock gesture; and

in response to the detection of the unlock gesture, execute an input unlock action to enable changes in the presentation of simulated elements to the user via the electronic display for those simulated elements associated with the input lock action.

16. A system comprising:

an imaging sensor configured to capture images of a user'"'"'s hands;

a head-mounted display (HMD) comprising;

an electronic display configured to display one or more simulated objects in accordance with display instructions;

wherein the HMD is further configured to;

receive the captured images from the imaging sensor;

extract joint information of the user'"'"'s hands from the captured images;

determine one or more poses based on the extracted joint information; and

in response to the determination that the poses indicate the user'"'"'s index finger positioned orthogonally to the user'"'"'s thumb, and the thumb is in contact with the index finger, detect a directional pad display gesture; and

in response to the detection of the direction pad display gesture, update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user'"'"'s thumb in a simulated environment that is presented to the user via the electronic display, the simulated directional pad indicating each of one or more selection directions each with a simulated arrow, the selection of the one or more selection directions detected based on a detection of further determined poses indicating a movement of a distal end of the user'"'"'s thumb.

17. The system of claim 16, wherein the HMD is further configured to:

in response to the determination that the poses indicate a distal end of the user'"'"'s thumb is moved in a selected direction, the selected direction being one of the selection directions indicated on the simulated direction pad, detect a directional pad selection gesture;

in response to the detection of the directional pad selection gesture, execute instructions associated with the selected direction; and

in response to the detection of the directional pad selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.

18. The system of claim 16, wherein the HMD is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger is extended, the user'"'"'s middle finger positioned orthogonally to the user'"'"'s thumb, and the thumb within a minimum distance to the index finger, detect a multifunction display gesture;

in response to the detection of the multifunction display gesture, generate a simulated display of multiple options for selection, the multiple options arranged radially and adjacent to the user'"'"'s thumb in the simulated environment; and

in response to the detection of the multifunction display gesture, update the display instructions to cause the electronic display to generate the simulated display of multiple options via the electronic display.

19. The system of claim 18, wherein the HMD is further configured to:

in response to the determination that the poses indicate the user'"'"'s index finger moving in a selected option, the selected option being one of the options indicated on the display of multiple options, detect a multifunction selection gesture;

in response to the detection of the multifunction selection gesture, execute instructions associated with the selected option; and

in response to the detection of the multifunction selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.

20. The system of claim 16, wherein the HMD is further configured to:

in response to the determination that the poses indicate the user'"'"'s thumb on the user'"'"'s dominant hand is substantially orthogonal to the lateral side of the user'"'"'s index finger, the distal portion of the thumb being within a minimum distance to the index finger, detect a pen-holding gesture;

in response to the detection of the pen-holding gesture, generate a simulated pen between the user'"'"'s thumb and the users index finger, the simulated pen capable of writing on a simulated pad, and

in response to the detection of the pen-holding gesture, update the display instructions to cause the electronic display to generate the simulated pen.