Two-hand interaction with natural user interface

• US 9,529,513 B2
• Filed: 08/05/2013
• Issued: 12/27/2016
• Est. Priority Date: 08/05/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru, Finocchio, Mark J., Lee, Kyungsuk David
• Primary Examiner(s): HARRIS, DOROTHY H
• Application Number:
  US13/959,555
• Publication Number:
  US 20150040040A1
• Time in Patent Office: 1,240 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Two-handed interactions with a natural user interface are disclosed. For example, one embodiment provides a method comprising detecting via image data received by the computing device a context-setting input performed by a first hand of a user, and sending to a display a user interface positioned based on a virtual interaction coordinate system, the virtual coordinate system being positioned based upon a position of the first hand of the user. The method further includes detecting via image data received by the computing device an action input performed by a second hand of the user, the action input performed while the first hand of the user is performing the context-setting input, and sending to the display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.

Claims

1. On a computing device, a method comprising:

detecting via image data received by the computing device a first posture of a context-setting input performed by a first hand of a user to trigger execution of a drawing program;

in response to detecting the first posture of the context-setting input, sending to a display a holographic stereoscopic user interface for the drawing program positioned based on a virtual interaction coordinate system, the virtual interaction coordinate system comprising a reference plane positioned based upon a position of the first hand of the user;

detecting via image data received by the computing device a change in the first posture to a second posture of the context-setting input performed by the first hand, and in response, triggering a sub-context for the drawing program to detect drawing input;

detecting via image data received by the computing device a drawing input performed by a second hand of the user relative to the holographic stereoscopic user interface and relative to the reference plane within the virtual reference coordinate system, the drawing input performed while the first hand of the user is performing the second posture of the context-setting input; and

displaying a drawing in the holographic stereoscopic user interface based on the drawing input relative to the reference plane within the virtual interaction coordinate system.

2. The method of claim 1, wherein detecting the drawing input further comprises detecting a movement of the second hand through the reference plane of the virtual interaction coordinate system from a position on one side of the reference plane to a position on another side of the reference plane from a viewpoint of the user, the reference plane located at a same apparent depth at which the user interface is displayed.

3. The method of claim 2, wherein detecting the second hand crossing through the reference plane triggers an image capture process via a camera.

4. The method of claim 1, further comprising, detecting a change in posture of the first hand from the second posture, and in response, triggering an end of the drawing input.

5. The method of claim 4, wherein the change in posture of the first hand triggers an end of a marking up of the user interface by the second hand.

6. The method of claim 1, further comprising sending to the display a user interface including one or more elements selectable by a cursor, detecting another context-setting input triggering a cursor control mode, detecting a movement of the second hand of the user across the reference plane within the virtual interaction coordinate system from a position in front of the reference plane to a position behind the reference plane from a viewpoint of the computing device, and sending the cursor to the display.

7. The method of claim 6, further comprising:

detecting a cursor-movement input comprising lateral movement of the second hand behind the reference plane;

in response to detecting the cursor-movement input, move the cursor in correspondence with the movement of the second hand behind the reference plane.

8. The method of claim 7, further comprising:

highlighting an element of the user interface when movement of the cursor intersects the element; and

selecting the element highlighted in response to detecting a cursor-select input comprising movement of the second hand from behind the reference plane to in front of the reference plane while the element is highlighted.

9. A near-eye display system, comprising:

a see-through near-eye display;

one or more image sensors;

a logic machine; and

a storage machine including instructions executable by the logic machine to;

receive image information from the one or more image sensors;

detect a first posture of a context-setting input performed by a first hand of a user based on the image information received to trigger execution of a drawing program;

in response to detecting the first posture of the context-setting input, define a virtual interaction coordinate system comprising a spatial region including a reference plane positioned based on a position of the first hand;

send to the see-through near-eye display an augmented reality user interface comprising one or more user interface elements positioned based upon the position of the first hand;

detect a second posture of the context-setting input performed by the first hand based on the image information received, and in response, trigger a sub-context for the drawing program to detect drawing input;

detect a drawing input performed by a second hand of the user based on the image information received, the drawing input performed at a location relative to the reference plane within the virtual interaction coordinate system and defined by a location of the first hand and while the first hand of the user is performing the second posture of the context-setting input; and

display via the see-through near-eye display a drawing based on the drawing input relative to the reference plane within the virtual interaction coordinate system.

10. The near-eye display system of claim 9, wherein the drawing input further comprises a movement of the second hand across the reference plane of the virtual interaction coordinate system from a position on one side of the reference plane to a position on another side of the reference plane from a viewpoint of the see-through near-eye display.

11. The near-eye display system of claim 10, wherein the instructions are executable to trigger an image capture process via a camera in response to the second hand crossing the reference plane.

12. The near-eye display system of claim 9, wherein the instructions are executable to detect a change in posture of the first hand from the second posture, and in response, trigger an end of the drawing input by the second hand.

13. The near-eye display system of claim 9, wherein the augmented user interface includes one or more elements selectable by a cursor, further comprising instructions executable to detect another context-setting input triggering a cursor control mode, detect an action input comprising a movement of the second hand of the user across the reference plane within the virtual interaction coordinate system, and sending the cursor to the see-through near-eye display.

14. A near-eye display system, comprising:

a see-through near-eye display;

one or more image sensors;

a logic machine; and

a storage machine including instructions executable by the logic machine to;

detect a first posture of a context-setting input triggering a drawing program, the first posture of the context-setting input performed by a first hand of a user;

in response to detecting the first posture of the context-setting input, define a virtual interaction coordinate system comprising a spatial region including a reference plane positioned based on a position of the first hand;

send to the see-through near-eye display a holographic stereoscopic augmented reality user interface positioned based on the position of the first hand;

detect a second posture of the context-setting input performed by the first hand, and in response, trigger a sub-context for the drawing program to detect drawing input;

detect a drawing input comprising a movement of a second hand of the user relative to the reference plane within the virtual interaction coordinate system while the first hand is performing the second posture of the context-setting input;

in response to detecting the drawing input, display via the see-through near-eye display a drawing based on movement of the second hand; and

detect a change in the second posture to the first posture of the context-setting input performed by the first hand, and in response, trigger an end of the drawing input by the second hand.

15. The near-eye display system of claim 14, wherein the instructions are further executable to display a drawing cursor based on a position of the second hand.

16. The near-eye display system of claim 15, wherein the instructions are further executable to display the drawing cursor at a position of a finger of the second hand.

17. The near-eye display system of claim 16, wherein the instructions are further executable to highlight an element of the augmented reality user interface when movement of the drawing cursor intersects the element.

18. The near-eye display system of claim 17, wherein the instructions are further executable to:

subsequent to selecting the highlighted element, detect an element manipulation input comprising movement of the first hand; and

in response to detecting the element manipulation input, adjusting a position of the selected highlighted element of the augmented reality user interface.

19. The near-eye display system of claim 18, wherein the instructions are further executable to deselect the selected element of the augmented reality user interface in response to a deselect input.