TWO-HAND INTERACTION WITH NATURAL USER INTERFACE

• US 20150040040A1
• Filed: 08/05/2013
• Published: 02/05/2015
• Est. Priority Date: 08/05/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru, Finocchio, Mark J., Lee, Kyungsuk David
• Granted Patent: US 9,529,513 B2
• Time in Patent Office: Days
• US Class Current: 715/762
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Two-handed interactions with a natural user interface are disclosed. For example, one embodiment provides a method comprising detecting via image data received by the computing device a context-setting input performed by a first hand of a user. and sending to a display a user interface positioned based on a virtual interaction coordinate system, the virtual coordinate system being positioned based upon a position of the first hand of the user. The method further includes detecting via image data received by the computing device an action input performed by a second hand of the user, the action input performed while the first hand of the user is performing the context-setting input, and sending to the display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.

Claims

1. On a computing device, a method comprising:

detecting via image data received by the computing device a context-setting input performed by a first hand of a user;

sending to a display a user interface positioned based on a virtual interaction coordinate system, the virtual coordinate system being positioned based upon a position of the first hand of the user;

detecting via image data received by the computing device an action input performed by a second hand of the user, the action input performed while the first hand of the user is performing the context-setting input; and

sending to the display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.

2. The method of claim 1, wherein detecting an action input comprises detecting a movement of the second hand across a plane of the virtual interaction coordinate system from a position on one side of the plane to a position on another side of the plane from a viewpoint of the user.

3. The method of claim 2, wherein detecting the second hand crossing the plane triggers an image capture process via a camera.

4. The method of claim 1, further comprising, detecting a change in posture of the first hand, and in response, triggering an end of the action input.

5. The method of claim 4, wherein the change in posture of the first hand triggers an end of a marking up of the user interface by the second hand.

6. The method of claim 1, wherein sending to the display a user interface comprises sending to the display a user interface including one or more elements selectable by a cursor, wherein the context-setting input triggers a cursor control mode, wherein detecting the action input comprises detecting a movement of the second hand of the user across a plane within the virtual interaction coordinate system from a position in front of the plane to a position behind the plane from a viewpoint of the computing device, and wherein sending the response comprises sending the cursor to the display.

7. The method of claim 6, further comprising:

detecting a cursor-movement input comprising lateral movement of the second hand behind the plane;

in response to detecting the cursor-movement input, move the cursor in correspondence with the movement of the second hand behind the plane.

8. The method of claim 7, further comprising:

highlighting an element of the user interface when movement of the cursor intersects the element; and

selecting the element highlighted in response to detecting a cursor-select input comprising movement of the second hand from behind the plane to in front of the plane while the element is highlighted.

9. A near-eye display system, comprising:

a see-through near-eye display;

one or more image sensors;

a logic machine; and

a storage machine including instructions executable by the logic machine to;

receive image information from the one or more image sensors;

detect a context-setting input performed by a first hand of a user based on the image information received;

in response to detecting the context-setting input, define a virtual interaction coordinate system comprising a spatial region positioned based on a position of the first hand;

send to the see-through near-eye display an augmented reality user interface comprising one or more user interface elements positioned based upon the position of the first hand;

detect an action input performed by a second hand of the user based on the image information received, the action input performed while the first hand of the user is performing the context-setting input; and

send to the see-through near-eye display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.

10. The near-eye display system of claim 9, wherein the action input comprises a movement of the second hand across a plane of the virtual interaction coordinate system from a position on one side of the plane to a position on another side of the plane from a viewpoint of the see-through near-eye display.

11. The near-eye display system of claim 10, wherein the instructions are executable to trigger an image capture process via a camera in response to the second hand crossing the plane.

12. The near-eye display system of claim 9, wherein the instructions are executable to detect a change in posture of the first hand, and in response, trigger an end of a marking up of the user interface by the second hand.

13. The near-eye display system of claim 9, wherein the augmented user interface includes one or more elements selectable by a cursor, wherein the context-setting input triggers a cursor control mode, wherein the action input comprises a movement of the second hand of the user across a plane within the virtual interaction coordinate system, and wherein the response comprises sending the cursor to the see-through near-eye display.

14. A near-eye display system, comprising:

a see-through near-eye display;

one or more image sensors;

a logic machine; and

a storage machine including instructions executable by the logic machine to;

detect a context-setting input triggering a cursor control mode, the context-setting input performed by a first hand of a user;

in response to detecting the context-setting input, define a virtual interaction coordinate system comprising a spatial region positioned based on a position of the first hand;

send to the see-through near-eye display an augmented reality user interface comprising one or more elements selectable by a cursor, the augmented reality user interface positioned based on the position of the first hand;

detect a cursor-initiating input comprising a movement of a second hand of the user across a plane within the virtual interaction coordinate system; and

in response to detecting the cursor-initiating input, send to the see-through near-eye display the cursor.

15. The near-eye display system of claim 14, wherein the cursor-initiating input comprises the second hand of the user crossing a plane extending from the first hand from a position in front of the plane to a position behind the plane from a viewpoint of the near-eye display system.

16. The near-eye display system of claim 15, wherein the instructions are further executable to:

detect a cursor-movement input comprising lateral movement of the second hand behind the plane;

in response to detecting the lateral input, move the image of the cursor in correspondence with the movement of the second hand behind the plane

17. The near-eye display system of claim 16, wherein the instructions are further executable to highlight an element of the augmented reality user interface when movement of the cursor intersects the element.

18. The near-eye display system of claim 17, wherein instructions are further executable to:

detect a cursor select input comprising movement of the second hand from behind the plane to in front of the plane; and

in response to detecting the cursor select input, select the highlighted element of the augmented reality user interface.

19. The near-eye display system of claim 18, wherein the instructions are further executable to:

subsequent to selecting the highlighted element, detect an element manipulation input comprising movement of the first hand; and

in response to detecting the element manipulation input, adjusting a position of the selected highlighted element of the augmented reality user interface.

20. The near-eye display system of claim 19, wherein the instructions are further executable to deselect the selected element of the augmented reality user interface in response to a deselect input comprising movement of the second hand from in front of the plane to behind the plane.