Using tracking to simulate direct tablet interaction in mixed reality

• US 10,445,935 B2
• Filed: 05/26/2017
• Issued: 10/15/2019
• Est. Priority Date: 05/26/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schwarz, Julia, Calabrese, David J., Sheri, Yasaman, Witriol, Daniel B.
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US15/607,276
• Publication Number:
  US 20180342103A1
• Time in Patent Office: 872 Days
• CPC Class Codes: G03H 2210/20 2D object
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Optimizations are provided for facilitating interactions with virtual objects included within an augmented-reality scene. Initially, an augmented-reality scene is rendered for a user. Within that scene, an interactive virtual object of an application is rendered. Then, the position of the user'"'"'s actual hand is determined relative to the interactive virtual object. When the user'"'"'s actual hand is within a target threshold distance to the interactive virtual object, then a target visual cue is projected onto the interactive virtual object. When the user'"'"'s actual hand is within an input threshold distance to the interactive virtual object, then an input visual cue is projected onto the interactive virtual object. Once the user'"'"'s hand is within the input threshold distance to the interactive virtual object, then input may be provided to the application via the interactive object.

Claims

1. A computer system comprising:

one or more processor(s); and

one or more computer-readable hardware storage media having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the computer system to;

on a wearable display of the computer system, render an augmented-reality scene for a user who is wearing the wearable display;

within the augmented-reality scene, render an interactive virtual object;

detect a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object such that the part of the user'"'"'s actual hand is identified as approaching the portion of the interactive virtual object;

anticipate a type of gesture the user will subsequently use to interact with the interactive virtual object, wherein the anticipated gesture type is anticipated based at least partially on an identified type of the interactive virtual object, where the identified type is based on which interactive abilities the interactive virtual object is identified as having;

in response to determining that the part of the user'"'"'s actual hand is within a target threshold distance to the portion of the interactive virtual object, display a target visual cue on the portion of the interactive virtual object;

in response to determining that the part of the user'"'"'s actual hand is within an input threshold distance to the portion of the interactive virtual object, display an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed; and

upon a condition in which the part of the user'"'"'s actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either a corresponding target threshold distance or a corresponding input threshold distance to a corresponding different portion of a different interactive virtual object;

determine which interactive virtual object the user is attempting to interact with;

in response to determining that the user is attempting to interact with the corresponding different portion of the different interactive virtual object, display either the corresponding target visual cue or the corresponding input visual cue on the corresponding different portion of the different interactive virtual object; and

based on a subsequent movement of the part of the user'"'"'s actual hand while the part of the user'"'"'s actual hand is within the corresponding input threshold distance, provide input to an application via the different interactive virtual object.

2. The computer system of claim 1, wherein the target visual cue is displayed at an anticipated ending location on the interactive virtual object where the part of the user'"'"'s actual hand is anticipated to be moving towards.

3. The computer system of claim 1, wherein the input visual cue displaces the target visual cue as soon as the part of the user'"'"'s actual hand is within the input threshold distance.

4. The computer system of claim 1, wherein, as the part of the user'"'"'s actual hand progressively moves from the target threshold distance to the input threshold distance, the target visual cue progressively changes in appearance until the input visual cue is displayed.

5. The computer system of claim 1, wherein the part of the user'"'"'s actual hand is an actual finger of the user'"'"'s actual hand.

6. The computer system of claim 1, wherein execution of the computer-executable instructions further causes the computer system to:

detect that the part of the user'"'"'s actual hand has passed through the interactive virtual object in the augmented-reality scene; and

in response to determining that the part of the user'"'"'s actual hand exceeds the input threshold distance in relation to a back side of the portion of the interactive virtual object, stop providing input to the application via the interactive virtual object.

7. The computer system of claim 1, wherein providing the input to the application via the interactive virtual object includes a scroll input gesture.

8. The computer system of claim 1, wherein providing the input to the application via the interactive virtual object includes a zoom input gesture.

9. The computer system of claim 1, wherein providing the input to the application via the interactive virtual object includes a tap input gesture.

10. The computer system of claim 1, wherein providing the input to the application via the interactive virtual object includes a multiple touch input gesture.

11. A method implemented by one or more processor(s) of a computer system, the method comprising:

on a wearable display of the computer system, rendering an augmented-reality scene for a user who is wearing the wearable display;

within the augmented-reality scene, rendering an interactive virtual object;

detecting a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object such that the part of the user'"'"'s actual hand is identified as approaching the portion of the interactive virtual object;

anticipate a type of gesture the user will subsequently use to interact with the interactive virtual object, wherein the anticipated gesture type is anticipated based at least partially on an identified type of the interactive virtual object, where the identified type is based on which interactive abilities the interactive virtual object is identified as having;

in response to determining that the part of the user'"'"'s actual hand is within a target threshold distance to the portion of the interactive virtual object, displaying a target visual cue on the portion of the interactive virtual object;

in response to determining that the part of the user'"'"'s actual hand is within an input threshold distance to the portion of the interactive virtual object, displaying an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed; and

upon a condition in which the part of the user'"'"'s actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either a corresponding target threshold distance or a corresponding input threshold distance to a corresponding different portion of a different interactive virtual object;

determine which interactive virtual object the user is attempting to interact with;

in response to determining that the user is attempting to interact with the corresponding different portion of the different interactive virtual object, display either the corresponding target visual cue or the corresponding input visual cue on the corresponding different portion of the different interactive virtual object; and

based on a subsequent movement of the part of the user'"'"'s actual hand while the part of the user'"'"'s actual hand is within the corresponding input threshold distance, providing input to an application via the different interactive virtual object.

12. The method of claim 11, wherein the target threshold distance and the input threshold distance are dynamically adjustable based on a size of the interactive virtual object within the augmented-reality scene.

13. The method of claim 11, wherein determining which interactive virtual object the user is attempting to interact with is performed at least in part by one or more of (i) tracking an eye movement, (ii) tracking a vocal cue, or (iii) tracking one or more events occurring within the augmented-reality scene.

14. The method of claim 11, wherein the target threshold distance is about 10 centimeters.

15. The method of claim 11, wherein the interactive virtual object is a two-dimensional object, and wherein providing input to the application via the interactive object includes:

monitoring a three-dimensional movement of the part of the user'"'"'s actual hand;

translating the three-dimensional movement into a two-dimensional movement relative to the two-dimensional interactive object; and

providing the two-dimensional movement as the input to the application.

16. One or more hardware storage devices having stored thereon computer-executable instructions that are executable by one or more processor(s) of a computer system to cause the computer system to:

on a wearable display of the computer system, render an augmented-reality scene for a user who is wearing the wearable display;

within the augmented-reality scene, render an interactive virtual object;

detect a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object such that the part of the user'"'"'s actual hand is identified as approaching the portion of the interactive virtual object;

anticipate a type of gesture the user will subsequently use to interact with the interactive virtual object, wherein the anticipated gesture type is anticipated based at least partially on an identified type of the interactive virtual object, where the identified type is based on which interactive abilities the interactive virtual object is identified as having;

in response to determining that the part of the user'"'"'s actual hand is within a target threshold distance to the portion of the interactive virtual object, display a target visual cue on the portion of the interactive virtual object;

in response to determining that the part of the user'"'"'s actual hand is within an input threshold distance to the portion of the interactive virtual object, display an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed; and

upon a condition in which the part of the user'"'"'s actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either a corresponding target threshold distance or a corresponding input threshold distance to a corresponding different portion of a different interactive virtual object;

determine which interactive virtual object the user is attempting to interact with;

in response to determining that the user is attempting to interact with the corresponding different portion of the different interactive virtual object, display either the corresponding target visual cue or the corresponding input visual cue on the corresponding different portion of the different interactive virtual object; and

based on a subsequent movement of the part of the user'"'"'s actual hand while the part of the user'"'"'s actual hand is within the corresponding input threshold distance, provide input to an application via the different interactive virtual object.

17. The one or more hardware storage devices of claim 16, wherein the interactive virtual object is rendered as a part of a two-dimensional planar surface within the augmented-reality scene.

18. The one or more hardware storage devices of claim 16, wherein the interactive virtual object is rendered as part of a simulated three-dimensional object within the augmented-reality scene.

19. The one or more hardware storage devices of claim 16, wherein an audio cue is played when either the target visual cue or the input visual cue is displayed.

20. A computer system comprising:

one or more processors; and

one or more computer-readable hardware storage media having stored thereon computer-executable instructions, the computer-executable instructions being executable by the one or more processors to cause the computer system to;

on a wearable display of the computer system, render an augmented-reality scene for a user who is wearing the wearable display;

within the augmented-reality scene, render an interactive virtual object;

detect a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object;

in response to determining that the part of the user'"'"'s actual hand is within a target threshold distance to the portion of the interactive virtual object, display a target visual cue on the portion of the interactive virtual object;

in response to determining that the part of the user'"'"'s actual hand is within an input threshold distance to the portion of the interactive virtual object, display an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed, wherein the part of the user'"'"'s actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either the target threshold distance or the input threshold distance to a different portion of a different interactive virtual object;

determine which interactive virtual object the user is attempting to interact with;

in response to determining that the user is attempting to interact with the different portion of the different interactive virtual object, display either the target visual cue or the input visual cue on the different portion of the different interactive virtual object; and

based on a subsequent movement of the part of the user'"'"'s actual hand while the part of the user'"'"'s actual hand is within the input threshold distance, provide input to an application via the interactive virtual object.