Contextual cursor display based on hand tracking

• US 10,409,443 B2
• Filed: 06/24/2015
• Issued: 09/10/2019
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wright, Shawn Crispin, Osborn, Dan, Thompson, Joe, Ramsby, Scott Robert, Gouin, Forest Woodcroft, Saunders, Megan, Cottle, Aaron David
• Primary Examiner(s): Pan, Yongjia
• Application Number:
  US14/748,646
• Publication Number:
  US 20160378294A1
• Time in Patent Office: 1,539 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A head mounted display device and method for contextual cursors comprising a camera configured to capture a series of images, a display, and a processor configured to: display, to a user of the head mounted display device, a virtual cursor on display so as to appear at a location in a three dimensional environment, determine whether motion of a hand of the user in the images is trackable, and in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate that motion of the hand is trackable, and in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

Claims

1. A head mounted display device for contextual cursors in a virtual or augmented reality environment comprising:

a camera configured to capture a series of images;

a display; and

a processor configured to;

display, to a user of the head mounted display device, a virtual cursor on the display so as to appear at a location in a three dimensional environment;

determine a context based on the captured series of images by detecting a recognized virtual object in the three dimensional environment based on a gaze direction of the user;

determine whether an intersection between the recognized virtual object and the gaze direction of the user is located at a planar surface, a resizing area, or a moving area of the recognized virtual object;

select a programmatic function based on the determined context and whether the gaze direction is located at the planar surface, the resizing area, or the moving area of the recognized virtual object, wherein;

when the intersection is located at the planar surface, a planar surface interaction function is selected, the planar surface interaction function being a drawing function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object;

when the intersection is located at the resizing area, a resizing function is selected, the resizing function being a programmatic function to change a size of the recognized object; and

when the intersection is located at the moving area, a moving function is selected, the moving function being a programmatic function to move an object location of the recognized object to a different location in the three dimensional environment;

determine whether motion of a hand of the user in the images is trackable;

in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate the selected programmatic function and that motion of the hand is trackable;

track motion of the hand in the images to identify a hand gesture of the user;

in response to at least identifying the hand gesture, execute the programmatic function; and

in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

2. The head mounted display device of claim 1, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor.

3. The head mounted display device of claim 2, wherein to modify the visual appearance of the holographic cursor, the processor is further configured to:

change a color, change a shape, add or remove an icon, or change a size of the holographic cursor.

4. The head mounted display device of claim 2, wherein in response to at least identifying the hand gesture, the processor is further configured to:

determine whether the hand gesture is being maintained based on the tracked motion of the hand;

in response to at least determining that the hand gesture is being maintained, continue executing the selected programmatic function and modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed; and

in response to at least determining that the hand gesture is not maintained, modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed.

5. The head mounted display device of claim 2, wherein the processor is further configured to:

in response to at least selecting the programmatic function based on the determined context, modify the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context.

6. A method for contextual cursors in a virtual or augmented reality environment comprising:

displaying a virtual cursor to a user on a display of a head mounted display device so as to appear at a location in a three dimensional environment;

determining whether motion of a hand of the user in a series of images captured by a camera of the head mounted display device is trackable;

determining a context based on the captured series of images by detecting a recognized virtual object in the three dimensional environment based on a gaze direction of the user;

determining whether an intersection between the recognized virtual object and the gaze direction of the user is located at a planar surface, a resizing area, or a moving area of the recognized virtual object;

selecting a programmatic function based on the determined context and whether the gaze direction is located at the planar surface, the resizing area, or the moving area of the recognized virtual object, wherein;

when the intersection is located at the planar surface, a planar surface interaction function is selected, the planar surface interaction function being a drawing function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object;

when the intersection is located at the resizing area, a resizing function is selected, the resizing function being a programmatic function to change a size of the recognized object; and

when the intersection is located at the moving area, a moving function is selected, the moving function being a programmatic function to move an object location of the recognized object to a different location in the three dimensional environment;

in response to at least determining that motion of the hand is trackable, modifying a visual appearance of the virtual cursor to indicate the selected programmatic function and that motion of the hand is trackable;

tracking motion of the hand in the images to identify a hand gesture of the user;

in response to at least identifying the hand gesture, executing the programmatic function; and

in response to at least determining that motion of the hand is not trackable, modifying the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

7. The method of claim 6, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor.

8. The method of claim 7, wherein modifying the visual appearance of the holographic cursor further comprises:

changing a color, changing a shape, adding or removing an icon, or changing a size of the holographic cursor.

9. The method of claim 7, wherein in response to at least identifying the hand gesture, the method further comprises:

determining whether the hand gesture is being maintained based on the tracked motion of the hand;

in response to at least determining that the hand gesture is being maintained, continuing to execute the selected programmatic function and modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed; and

in response to at least determining that the hand gesture is not maintained, modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed.

10. The method of claim 7, further comprising:

in response to at least selecting the programmatic function based on the determined context, modifying the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context.

11. A head mounted display device for contextual cursors in a holographic environment comprising:

a camera configured to capture a series of images;

an at least partially see-through display configured to display holograms; and

a processor configured to;

display, to a user of the head mounted display device, a holographic cursor on the at least partially see-though display so as to appear at a location in a three dimensional environment;

determine a context based on the captured series of images by detecting a recognized virtual object in the three dimensional environment based on a gaze direction of the user;

determine whether an intersection between the recognized virtual object and the gaze direction of the user is located at a planar surface, a resizing area, or a moving area of the recognized virtual object;

select a programmatic function based on the determined context and whether the gaze direction is located at the planar surface, the resizing area, or the moving area of the recognized virtual object, wherein;

when the intersection is located at the planar surface, a planar surface interaction function is selected, the planar surface interaction function being a drawing function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object;

when the intersection is located at the resizing area, a resizing function is selected, the resizing function being a programmatic function to change a size of the recognized object; and

when the intersection is located at the moving area, a moving function is selected, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment;

modify a visual appearance of the holographic cursor to indicate the selected programmatic function; and

track motion of the hand in the images to identify a hand gesture of the user; and

in response to at least identifying the hand gesture, execute the selected programmatic function.