CONTEXTUAL CURSOR DISPLAY BASED ON HAND TRACKING

• US 20160378294A1
• Filed: 06/24/2015
• Published: 12/29/2016
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ramsby, Scott Robert, Cottle, Aaron David, Wright, Shawn Crispin, Osborn, Dan, Thompson, Joe, Gouin, Forest Woodcroft, Saunders, Megan
• Granted Patent: US 10,409,443 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A head mounted display device and method for contextual cursors comprising a camera configured to capture a series of images, a display, and a processor configured to: display, to a user of the head mounted display device, a virtual cursor on display so as to appear at a location in a three dimensional environment, determine whether motion of a hand of the user in the images is trackable, and in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate that motion of the hand is trackable, and in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

Claims

1. A head mounted display device for contextual cursors in a virtual or augmented reality environment comprising:

a camera configured to capture a series of images;

a display; and

a processor configured to;

display, to a user of the head mounted display device, a virtual cursor on the display so as to appear at a location in a three dimensional environment;

determine whether motion of a hand of the user in the images is trackable;

in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate that motion of the hand is trackable; and

in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

2. The head mounted display device of claim 1, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor.

3. The head mounted display device of claim 2, wherein to modify the visual appearance of the holographic cursor, the processor is further configured to:

change a color, change a shape, add or remove an icon, or change a size of the holographic cursor.

4. The head mounted display device of claim 2, wherein in response to at least determining that motion of the hand is trackable, the processor is further configured to:

track motion of the hand in the images to identify a user hand gesture; and

in response to at least identifying the hand gesture, execute a programmatic function.

5. The head mounted display device of claim 4, wherein the processor is further configured to:

determine a context based on the captured series of images; and

select the programmatic function based on the determined context.

6. The head mounted display device of claim 5, wherein the processor is further configured to:

modify the visual appearance of the holographic cursor to indicate the selected programmatic function.

7. The head mounted display device of claim 5, wherein to determine the context based on the captured series of images, the processor is further configured to:

detect a recognized object in the three dimensional environment;

determine a gaze direction of the user of the head mounted display device;

determine an intersection between the recognized object and the gaze direction of the user; and

determine the context based on the intersection and the recognized object.

8. The head mounted display device of claim 6, wherein in response to at least identifying the hand gesture, the processor is further configured to:

determine whether the hand gesture is being maintained based on the tracked motion of the hand;

in response to at least determining that the hand gesture is being maintained, continue executing the selected programmatic function and modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed; and

in response to at least determining that the hand gesture is not maintained, modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed.

9. The head mounted display device of claim 5, wherein the processor is further configured to:

in response to at least selecting the programmatic function based on the determined context, modify the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context.

10. The head mounted display device of claim 7, wherein the recognized object is a virtual object having an object location in the three dimensional environment, and the processor is further configured to:

determine whether the intersection between the recognized object and the gaze direction of the user is located at a planar surface of the recognized object, a resizing area of the recognized object, or a moving area of the recognized object;

in response to at least determining that the intersection is located at the planar surface of the recognized object, select a drawing function as the selected programmatic function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object;

in response to at least determining that the intersection is located at the resizing area of the recognized object, select a resizing function as the selected programmatic function, the resizing function being a programmatic function to change a size of the recognized object; and

in response to at least determining that the intersection is located at the moving area of the recognized object, select a moving function as the selected programmatic function, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment.

11. A method for contextual cursors in a virtual or augmented reality environment comprising:

displaying a virtual cursor to a user on a display of a head mounted display device so as to appear at a location in a three dimensional environment;

determining whether motion of a hand of the user in a series of images captured by a camera of the head mounted display device is trackable;

in response to at least determining that motion of the hand is trackable, modifying a visual appearance of the virtual cursor to indicate that motion of the hand is trackable; and

in response to at least determining that motion of the hand is not trackable, modifying the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.

12. The method of claim 11, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor.

13. The method of claim 12, wherein modifying the visual appearance of the holographic cursor further comprises:

changing a color, changing a shape, adding or removing an icon, or changing a size of the holographic cursor.

14. The method of claim 12, wherein in response to at least determining that motion of the hand is trackable, the method further comprises:

tracking motion of the hand in the images to identify a user hand gesture; and

in response to at least identifying the hand gesture, executing a programmatic function.

15. The method of claim 14, further comprising:

determining a context based on the captured series of images;

selecting the programmatic function based on the determined context; and

modifying the visual appearance of the holographic cursor to indicate the selected programmatic function.

16. The method of claim 15, wherein determining the context based on the captured series of images, further comprises:

detecting a recognized object in the three dimensional environment;

determining a gaze direction of the user of the head mounted display device;

determining an intersection between the recognized object and the gaze direction of the user; and

determining the context based on the intersection and the recognized object.

17. The method of claim 15, wherein in response to at least identifying the hand gesture, the method further comprises:

determining whether the hand gesture is being maintained based on the tracked motion of the hand;

in response to at least determining that the hand gesture is being maintained, continuing to execute the selected programmatic function and modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed; and

in response to at least determining that the hand gesture is not maintained, modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed.

18. The method of claim 15, further comprising:

in response to at least selecting the programmatic function based on the determined context, modifying the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context.

19. The method of claim 16, wherein the recognized object is a virtual object having an object location in the three dimensional environment, and further comprising:

determining whether the intersection between the recognized object and the gaze direction of the user is located at a planar surface of the recognized object, a resizing area of the recognized object, or a moving area of the recognized object;

in response to at least determining that the intersection is located at the planar surface of the recognized object, selecting a drawing function as the selected programmatic function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object;

in response to at least determining that the intersection is located at the resizing area of the recognized object, selecting a resizing function as the selected programmatic function, the resizing function being a programmatic function to change a size of the recognized object; and

in response to at least determining that the intersection is located at the moving area of the recognized object, selecting a moving function as the selected programmatic function, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment.

20. A head mounted display device for contextual cursors in a holographic environment comprising:

a camera configured to capture a series of images;

an at least partially see-through display configured to display holograms; and

a processor configured to;

display, to a user of the head mounted display device, a holographic cursor on the at least partially see-though display so as to appear at a location in a three dimensional environment;

determine a context based on the captured series of images;

select a programmatic function based on the determined context;

modify a visual appearance of the holographic cursor to indicate the selected programmatic function; and

track motion of the hand in the images to identify a user hand gesture; and

in response to at least identifying the hand gesture, execute the selected programmatic function.