Eclipse cursor for mixed reality displays

• US 10,540,941 B2
• Filed: 01/30/2018
• Issued: 01/21/2020
• Est. Priority Date: 01/30/2018
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Day, John Austin, Pazmino, Lorena, Petty, James Cameron, Hoover, Paul Armistead, Sorrell, Chris, Powderly, James M., Niles, Savannah
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/884,117
• Publication Number:
  US 20190237044A1
• Time in Patent Office: 721 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/08 Cursor circuits
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

Systems and methods for displaying a cursor and a focus indicator associated with real or virtual objects in a virtual, augmented, or mixed reality environment by a wearable display device are disclosed. The system can determine a spatial relationship between a user-movable cursor and a target object within the environment. The system may render a focus indicator (e.g., a halo, shading, or highlighting) around or adjacent objects that are near the cursor. The focus indicator may be emphasized in directions closer to the cursor and deemphasized in directions farther from the cursor. When the cursor overlaps with a target object, the system can render the object in front of the cursor (or not render the cursor at all), so the object is not occluded by the cursor. The cursor and focus indicator can provide the user with positional feedback and help the user navigate among objects in the environment.

Claims

1. A wearable display system comprising:

a display configured to be positioned in front of an eye of a user, the display configured to project virtual content toward the eye of the user;

a user input device configured to receive user input data associated with movement of a virtual cursor; and

a hardware processor in communication with the display and the user input device, the hardware processor programmed to;

identify a location of the virtual cursor in an environment of the user;

determine a spatial relationship between the virtual cursor and an object in the environment of the user; and

direct the display to render a focus indicator associated with the object, wherein to direct the display to render the focus indicator, the hardware processor is programmed to;

perform a first render pass to render a cursor glow to a first buffer, a location of the cursor glow based at least partly on the location of the virtual cursor,

perform a second render pass to render a shape mask representation of the object to a second buffer,

perform a third render pass to render a glow mask associated with the object to a third buffer, and

perform a fourth render pass configured to combine at least the first buffer, the second buffer, and the third buffer for presentation to the user.

2. The wearable display system of claim 1, wherein the user input device comprises one or more of:

a handheld totem comprising a touch sensitive surface, an outward-facing imaging system configured to detect a user gesture, an inward-facing imaging system configured to detect an eye pose of the user, or an inertial measurement unit configured to detect a head pose of the user.

3. The wearable display system of claim 1, wherein to determine the spatial relationship between the virtual cursor and the object, the hardware processor is programmed to:

determine a distance between the location of the virtual cursor and a portion of the object; and

direct the display to render the focus indicator based at least partly on the determined distance.

4. The wearable display system of claim 1, wherein the focus indicator comprises one or more of:

a glow or a halo at least partially surrounding the object, a size or depth change of the object, or graphical highlighting.

5. The wearable display system of claim 1, wherein to perform the fourth render pass, the hardware processor is programmed to combine at least the first buffer, the second buffer, and the third buffer with a fourth buffer comprising virtual scene content.

6. The wearable display system of claim 1, wherein the hardware processor is further programmed to:

determine a distance between the location of the virtual cursor and the object; and

update the location of the virtual cursor to be a location representative of the object if the distance is below a threshold distance.

7. The wearable display system of claim 1, wherein the hardware processor is programmed to:

determine a distance between the location of the virtual cursor and a portion of the object; and

in response to determining the distance is less than a threshold;

render the object in front of the virtual cursor;

or

cease rendering the virtual cursor.

8. The wearable display system of claim 1, wherein to determine the spatial relationship between the virtual cursor and the object, the hardware processor is programmed to determine a relative orientation between the virtual cursor and the object;

and direct the display to render the focus indicator based at least partly on the determined relative orientation.

10. The wearable display system of claim 1, wherein the hardware processor is programmed to:

update the location of the virtual cursor based at least partly on an earlier motion path of the virtual cursor.

11. The wearable display system of claim 10, wherein the hardware processor is programmed to update the location of the virtual cursor based at least partly on the earlier motion path in response to a cessation of user input from the user input device.

9. A wearable display system comprising:

a display configured to be positioned in front of an eye of a user, the display configured to project virtual content toward the eye of the user;

a user input device configured to receive user input data associated with movement of a virtual cursor; and

a hardware processor in communication with the display and the user input device, the hardware processor programmed to;

identify a location of the virtual cursor in an environment of the user;

determine a spatial relationship between the virtual cursor and an object in the environment of the user;

determine an orientation between the location of the virtual cursor and the object; and

based at least partly on the spatial relationship, direct the display to render the focus indicator preferentially toward the orientation of the virtual cursor, wherein the focus indicator is associated with the object.

12. The wearable display system of claim 9, wherein the user input device comprises one or more of:

a handheld totem comprising a touch sensitive surface, an outward-facing imaging system configured to detect a user gesture, an inward-facing imaging system configured to detect an eye pose of the user, or an inertial measurement unit configured to detect a head pose of the user.

13. The wearable display system of claim 9, wherein to determine the spatial relationship between the virtual cursor and the object, the hardware processor is programmed to determine a distance between the location of the virtual cursor and a portion of the object, and direct the display to render the focus indicator based at least partly on the determined distance.

14. The wearable display system of claim 9, wherein to determine the spatial relationship between the virtual cursor and the object, the hardware processor is programmed to determine a relative orientation between the virtual cursor and the object, and direct the display to render the focus indicator based at least partly on the determined relative orientation.

15. The wearable display system of claim 9, wherein the focus indicator comprises one or more of:

a glow or a halo at least partially surrounding the object, a size or depth change of the object, or graphical highlighting.

16. The wearable display system of claim 9, wherein the hardware processor is further programmed to:

determine a distance between the location of the virtual cursor and the object; and

update the location of the virtual cursor to be a location representative of the object if the distance is below a threshold distance.

17. The wearable display system of claim 9, wherein the hardware processor is further programmed to:

update the location of the virtual cursor based at least partly on an earlier motion path of the virtual cursor.

18. The wearable display system of claim 17, wherein the hardware processor is programmed to update the location of the virtual cursor based at least partly on the earlier motion path in response to a cessation of user input from the user input device.

19. The wearable display system of claim 9, wherein the hardware processor is programmed to:

determine a distance between the location of the virtual cursor and a portion of the object; and

in response to determining the distance is less than a threshold, render the object in front of the virtual cursor.

20. The wearable display system of claim 9, wherein the hardware processor is programmed to:

determine a distance between the location of the virtual cursor and a portion of the object; and

in response to determining the distance is less than a threshold, cease rendering the virtual cursor.

21. An augmented reality display system comprising:

a user input device configured to receive user input related to a position of a cursor in an environment of the user;

a display through which a user can perceive virtual objects in the environment of the user;

a hardware processor in communication with the user input device and the display, the hardware processor programmed to;

cause first and second virtual objects to be rendered via the display;

track, based at least in part on the user input, a position of the cursor in the environment;

identify a position of the first virtual object;

identify a position of the second virtual object;

determine whether a first distance between the first virtual object and the position of the cursor satisfies a first distance threshold;

determine whether a second distance between the second virtual object and the position of the cursor satisfies a second distance threshold;

in response to a determination that the first distance satisfies the first distance threshold, cause a first focus indicator proximate the first virtual object to be rendered via the display; and

in response to a determination that the second distance satisfies the second distance threshold, cause a second focus indicator proximate the second virtual object to be rendered via the display.

22. The system of claim 21, wherein in response to a determination that the first distance does not satisfy the first distance threshold, the hardware processor is further programmed to cause a reticle indicating the position of the cursor to be rendered via the display.

23. The system of claim 21, wherein the first virtual object comprises a selectable object such that in response to a user input selecting the first virtual object, the hardware processor performs an action associated with the first virtual object.

24. The system of claim 21, wherein the first distance satisfies the first distance threshold when the position of the cursor overlaps with at least a portion of the first virtual object.

25. The system of claim 21, wherein the first distance satisfies the first distance threshold when the position of the cursor and at least a portion of the first virtual object are mapped to a same pixel on the display.

26. The system of claim 21, wherein the first distance satisfies the distance threshold when the position of the cursor and at least a portion of the first virtual object would be co-located when rendered on the display.

27. The system of claim 21, wherein the hardware processor is further programmed to:

in response to a determination that the first distance is less than the second distance;

cause the first focus indicator to be rendered more prominently than the second focus indicator.

28. The system of claim 27, wherein to cause the first focus indicator to be rendered more prominently than the second focus indicator, the hardware processor is programmed to cause the first focus indicator to be rendered brighter than the second focus indicator.

29. The system of claim 21, wherein the hardware processor is programmed to:

identify a shape of the first virtual object;

determine an orientation of the cursor relative to the first virtual object; and

cause the first focus indicator to be rendered based at least partly on the shape of the first virtual object and the orientation of the cursor relative to the first virtual object.

30. The system of claim 29, wherein the hardware processor is programmed to cause the first focus indicator to be rendered more prominently along a line between the cursor and the first virtual object.

31. The system of claim 27, wherein to cause the first focus indicator to be rendered more prominently than the second focus indicator, the hardware processor is programmed to cause the first focus indicator to be rendered larger than the second focus indicator.

32. The system of claim 27, wherein to cause the first focus indicator to be rendered more prominently than the second focus indicator, the hardware processor is programmed to cause the first focus indicator to be rendered more prominently in a direction toward the cursor than in a direction away from the cursor.