IMAGING MODIFICATION, DISPLAY AND VISUALIZATION USING AUGMENTED AND VIRTUAL REALITY EYEWEAR

• US 20190011703A1
• Filed: 07/24/2017
• Published: 01/10/2019
• Est. Priority Date: 07/25/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Robaina, Nastasja U., Samec, Nicole Elizabeth, Harrises, Christopher M., Abovitz, Rony, Baerenrodt, Mark, Schmidt, Brian Lloyd
• Granted Patent: US 10,838,210 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/254 being adapted depending on the stage of the surgical procedure
• CPC Class Codes: A61B 2034/258 providing specific settings for specific users
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture

Abstract

A display system can include a head-mounted display configured to project light to an eye of a user to display augmented reality image content to the user. The display system can include one or more user sensors configured to sense the user and can include one or more environmental sensors configured to sense surroundings of the user. The display system can also include processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors. The processing electronics can be configured to sense a situation involving user focus, determine user intent for the situation, and alter user perception of a real or virtual object within the vision field of the user based at least in part on the user intent and/or sensed situation involving user focus. The processing electronics can be configured to at least one of enhance or de-emphasize the user perception of the real or virtual object within the vision field of the user.

Claims

1. A head-mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user, said head-mounted display system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured to project light into said user'"'"'s eye to display augmented reality image content to the user'"'"'s vision field at different amounts of divergences as if projected from different distances from the user'"'"'s eye, at least a portion of said display being transparent and disposed at a location in front of the user'"'"'s eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user'"'"'s eye to provide a view of said portion of the environment in front of the user and said head-mounted display;

one or more user sensors configured to sense the user;

one or more environmental sensors configured to sense surroundings of the user;

processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors, the processing electronics configured to;

sense a situation involving increased user focus; and

alter user perception of a real or virtual object within the vision field of the user based at least in part on sensing the increased focus.

2. The display system of claim 1, wherein the processing electronics are configured to alter the user perception of the real or virtual object within the vision field of the user by altering image content comprising the object.

3. The display system of claim 2, wherein altering image content comprises moving image content of the object from a first location to a second location.

4. The display system of claim 3, wherein the first location is the peripheral region and the second location is the central region or wherein the first location is the central region and the second location is the peripheral region.

5. The display system of claim 2, wherein altering image content comprises altering at least one of contrast, opacity, color, color saturation, color balance, size, background, brightness, edges, or sharpness of image content comprising the object.

6. The display system of claim 1, wherein the one or more environmental sensors comprise a depth sensor, a pair of binocular world cameras, a geolocation sensor, a proximity sensor, or a GPS.

7. The display system of claim 1, wherein the one or more user sensors comprise one or more cameras.

8. The display system of claim 1, wherein the processing electronics is further configured to determine the user intent for the situation and alter the user perception of the real or virtual object within the vision field of the user based at least in part on sensing the increased focus.

9. The display system of claim 1, wherein said situation involving increased user focus comprises driving a motor vehicle.

10. The display system of claim 9, wherein said one or more environmental sensors comprise a sensor configured to detect a radio signal.

11. The display system of claim 10, wherein said one or more environmental sensors comprise a sensor configured to detect a blue tooth signal from an automobile.

12. The display system of claim 9, wherein said processing electronics are configured to alter user perception of said real or virtual object within the vision field of the user based at least in part on one or more data records regarding the user, said one or more data records comprising a driving record of said user.

13. A head-mounted display system configured to project light to an eye of a user to display augmented reality image content, said user'"'"'s eye having a vision field having a central region and a peripheral region disposed about said central region, said head-mounted display system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured to project light into said user'"'"'s eye so as to present image content at said central region of said user'"'"'s vision field, at least a portion of said display being transparent and disposed at a location in front of the user'"'"'s eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user'"'"'s eye to provide a view of said portion of the environment in front of the user and said head-mounted display;

processing electronics in communication with said display to control presentation of image content on said display,

wherein said head-mounted display system is configured to present image content to said central region of the user'"'"'s vision field that is enhanced in comparison to image content presented to the peripheral region of the user'"'"'s vision field.

14. The system of claim 13, further comprising one or more sensors configured to monitor the environment.

15. The system of claim 14, wherein said one or more sensors comprise one or more outward-facing image capture devices configured to image said environment.

16. The system of claim 14, wherein said one or more sensors comprise a distance measuring device.

17. The system of claim 13, further comprising an eye tracking device configured to track position and/or movement of said user'"'"'s eye.

18. The system of claim 13, wherein the head-mounted display device is configured to process image content presented to at least a portion of said central region of the user'"'"'s vision field differently in comparison to image content presented to the peripheral region of the user'"'"'s vision field.

19. The system of claim 18, wherein the head-mounted display device is configured to process image content differently by magnifying image content presented to at least a portion of said central region of the user'"'"'s vision field in comparison to image content presented to the peripheral region of the user'"'"'s vision field.

20. The system of claim 18, wherein the head-mounted display device is configured to process image content differently by increasing brightness in image content presented to at least a portion of said central region of the user'"'"'s vision field in comparison to image content presented to the peripheral region of the user'"'"'s vision field.