Selective light transmission for augmented or virtual reality

• US 10,163,265 B2
• Filed: 05/02/2015
• Issued: 12/25/2018
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): Stone, Robert M
• Application Number:
  US14/702,707
• Publication Number:
  US 20150235429A1
• Time in Patent Office: 1,333 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A method, comprising:

tracking a movement of a first user'"'"'s eye using a first head mounted display device;

estimating a depth of focus of the first user'"'"'s eye based on the tracked eye movement;

modifying a light beam associated with a display object based on the estimated depth of focus such that the display object appears in focus;

projecting the light beam toward a display lens of the first head mounted display device;

directing the light beam into the first user'"'"'s eye using the display lens;

selectively allowing a transmission of light from a local environment of the first user based on at least a selection of an augmented reality mode of the first head mounted display device;

capturing a field-of-view image by the first head mounted display device at the local environment of the first user;

generating a rendered physical object using the field-of-view image, the rendered physical object corresponding to a physical object present in the local environment of the first user, and the rendered physical object representing the physical object as it appears in the local environment; and

transmitting at least a portion of virtual world data associated with the display object and the rendered physical object to a second head mounted display device associated with a second user at a second location, the second head mounted display device projects the display object and the rendered physical object at the second location based at least in part on the virtual world data, wherein the first user and the second user interface with a shared virtual reality, the first head mounted display device of the first user operates in an augmented reality mode and the second head mounted display device of a second user operates in a virtual reality mode, and the second head mounted display device displaying the rendered physical object as the physical object appears in the local environment of the first user.

2. The method of claim 1, wherein the field-of-view image corresponds to a field-of-view for each of the user'"'"'s eyes.

3. The method of claim 2, further comprising:

estimating a head pose of the user based on the field-of-view image of the user'"'"'s eyes.

4. The method of claim 2, further comprising:

converting a second physical object in the captured field-of-view image to a physically rendered virtual object; and

displaying the physically rendered virtual object to the user.

5. The method of claim 2, further comprising:

extracting a set of points in the captured field-of-view image; and

creating a fiducial for a second physical object in the captured field-of-view image based on the extracted set of points.

6. The method of claim 5, further comprising:

transmitting at least one of the extracted set of points and the fiducial to a cloud computer; and

tagging at least one of the extracted set of points and the fiducial to a type of object.

7. The method of claim 6, further comprising:

recognizing a different physical object as belonging to the type of object based on at least one of the tagged set of points associated with the type of object and the fiducial associated with the type of object.

8. The method of claim 1, wherein the projected light is a light beam having a diameter less than 0.7 mm.

9. The method of claim 1, further comprising:

sensing, by at least one sensor, at least one of a movement of the user, a location of the user, a direction of the user, and an orientation of the user; and

calculating a pose of the user based on the at least one sensed movement, sensed location, sensed direction, and sensed orientation.

10. The method of claim 9, wherein the at least one sensor is at least one of an accelerometer, a compass and a gyroscope.

11. The method of claim 1, further comprising:

processing the virtual world data associated with the display object using a computer network; and

transmitting at least a portion of the virtual world data associated with the display object to a second user located at a second location such that the second user may experience the portion of the virtual world data associated with the display object at the second location.

12. The method of claim 11, further comprising:

sensing a second physical object; and

modifying, based on a predetermined relationship with the sensed second physical object, at least a portion of the virtual world data associated with the display object.

13. The method of claim 12, presenting the modified virtual world data to the second user.

14. The method of claim 1, further comprising:

modifying another light associated with another object that is not the display object such that the other object appears blurred.

15. The method of claim 1, further comprising;

receiving user input through a user interface; and

modifying the display object based on the received user input.

16. The method of claim 15, wherein the user interface is at least one of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device and an audio input device.

17. The method of claim 1, wherein the display object is at least one of a virtual object, a rendered physical object, an image and a video.

18. The method of claim 1, wherein the selectively allowing the transmission of light from the local environment of the user further comprises blocking, with a cover, a lens of a head-mounted display device to selectively allow the transmission of light from the local environment.

19. The method of claim 1, wherein the selectively allowing a transmission of light from the local environment of the user is based on a visualization mode of the display object, and

wherein the visualization mode is one of an augmented reality mode, a virtual reality mode, or a combination of augmented and virtual reality modes.

20. A computer program product embodied in a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method, the method comprising:

tracking a movement of a first user'"'"'s eye using a first head mounted display device;

estimating a depth of focus of the first user'"'"'s eye based on the tracked eye movement;

modifying a light beam associated with a display object based on the estimated depth of focus such that the display object appears in focus;

projecting the light beam toward a display lens of the first head mounted display device;

directing the light beam into the first user'"'"'s eye using the display lens;

selectively allowing a transmission of light from a local environment of the first user based on at least a selection of an augmented reality mode of the first head mounted display device;

capturing a field-of-view image by the first head mounted display device at the local environment of the first user;

generating a rendered physical object using the field-of-view image, the rendered physical object corresponding to a physical object present in the local environment of the first user, and the rendered physical object representing the physical object as it appears in the local environment; and

transmitting at least a portion of virtual world data associated with the display object and the rendered physical object to a second head mounted display device associated with a second user at a second location, the second head mounted display device projects the display object and the rendered physical object at the second location based at least in part on the virtual world data, wherein the first user and the second user interface with a shared virtual reality, the first head mounted display device of the first user operates in an augmented reality mode and the second head mounted display device of a second user operates in a virtual reality mode, and the second head mounted display device displaying the rendered physical object as the physical object appears in the local environment of the first user.