Interacting with a network to transmit virtual image data in augmented or virtual reality systems

• US 10,282,907 B2
• Filed: 05/04/2015
• Issued: 05/07/2019
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): Stone, Robert M
• Application Number:
  US14/703,854
• Publication Number:
  US 20150235610A1
• Time in Patent Office: 1,464 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A method, comprising:

capturing an image of a real physical object, the real physical object located in a field of view of a user, wherein a location of the real physical object is known;

rendering from the captured image of the real physical object a rendered physical object, wherein the rendered physical object appears substantially identical to the real physical object; and

displaying to the user, in a blended reality mode selected by the user from a plurality of visualization modes, the rendered physical object at a display location in the display that corresponds to an area of the display through which the real physical object is viewable, wherein the blended reality mode comprises displaying virtual objects and the rendered physical object at a physical environment of the user, wherein the rendered physical object is displayed in place of the real physical object in the display.

2. The method of claim 1, wherein the plurality of visualization modes is one of a virtual reality mode, an augmented reality mode, a blended reality mode, or a combination of the virtual reality mode and the augmented reality mode.

3. The method of claim 1, further comprising:

receiving image data associated with another display object through a computer network; and

converting the image data to a data format compatible with the user inputted visualization mode such that the user can view the other display object in the user inputted visualization mode.

4. The method of claim 1, wherein rendering the rendered physical object comprises converting the captured image of the real physical object into digital object data.

5. The method of claim 1, further comprising rendering a virtual display object in the field of view of the user such that the user views the rendered physical object simultaneously along with the virtual display object.

6. The method of claim 1, wherein rendering the rendered physical object comprises forming a virtual object from at least the captured image.

7. The method of claim 1, wherein the rendered physical object is displayed on a head-mounted augmented reality display.

8. The method of claim 1, wherein displaying to the user the rendered physical object further comprises displaying a bolded outline of the rendered physical object for differentiating between rendered physical objects and virtual objects.

9. The method of claim 1, wherein the rendered physical object appears identical to the real physical object.

10. An augmented reality display system, comprising:

a camera to capture an image of a real physical object in a field of view of a user, wherein a location of the real physical object in a real world is known;

a module for processing data, wherein the module is stored in a memory, the module rendering from the captured image of the real physical object a rendered physical object, wherein the rendered physical object appears substantially identical to the real physical object; and

a display for displaying to the user, in a blended reality mode selected by the user from a plurality of visualization modes, the rendered physical object at a display location in the display that corresponds to an area of the display through which the real physical object is viewable, wherein the blended reality mode comprises displaying virtual objects and the rendered physical object at a physical environment of the user, wherein the rendered physical object is displayed in place of the real physical object in the display.

11. The augmented reality display system of claim 10, wherein the plurality of visualization modes is one of a virtual reality mode, an augmented reality mode, a blended reality mode, or a combination of the virtual reality mode and the augmented reality mode.

12. The augmented reality display system of claim 10, wherein the real physical object is substantially not visible when the rendered physical object is displayed.

13. The augmented reality display system of claim 10, wherein the display further displays at least one virtual object in addition to the rendered physical object, wherein the at least one virtual object moves relative to one or more real physical objects in the real world.

14. The system of claim 10, wherein the display displays a bolded outline of the rendered physical object for differentiating between rendered physical objects and virtual objects.

15. The system of claim 10, wherein the module converts the captured image of the real physical object into digital object data.

16. The system of claim 10, wherein the module forms a virtual object from at least the captured image.

17. The system of claim 10, wherein the rendered physical object appears identical to the real physical object.