Interacting with a network to transmit virtual image data in augmented or virtual reality systems

• US 10,126,812 B2
• Filed: 05/02/2015
• Issued: 11/13/2018
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): Stone, Robert
• Application Number:
  US14/702,709
• Publication Number:
  US 20150234462A1
• Time in Patent Office: 1,291 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A method, comprising:

sensing a physical object at a first location using at least one or more outward facing cameras of a head-mounted user display device and recognizing a type of the physical object sensed at the first location by the head-mounted user display device by;

capturing one or more field-of-view images,

extracting one or more sets of points from the one or more field-of-view images,

extracting one or more fiducials for at least one physical object in the one or more field-of-view images based on at least some of the one or more sets of points,

processing at least some of the one or more fiducials for the at least one physical object to identify the type of the physical object sensed at the first location, wherein processing at least some of the one or more fiducials comprises comparing the one or more fiducials to sets of previously stored fiducials;

associating a virtual object with the sensed physical object based on the type of the physical object as a result of both recognizing the type of the sensed physical object at the first location by the head-mounted user display device using the at least one or more outward facing cameras and identifying a predetermined relationship between the virtual object and the type of the sensed physical object recognized at the first location by the head-mounted user display device;

receiving virtual world data representing a virtual world, the virtual would data including at least data corresponding to manipulation of the virtual object in the virtual world by a first user at the first location;

transmitting at least the virtual world data corresponding to manipulation of the virtual object by the first user at the first location to a head-mounted user display device, wherein the head-mounted user display device renders a display image associated with at least a portion of the virtual world data including at least the virtual object to the first user based on at least an estimated depth of focus of a first user'"'"'s eyes;

creating additional virtual world data originating from the manipulation of the virtual object by the first user at the first location; and

transmitting the additional virtual world data to a second user at a second location different from the first location for presentation to the second user, such that the second user experiences the additional virtual world data from the second location.

2. The method of claim 1, wherein the additional virtual world data is associated with a field-of-view image captured through the head-mounted user display device.

3. The method of claim 1, wherein the additional virtual world data is associated with at least one of a sensed movement of the first user, a sensed location of the first user, a sensed direction of the first user and a sensed orientation of the first user.

4. The method of claim 1, further comprising:

selecting, based on user input, an interface for displaying the display image to the first user to identify a selected interface; and

rendering the display image using the selected interface.

5. The method of claim 4, wherein the selected interface is one of a virtual reality mode, an augmented reality mode, a blended reality mode, and a combination of the virtual reality mode and augmented reality mode.

6. The method of claim 1, wherein the virtual world is presented in a two-dimensional format.

7. The method of claim 1, wherein the virtual world is presented in a three-dimensional format.

8. The method of claim 1, wherein the head-mounted user display device displays the display image to the first user in a first visualization mode, and

wherein the additional virtual world data is presented to the second user in a second visualization mode.

9. The method of claim 8, wherein the first visualization mode is different from the second visualization mode.

10. The method of claim 8, wherein the first visualization mode is at least one of an augmented reality mode, a virtual reality mode, a blended reality mode, and a combination of the virtual reality mode and augmented reality mode.

11. The method of claim 8, wherein the second visualization mode is at least one of an augmented reality mode, a virtual reality mode, a blended reality mode, and a combination of the virtual reality mode and augmented reality mode.

12. A system, comprising:

a first head-mounted user display device associated with a first user at a first location;

a second head-mounted user display device associated with a second user at a second location different from the first location; and

a server having a processor, the server operatively coupled to the first head-mounted user display device and the second head-mounted user display device,

wherein the server is configured to;

sense a physical object at a first location using at least one or more outward facing cameras of a head-mounted user display device and recognize a type of the physical object sensed at the first location by the head-mounted user display device by;

capturing one or more field-of-view images,

extracting one or more sets of points from the one or more field-of-view images,

extracting one or more fiducials for at least one physical object in the one or more field-of-view images based on at least some of the one or more sets of points,

processing at least some of the one or more fiducials for the at least one physical object to identify the type of the physical object sensed at the first location, wherein processing at least some of the one or more fiducials comprises comparing the one or more fiducials to sets of previously stored fiducials;

associate a virtual object with the sensed physical object based on the type of the physical object as a result of both recognizing the type of the sensed physical object at the first location by the head-mounted user display device using the at least one or more outward facing cameras and identifying a predetermined relationship between the virtual object and the type of the sensed physical object recognized at the first location by the head-mounted user display device;

receive virtual world data representing a virtual world, the virtual would data including at least data corresponding to manipulation of the virtual object in the virtual world by a first user at the first location;

transmit at least the virtual world data corresponding to manipulation of the virtual object by the first user at the first location to a head-mounted user display device, wherein the head-mounted user display device renders a display image associated with at least a portion of the virtual world data including at least the virtual object to the first user based on at least an estimated depth of focus of a first user'"'"'s eyes;

create additional virtual world data originating from the manipulation of the virtual object by the first user at the first location; and

transmit the additional virtual world data to a second user at a second location different from the first location for presentation to the second user, such that the second user experiences the additional virtual world data from the second location.

13. The system of claim 12, wherein the additional virtual world data is associated with a field-of-view image captured through the first head-mounted user display device.

14. The system of claim 12, wherein the additional virtual world data is associated with at least one of a sensed movement of the first user, a sensed location of the first user, a sensed direction of the first user and a sensed orientation of the first user.

15. The system of claim 12, further comprising:

selecting, based on user input, an interface for displaying the display image to the first user to identify a selected interface; and

rendering the display image using the selected interface.

16. The system of claim 15, wherein the selected interface is one of a virtual reality mode, an augmented reality mode, a blended reality mode, and a combination of the virtual reality mode and augmented reality mode.

17. The system of claim 12, wherein the virtual world is presented in a two-dimensional format.

18. The system of claim 12, wherein the virtual world is presented in a three-dimensional format.

19. The system of claim 12, wherein the first head-mounted user display device displays the display image to the first user in a first visualization mode, and

wherein the additional virtual world data is presented to the second user in a second visualization mode.

20. The system of claim 19, wherein the first visualization mode is different from the second visualization mode.