Systems and methods for a plurality of users to interact with an augmented or virtual reality systems

• US 10,068,374 B2
• Filed: 05/04/2015
• Issued: 09/04/2018
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): Stone, Robert
• Application Number:
  US14/703,867
• Publication Number:
  US 20150235434A1
• Time in Patent Office: 1,219 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A method for a first user located in a first real geographic location and a second user located in a second real geographic location different from the first real geographic location to interact within a virtual world comprising virtual world data, the method comprising:

sensing a plurality of inanimate real world objects at the first real geographic location of a first user device, the first user device comprising at least a first head mounted display device having an environment-sensing system, the plurality of inanimate real world objects being sensed in one or more fields of view of the first user;

creating an avatar for a first user accessing a virtual world through the first user device at the first real geographic location, wherein the virtual world includes a virtual representation of the plurality of inanimate real world objects as the plurality of inanimate real world objects appear in the first real geographical location;

placing the avatar in the virtual world;

displaying the avatar in an augmented reality view of the virtual world to a second user accessing the virtual world through a second user device from a second real geographical location, the second user device comprising at least a second head mounted display device, wherein the avatar is animated based on at least input from the first head mounted display device;

identifying one or more inanimate real world objects of the plurality of real world objects for display by at least receiving one or more inputs from the first user to select one or more inanimate real world objects of the plurality of inanimate real world objects for display or receiving one or more inputs from the first user to remove one or more inanimate real world objects of the plurality of inanimate real world objects from display;

displaying one or more virtual representations of the one or more inanimate real world objects as the one or more inanimate real world objects appear in the first real geographical location to the second user through the second user device, the one or more inanimate real world object sensed at the first real geographic location by the first user device appears to be physically present in the augmented reality view of the virtual world displayed to the second user using the second user device, thereby allowing the second user to experience the first real geographical location through the second user device from the second real geographical location; and

facilitating interaction between the first user and second user in the virtual world through the first user device and the second user device.

2. The method of claim 1, wherein the avatar has a predetermined physical relationship to a physical object at the first real geographical location.

3. The method of claim 1, wherein the avatar responds to an environmental cue at the first real geographical location.

4. The method of claim 1, wherein a movement of the avatar is controlled by the first user.

5. The method of claim 1, wherein the avatar can interact with a second user at the second real geographical location.

6. The method of claim 1, wherein the avatar is placed in the virtual world based at least in part on a permission granted by the second user.

7. The method of claim 1, further comprising creating a dense digital model, wherein the dense digital model comprises a group of cameras viewing the first real geographical location.

8. The method of claim 1, wherein the input from the first head mounted display device is based at least in part on an interaction the first user has with the first user device and processed using an avatar emotion engine, the interaction comprising voice inflections, head movements, body temperature, heart rate, facial recognition data.

9. The method of claim 8, wherein digitizing input from the first head mounted display device and passing that to remote avatar uses less bandwidth then passing captured video.

10. The method of claim 8, wherein the avatar comprises a non-human form and the input from the first head mounted user display device is mapped to the non-human form.

11. The method of claim 1, further comprising displaying the virtual representation of the inanimate real world object as the inanimate real world object appears in a blended mode view of the virtual world to the first user accessing the virtual world through the first user device.

12. A system for implementing interaction between a first user located in a first real geographic location and a second user located in a second real geographic location different from the first real geographic location within a virtual world comprising virtual world data, the system comprising:

a first user device having a first processor; and

a second user device having a second processor, the first and second user devices being operatively coupled;

wherein the first processor is configured to;

sense a plurality of inanimate real world objects at the first real geographic location of a first user device, the first user device comprising at least a first head mounted display device having an environment-sensing system, the plurality of inanimate real world objects being sensed in one or more fields of view of the first user;

create an avatar for a first user accessing a virtual world through the first user device at the first real geographic location, wherein the virtual world includes a virtual representation of the plurality of inanimate real world objects as the plurality of inanimate real world objects appear in the first real geographical location;

place the avatar in the virtual world; and

identify one or more inanimate real world objects of the plurality of real world objects for display by at least receiving one or more inputs from the first user to select one or more inanimate real world objects of the plurality of inanimate real world objects for display or receiving one or more inputs from the first user to remove one or more inanimate real world objects of the plurality of inanimate real world objects from display;

wherein the second processor is configured to instruct the second user device to;

display the avatar in an augmented reality view of the virtual world to a second user accessing the virtual world through a second user device from a second real geographical location, the second user device comprising at least a second head mounted display device, wherein the avatar is animated based on at least input from the first head mounted display device;

displaying one or more virtual representations of the one or more inanimate real world objects as the one or more inanimate real world objects appear in the first real geographical location to the second user through the second user device, the one or more inanimate real world object sensed at the first real geographic location by the first user device appears to be physically present in the augmented reality view of the virtual world displayed to the second user using the second user device, thereby allowing the second user to experience the first real geographical location through the second user device from the second real geographical location; and

wherein the first and second processors are configured to facilitate interaction between the first user and second user in the virtual world through the first user device and the second user device.

13. The system of claim 12, further comprising a group of cameras configured to view the first real geographical location to facilitate creation of a dense digital model.

14. The system of claim 12, wherein the first head mounted display further comprises a camera configured to capture an image of a face of the first user and a microphone configured to capture a voice of the first user, wherein the first processor is configured to animate the avatar based at least in part on an interaction the first user has with the first user device and processed using an avatar emotion engine, the interaction comprising voice inflections, head movements, body temperature, heart rate, facial recognition data.

15. The system of claim 12, wherein digitizing input from the first head mounted display device and passing that to remote avatar uses less bandwidth then passing captured video.

16. The system of claim 12, wherein the first processor is configured to instruct the first user device to display the virtual representation of the inanimate real world object as the inanimate real world object appears in a blended mode view of the virtual world to the first user.

17. A system for implementing interaction between a first user located in a first real geographic location and a second user located in a second real geographic location different from the first real geographic location in a virtual world comprising virtual world data, the system comprising:

a server having a server processor and being operatively coupled to a first user device and a second user device, wherein the server operatively couples the first user device and the second user device,

wherein the server processor is configured to;

sense a plurality of inanimate real world objects at the first real geographic location of a first user device, the first user device comprising at least a first head mounted display device having an environment-sensing system, the plurality of inanimate real world objects being sensed in one or more fields of view of the first user;

create an avatar for a first user accessing a virtual world through the first user device at the first real geographic location, wherein the virtual world includes a virtual representation of the plurality of inanimate real world objects as the plurality of inanimate real world objects appear in the first real geographical location;

place the avatar in the virtual world;

instruct the second user device to display the avatar in an augmented reality view of the virtual world to a second user accessing the virtual world through a second user device from a second real geographical location, the second user device comprising at least a second head mounted display device, wherein the avatar is animated based on at least input from the first head mounted display device;

identify one or more inanimate real world objects of the plurality of real world objects for display by at least receiving one or more inputs from the first user to select one or more inanimate real world objects of the plurality of inanimate real world objects for display or receiving one or more inputs from the first user to remove one or more inanimate real world objects of the plurality of inanimate real world objects from display;

instruct the second user device to display one or more virtual representations of the one or more inanimate real world objects as the one or more inanimate real world objects appear in the first real geographical location to the second user through the second user device, the one or more inanimate real world object sensed at the first real geographic location by the first user device appears to be physically present in the augmented reality view of the virtual world displayed to the second user using the second user device, thereby allowing the second user to experience the first real geographical location through the second user device from the second real geographical location; and

facilitate interaction between the first user and second user in the virtual world through the first user device and the second user device.

18. The system of claim 17, further comprising a group of cameras configured to view the first real geographical location to facilitate creation of a dense digital model.

19. The system of claim 17, wherein the first head mounted display further comprises a camera configured to capture an image of a face of the first user and a microphone configured to capture a voice of the first user, wherein the server processor is configured to animate the avatar based at least in part on an interaction the first user has with the first user device and processed using an avatar emotion engine, the interaction comprising voice inflections, head movements, body temperature, heart rate, facial recognition data.

20. The system of claim 17, wherein digitizing input from the first head mounted display device and passing that to remote avatar uses less bandwidth then passing captured video.