Systems and methods for a shared mixed reality experience

• US 9,940,897 B2
• Filed: 05/23/2014
• Issued: 04/10/2018
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: AWE Company Limited, Ryerson University
• Original Assignee: AWE Company Limited, Reyerson University
• Inventors: Guan, Ling, Kyan, Matthew, Krishna, Srinivas, Bond, Jason Raymond Thomas, Dong, Nan, Khan, Naimul Mefraz, Nan, Xiaoming, He, Yifeng, Biggs, Edward
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US14/893,630
• Publication Number:
  US 20160104452A1
• Time in Patent Office: 1,418 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/847 Cooperative playing, e.g. requiring coordinated actions from several players to achieve a common goal
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: G01S 5/0284 Relative positioning
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

A method for sharing a mixed reality experience (mixed reality content, mixed reality event) between one or more computing devices is disclosed. The method includes: determining a spatial location and a spatial orientation (spatial data) of the one or more computing devices each having a camera; mapping the (spatial) location and/or the spatial orientation (spatial data) of each of the one or more computing devices into a mixed reality manager; and presenting an event that is shared among the one or more computing devices, and, the presenting of the event is experienced simultaneously and varies among each of the one or more computing devices depending on the location or the orientation or both.

Claims

1. A method for sharing a mixed reality between one or more computing devices, comprising:

determining a location data and an orientation data of a camera input of the one or more computing devices;

improving the accuracy of the location data and the orientation data by using a secondary tracking system;

mapping the location and the orientation of each of the one or more computing devices into a mixed reality manager;

setting a state of a state machine that is configured to trigger the presentation of an event on the one or more computing devices once a condition is met;

changing the state of the state machine once the condition is met, thereby triggering the event on the one or more computing devices; and

presenting the event on the one or more computing devices, wherein the event is experienced on the one or more computing devices depending on the location or the orientation or both.

2. The method of claim 1, wherein:

the step of presenting further comprises;

registering the camera input with the event for aligning a real world view from the camera input with a virtual world view of the event.

3. The method of claim 1, wherein:

the step of presenting further comprises;

determining occlusion between one or more real objects in a real world view from a camera providing the camera input and one or more virtual objects in a virtual world view of the event.

4. The method of claim 1, wherein:

the step of presenting further comprises;

matching lighting between a real world view from the camera input and a virtual world view from the event.

5. The method of claim 1, wherein:

the event is an interactive performance.

6. The method of claim 1, wherein:

the one or more computing devices comprise one or more input devices for interacting with the event.

7. The method of claim 1, further comprising:

performing an audio content associated with the virtual world content.

8. The method of claim 1, further comprising:

performing an audio content.

9. The method of claim 1, further comprising:

determining spatial data of a device having a real world camera; and

aligning a virtual world camera in a virtual world with the real world camera using the spatial data of the device so that a point of view of the virtual world camera in the virtual world is aligned with a point of view of the real world camera.

10. The method of claim 1, wherein the secondary tracking system comprises a radio frequency (RF) receiver array deployed in a space and a RF transmitter on each of the one or more computing devices for tracking the computing device in the space.

11. The method of claim 1, wherein the secondary tracking system is configured to use simultaneous location and mapping (SLAM) to track the computing device in the space.

12. The method of claim 11, wherein the secondary tracking system is further configured to predict a changing spatial location data and spatial orientation data.

13. A system for sharing a mixed reality between one or more computing devices, each of the one or more computing devices providing an output, a camera input, a location data, and an orientation data of the camera input, the system comprising:

a non-transitory machine readable memory having machine executable instructions; and

one or more processors that access the memory and execute the machine readable instructions, the machine readable instructions comprising;

a mixed reality manager for coordinating an event among the one or more computing devices based on the location and a point of view for each of the one or more computing devices, the mixed reality manager having a state machine that is configured to trigger the presentation of an event on the one or more computing devices once a condition is met;

a mixed reality renderer for rendering the event on the output of the one or more computing devices once the condition is met, and the rendering of the event varies among each of the one or more computing devices depending on the location or the point of view or both; and

a secondary tracking system for improving the accuracy of the location data and the orientation provided by the computing devices.

14. The system of claim 13, wherein:

the mixed reality renderer is further configured to;

register the camera input with the event for aligning a real world view from the camera input with a virtual world view of the event.

15. The system of claim 13, wherein:

the mixed reality renderer is further configured to;

determine occlusion between one or more real world objects in a real world view and one or more virtual world objects in a virtual world view.

16. The system of claim 13, wherein:

the mixed reality renderer is further configured to;

match lighting between a real world view from the camera input and a virtual world view from the event.

17. The system of claim 13, wherein:

the event is an interactive performance.

18. The system of claim 13, wherein:

the one or more computing devices comprise one or more input devices for interacting with the event.

19. The system of claim 13, wherein:

a device is configured to perform an audio content associated with the virtual world content.

20. The system of claim 13, wherein:

a device is configured to perform an audio content.

21. The system of claim 13, further comprising:

an experience system configured to aligning a virtual world camera in a virtual world with the real world camera using the spatial data of the device so that a point of view of the virtual world camera in the virtual world is aligned with a point of view of the real world camera.

22. The system of claim 13, wherein the secondary tracking system comprises a radio frequency (RF) receiver array deployed in a space and a RF transmitter on each of the one or more computing devices for tracking the computing device in the space.

23. The system of claim 14, wherein the secondary tracking system is configured to use simultaneous location and mapping (SLAM) to track the computing device in the space.

24. The system of claim 23, wherein the secondary tracking system is further configured to predict a changing spatial location data and spatial orientation data.