Real-time immersive mediated reality experiences

• US 10,089,785 B2
• Filed: 07/02/2015
• Issued: 10/02/2018
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Mindhive Inc.
• Original Assignee: Mindhive Inc.
• Inventors: Demirli, Oya, Dalke, George
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US14/790,641
• Publication Number:
  US 20160027209A1
• Time in Patent Office: 1,188 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

The invention relates to creating real-time, immersive mediated reality environments using real data collected from a physical event or venue. The invention provides a virtual participant with the ability to control their viewpoint and freely explore the venue, in real time by synthesizing virtual data corresponding to a requested virtual viewpoint using real images obtained from data collectors or sources at the venue. By tracking and correlating real and virtual viewpoints of virtual participants, physical objects, and data sources, systems and methods of the invention can create photo-realistic images for perspective views for which there is not physically present data source. Systems and methods of the invention also relate to applying effect objects to enhance the immersive experience including virtual guides, docents, text or audio information, expressive auras, tracking effects, and audio.

Claims

1. A system for creating a mediated reality environment, said system comprising a server computing system comprising a processor coupled to a tangible, non-transitory memory, the system operable to:

receive, in real-time, real viewpoint information for a plurality of data collectors located at a venue wherein one or more of the one or more data collectors comprise a computing device of a real participant, said computing device comprising a sensor selected from the group consisting of a GPS sensor, a wireless network sensor, an accelerometer, and a gyroscope wherein the real viewpoint information is determined using the sensor;

receive a virtual viewpoint from a computing device of a virtual participant, said computing device comprising a processor coupled to a tangible, non-transitory memory;

identify one or more of the one or more data collectors having a real viewpoint which intersects the virtual viewpoint;

receive one or more real-time images from the identified data collectors, said one or more real-time images comprising a plurality of real pixels;

create, using the server'"'"'s processor, a real-time virtual image comprising a plurality of virtual pixels and corresponding to the virtual viewpoint by using pixel information from the one or more real-time images wherein the virtual viewpoint'"'"'s location within the venue is different than the real viewpoint of the identified data collectors; and

cause the computing device of the virtual participant to display the real-time virtual image.

2. The system of claim 1, further operable to:

identify, using the server'"'"'s processor, one or more archived images comprising a plurality of archived pixels, said archived images having a corresponding real viewpoint which intersects with the virtual viewpoint;

retrieve one or more of the archived pixels;

use the one or more archived pixels to supplement the real-time virtual image using the server'"'"'s processor; and

cause the computing device of the virtual participant to display the supplemented real-time virtual image.

3. The system of claim 1 wherein the one or more data collectors comprise a microphone, said system further operable to:

receive real-time audio data from the microphone;

correlate the real-time audio data with the real viewpoint information for the data collector that generated it;

create, using the server'"'"'s processor, real-time virtual audio corresponding to the virtual viewpoint; and

cause the computing device of the virtual participant to emit the real-time virtual audio.

4. The system of claim 1 wherein one or more of the one or more data collectors comprise an optical sensor.

5. The system of claim 1 wherein the real viewpoint information and the virtual viewpoint comprise a latitudinal coordinate, a longitudinal coordinate, a field of view, and an orientation.

6. The system of claim 1 further operable to:

determine a stereoscopic virtual viewpoint capable of being combined with the virtual viewpoint in order to create a stereoscopic viewpoint;

create a stereoscopic real-time virtual image corresponding to the stereoscopic virtual viewpoint; and

cause the computing device of the virtual participant to simultaneously display the real-time virtual image and the stereoscopic real-time virtual image.

7. The system of claim 1 wherein the real-time virtual image is a 3-dimensional image.

8. The system of claim 3 wherein the real-time virtual audio includes a plurality of channels.

9. The system of claim 1 further operable to:

create an effect object;

assign a virtual location to the effect object which corresponds with a physical location at the venue; and

where the virtual location is observable from the virtual viewpoint, overlay the effect object onto the real-time virtual image.

10. The system of claim 9 wherein the effect object comprises an image of a real object or a computer generated image.

11. The system of claim 9 wherein the effect object is generated by the virtual participant.

12. The system of claim 9 wherein the effect object comprises a virtual representation of a virtual participant.

13. The system of claim 9 wherein the effect object may be manipulated by the virtual participant.

14. The system of claim 1 wherein the virtual viewpoint is manipulated through movement of the computing device of the virtual participant.

15. The system of claim 1 further operable to process the one or more real-time images using the server'"'"'s processor to enhance a feature of the one or more real-time images.

16. The system of claim 15 wherein the feature is selected from the group consisting of resolution, color, brightness, contrast, signal to noise ratio, smoothness, edge preservation, luminosity, and dynamic range.

17. The system of claim 1 wherein the server'"'"'s processor comprises a distributed computing system comprising a plurality of computing device processors in communication with each other through a communication network.

18. The system of claim 9, said system further operable to receive real-time biometric data for the virtual participant or a real participant from a biometric sensor;

wherein the effect object is determined by the real-time biometric data.

19. The system of claim 18 wherein the real-time biometric data is selected from the group consisting of temperature, heart rate, breathing rate, pupil dilation, brain activity, and bodily concentration of an analyte.

20. The system of claim 19 wherein the analyte is selected from the group consisting of oxygen, carbon dioxide, glucose, alcohol, and THC.