SYSTEMS AND ASSOCIATED METHODS FOR CREATING A VIEWING EXPERIENCE

• US 20180350136A1
• Filed: 05/31/2018
• Published: 12/06/2018
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Live CGI, Inc.
• Original Assignee: Temavr, LLC
• Inventors: Rowley, Marc
• Granted Patent: US 10,789,764 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Systems and processes generate a viewing experience by determining location data and movement data of (a) at least one object and (b) at least one participant within an event area. A three-dimensional model of the event area, the participant and the object is determined based upon the location data and the movement data. A viewpoint of a spectator defines an origin, relative to the three-dimensional model, and a direction of the viewing experience. The viewing experience is generated for the viewpoint at least in part from the three-dimensional model to include one or more of augmented reality, mixed reality, extended reality, and virtual reality.

Claims

1. A process for generating a viewing experience, comprising steps of:

determining location data and movement data of (a) an object and (b) a participant within an event area;

determining a three-dimensional model of the event area, the participant and the object based upon the location data and the movement data;

determining a viewpoint of a spectator, the viewpoint defining an origin, relative to the three-dimensional model, and a direction of the viewing experience; and

generating the viewing experience for the viewpoint at least in part from the three-dimensional model.

2. The process of claim 1, the step of generating comprising blurring parts of the viewing experience that are less important to reduce latency of generating the viewing experience.

3. The process of claim 1, the step of determining the location data and the movement data further comprising capturing light-field data relative to the object and the participant to enhance the three-dimensional model.

4. The process claim 1, the step of determining the viewpoint further comprising capturing light-field data relative to the viewpoint to enhance the three-dimensional model, wherein the light-field data comprises light intensity, light direction, and light color.

5. The process of claim 1, the step of determining the location data and the movement data further comprising determining relative location data, of the object and the participant, with respect to one or more of (i) a permanent object location at the event area, (ii) a second object at the event area, (iii) a second participant at the event area, and (iv) a secondary grid at the event area.

6. The process of claim 1, the step of determining the viewpoint further comprising determining relative location data, of the viewpoint, with respect to one or more of (i) a permanent object location at the event area, (ii) the at least one object, (iii) the at least one participant, and (iv) a secondary grid at the event area.

7. The process of claim 6, the secondary grid being a secondary virtual grid positioned between the viewpoint and the object or the participant.

8. The process of claim 1, further comprising:

receiving primary images from a plurality of cameras positioned at the event area; and

mapping at least one of the primary images to the three-dimensional model.

9. The process of claim 8, the step of mapping further comprising mapping light field data to the three-dimensional model.

10. The process of claim 8, the step of determining the viewpoint further comprising determining, based on viewing directives received from the spectator, a virtual camera defining a virtual origin, relative to the three-dimensional model, and a virtual direction of the viewing experience.

11. The process of claim 10, further comprising:

generating a virtual image, having the object and/or the participant, based upon (i) the three-dimensional model and (ii) the viewpoint or the virtual camera; and

sending one or both of (i) the three-dimensional model and (ii) at least a portion of the virtual image to a viewing device configured to provide the viewing experience.

12. The process of claim 11, further comprising:

determining when an obstruction is located between (i) one of the viewpoint and the virtual camera and (ii) one of the object and the participant; and

adding at least a portion of the virtual image, corresponding to a location of the obstruction, to the viewing experience to at least partially remove the obstruction from the viewing experience.

13. The process of claim 8, the process further comprising:

within the three-dimensional model, determining, around each of the participant and the object, a virtual grid having a plurality of cells; and

the step of mapping further comprising;

mapping at least a portion of one of the primary images identified by at least one cell of the virtual grid corresponding to the participant or object.

14. The process of claim 13, the step of mapping further comprising mapping at least a portion of one of the primary images identified by a section of a secondary grid corresponding to the participant or object.

15. The process of claim 10, further comprising adding visual special effects and audible special effects to the viewing experience, the special effects being generated based upon one or both of (i) the location data and the movement data of the object and/or the participant and (ii) an occurrence of interest detected within the event area.

16. The process of claim 10, further comprising:

receiving sound feeds from a plurality of microphones positioned at the event area;

mapping the sound feeds to the three-dimensional model; and

generating the viewing experience to include sounds based upon the three-dimensional model.

17. The process of claim 10, further comprising providing haptic feedback to the spectator based at least in part upon one or more of (a) the virtual camera and the location data of the object and the participant, (b) an occurrence of interest detected within the event area and the visual and audio special effects, and (c) feedback from other spectators sharing the viewing experience.

18. A system for generating a free-viewpoint experience for a spectator, comprising:

a plurality of cameras positioned at an event area to capture primary images of the event area;

tracking apparatus configured to determine location data and movement data of (a) at least one object and (b) at least one participant within the event area; and

a server having a processor and memory storing machine readable instructions that when executed by the processor are capable of;

receiving the primary images from the plurality of cameras;

determining a three-dimensional model of the event area, the participant and the object based upon the location data and the movement data of the participant and the object; and

sending an output to a viewing device for providing the free-viewpoint experience, having at least one virtual image, to the spectator.

19. The system of claim 18, further comprising machine readable instructions that, when processed by the server, are capable of:

determining, based on viewing directives received from the spectator, a virtual camera defining an origin within the three-dimensional model and a direction of the free-viewpoint experience; and

generating the at least one virtual image having a portion of the three-dimensional model, based upon the virtual camera.

20. The system of claim 19, the output comprising one or both of the virtual image, and the three-dimensional model.