IMMERSIVE VIRTUAL REALITY PRODUCTION AND PLAYBACK FOR STORYTELLING CONTENT

• US 20160191893A1
• Filed: 12/05/2015
• Published: 06/30/2016
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Warner Bros Entertainment Incorporated (Warner Bros. Discovery, Inc.)
• Original Assignee: Warner Bros Entertainment Incorporated (Warner Bros. Discovery, Inc.)
• Inventors: Gewickey, Gregory I., Ostrover, Lewis S., Smith, Michael, Zink, Michael
• Granted Patent: US 9,997,199 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/388 Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume

Abstract

Methods for digital content production and playback of an immersive stereographic video work provide or enhance interactivity of immersive entertainment using various different playback and production techniques. “Immersive stereographic” may refer to virtual reality, augmented reality, or both. The methods may be implemented using specialized equipment for immersive stereographic playback or production. Aspects of the methods may be encoded as instructions in a computer memory, executable by one or more processors of the equipment to perform the aspects.

Claims

1. A method for controlling a story line of immersive stereographic content, the method comprising:

collecting, in a computer memory, sensor data from a plurality of immersive stereographic client devices indicating at least one user biometric during performance of a prerecorded stereographic video work playing on the immersive stereographic client devices;

correlating, by at least one computer processor, the sensor data to ones of segment identifiers recorded in a data structure, each of the segment identifiers correlated to a different one of prerecorded immersive stereographic video segments comprising the prerecorded immersive stereographic video work; and

defining, by the at least one computer processor, an alternative story line in the form of an electronic data file for a modified version of the prerecorded immersive stereographic video, based on the correlating and on a library of immersive stereographic video segments each associated with an alternative story line branch.

2. The method of claim 1, further comprising selecting the at least one user body metric from the group consisting of:

a user head orientation, a user eye orientation, or a user eye surface shape.

3. The method of claim 1, further comprising selecting the at least one user body metric from the group consisting of:

heart rate or skin temperature.

4. The method of claim 1, further comprising determining a region of high audience interest based on the sensor data, and increasing rendering resources allocated to the region based on a measure of the audience interest.

5. The method of claim 4, further comprising determining the measure of the audience interest based on the sensor data from the plurality of immersive stereographic client devices while in use by a corresponding plurality of different users.

6. The method of claim 1, further comprising distributing the prerecorded immersive stereographic video work as part of a real-time rendered application package, wherein playing of the prerecorded video work is triggered by a user-controllable event defined by the rendered application.

7. The method of claim 6, further comprising altering an amount or timing that the prerecorded immersive stereographic video work is played relative to the real-time rendered application package, based on the sensor data.

8. The method of claim 1, further comprising storing a series of versions of the prerecorded immersive stereographic video work in an electronic data storage component, wherein each of the versions is based on a different set of the sensor data.

9. A method for controlling a display of an immersive stereographic content display apparatus, the method comprising:

processing, by a processor of the apparatus, sensor data from at least one sensor coupled to the processor during performance of a immersive stereographic video work playing on the apparatus for a user;

determining, during the performance, a virtual position of a region of visual focus in a virtual environment defined by the immersive stereographic video work, based on the sensor data; and

controlling output of the apparatus in response to the determining.

10. The method of claim 9, wherein the controlling output comprises adjusting at least one of a depth of focus of the region of visual focus or an apparent focal length on a display of the immersive stereographic content display apparatus, based on the sensor data indicating that the region of visual focus exists.

11. The method of claim 10, wherein adjusting at least one of the depth of focus or apparent focal length comprises transitioning from a deep focus to a shallow focus, holding the shallow focus for a period of time, and then transitioning back to a deep focus.

12. The method of claim 10, wherein adjusting at least one of the depth of focus or apparent focal length comprises shifting a focus point from an original depth towards a foreground object, holding the focus point on the foreground object for a period of time, and then shifting the focus point to a predetermined depth.

13. The method of claim 10, wherein adjusting at least one of the depth of focus or apparent focal length comprises shifting a focus point from an original depth towards a background object, holding the focus point on the background object for a period of time, and then shifting the focus point to a predetermined depth.

14. The method of claim 10, wherein adjusting at least one of the depth of focus or apparent focal length comprises changing the apparent focal length from an initial length to a second length, holding the apparent focal length at the second length for a period of time, and then changing the apparent focal length to a predetermined length.

15. The method of claim 10, wherein adjusting at least one of the depth of focus or apparent focal length comprises at least one of constructing and displaying a video segment based on photogrammetry metadata of the immersive stereographic video work for the predetermined period, or displaying prerecorded video segments characterized by an adjusted depth of focus or focal length for the predetermined period.

16. The method of claim 9, wherein determining the virtual position of the region of visual focus comprises processing the sensor data indicating at least one of a user head orientation, a user eye orientation, or a user eye surface shape.

17. The method of claim 16, further comprising receiving the sensor data from the sensor comprising at least one of:

an image sensor, an accelerometer, an interferometer, or a microphone.

18. The method of claim 9, wherein the determining the virtual position of the region of visual focus comprises sensing that the user is viewing a limited portion of the display for a qualifying period of time, based on the sensor data.

19. The method of claim 9, wherein controlling the output comprises selecting a video segment from a plurality of possible video segments, wherein each of the possible video segments depicts a different narrative element of a story line.

20. The method of claim 9, wherein controlling the output comprises selecting a video segment from a plurality of possible video segments, wherein each of the possible video segments depicts an event happening at a corresponding one of different locations of the virtual environment.

21. The method of claim 9, wherein controlling the output comprises driving an audio transducer with a signal that produces a directional cue indicating a direction of interest for the user.

22. The method of claim 21, wherein the directional cue is selected from the group consisting of spoken language or a sound field configured to be perceived as originating from a particular location of the virtual environment.

23. A method for controlling a display of an immersive stereographic content display apparatus, the method comprising:

playing a prerecorded immersive stereographic video work on an immersive stereographic content display apparatus;

receiving, during the playing, user input indicating an interest in an object appearing in a display of the immersive stereographic content display apparatus;

identifying at least one of a three-dimensional model or a rendered image of the three-dimensional model from a computer memory, based on an identity of the object; and

combining a rendered image of the three-dimensional model with video data of the prerecorded immersive stereographic video work, wherein the rendered image replaces an image of the object.

24. The method of claim 23, wherein receiving the input comprises processing sensor data indicating at least one of a gaze direction or an intended a view direction of a user of the immersive stereographic content display apparatus.

25. The method of claim 23, wherein receiving the input comprises receiving input from a pointing device manually operated by the user.

26. The method of claim 23, further comprising changing a position, an orientation, a position relative to user viewpoint, or an orientation relative to user viewpoint, of the three-dimensional model in response to user input, whereby the rendered image of the three-dimensional model reveals a view of the object that is not included in the prerecorded immersive stereographic video work without the rendered image.

27. The method of claim 26, wherein the three-dimensional model is selected from one of a prop for replacing a static object or a figure movable by an armature for replacing a character.

28. A method for producing video content for output by an immersive stereographic content display apparatus, the method comprising:

capturing a scene using an array of video cameras, providing time-correlated video data;

creating an image map based on each frame of the time-correlated video data, and combining each image map into a mapped frame set;

generating metadata describing a mapping scheme used for the mapped frame set; and

configuring immersive stereographic video data comprising the mapped frame set and the metadata for play in an immersive stereographic content display apparatus.

29. The method of claim 28, further comprising configuring the array as a close-packed planar array.

30. The method of claim 28, wherein creating the image map comprising projecting a combination of each frame of the time-correlated video data into at least a portion of a spherical map.

31. The method of claim 28, wherein creating the image map comprising defining separate tracks of combined image data.

32. A method for producing video content for output by an immersive stereographic content display apparatus, the method comprising:

capturing a scene using an array of video cameras, providing time-correlated video data;

defining a framespace based on combining frame sets from the array of video cameras, wherein each of the frame sets is correlated based on time and on a position and orientation of a corresponding one of the array of video cameras;

defining a framespace path across different frame sets of the framespace, the framespace path configured for providing a video stream based on the framespace.

33. The method of claim 32, further comprising generating a video stream based on the framespace and the framespace path.

34. The method of claim 33, further comprising generating the video stream configured for output by an immersive stereographic content display apparatus.

35. The method of claim 32, further comprising interpolating video frames for transitioning between ones of the frame sets.