Immersive virtual reality production and playback for storytelling content

• US 10,276,211 B2
• Filed: 09/27/2017
• Issued: 04/30/2019
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Warner Bros Entertainment Incorporated (Warner Bros. Discovery, Inc.)
• Original Assignee: Warner Bros Entertainment Incorporated (Warner Bros. Discovery, Inc.)
• Inventors: Gewickey, Gregory I., Ostrover, Lewis S., Smith, Michael, Zink, Michael
• Primary Examiner(s): Adams, Eileen M
• Application Number:
  US15/717,561
• Publication Number:
  US 20180025753A1
• Time in Patent Office: 580 Days
• Field of Search: None
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/388 Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume

Abstract

Methods for digital content production and playback of an immersive stereographic video work provide or enhance interactivity of immersive entertainment using various different playback and production techniques. “Immersive stereographic” may refer to virtual reality, augmented reality, or both. The methods may be implemented using specialized equipment for immersive stereographic playback or production. Aspects of the methods may be encoded as instructions in a computer memory, executable by one or more processors of the equipment to perform the aspects.

Claims

1. A method for controlling a display of an immersive stereographic content display apparatus, the method comprising:

processing, by a processor of the apparatus, sensor data indicating at least one user biometric from at least one sensor coupled to the processor during performance of an immersive stereographic video work playing on the apparatus for a user;

determining, during the performance based on the at least one user biometric, a virtual position of a region of visual focus of the user in a virtual environment defined by the immersive stereographic video work, at least in part by sensing that the user is viewing a limited portion of the display for a qualifying period of time based on the sensor data; and

controlling output of the apparatus in response to the determining.

2. The method of claim 1, wherein the controlling output comprises adjusting during the performance at least one of a depth of focus of the region of visual focus or an apparent focal length on a display of the immersive stereographic content display apparatus, based on the sensor data indicating that the region of visual focus exists.

3. The method of claim 2, wherein adjusting at least one of the depth of focus or apparent focal length comprises transitioning from a deep focus to a shallow focus, holding the shallow focus for a period of time, and then transitioning back to a deep focus.

4. The method of claim 2, wherein adjusting at least one of the depth of focus or apparent focal length comprises shifting a focus point from an original depth towards a foreground object, holding the focus point on the foreground object for a period of time, and then shifting the focus point to a predetermined depth.

5. The method of claim 2, wherein adjusting at least one of the depth of focus or apparent focal length comprises shifting a focus point from an original depth towards a background object, holding the focus point on the background object for a period of time, and then shifting the focus point to a predetermined depth.

6. The method of claim 2, wherein adjusting at least one of the depth of focus or apparent focal length comprises changing the apparent focal length from an initial length to a second length, holding the apparent focal length at the second length for a period of time, and then changing the apparent focal length to a predetermined length.

7. The method of claim 2, wherein adjusting at least one of the depth of focus or apparent focal length comprises at least one of constructing and displaying a video segment based on photogrammetry metadata of the immersive stereographic video work for the predetermined period, or displaying prerecorded video segments characterized by an adjusted depth of focus or focal length for the predetermined period.

8. The method of claim 1, wherein determining the virtual position of the region of visual focus comprises processing the sensor data indicating at least one of a user head orientation, a user eye orientation, or a user eye surface shape.

9. The method of claim 8, further comprising receiving the sensor data from the sensor comprising at least one of:

an image sensor, an accelerometer, an interferometer, or a microphone.

10. The method of claim 1, wherein the determining the virtual position of the region of visual focus comprises sensing that the user is viewing a limited portion of the display for a qualifying period of time, based on the sensor data.

11. The method of claim 1, wherein controlling the output comprises selecting a video segment from a plurality of possible video segments, wherein each of the possible video segments depicts a different narrative element of a story line.

12. The method of claim 1, wherein controlling the output comprises selecting a video segment from a plurality of possible video segments, wherein each of the possible video segments depicts an event happening at a corresponding one of different locations of the virtual environment.

13. An apparatus for controlling a display of an immersive stereographic content display apparatus, the apparatus comprising a processor coupled to a memory and to at least one sensor, the memory holding program instructions that when executed by the processor cause the apparatus to perform:

processing sensor data indicating at least one user biometric from at least one sensor coupled to the processor during performance of an immersive stereographic video work playing on the apparatus for a user;

determining, during the performance based on the at least one user biometric, a virtual position of a region of visual focus of the user in a virtual environment defined by the immersive stereographic video work, at least in part by sensing that the user is viewing a limited portion of the display for a qualifying period of time based on the sensor data; and

controlling output of the apparatus in response to the determining.

14. The apparatus of claim 13, wherein the memory holds further instructions for controlling the output during the performance at least in part by adjusting at least one of a depth of focus of the region of visual focus or an apparent focal length on a display of the immersive stereographic content display apparatus, based on the sensor data indicating that the region of visual focus exists.

15. The apparatus of claim 13, wherein the memory holds further instructions for determining the virtual position of the region of visual focus at least in part by processing the sensor data indicating at least one of a user head orientation, a user eye orientation, or a user eye surface shape.

16. The apparatus of claim 13, wherein the memory holds further instructions for determining the virtual position of the region of visual focus at least in part by sensing that the user is viewing a limited portion of the display for a qualifying period of time, based on the sensor data.

17. The apparatus of claim 13, wherein the memory holds further instructions for controlling the output at least in part by selecting a video segment from a plurality of possible video segments, wherein each of the possible video segments depicts a different narrative element of a story line.

18. A method for controlling a display of an immersive stereographic content display apparatus, the method comprising:

processing, by a processor of the apparatus, sensor data from at least one sensor coupled to the processor during performance of an immersive stereographic video work playing on the apparatus for a user;

determining, during the performance, a virtual position of a region of visual focus of the user in a virtual environment defined by the immersive stereographic video work, at least in part by sensing that the user is viewing a limited portion of the display for a qualifying period of time based on the sensor data; and

controlling output of the apparatus in response to the determining, wherein controlling the output comprises driving an audio transducer with a signal that produces a directional cue indicating a direction of interest for the user.

19. The method of claim 18, wherein the directional cue is selected from the group consisting of spoken language or a sound field configured to be perceived as originating from a particular location of the virtual environment.

20. An apparatus for controlling a display of an immersive stereographic content display apparatus, the apparatus comprising a processor coupled to a memory and to at least one sensor, the memory holding program instructions that when executed by the processor cause the apparatus to perform:

processing sensor data from at least one sensor coupled to the processor during performance of an immersive stereographic video work playing on the apparatus for a user;

determining, during the performance, a virtual position of a region of visual focus of the user in a virtual environment defined by the immersive stereographic video work, at least in part by sensing that the user is viewing a limited portion of the display for a qualifying period of time based on the sensor data; and

controlling output of the apparatus in response to the determining at least in part by driving an audio transducer with a signal that produces a directional cue indicating a direction of interest for the user.