Methods and apparatus for delivering content and/or playing back content

• US 9,918,136 B2
• Filed: 05/29/2015
• Issued: 03/13/2018
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Primary Examiner(s): Bibbins, Latanya
• Application Number:
  US14/726,431
• Publication Number:
  US 20150346832A1
• Time in Patent Office: 1,019 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/371 for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis
• CPC Class Codes: H04N 19/134 characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/39 involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72

Abstract

Content delivery and playback methods and apparatus are described. The methods and apparatus are well suited for delivery and playback of content corresponding to a 360 degree environment and can be used to support streaming and/or real time delivery of 3D content corresponding to an event, e.g., while the event is ongoing or after the event is over. Portions of the environment are captured by cameras located at different positions. The content captured from different locations is encoded and made available for delivery. A playback device selects the content to be received based in a user'"'"'s head position.

Claims

1. A method, comprising:

operating a content playback device including an interface including a receiver to receive content stream information including information identifying a plurality of content streams, said plurality of content streams including at least a first content stream providing content corresponding to a first portion of an environment and a second content stream corresponding to a second portion of the environment, said second portion of the environment being a different portion of the environment than said first portion;

operating the content playback device to store, in memory, said content stream information;

operating a processor in said content playback device to perform selecting, based on a head position of a user, which of the plurality of content streams to receive for use in playback at a first time; and

operating the content playback device to receive one or more selected content streams to use for playback.

2. The method of claim 1, wherein said selecting includes:

operating the playback device to perform prioritizing content streams based on the head position of the user.

3. The method of claim 2, wherein said prioritizing includes:

identifying one or more content streams communicating content corresponding to a current field of view of the user, said field of view being a field of view in said environment; and

prioritizing one or more content streams providing portions of a scene corresponding to the current field of view based on the size of portions of the environment individual content streams provide.

4. The method of claim 3, wherein prioritizing content streams further includes:

prioritizing one or more additional content streams communicating content corresponding to portions outside said current field of view based on at least one of proximity of communicated image portions to the current field of view or direction of head rotation.

5. The method of claim 4,

wherein said prioritizing one or more additional content streams communicating content corresponding to portions outside said current field of view includes:

prioritizing at least one stream corresponding to a portion of said environment which is not displayed to a user of said playback device and which is outside said current field of view; and

prioritizing one or more additional content streams communicating content corresponding to portions outside said current field of view based on proximity of communicated image content to the current field of view, a content stream communicating image content in close proximity to an area of the current field of view being assigned a higher priority than a content stream outside and further away from the current field of view.

6. The method of claim 4, wherein said prioritizing one or more additional content streams is based on a direction of head rotation of the user, a content stream providing image content outside the current field of view but in the direction of head rotation being assigned a higher priority than another content stream providing image content outside the current field of view in a direction away from the direction of head rotation.

7. The method of claim 2, wherein at least some of said plurality of content streams communicate separate encoded versions of different portions of a 360 degree environment, said plurality of content streams including at least one content stream corresponding to each portion of the 360 degree environment;

and

wherein selecting content streams includes selecting multiple content streams corresponding to different portions of said 360 degree environment.

8. The method of claim 2, wherein said step of selecting further comprises:

determining, at the playback device, bandwidth available for receiving content streams; and

selecting from the plurality of content streams a single content stream corresponding to the first portion of the environment and a single content stream corresponding to the second portion of the environment, based on the determined amount of available bandwidth, the plurality of content streams including multiple content streams corresponding to the first portion of the environment and multiple content streams corresponding to the second portion of the environment.

9. The method of claim 8, further comprising:

determining, at the playback device, bandwidth for at least one content stream based on a bandwidth constraint communicated to said playback device; and

wherein said selecting from a plurality of content streams is based on the determined bandwidth for said at least one content stream.

10. The method of claim 2, wherein said playback device includes a head mounted display, the position of the head mounted display corresponding to the head position of the user.

11. The method of claim 1,

wherein receiving content stream information including information identifying a plurality of content streams includes receiving, at the playback device, guide information including information on content streams that the playback device may select to receive, said guide information including information listing content streams for a plurality of different events, said information including stream identifiers for multiple content streams corresponding to a first event, said stream identifiers including a first content stream identifier identifying a first content stream corresponding to a first portion of the environment at which said first event occurs and a second stream identifier identifying a second content stream corresponding to a second portion of the environment where said first event occurs and a third content stream identifier corresponding to another content stream corresponding to the second portion of said environment where the first event occurs, said information further including information indicating a data rate of the second content stream corresponding to a second portion of the environment where said first event occurs and another data rate, said another data rate being a data rate of the another stream identified by the third content stream identifier which corresponds to the second portion of said environment where the first event occurs, said data rate of the second content stream and said another data rate being different.

12. The method of claim 11, wherein said guide information includes, for a first content stream, one of a multicast address of a multicast group which can be joined to receive the first content stream, information which can be used to request access to a switched digital video channel used to provide the first content stream, or channel tuning information which can be used to control a tuner of the playback device to tune to a broadcast channel on which said first content stream is broadcast.

13. The method of claim 1, wherein said environment is an environment in which cameras captured images which were encoded to produce said plurality of content streams.

14. The method of claim 1, wherein said playback device is a head mounted playback device.

15. The method of claim 1,

wherein the environment is a 360 degree environment;

and

wherein the method further comprises;

decoding received content to generate an image for each portion of the 360 degree environment;

storing an image for each portion of the 360 degree environment in a buffer; and

rendering, using the buffered images, a final output image as needed based on the current field of view of a user of the playback device, the final output image including less the full 360 degree environment.

16. The method of claim 15, wherein the final rendered image is a left eye image, the method further comprising:

rendering a different right eye image; and

displaying both said left and right eye images to a user of the playback device.

17. The method of claim 1, further comprising:

operating the playback device to initialize, during an initialization phase of operation, a viewing position of the user to be a forward zero degree environmental viewing position by setting a detected user'"'"'s head position to be the forward zero degree viewing position.

18. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control a playback system to:

operate an interface including a receiver to receive content stream information including information identifying a plurality of content streams, said plurality of content streams including at least a first content stream providing content corresponding to a first portion of an environment and a second content stream corresponding to a second portion of the environment, said second portion of the environment being a different portion of the environment than said first portion;

store, in memory, said content stream information;

select, based on a head position of a user, which of the plurality of content streams to receive for use in playback at a first time; and

receive one or more selected content streams to use for playback.

19. A content playback device, comprising:

a receiver;

memory including information identifying a plurality of content streams received by said receiver, said plurality of content streams including at least a first content stream providing content corresponding to a first portion of an environment and a second content stream corresponding to a second portion of the environment, said second portion of the environment being a different portion of the environment than said first portion;

a processor in said content playback device configured to perform the steps of;

selecting, based on a head position of a user, which of the plurality of content streams to receive for use in playback at a first time; and

controlling the content playback device to receive one or more selected content streams to use for playback; and

controlling the content playback device to display a portion of the received content to a user of the playback device.

20. The content playback device of claim 19,

further comprising:

a head mounted display; and

wherein the processor is further configured to perform, as part of said selecting;

prioritizing content streams based on the head position of the user, said head position of the user corresponding to the position of the head mounted display.

21. The content playback device of claim 20, wherein said processor is configured to perform as part of said prioritizing:

identifying one or more content streams communicating content corresponding to a current field of view of the user, said field of view being a field of view in said environment; and

prioritizing one or more content streams providing portions of a scene corresponding to the current field of view based on the size of portions of the environment that individual content streams provide.

22. The content playback device of claim 21, wherein the processor is further configured to perform as part of prioritizing content streams:

prioritizing one or more additional content streams communicating content corresponding to portions outside said current field of view based on at least one of proximity of communicated image portions to the current field of view or direction of head rotation.

23. The content playback device of claim 22,

wherein said processor is configured as part of said prioritizing one or more additional content streams communicating content corresponding to portions outside said current field to perform the steps of:

prioritizing at least one stream corresponding to a portion of said environment which is not displayed to a user of said playback device and which is outside said current field of view; and

prioritizing one or more additional content streams communicating content corresponding to portions outside said current field of view based on proximity of communicated image content to the current field of view, a content stream communicating image content in close proximity to an area of the current field of view being assigned a higher priority than a content stream outside and further away from the current field of view.