Methods and apparatus for streaming content

• US 9,699,437 B2
• Filed: 04/14/2014
• Issued: 07/04/2017
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Moss, Alan McKay, Cole, David Michael
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): TEKLE, DANIEL T
• Application Number:
  US14/252,289
• Publication Number:
  US 20150249813A1
• Time in Patent Office: 1,177 Days
• Field of Search: 386248
• CPC Class Codes: G09G 2340/0435 Change or adaptation of the frame rate of the video stream
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G11B 27/30 on the same track as the main recording
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Methods and apparatus for streaming content corresponding to a 360 degree field of view are described. The methods and apparatus of the present invention are well suited for use with 3D immersion systems and/or head mounted display which allow a user to turn his or her head and see a corresponding scene portion. The methods and apparatus can support real or near real time streaming of 3D image content corresponding to a 360 degree field of view.

Claims

1. A method of delivering content corresponding to a 360 degree scene, the method comprising:

operating an encoding system including at least one encoder to encode N different portions of a 360 degree scene to generate at least N different encoded portions of the 360 degree scene, at least one of said N different encoded portions of the 360 degree scene corresponding to a front portion of said 360 degree scene and another one of said N different encoded portions of the 360 degree scene corresponding to another portion of said 360 degree scene;

storing, in a storage device, the N different encoded portions;

receiving, at a content server, a request for content from a playback device corresponding to a user, said playback device including a head mounted display;

selecting, at the content server, which of said N different encoded portions to stream to the playback device corresponding to said user during a first time interval based on the request for content, a head of said user being directed in a first direction during the first time interval, the selected different encoded portions including a scene portion to which the head of the user is directed during the first time interval; and

operating the content server to stream the selected encoded portions to the playback device.

2. The method of claim 1,

wherein selecting which of said N different encoded portions to stream to a playback device corresponding to said user during a first time interval includes:

selecting at least two different encoded portions of said 360 degree scene to be streamed to said user during said first time interval, said at least two different encoded portions including a scene portion to which the head of the user is directed and at least one scene portion adjacent the scene portion to which the head of the user is directed.

3. The method of claim 2, wherein selecting which of said N different encoded portions to stream to a playback device corresponding to said user during a first time interval is based on the current viewing angle of said user and includes:

determining a number of different encoded portions to stream based on a data rate available for streaming content to the user.

4. The method of claim 3, further comprising:

determining which of a plurality of versions of said second scene portion to stream based on the current data rate available for streaming content.

5. The method of claim 2, wherein selecting at least two different encoded portions of said 360 degree scene to be streamed to said user during said first time interval includes:

selecting a first scene portion to which a current viewing angle of the user corresponds; and

selecting a second scene portion which is directly adjacent said first scene portion.

6. The method of claim 5, wherein said second scene portion is selected as a function of the current viewing angle of the user, said second scene portion being a scene portion adjacent said first scene portion which is closest to the angle within said first scene portion indicated by said viewing angle.

7. The method of claim 5, wherein said first scene portion corresponds to at least 120 degrees of the 360 degree scene;

and

wherein said second scene portion corresponds to at least a 30 degree scene portion.

8. The method of claim 3, wherein said step of determining a number of different encoded portions to stream includes determining to stream at least three different portions;

and

wherein said at least three different portions in combination cover at least 170 degrees of the 360 degree scene.

9. The method of claim 8, further comprising:

identifying scene portions which have not been transmitted to the playback device during a refresh interval; and

transmitting an updated version of the identified scene portions which were not transmitted to the playback device during the refresh interval.

10. The method of claim 1, further comprising:

communicating at least a sufficient number of the N different encoded portions to said playback device on a periodic basis to allow the playback device to fully refresh a 360 degree version of said scene at least once during each refresh period.

11. The method of claim 1, further comprising:

receiving, at the content server, user head position information; and

wherein selecting, at the content server, which of said N different encoded portions to stream to the playback device is based on the received user head position information.

12. The method of claim 1, wherein

operating the content server to stream the selected encoded portions to the playback device includes operating the content server to stream different encoded portions to refresh content corresponding to different portions of the environment at different rates.

13. The method of claim 1, further comprising:

wherein less than the entire 360 degree scene is streamed during at least some period of time periods; and

wherein the method further includes sending a 360 degree scene update to the playback device at predetermined time intervals.

14. The method of claim 13, wherein the 360 degree scene update includes portions of the scene which were not updated since the last 360 degree scene update but does not include portions of the 360 degree scene which were updated since a last 360 degree scene update.

15. A content delivery apparatus for delivering content corresponding to a 360 degree scene, comprising:

an encoding system including at least one encoder, said encoding system encoding N different portions of a 360 degree scene to generate at least N different encoded portions of the 360 degree scene, at least one of said N different encoded portions of the 360 degree scene corresponding to a front portion of said 360 degree scene and another one of said N different encoded portions of the 360 degree scene corresponding to another portion of said 360 degree scene;

a storage device to store the N different encoded portions;

a receiver to receive a request for content from a playback device corresponding to a user, said playback device including a head mounted display;

a selection module to select which of said N different encoded portions to stream to the playback device corresponding to said user during a first time interval based on the request for content, a head of said user being directed in a first direction during the first time interval, the selected different encoded portions including a scene portion to which the head of the user is directed during the first time interval; and

a transmitter to stream the selected encoded portions to the playback device.

16. The content delivery apparatus of claim 15,

wherein said selection module is further configured to select at least two different portions of said 360 degree scene to be streamed to said user during said first time interval, as part of selecting which of said N different encoded portions to stream to the playback device corresponding to said user during the first time interval.

17. The content delivery apparatus of claim 15, wherein said selection module is further configured to determine a number of different portions to stream based on a data rate available for streaming content to the user and select the different portions based on the direction of the head of said user.

18. A non-transitory computer readable medium for use in a content delivery apparatus for delivering content corresponding to a 360 degree scene, the non-transitory computer readable medium comprising:

instructions for controlling a processor to control one or more encoders to encode N different portions of a 360 degree scene to generate at least N different encoded portions of the 360 degree scene, at least one of said N different encoded portions of the 360 degree scene corresponding to a front portion of said 360 degree scene and another one of said N different encoded portions of the 360 degree scene corresponding to another portion of said 360 degree scene;

instructions for controlling the processor to store, in a memory, the N different encoded portions;

instructions for controlling the processor to control an interface that receives a request for content from a playback device corresponding to a user, said playback device including a head mounted display;

instructions for controlling the processor to select which of said N different encoded portions to stream to the playback device corresponding to said user during a first time interval based on the request for content, a head of said user being directed in a first direction during the first time interval, the selected different encoded portions including a scene portion to which the head of the user is directed during the first time interval; and

instructions for controlling the processor to control a transmitter to stream the selected encoded portions to the playback device.