METHODS AND APPARATUS FOR PROCESSING CONTENT BASED ON VIEWING INFORMATION AND/OR COMMUNICATING CONTENT

• US 20160360180A1
• Filed: 08/17/2016
• Published: 12/08/2016
• Est. Priority Date: 02/17/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M.
• Granted Patent: US 10,362,290 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/234381 : by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/44016 : involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4728 : for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 : involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Methods and apparatus for collecting user feedback information from viewers of content are described. Feedback information is received from viewers of content. The feedback indicates, based on head tracking information in some embodiments, where users are looking in a simulated environment during different times of a content presentation, e.g., different frame times. The feedback information is used to prioritize different portions of an environment represented by the captured image content. Resolution allocation is performed based on the feedback information and the content re-encoded based on the resolution allocation. The resolution allocation may and normally does change as the priority of different portions of the environment change.

Claims

1. A method comprising the steps of:

receiving video corresponding to an environment;

storing the video in a storage device;

transmitting at least some of the video to playback devices;

receiving, from the playback devices, viewing information indicating, on a per user basis, which portion of the environment a user was viewing at a given time during playback of the video;

determining, based on the received viewing information, relative importance of different portions of the environment at different times during playback of the video;

processing the stored video to control allocation of resolution to different portions of said environment based on the determined relative importance of different portions of the environment at different times, said processing producing encoded video; and

storing the encoded video or transmitting the encoded video to at least one playback device.

2. The method of claim 1, wherein the determining, based on the received viewing information, relative importance of different portions of the environment at different times during playback of the video includes:

determining, on a per frame time basis, for each of a plurality of different individual portions of the environment, a number of viewers having their view directed to the individual portion of the environment.

3. The method of claim 2, wherein the determining, based on the received viewing information, relative importance of different portions of the environment at different times during the video further includes:

ranking the plurality of different individual portions of the environment corresponding to a first frame time based on the determined number of viewers watching the plurality of different portions of the environment during the first frame time.

4. The method of claim 3, wherein ranking the different portions of the environment based on the determined numbers of viewers, includes generating, for the first frame time, a weighted priority ranking value for each of the different portions of the environment, said weighted priority value being based on viewing information corresponding to predetermined viewers whose identity is known and viewers of unknown identify, viewing information from predetermined viewers whose identity is known being weighted more heavily than viewing information from viewers whose identify is unknown.

5. The method of claim 4, further comprising:

selecting, based on a first data rate to be supported and weighted priority ranking values of different portions of the environment for the first frame time, a first resolution allocation pattern to use on a first video frame corresponding to the first frame; and

wherein processing the stored video includes;

performing a first resolution reduction operation on the first video frame in accordance with the selected first resolution allocation pattern on one or more portions of the first video frame to reduce the resolution of at least a portion of said first video frame in accordance with the selected first resolution allocation pattern to produce a first reduced resolution version of said first video frame; and

encoding the first reduced resolution version of the first video frame to generate a first encoded version of said first video frame.

6. The method of claim 5, wherein said first video frame is a left frame of a stereoscopic frame pair;

and wherein processing the stored video includes;

performing the first resolution reduction operation in accordance with the selected first resolution allocation pattern on one or more portions of a second video frame of said video to reduce the resolution of at least a portion of said second video frame in accordance with the selected first resolution allocation pattern to produce a first reduced resolution version of the second video frame, said first and second video frames being frames of a stereoscopic frame pair corresponding to the first frame time; and

encoding the first reduced resolution version of the second video frame.

7. The method of claim 6, further comprising:

prior to encoding the first reduced resolution version of the first video frame and the first reduced resolution version of the second video frame, packing the first reduced resolution version of the first video frame and the first reduced resolution version of the second video frame into a single frame.

8. The method of claim 5, further comprising:

selecting, based on a second data rate to be supported and weighted priority ranking values of different portions of the environment for the first frame time, a second resolution allocation pattern to use on the first video frame corresponding to the first frame time; and

and wherein processing the stored video further includes;

performing a second resolution reduction operation on the first video frame in accordance with the selected second resolution allocation pattern on one or more portions of the first video frame of said video to reduce the resolution of at least a portion of said first video frame in accordance with the second selected resolution allocation pattern to produce a second reduced resolution version of said first video frame; and

encoding the second reduced resolution version of the first video frame to generate a second encoded version of said first video frame.

9. The method of claim 8, wherein said first video frame is a left frame of a stereoscopic frame pair;

and wherein processing the stored video includes;

performing the second resolution reduction operation in accordance with the selected second resolution allocation pattern on one or more portions of the second video frame of said video to reduce the resolution of at least a portion of said second video frame in accordance with the selected second resolution allocation pattern to produce a second reduced resolution version of the second video frame; and

encoding the second reduced resolution version of the second video frame.

10. The method of claim 1, further comprising:

determining, based on the received viewing information, a frame time to switch from a first resolution allocation pattern to another resolution allocation pattern.

11. The method of claim 10, wherein said frame time to switch from said first resolution allocation pattern to another resolution allocation pattern corresponds to a key frame, with said another resolution allocation pattern to be used with said key frame.

12. The method of claim 11 wherein said key frame is an Intra-coded frame.

13. A system comprising:

a receiver configured to receive video corresponding to an environment;

storage device configured to store the video in a storage device;

a transmitter configured to transmit at least some of the video to playback devices;

wherein said receiver is further configured to receive, from the playback devices, viewing information indicating, on a per user basis, which portion of the environment a user was viewing at a given time during playback of the video;

a prioritizer configured to determine, based on the received viewing information, relative importance of different portions of the environment at different times during playback of the video;

a video processor configured to process the stored video to control allocation of resolution to different portions of said environment based on the determined relative importance of different portions of the environment at different times, said video processor producing encoded video;

wherein said storage device is further configured to store the encoded video; and

wherein said transmitter is further configured to transmit the encoded video to at least one playback device.

14. The system of claim 13, wherein said prioritizer is further configured, as part of being configured to determine relative importance of different portions of the environment at different times during playback of the video, to determine, on a per frame time basis, for each of a plurality of different individual portions of the environment, a number of viewers having their view directed to the individual portion of the environment.

15. The system of claim 14, wherein said prioritizer is further configured, as part of being configured to determine relative importance of different portions of the environment at different times during playback of the video, to rank the plurality of different individual portions of the environment corresponding to a first frame time based on the determined number of viewers watching the plurality of different portions of the environment during the first frame time.

16. The system of claim 15, wherein said prioritizer is further configured, as part of being configured to rank the plurality of different portions of the environment based on the determined numbers of viewers, to generate for the first frame time, a weighted priority ranking value for each of the different portions of the environment, said weighted priority value being based on viewing information corresponding to predetermined viewers whose identity is known and viewers of unknown identify, viewing information from predetermined viewers whose identity is known being weighted more heavily than viewing information from viewers whose identify is unknown.

17. The system of claim 16, further comprising:

a first resolution allocation selector configured to select, based on a first data rate to be supported and weighted priority ranking values of different portions of the environment for the first frame time, a first resolution allocation pattern to use on a first video frame corresponding to the first frame; and

a first resolution reduction device configured to perform a first resolution reduction operation on the first video frame in accordance with the selected first resolution allocation pattern on one or more portions of the first video frame to reduce the resolution of at least a portion of said first video frame in accordance with the selected first resolution allocation pattern to produce a first reduced resolution version of said first video frame; and

a first encoder configured to encode the first reduced resolution version of the first video frame to generate a first encoded version of said first video frame.

18. The system of claim 17, wherein said first video frame is a left frame of a stereoscopic frame pair;

and

wherein said first resolution allocation selector is further configured to perform the first resolution reduction operation in accordance with the selected first resolution allocation pattern on one or more portions of a second video frame of said video to reduce the resolution of at least a portion of said second video frame in accordance with the selected first resolution allocation pattern to produce a first reduced resolution version of the second video frame, said first and second video frames being frames of a stereoscopic frame pair corresponding to the first frame time; and

wherein said first encoder is further configured to encode the first reduced resolution version of the second video frame.

19. The system of claim 17, further comprising:

a second resolution allocation selector configured to select, based on a second data rate to be supported and weighted priority ranking values of different portions of the environment for the first frame time, a second resolution allocation pattern to use on the first video frame corresponding to the first frame time; and

a second resolution reduction device configured to perform a second resolution reduction operation on the first video frame in accordance with the selected second resolution allocation pattern on one or more portions of the first video frame of said video to reduce the resolution of at least a portion of said first video frame in accordance with the second selected resolution allocation pattern to produce a second reduced resolution version of said first video frame; and

a second encoder configured to encode the second reduced resolution version of the first video frame to generate a second encoded version of said first video frame.

20. A non-transitory computer readable medium comprising processor executable instructions, which when executed by a processor, control a content processing and delivery system to:

receive video corresponding to an environment;

store the video in a storage device;

transmit at least some of the video to playback devices;

receive, from the playback devices, viewing information indicating, on a per user basis, which portion of the environment a user was viewing at a given time during playback of the video;

determine, based on the received viewing information, relative importance of different portions of the environment at different times during playback of the video;

process the stored video to control allocation of resolution to different portions of said environment based on the determined relative importance of different portions of the environment at different times, said processing producing encoded video; and

perform at least one of storing the encoded video or transmitting the encoded video to at least one playback device.