Methods and apparatus for receiving and/or using reduced resolution images

• US 9,729,850 B2
• Filed: 02/17/2016
• Issued: 08/08/2017
• Est. Priority Date: 02/17/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Medina, Hector M., Cole, David, Moss, Alan McKay
• Primary Examiner(s): ADAMS, EILEEN M
• Application Number:
  US15/046,327
• Publication Number:
  US 20160241836A1
• Time in Patent Office: 538 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/42653 for processing graphics
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/437 : Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/44016 : involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/472 : End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/8146 : involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 : involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.

Claims

1. A method of operating a content playback device, the method comprising:

receiving a first encoded image;

receiving a second encoded image, said first encoded image and said second encoded image being images in a first stereoscopic image pair including a left eye view and a right eye view;

receiving a first indicator indicating which of a plurality of texture maps corresponding to different resolution allocations is to be used with the first encoded image, said first indicator identifying a first texture map corresponding to a first resolution allocation;

decoding the first encoded image to recover a first image;

decoding the second encoded image to recover a second image;

using, as part of a first rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said first image to a surface of a first portion of a model of an environment to generate a first rendered image;

using, as part of a second rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said second image to the surface of the first portion of the model of an environment to generate a second rendered image;

outputting the first rendered image to a display device;

receiving a third encoded image;

receiving a second indicator indicating which of the plurality of texture maps corresponding to different resolution allocations is to be used with the third encoded image, said second indicator identifying a second texture map corresponding to a second resolution allocation;

decoding the third encoded image to recover a third image; and

using, as part of a third rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said third image to a surface of the first portion of the model of the environment to generate a third rendered image; and

outputting the third rendered image to the display device.

2. The method of claim 1, further comprising:

displaying the first rendered image to a first one of a user'"'"'s left and right eyes; and

displaying the second rendered image to a second one of the user'"'"'s left and right eyes.

3. The method of claim 1, further comprising:

storing said first texture map and said second texture map in memory.

4. The method of claim 1, further comprising:

receiving a fourth encoded image, said fourth encoded image and said third encoded image being images in a second stereoscopic image pair including a second left eye view and a second right eye view;

decoding the fourth encoded image to recover a fourth image; and

using, as part of a fourth rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said fourth image to the surface of the first portion of the model of said environment to generate a fourth rendered image;

displaying the third rendered image to a first one of the user'"'"'s left and right eyes; and

displaying the fourth rendered image to a second one of the user'"'"'s left and right eyes while said third rendered image is being displayed.

5. The method of claim 4, wherein said first, second, third and fourth encoded images are received and decoded while an event to which the first, second, third and fourth encoded images correspond is ongoing.

6. A content playback device comprising:

a receiver;

a decoder;

a memory, said memory including a first texture map and a second texture map;

a processor, said processor configured to control said content playback device to;

receive a first encoded image;

receive a second encoded image, said first encoded image and said second encoded image being images in a first stereoscopic image pair including a left eye view and a right eye view;

receive a first indicator indicating which of a plurality of texture maps corresponding to different resolution allocations is to be used with the first encoded image, said first indicator identifying the first texture map corresponding to a first resolution allocation;

decode the first encoded image to recover a first image;

decode the second encoded image to recover a second image;

use, as part of a first rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said first image to a surface of a first portion of a model of an environment to generate a first rendered image;

use, as part of a second rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said second image to the surface of the first portion of the model of an environment to generate a second rendered image;

receive a third encoded image;

receive a second indicator indicating which of the plurality of texture maps corresponding to different resolution allocations is to be used with the third encoded image, said second indicator identifying the second texture map corresponding to a second resolution allocation;

decode the third encoded image to recover a third image; and

use, as part of a third rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said third image to a surface of the first portion of the model of the environment; and

a display for displaying images rendered by the processor.

7. The content playback device of claim 6, wherein the processor is further configured to control the content playback device to:

display the first rendered image to a first one of a user'"'"'s left and right eyes, and

display the second rendered image to a second one of the user'"'"'s left and right eyes.

8. The content playback device of claim 6, wherein the processor is further configured to control the content playback device to:

receive a fourth encoded image, said fourth encoded image and said third encoded image being images in a second stereoscopic image pair including a second left eye view and a second right eye view;

decode the fourth encoded image to recover a fourth image; and

use, as part of a fourth rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said fourth image to the surface of the first portion of the model of said environment to generate a fourth rendered image;

display the third rendered image to a first one of the user'"'"'s left and right eyes; and

display the fourth rendered image to a second one of the user'"'"'s left and right eyes while said third rendered image is being displayed.

9. The content playback device of claim 8, wherein said first, second, third and fourth encoded images are received and decoded while an event to which the first, second, third and fourth encoded images correspond is ongoing.

10. A non-transitory computer readable medium comprising processor executable instructions, which when executed by a processor, control a content playback device to:

receive a first encoded image;

receive a second encoded image, said first encoded image and said second encoded image being images in a first stereoscopic image pair including a left eye view and a right eye view;

receive a first indicator indicating which of a plurality of texture maps corresponding to different resolution allocations is to be used with the first encoded image, said first indicator identifying a first texture map corresponding to a first resolution allocation;

decode the first encoded image to recover a first image;

decode the second encoded image to recover a second image;

use, as part of a first rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said first image to a surface of a first portion of a model of an environment to generate a first rendered image;

use, as part of second rendering operation, the first texture map corresponding to the first resolution allocation to apply at least a portion of said second image to the surface of the first portion of the model of an environment to generate a second rendered image;

output the first rendered image to a display device;

receive a third encoded image;

receive a second indicator indicating which of the plurality of texture maps corresponding to different resolution allocations is to be used with the third encoded image, said second indicator identifying a second texture map corresponding to a second resolution allocation;

decode the third encoded image to recover a third image; and

use, as part of a third rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said third image to a surface of the first portion of the model of the environment to generate a third rendered image;

output the third rendered image to the display device; and

display the first rendered image on a display.

11. The non-transitory computer readable medium of claim 10, wherein said processor executable instructions, which when executed by said processor, further control said content playback device to:

display the first rendered image to a first one of a user'"'"'s left and right eyes; and

display the second rendered image to a second one of the user'"'"'s left and right eyes.

12. The non-transitory computer readable medium of claim 10, wherein said processor executable instructions, which when executed by said processor, further control said content playback device to:

receive a fourth encoded image, said fourth encoded image and said third encoded image being images in a second stereoscopic image pair including a second left eye view and a second right eye view;

decode the fourth encoded image to recover a fourth image; and

use, as part of a fourth rendering operation, the second texture map corresponding to the second resolution allocation to apply at least a portion of said fourth image to the surface of the first portion of the model of said environment to generate a fourth rendered image;

display the third rendered image to a first one of the user'"'"'s left and right eyes; and

display the fourth rendered image to a second one of the user'"'"'s left and right eyes while said third rendered image is being displayed.