Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device

• US 9,832,450 B2
• Filed: 02/17/2016
• Issued: 11/28/2017
• Est. Priority Date: 02/17/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, McKay Moss, Alan, Medina, Hector M.
• Primary Examiner(s): Chowdhury, Sumaiya A
• Application Number:
  US15/046,311
• Publication Number:
  US 20160241892A1
• Time in Patent Office: 650 Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/42653 for processing graphics
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/437 : Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/44016 : involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/472 : End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/8146 : involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 : involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.

Claims

1. A method comprising the steps of:

storing texture maps corresponding to a first portion of an environment in memory;

operating a processor to select a first resolution allocation to be used for at least one image corresponding to said first portion of an environment;

performing a resolution reduction operation on a first image of said first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image, wherein said first image is one image of an image pair including a left eye image and a right eye image;

performing a resolution reduction operation on a second image of said image pair in accordance with the selected first resolution allocation to generate a second reduced resolution image;

communicating to a playback device information indicating a first texture map to be used to map portions of images generated in accordance with the first resolution allocation to a surface of a model of said environment; and

communicating a first stereoscopic image pair to the playback device, said first stereoscopic image pair including the said first reduced resolution image and said second reduced resolution image.

2. The method of claim 1,

wherein said information is said first texture map or an identifier identifying said first texture map;

and

wherein the size of a first segment in said first texture map is a function of the amount of resolution reduction applied to a corresponding first area of the first image to generate a first segment of said first reduced resolution image.

3. The method of claim 2, wherein the first texture map includes a second segment corresponding to a portion of the first image which was not subject to a resolution reduction operation, the size of the second segment in said first texture map being the same as the size of the corresponding portion of the first image.

4. The method of claim 2, wherein the size of the first segment in said texture map is reduced from the size of a source of a corresponding area in the first image by an amount which is based on the amount of resolution reduction applied to the corresponding first area of the first image.

5. The method of claim 4, further comprising:

communicating to said playback device an environmental model; and

wherein said first texture map corresponds to a portion of said environmental model, said first texture map providing information indicating how to map portions of images subject to said first resolution allocation to a portion of said environmental model.

6. The method of claim 1, further comprising:

selecting a second resolution allocation to be used for another image corresponding to the first portion of the environment, said second resolution allocation being different from said first resolution allocation, said another image being a third image;

performing a resolution reduction operation on the third image in accordance with the selected second resolution allocation to generate a third reduced resolution image; and

communicating said third reduced resolution image to said playback device.

7. The method of claim 6, further comprising:

communicating to the playback device information indicating a second texture map to be used to map portions of images generated in accordance with the second resolution allocation to said surface of said model of said environment, said second texture map being different from said first texture map.

8. The method of claim 7, wherein the size of a first segment in said second texture map is a function of the amount of resolution reduction applied to a corresponding first area of the third image to generate a first segment of said third reduced resolution image.

9. The method of claim 8, wherein the second texture map includes a third segment corresponding to a portion of the third image which was not subject to a resolution reduction operation, the size of the third segment in said second texture map being the same as the size of the corresponding portion of the third image.

10. The method of claim 1, wherein the first resolution reduction operation includes a downsampling operation.

11. The method of claim 1, wherein said first image of said first portion of the environment is an image which was captured by a camera.

12. A system comprising:

a processor configured to;

select a first resolution allocation to be used for at least one image corresponding to a first portion of an environment;

perform a resolution reduction operation on a first image of said first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image, wherein said first image is one image of an image pair including a left eye image and a right eye image; and

perform a resolution reduction operation on a second image of said image pair in accordance with the selected first resolution allocation to generate a second reduced resolution image; and

a transmitter configured to;

communicate to a playback device a first texture map to be used to map portions of said images generated in accordance with the first resolution allocation to a surface of a model of said environment; and

communicate a first stereoscopic image pair to the playback device, said first stereoscopic image pair including said first reduced resolution image and said second reduced resolution image.

13. The system of claim 12, wherein the size of a first segment in said first texture map is a function of the amount of resolution reduction applied to a corresponding first area of the first image to generate a first segment of said first reduced resolution image.

14. The system of claim 13, wherein the first texture map includes a second segment corresponding to a portion of the first image which was not subject to a resolution reduction operation, the size of the second segment in said second texture map being the same as the size of the segment in said first image.

15. The system of claim 13, wherein the size of the first segment in said texture map is reduced from the size of the source of the corresponding area in the first image by an amount which is based on the amount of resolution reduction applied to the corresponding first area of the first image.

16. The system of claim 15,

wherein said transmitter is further configured to communicate to said playback device an environmental model;

and

wherein said first texture map corresponds to a portion of said environmental model, said first texture map providing information indicating how to map portions of images subject to said first resolution allocation to a portion of said environmental model.

17. A non-transitory computer readable medium comprising processor executable instructions, which when executed by a processor, control a system to:

select a first resolution allocation to be used for at least one image corresponding to a first portion of an environment;

perform a resolution reduction operation on a first image of said first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image, wherein said first image is one image of an image pair including a left eye image and a right eye image;

perform a resolution reduction operation on a second image of the image pair in accordance with the selected first resolution allocation to generate a second reduced resolution image;

communicate to a playback device information indicating a first texture map to be used to map portions of said images generated in accordance with the first resolution allocation to a surface of a model of said environment; and

communicate a first stereoscopic image pair to the playback device, said first stereoscopic image pair including said first reduced resolution image and said second reduced resolution image.

18. A method, comprising:

operating a device including a processor and memory to;

select, from a plurality of supported resolution allocations corresponding to different supported UV maps, a first resolution allocation to be used for at least one image corresponding to a first portion of an environment;

perform a resolution reduction operation on a first image of said first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image, wherein said first image is a first image of an image pair including a left eye image and a right eye image;

perform a resolution reduction operation on a second image of the image pair in accordance with the selected first resolution allocation to generate a second reduced resolution image;

communicate a first UV map corresponding to the first resolution allocation to a playback device; and

communicate a first stereoscopic image pair to the playback device, said first stereoscopic image pair including said first reduced resolution image and said second reduced resolution image.

19. A method comprising the steps of:

storing in memory a mesh model of a first portion of an environment;

storing in memory a plurality of different texture maps corresponding to the first portion of the environment, different texture maps in said plurality of different texture maps corresponding to different resolution allocations;

operating a processor to select a first resolution allocation from a plurality of different resolution allocations to be used for at least one image corresponding to the first portion of environment;

performing a resolution reduction operation on a first image of said first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image, wherein said first image is one image of an image pair, said image pair including said first image and a second image, the first image being one of a left and right eye image pair, the second image being a second one of a left and right eye image pair;

performing a resolution reduction operation on the second image in accordance with the selected first resolution allocation to generate a second reduced resolution image; and

communicating, via a network interface, said mesh model, a first texture map from said plurality of different texture maps, said first reduced resolution image and said second reduced resolution image to a playback device, said first texture map corresponding to the selected first resolution allocation, said first reduced resolution image and said second reduced resolution image being part of a stereoscopic image pair.