IMAGE ENCODING AND DISPLAY

• US 20160366392A1
• Filed: 02/25/2015
• Published: 12/15/2016
• Est. Priority Date: 02/26/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardajal, Sharwin Winesh, Bickerstaff, Ian Henry
• Granted Patent: US 10,257,492 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0074 Stereoscopic image analysis
• CPC Class Codes: H04N 21/440245 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 2213/008 Aspects relating to glasses for viewing stereoscopic images

Abstract

An image encoding method includes, for each of left and right images of an image stereo pair, identifying a region of interest (ROI) in an input image, generating multiple representations of that input image such that one representation is a high field of view (FOV) representation including substantially all of the input image and another representation is a low FOV representation having a lower FOV than the high FOV representation and including at least the ROI. The relationship between pixel resolution and image size for the high and low FOV representations is such that, in relation to the ROI, the low FOV representation provides more image detail than the high FOV representation. The method also includes generating one or more multiplexed images by juxtaposing portions of the left and right, high and low FOV representations so as to include all image content thereof in fewer than four multiplexed images.

Claims

1. An image encoding method comprising:

for each of a left and a right image of a stereo pair of images, identifying, in an input image, a region of interest;

generating, by one or more processors, at least two representations of that input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation; and

including at least the region of interest;

in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation;

generating, by the one or more processors, one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and

performing one or both of transmitting or storing the multiplexed images.

2. A method according to claim 1, in which the step of generating the one or more multiplexed images comprises generating a first multiplexed image by juxtaposing a whole of a first one of the left and right, high FOV and low FOV representations next to a whole of a second one of the left and right, high FOV and low FOV representations, and generating a second multiplexed image by juxtaposing a whole of a third one of the left and right, high FOV and low FOV representations next to a whole of a fourth one of the left and right, high FOV and low FOV representations.

3. A method according to claim 1, in which the performing step comprises one or both of transmitting or storing the multiplexed images at a higher image repetition rate than an image repetition rate of the input images.

4. A method according to claim 1, in which performing step comprises one or both of transmitting or storing the multiplexed images as respective left and right dummy 3D images.

5. An image display method comprising:

receiving multiplexed images, each multiplexed image including juxtaposed portions of left and right, high field of view (FOV) and low FOV representations of an input stereo image pair, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images, in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to a region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation;

demutiplexing, by one or more processors, the multiplexed images to generate sets of left and right, high FOV and low FOV representations; and

combining, by the one or more processors, the low FOV representation and the high FOV representation of each of the left and right images to provide respective left and right display images.

6. A method according to claim 5, in which the combining step comprises:

using primarily image content from the low FOV representation in respect of image regions represented by the low FOV representation, and using image content from the respective high FOV representation for other image regions.

7. A method according to claim 6, in which the combining step comprises:

for at least a central region of the low FOV representation, replacing image content of the high FOV representation by corresponding image content of the low FOV representation.

8. A method according to claim 7, in which the combining step comprises:

for at least a peripheral region of the low FOV representation, mixing image content of the high and low FOV representations.

9. A method according to claim 5, in which the combining step comprises re-projecting to form a respective display image, according to a desired display viewpoint.

10. A method according to claim 9, comprising displaying the re-projected image using a display.

11. A method according to claim 10, in which the display is a head-mountable display and the desired display viewpoint is detected by detecting one or both of a position and an orientation of the head-mountable display.

12. (canceled)

13. A non-transitory computer readable storage medium storing computer readable instructions thereon, the instructions, when executed by one or more processors, casing the one or more processors to perform an image encoding method, the method comprising:

for each of a left and a right image of a stereo pair of images, identifying, in an input image, a region of interest;

generating at least two representations of that input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation and including at least the region of interest;

in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation;

generating one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and

performing one or both of transmitting or storing the multiplexed images.

14-17. -17. (canceled)

18. An image encoding apparatus comprising:

a representation generator configured, for each of a left and a right image of a stereo pair of images, to;

identify, in an input image, a region of interest; and

generate at least two representations of the input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation; and

including at least the region of interest;

in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation;

a multiplexer operatively coupled to the representation generator and configured to generate one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and

an image output operatively coupled to the multiplexer and configured to output the multiplexed images.

19. The image encoding apparatus of claim 18, wherein the image encoding apparatus is a videogame machine.

20. An image display apparatus comprising:

an image input configured to receive multiplexed images output by an image encoding apparatus, each multiplexed image including juxtaposed portions of left and right, high field of view (FOV) and low FOV representations of an input stereo image pair, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images, in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to a region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation;

a demultiplexer configured to demultiplex the multiplexed images to generate sets of left and right, high FOV and low FOV representations; and

a combiner configured to combine the low FOV representation and the high FOV representation of each of the left and right images to provide respective left and right display images.

21. The image display apparatus of claim 20, wherein the image display apparatus is a head-mountable display system.