Stereoscopic image processing methods and apparatus

• US 9,204,127 B1
• Filed: 01/17/2013
• Issued: 12/01/2015
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, Moss, Alan McKay
• Primary Examiner(s): Lee, Michael
• Application Number:
  US13/744,400
• Time in Patent Office: 1,048 Days
• Field of Search: 348/42, 348/43, 348/46
• US Class Current: 1/1
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Stereoscopic image processing methods and apparatus are described. Left and right eye images of a stereoscopic frame are examined to determine if one or more difference reduction operations designed to reduce the luminance and/or chrominance differences between the left and right frames is within a range used to trigger a difference reduction operation. A difference reduction operation may involve assigning portions of the left and right frames to different depth regions and/or other region categories. A decision on whether or not to perform a difference reduction operation is then performed on a per regions basis with at the difference between the left and right eye portions of at least one region being reduced when a difference reduction operation is to be performed. The difference reduction process may, and in some embodiments is, performed in a precoder which processes left and right eye images of stereoscopic frames prior to stereoscopic encoding.

Claims

1. A method of processing a pair of images including a left eye image and a right eye image, each of said left and right eye images including image values of a first type, the method comprising:

generating a first overall disparity value indicating a first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image;

determining if said first overall disparity value is in a first difference range used to control performing of a first difference reduction operation, said first difference reduction operation being an operation on values of said first type included in at least one of said left eye image or said right eye image to reduce the overall difference between values of said first type in said left eye image and values of said first type in said right eye image; and

performing said first difference reduction operation to reduce the first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image when said first overall disparity value is determined to be in said first difference range and skipping performing said first difference reduction operation when said first overall disparity value is above a second threshold which is an upper boundary of said first difference range.

2. The method of claim 1 wherein said values of said first type are luminance values and wherein the first difference range is defined by a lower threshold which is a first value that is a first percentage of the maximum possible difference in luminance values between the left and right eye image and an upper threshold that is a second percentage of the maximum possible difference in luminance values between the left and right eye image.

3. The method of claim 2, further comprising:

determining if a second difference reduction operation to reduce an overall level of difference between values of a second type in said left eye image and values of said second type in said right eye image should be performed, said values of the second type being chrominance values, said determining if a second difference reduction operation should be performed being made independent of any determination as to whether said first difference reduction operation is to be performed.

4. The method of claim 3,

wherein determining if said first overall disparity value is in the first difference range used to control performing of the first difference reduction operation includes:

comparing said first overall disparity value to a first threshold corresponding to a lower boundary of said first difference range; and

comparing said first overall disparity value to said second threshold corresponding to said upper boundary of said first difference range.

5. The method of claim 4, wherein determining if said first overall disparity value is in the first difference range used to control performing of the first difference reduction operation further includes:

determining that said first overall disparity value is in the first difference range when said comparing steps indicate that said first overall disparity value is above said first threshold and below said second threshold.

6. The method of claim 1, wherein performing said first difference reduction operation includes:

modifying values of the first type in a first one of said left and right eye images to reduce an overall disparity between the values of the first type in a first region of the first one of said left and right eye images and the values of the first type included in a second one of said left and right eye images, said second one of said left and right eye images being different from said first one of said left and right eye images.

7. The method of claim 6, wherein performing said first difference reduction operation further includes:

generating a depth map for each of said left and right eye images; and

identifying different regions in said left and right eye images based on depths to which the regions are determined to correspond, said depths including at least a first image depth and a second image depth; and

applying a disparity operation to at least one region of a frame identified based on depth information from said depth map but not another region of the frame corresponding to a different depth.

8. The method of claim 7, wherein said first image depth corresponds to a first distance range from a camera to objects in an image region identified as corresponding to said first image depth and a second image depth corresponds to a second distance range from the camera to objects in an image region identified as corresponding to said second image depth, said first and second image depths being different.

9. The method of claim 1, further comprising:

dividing said left and right eye images into regions based on depth information, said left and right eye images being divided into regions based on at least a first depth corresponding to a foreground region and second depth corresponding to a background region; and

making separate decisions whether to apply a disparity reduction operation on said foreground and background regions based on the content of the separate foreground and background regions.

10. An apparatus for processing a pair of images including a left eye image and a right eye image, each of said left and right eye images including image values of a first type, comprising:

a disparity value generation module configured to generate a first overall disparity value indicating a first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image;

a determination module configured to determine if said first overall disparity value is in a first difference range used to control performing of a first difference reduction operation, said first difference reduction operation being an operation on values of said first type included in at least one of said left eye image or said right eye image to reduce the overall difference between values of said first type in said left eye image and values of said first type in said right eye image; and

a difference reduction operation module configured to perform said first difference reduction operation to reduce the first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image when said first overall disparity value is determined to be in said first difference range and skip performing said first difference reduction operation when said first overall disparity value is above a second threshold which is an upper boundary of said first difference range.

11. The apparatus of claim 10, wherein said values of said first type are one of luminance values, chrominance values corresponding to a first color, or chrominance values corresponding to a second color.

12. The apparatus of claim 10, wherein said determination module includes:

a comparison module configured to compare said first overall disparity value to a first threshold corresponding to a lower boundary of said first difference range.

13. The apparatus of claim 12, wherein said comparison module is further configured to compare said first overall disparity value to said second threshold corresponding to said upper boundary of said first difference range.

14. The apparatus of claim 13, wherein said determination module is further configured to determine that said first overall disparity value is in the first difference range, when a result of a comparison performed by said comparison module indicates that said first overall disparity value is above said first threshold and below said second threshold.

15. The apparatus of claim 10, wherein said difference reduction operation module includes:

a modification module configured to modify values of the first type in a first one of said left and right eye images to reduce an overall disparity between the values of the first type in a first region of the first one of said left and right eye images and the values of the first type included in a second one of said left and right eye images, said second one of said left and right eye images being different from said first one of said left and right eye images.

16. The apparatus of claim 15, wherein said difference reduction operation module further includes:

a depth map generation module configured to generate a depth map for each of said left and right eye images; and

an identification module configured to identify different regions in said left and right eye images based on depths to which the regions are determined to correspond, said depths including at least a first image depth and a second image depth.

17. The apparatus of claim 16, wherein said first image depth corresponds to a first distance range from a camera to objects in an image region identified as corresponding to said first image depth and a second image depth corresponds to a second distance range from the camera to objects in an image region identified as corresponding to said second image depth, said first and second image depths being different.

18. The apparatus of claim 16, wherein said first and second image depths correspond to foreground and background image regions, respectively.

19. A non-transitory computer readable medium for use in an apparatus for processing a pair of images including a left eye image and a right eye image, each of said left and right eye images including image values of a first type, the non-transitory computer readable medium comprising:

instructions which when executed by a processor cause said processor to generate a first overall disparity value indicating a first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image;

instructions which when executed by said processor cause said processor to determine if said first overall disparity value is in a first difference range used to control performing of a first difference reduction operation, said first difference reduction operation being an operation on values of said first type included in at least one of said left eye image or said right eye image to reduce the overall difference between values of said first type in said left eye image and values of said first type in said right eye image; and

instructions which when executed by said processor cause said processor to perform the first difference reduction operation to reduce the first overall level of difference between values of said first type in said left eye image and values of said first type in said right eye image when said first overall disparity value is determined to be in said first difference range and skip performing said first difference reduction operation when said first overall disparity value is above a second threshold which is an tipper boundary of said first difference range.

20. The non-transitory computer readable medium of claim 19, further comprising:

instructions which when executed by said processor cause said processor to compare said first overall disparity value to a first threshold corresponding to a lower boundary of said first difference range; and

instructions which when executed by said processor cause said processor to compare said first overall disparity value to a second threshold corresponding to an upper boundary of said first difference range.