Methods and apparatus for reducing noise in images

• US 9,967,535 B2
• Filed: 09/18/2015
• Issued: 05/08/2018
• Est. Priority Date: 04/17/2015
• Status: Active Grant

• Current Assignee: Blue River Technology Inc. (Deere & Company)
• Original Assignee: Light Labs Inc.
• Inventors: Laroia, Rajiv, Shroff, Nitesh, Shroff, Sapna A
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Matt, Marnie
• Application Number:
  US14/859,103
• Publication Number:
  US 20160309133A1
• Time in Patent Office: 963 Days
• Field of Search: 348 49
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/685 performed by mechanical compensation
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

Various features relating to reducing and/or eliminating noise from images are described. In some embodiments depth based denoising is used on images captured by one or more camera modules based on depth information of a scene area and optical characteristics of the one or more camera modules used to captures the images. In some embodiments by taking into consideration the camera module optics and the depth of the object included in the image portion, a maximum expected frequency can be determined and the image portion is then filtered to reduce or remove frequencies above the maximum expected frequency. In this way noise can be reduced or eliminated from image portions captured by one or more camera modules. The optical characteristic of different camera modules may be different. In some embodiments a maximum expected frequency is determined on a per camera module and depth basis.

Claims

1. A method of generating an image, the method comprising:

determining a first plurality of maximum expected frequencies, each of the first plurality of maximum expected frequencies corresponding to a first camera module and a different depth, said first plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said first camera module and a first depth and a second maximum expected frequency corresponding to the first camera module and a second depth;

receiving portions of a first image captured by said first camera module; and

performing first filtering on individual portions of said first image captured by said first camera module, said first filtering including filtering individual portions of said first image corresponding to different depths using different filters, the filter being used on an individual portion being based on a depth to which the individual portion of said first image, to which the filter is applied, corresponds.

2. The method of claim 1, wherein said first filtering includes filtering a first portion of said first image corresponding to the first depth with a filter which removes or reduces frequency content above said first maximum expected frequency corresponding to said first camera module.

3. The method of claim 2, wherein said step of determining a first plurality of maximum expected frequencies is performed prior to said first filtering.

4. The method of claim 2,

wherein said first filtering further includes filtering a second portion of said first image corresponding to the second depth with a second filter which removes or reduces frequency content above said second maximum expected frequency corresponding to the first camera module;

and

wherein said first filtering further includes filtering a third portion of said first image corresponding to a third depth with a third filter which removes or reduces frequency content above a third maximum expected frequency.

5. The method of claim 4, further comprising:

prior to performing said first filtering, determining depths to which different portions of said first image correspond to; and

applying said first filtering to portions of said first image based on the determined depth of the individual image portions.

6. The method of claim 5, wherein determining depths to which different portions of said first image correspond to include includes:

generating a depth map for a scene area corresponding to said first image; and

determining the depths to which portions of said first image correspond based on said depth map.

7. The method of claim 4, further comprising:

determining a second plurality of maximum expected frequencies, each of the second plurality of maximum expected frequencies corresponding to a second camera module and a different depth, said second plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said second camera module and the first depth and a second maximum expected frequency corresponding to the second camera module and the second depth;

receiving portions of a second image captured by said second camera module; and

performing second filtering on individual portions of said second image captured by said second camera module based on a depth to which each individual portion being filtered corresponds.

8. The method of claim 7, wherein said second filtering includes filtering a first portion of said second image corresponding to the first depth with a filter which removes or reduces frequency content above said first maximum expected frequency corresponding to said second camera module.

9. The method of claim 8, wherein said second filtering further includes filtering a second portion of said second image corresponding to the second depth with a second filter which removes or reduces frequency content above said second maximum expected frequency corresponding to said second camera module.

10. The method of claim 9, wherein said second filtering further includes filtering a third portion of said second image corresponding to a third depth with a third filter which removes or reduces frequency content above a third maximum expected frequency corresponding to the second camera module.

11. The method of claim 1, further comprising:

prior to performing said first filtering, determining depths to which different portions of said first image correspond; and

wherein performing first filtering on individual portions of said first image captured by said first camera module includes;

applying filters having different cut off frequencies to individual image portions corresponding to different depths.

12. The method of claim 1, wherein performing first filtering on individual portions of said first image captured by said first camera module includes:

using filters having different filter cut off frequencies to filter different portions of the first image where said different portions correspond to different depths, which of the plurality of different filters to be used for an individual portion of the first image being based on the depth to which the individual portion of the first image corresponds and a noise threshold determined for the depth.

13. The method of claim 12, wherein the filtering applied by one of said different filters removes or smoothes frequencies above the noise threshold determined for the depth of the individual image portion to which the filter is applied.

14. A camera device, comprising:

a first camera module; and

a processor configured to;

determine a first plurality of maximum expected frequencies, each of the first plurality of maximum expected frequencies corresponding to the first camera module and a different depth, said first plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said first camera module and a first depth and a second maximum expected frequency corresponding to the first camera module and a second depth;

receive portions of a first image captured by said first camera module; and

perform first filtering on individual portions of said first image captured by said first camera module, said first filtering including filtering individual portions of said first image corresponding to different depths using different filters, the filter being used on an individual portion being based on a depth to which the individual portion of said first image, to which the filter is applied, corresponds.

15. An apparatus, comprising:

a maximum expected frequency determination module configured to determine a first plurality of maximum expected frequencies, each of the first plurality of maximum expected frequencies corresponding to a first camera module and a different depth, said first plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said first camera module and a first depth and a second maximum expected frequency corresponding to the first camera module and a second depth;

a processing module configured to receive portions of a first image captured by the first camera module; and

a filtering module configured to perform first filtering on individual portions of said first image captured by said first camera module, said first filtering including filtering individual portions of said first image corresponding to different depths using different filters, the filter being used on an individual portion being based on a depth to which the individual portion of said first image, to which the filter is applied, corresponds.

16. The apparatus of claim 15, wherein said filtering module is configured to filter a first portion of said first image corresponding to the first depth to remove or reduce frequency content above said first maximum expected frequency corresponding to said first camera module, as part of performing said first filtering.

17. The apparatus of claim 16, wherein said filtering module is configured to filter a second portion of said first image corresponding to the second depth to remove or reduce frequency content above said second maximum expected frequency corresponding to the first camera module, as part of performing said first filtering.

18. The apparatus of claim 17, wherein said filtering module is configured to filter a third portion of said first image corresponding to a third depth to remove or reduce frequency content above a third maximum expected frequency, as part of performing said first filtering.

19. The apparatus of claim 18, further comprising:

a processor configured to determine depths to which different portions of said first image correspond to; and

wherein said filtering module is configured to apply said first filtering to portions of said first image based on the determined depth of the individual image portions.

20. The apparatus of claim 19, wherein said processor is configured to:

generate a depth map for a scene area corresponding to said first image; and

determine depths to which different portions of said first image correspond based on said depth map.

21. The apparatus of claim 20, wherein said processor is further configured to generate said depth map from multiple images of said scene area captured by different camera modules.

22. The apparatus of claim 18,

wherein said maximum expected frequency determination module is further configured to determine a second plurality of maximum expected frequencies, each of the second plurality of maximum expected frequencies corresponding to a second camera module and said different depth, said second plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said second camera module and the first depth and a second maximum expected frequency corresponding to the second camera module and the second depth;

wherein said image processing module is further configured to receive portions of a second image captured by said second camera module; and

wherein said filtering module is further configured to perform second filtering on individual portions of said second image captured by said second camera module based on a depth to which each individual portion being filtered corresponds.

23. A non-transitory machine readable medium for use in a camera including a first image sensor and a second image sensor, the non-transitory machine readable medium including processor executable instructions which when executed control a processor to:

determine a first plurality of maximum expected frequencies, each of the first plurality of maximum expected frequencies corresponding to a first camera module and a different depth, said first plurality of maximum expected frequencies including a first maximum expected frequency corresponding to said first camera module and a first depth and a second maximum expected frequency corresponding to the first camera module and a second depth;

receive portions of a first image captured by said first camera module; and

perform first filtering on individual portions of said first image captured by said first camera module, said first filtering including filtering individual portions of said first image corresponding to different depths using different filters, the filter being used on an individual portion being based on a depth to which the individual portion of said first image, to which the filter is applied, corresponds.