Method, apparatus and computer program product for motion deblurring of images

• US 10,176,558 B2
• Filed: 12/01/2015
• Issued: 01/08/2019
• Est. Priority Date: 12/29/2014
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Uliyar, Mithun, Govindarao, Krishna, Putraya, Gururaj, S V, Basavaraja
• Primary Examiner(s): Hsu, Amy R
• Application Number:
  US15/539,744
• Publication Number:
  US 20170352136A1
• Time in Patent Office: 1,134 Days
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/53 Control of the integration time
• CPC Class Codes: H04N 5/211 Ghost signal cancellation H04N23/81 takes precedence

Abstract

In an example embodiment a method, apparatus and computer program product are provided. The method includes determining presence of at least one moving object in a scene based on two or more burst images corresponding to the scene captured by a first camera. One or more portions of the scene associated with the at least one moving object are identified, and, information related to the one or more portions is provided to a second camera. An image of the scene captured by the second camera second camera is received, where a pixel level shutter disposed in front of an image sensor of the second camera is programmed to periodically open and close, throughout a duration of said image capture, for pixels of the image sensor corresponding to the one or more portions of the scene. A deblurred image corresponding to the scene is generated based on the image.

Claims

1. A method comprising:

determining presence of at least one moving object in a scene based on two or more burst images corresponding to the scene captured by a first camera, the two or more burst images are associated with a first exposure time;

identifying one or more portions of the scene associated with the at least one moving object if the at least one moving object is determined to be present in the scene;

providing information related to the one or more portions of the scene to a second camera, the second camera configured to capture single exposure images associated with a second exposure time, the second exposure time being greater than the first exposure time, wherein the information related to the one or more portions of the scene associated with the at least one moving object is provided substantially instantaneously to the second camera subsequent to the capturing of the two or more burst images;

receiving an image of the scene captured by the second camera, wherein a pixel level shutter disposed in front of an image sensor of the second camera is programmed to periodically open and close, throughout a duration of said image capture, for pixels of the image sensor corresponding to the one or more portions of the scene; and

generating a deblurred image corresponding to the scene based on the image.

2. The method as claimed in claim 1, wherein the pixels of the image sensor corresponding to the one or more portions of the scene are identified based on maximum permissible disparity between the first camera and the second camera.

3. The method as claimed in claim 1, further comprising:

performing deconvolution of the image to remove motion blur caused by the at least one moving object in the scene, wherein removal of the motion blur from the image generates the deblurred image.

4. The method as claimed in claim 3, wherein information related to a portion of the scene from among the one or more portions of the scene comprises a velocity estimate of a corresponding moving object from among the at least one moving object.

5. The method as claimed in claim 1, further comprising performing, upon determining an absence of the at least one moving object in the scene:

providing indication of the absence of the at least one moving object in the scene to the second camera; and

receiving a final image of the scene captured by the second camera, wherein the pixel level shutter is programmed to stay open, for all pixels of the image sensor throughout the duration of the image capture.

6. The method as claimed in claim 5, further comprising:

receiving a plurality of burst images corresponding to the scene captured within a time period equal to the second exposure time by the first camera, the plurality of burst images comprising the two or more burst images; and

generating a secondary image corresponding to the scene based on the plurality of burst images by utilizing one or more deghosting algorithms.

7. An apparatus comprising:

at least one processor; and

at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform;

determine presence of at least one moving object in a scene based on two or more burst images corresponding to the scene captured by a first camera, the two or more burst images are associated with a first exposure time;

identify one or more portions of the scene associated with the at least one moving object if the at least one moving object is determined to be present in the scene;

provide information related to the one or more portions of the scene to a second camera, the second camera configured to capture single exposure images associated with a second exposure time, the second exposure time being greater than the first exposure time, wherein the information related to the one or more portions of the scene associated with the at least one moving object is provided substantially instantaneously to the second camera subsequent to the capturing of the two or more burst images;

receive an image of the scene captured by the second camera, wherein a pixel level shutter disposed in front of an image sensor of the second camera is programmed to periodically open and close, throughout a duration of said image capture, for pixels of the image sensor corresponding to the one or more portions of the scene; and

generate a deblurred image corresponding to the scene based on the image.

8. The apparatus as claimed in claim 7, wherein the pixels of the image sensor corresponding to the one or more portions of the scene are identified based on maximum permissible disparity between the first camera and the second camera.

9. The apparatus as claimed in claim 7, wherein the apparatus is further caused, at least in part to:

perform deconvolution of the image to remove motion blur caused by the at least one moving object in the scene, wherein removal of the motion blur from the image generates the deblurred image.

10. The apparatus as claimed in claim 9, wherein information related to a portion of the scene from among the one or more portions of the scene comprises a velocity estimate of a corresponding moving object from among the at least one moving object.

11. The apparatus as claimed in claim 7, wherein the apparatus upon determining an absence of the at least one moving object in the scene is further caused, at least in part to:

provide indication of the absence of the at least one moving object in the scene to the second camera; and

receive a final image of the scene captured by the second camera, wherein the pixel level shutter is programmed to stay open, for all pixels of the image sensor throughout the duration of the image capture.

12. The apparatus as claimed in claim 11, wherein the apparatus is further caused, at least in part to:

receive a plurality of burst images corresponding to the scene captured within a time period equal to the second exposure time by the first camera, the plurality of burst images comprising the two or more burst images; and

generate a secondary image corresponding to the scene based on the plurality of burst images by utilizing one or more deghosting algorithms.

13. A computer program product comprising at least one non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a set of instructions, which, when executed by one or more processors, cause an apparatus to at least perform:

determine presence of at least one moving object in a scene based on two or more burst images corresponding to the scene captured by a first camera, the two or more burst images are associated with a first exposure time;

identify one or more portions of the scene associated with the at least one moving object if the at least one moving object is determined to be present in the scene;

provide information related to the one or more portions of the scene to a second camera, the second camera configured to capture single exposure images associated with a second exposure time, the second exposure time being greater than the first exposure time, wherein the information related to the one or more portions of the scene associated with the at least one moving object is provided substantially instantaneously to the second camera subsequent to the capturing of the two or more burst images;

receive an image of the scene captured by the second camera, wherein a pixel level shutter disposed in front of an image sensor of the second camera is programmed to periodically open and close, throughout a duration of said image capture, for pixels of the image sensor corresponding to the one or more portions of the scene; and

generate a deblurred image corresponding to the scene based on the image.

14. The computer program product as claimed in claim 13, wherein the pixels of the image sensor corresponding to the one or more portions of the scene are identified based on maximum permissible disparity between the first camera and the second camera.

15. The computer program product as claimed in claim 13, wherein the apparatus is further caused, at least in part to:

perform deconvolution of the image to remove motion blur caused by the at least one moving object in the scene, wherein removal of the motion blur from the image generates the deblurred image.

16. The computer program product as claimed in claim 15, wherein information related to a portion of the scene from among the one or more portions of the scene comprises a velocity estimate of a corresponding moving object from among the at least one moving object.

17. The computer program product as claimed in claim 13, wherein the apparatus upon determining an absence of the at least one moving object in the scene is further caused, at least in part to:

provide indication of the absence of the at least one moving object in the scene to the second camera; and

receive a final image of the scene captured by the second camera, wherein the pixel level shutter is programmed to stay open, for all pixels of the image sensor throughout the duration of the image capture.