SYSTEM AND METHOD FOR ACQUIRING A STILL IMAGE FROM A MOVING IMAGE

• US 20130321657A1
• Filed: 03/16/2013
• Published: 12/05/2013
• Est. Priority Date: 09/14/2009
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: McGarry, E. John, Silver, William M.
• Granted Patent: US 8,743,220 B2
• Time in Patent Office: Days
• US Class Current: 348/220.1
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 25/48 Increasing resolution by shifting the sensor relative to the scene
• CPC Class Codes: H04N 25/771 comprising storage means other than floating diffusion

Abstract

A system and method captures a moving image of a scene that can be more readily de-blurred as compared to images captured through other methods operating on equivalent exposure-time intervals. Rather than stopping and starting the light measurement during the exposure-time interval, photo-generated current is switched between multiple charge storage sites in accordance with a temporal switching pattern that optimizes the conditioning of the solution to the inverse blur transform. By switching the image intensity signal between storage sites all of the light energy available during the exposure-time interval is transduced to electronic charge and captured to form a temporally decomposed representation of the moving image. As compared to related methods that discard approximately half of the image intensity signal available over an equivalent exposure-time interval, such a temporally decomposed image is a far more complete representation of the moving image and more effectively de-blurred using simple linear de-convolution techniques.

Claims

1. A system for acquiring a still image from an input moving image, comprising:

a) an imaging assembly including a plurality of pixels, each of the pixels comprising a photosensitive element and a plurality of integrating storage sites, each of the pixels being constructed and arranged to direct a measurement of light impinging on the photosensitive element to any one of the plurality of integrating storage sites;

b) an image capture process that forms a temporally decomposed representation of the moving image in an exposure time interval by directing, according to a predetermined temporal switching pattern, the measurement of light impinging on each of the photosensitive elements among the plurality of integrating storage sites, such that the temporally decomposed image is formed with a temporally decomposed point-spread-function having an inverse transformation that is optimally well-conditioned; and

c) an image extraction process that applies the inverse transformation of the temporally decomposed point-spread-function to the temporally decomposed image to form a motion-blur free digital image.

2. The system as set forth in claim 1 wherein the photosensitive element is a photodiode connected to a first sense node through a first transfer gate and to a second sense node through a second transfer gate, each of the first transfer gate and the second transfer gate being responsive to the input image capture process.

3. The system as set forth in claim 1 wherein the plurality of integrating storage sites include at least a third integrating storage site, each of the pixels being constructed an arranged to direct the measurement of light impinging on the photosensitive element to the at least third storage site according to the temporal switching pattern.

4. The system as set forth in claim 1 wherein the temporal switching pattern is selected to at least one of (a) minimize attenuation of spatial frequencies in the temporally decomposed representation of the moving image;

(b) minimize, by the image extraction process, attenuation of spatial frequencies in the still image;

(c) provide, by the image capture process and the image extraction process, a still image that contains a filtered set of spatial frequencies; and

(d) provide, by the image capture process and the image extraction process a still image having reduced blur.

5. The system as set forth in claim 1 wherein the image extraction process is constructed and arranged to provide a point-spread-function having a plurality of coefficients based upon a linear combination of values stored in the each of the plurality of integrating storage sites.

6. The system as set forth in claim 5 wherein the image extraction process is constructed and arranged to provide a blur matrix based upon the point-spread-function, the blur matrix being based upon a best version of the temporal switching pattern in terms of a residual error and a condition number.

7. The system as set forth in claim 6 wherein the image extraction process is constructed and arranged to determine the best version of the temporal switching pattern by (a) using each of a plurality of candidate temporal switching patterns to respectively generate the blur matrix;

(b) solving for a set of blur transform coefficients with respect to each blur matrix respectively;

(c) computing respectively the residual error and the condition number with respect to each set of blur coefficients;

(d)comparing a current value of the residual error and a current value of the condition number to a previously existing minimum value of the residual error and a previously existing minimal value of the condition number to determine if a current one of the candidate temporal switching patterns represents a better solution for at least one of the residual error and the condition number, and if the current one of the candidate temporal switching patterns represents a better solution, saving the current one of the candidate switching patterns in association with the current value of the residual error and the current value of the condition number;

(e) and, when all of the candidate temporal switching patterns have been used, returning the best version of the temporal switching pattern.

8. The system as set forth in claim 5 wherein the coefficients include both positive and negative values.

9. The system as set forth in claim 1 wherein the moving image is acquired from a moving object operatively connected with an encoder that provides a signal related to movement of the object or scene, the temporal switching pattern being based upon the signal related to the movement.

10. The system as set forth in claim 9, further comprising an object-height-detection assembly constructed and arranged to measure a height of a surface facing the imaging assembly, and based upon the height, varying the temporal switching pattern in conjunction with the signal related to movement of the object provided from the encoder.

11. The system as set forth in claim 10 wherein the height-detection-assembly comprises a light curtain.

12. The system as set forth in claim 10 wherein the height-detection-assembly is constructed and arranged to measure the height based upon at least one of an optical and a sonar measurement.

13. The system as set forth in claim 1 wherein the moving image defines a blurred image and wherein blur therein is caused by at least one of object motion, camera motion, and a combination of object motion and camera motion.

14. A method for extracting a still image from an input moving image, comprising steps of:

a) providing an imaging assembly including a plurality of pixels, each of the pixels comprising a photosensitive element and a plurality of integrating storage sites, each of the pixels being constructed and arranged to direct a measurement of light impinging on the photosensitive element to any one of the plurality of integrating storage sites;

b) forming a temporally decomposed representation of the moving image in an exposure time interval by directing, according to a predetermined temporal switching pattern, the measurement of light impinging on each of the photosensitive elements among the plurality of integrating storage sites, such that the temporally decomposed image is formed with a temporally decomposed point-spread-function having an inverse transformation that is optimally well-conditioned; and

c) operating on the temporally decomposed representation of the moving image to extract the still image.

15. The method as set forth in claim 14, further comprising selecting the temporal switching pattern to, at least one of, (a) minimize attenuation of spatial frequencies in the temporally decomposed representation of the moving image;

(b) minimize, by the image extraction process, attenuation of spatial frequencies in the still image;

(c) provide, by the image capture process and the image extraction process, a still image that contains a filtered set of spatial frequencies; and

(d) provide, by the image capture process and the image extraction process a still image having reduced blur.

16. The method as set forth in claim 15 wherein the step of operating on the temporally decomposed representation includes providing a blur matrix based upon the point-spread-function, and basing the blur matrix upon a best version of the temporal switching pattern in terms of a residual error and a condition number.

17. The method as set forth in claim 16 wherein the step of operating on the temporally decomposed representation includes determining the best version of the temporal switching pattern by (a) using each of a plurality of candidate temporal switching patterns to respectively generate the blur matrix;

(b) solving for a set of blur transform coefficients with respect to each blur matrix respectively;

(c) computing respectively the residual error and the condition number with respect to each set of blur coefficients;

(d)comparing a current value of the residual error and a current value of the condition number to a previously existing minimum value of the residual error and a previously existing minimal value of the condition number to determine if a current one of the candidate temporal switching patterns represents a better solution for at least one of the residual error and the condition number, and if the current one of the candidate temporal switching patterns represents a better solution, saving the current one of the candidate switching patterns in association with the current value of the residual error and the current value of the condition number;

(e) and, when all of the candidate temporal switching patterns have been used, returning the best version of the temporal switching pattern.

18. The method as set forth in claim 14, further comprising capturing the moving image from a moving object operatively connected with an encoder that provides a signal related to movement of the object or scene, and basing the temporal switching pattern upon the signal related to the movement.

19. The method as set forth in claim 18, further comprising measuring, with an object-height-detection assembly, a height of a surface facing the imaging assembly, and based upon the height, varying the temporal switching pattern in conjunction with the signal related to movement of the object provided from the encoder.

20. A system for acquiring a still image from an input moving image, comprising:

a) an imaging assembly including a plurality of pixels, each of the pixels comprising a photosensitive element and a plurality of integrating storage sites, each of the pixels being constructed and arranged to direct a measurement of light impinging on the photosensitive element to any one of the plurality of integrating storage sites;

b) an image capture process that forms a temporally decomposed representation of the moving image in an exposure time interval by repeatedly switching the measurement of light impinging on each of the photosensitive elements among the plurality of integrating storage sites according to a predetermined temporal switching pattern, wherein the predetermined temporal switching pattern is the temporal switching pattern that results in a temporally decomposed point-spread-function whose mathematical inverse is maximally sparse; and

c) an image extraction process that operates on the temporally decomposed representation of the moving image to extract the still image.