Method and apparatus for uniform lineal motion blur estimation using multiple exposures

• US 20010021224A1
• Filed: 12/14/2000
• Published: 09/13/2001
• Est. Priority Date: 12/14/1999
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Larkin, Kieran Gerard, Gibson, Ian Richard, Yip, Dominic
• Granted Patent: US 6,888,566 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method and apparatus (9) for estimating the blur parameters of blurred images (g₁, g₂) are disclosed. The apparatus (9) has one or more image sensors (10) for capturing the blurred images (g₁, g₂), a plurality of correlators (20, 30) for performing autocorrelation of the blurred image (g₁) and cross-correlation between the two images (g₁, g₂) respectively, and an error function calculator (40) for evaluating an error function over all possible displacements using the results from the correlators (20, 30). The apparatus (9) further includes an extreme locater (50) for finding the displacement with the minimum value for the error function.

Claims

1. A method of estimating motion of a moving object, said method comprising the steps of:

capturing at least first and second blurred images of said moving object, the blur in said images arising from at least motion blur of said object, wherein exposure durations of said first and second blurred images overlap at least partially;

generating an error function, said error function being a function of said first blurred image and said second blurred image;

minimising said error function; and

estimating said motion of said object from said minimised error function.

2. A method according to

claim 1

, wherein said images are captured with a time difference between a start time of capture of said first blurred image and a start time of capture of said second blurred image.

3. A method according to

claim 1

, wherein a start time of capture of the first blurred image and a start time of capture of the second blurred image are concurrent.

4. A method according to

claim 2

, wherein said exposure duration of said first blurred image is substantially equal to said exposure duration of the second blurred image.

5. A method according to

claim 2

or

3

, wherein said exposure duration of said second blurred image is a predetermined integer multiple of said exposure duration of said first blurred image.

6. A method according to

claim 1

, wherein an exposure pattern (profile) of said exposure duration of at least one of said blurred images is non-uniform.

7. A method according to

claim 6

, wherein said exposure pattern (profile) comprises a triangular profile.

8. A method of estimating motion of a moving object, said method comprising the steps of:

capturing at least first and second blurred images of said moving object, the blur in said images arising from at least motion blur of said object, wherein exposure durations of said first and second blurred images overlap at least partially;

generating an error function, said error function comprising a cross-correlation term being a cross-correlation between said first blurred image and said second blurred image;

minimising said generated error function; and

estimating said object motion from said minimised error function.

9. A method according to

claim 8

, wherein said error function further comprises an auto-correlation term being an auto-correlation of said first blurred image.

10. Apparatus for estimating motion of a moving object, said apparatus comprising:

one or more capture devices for capturing at least first and second blurred images of said moving object, the blur in said blurred images arising from at least motion blur of said object, wherein exposure durations of said blurred images overlap at least partially;

means for generating an error function, said error function being a function of said first blurted image and said second blurred image;

means for minimising said error function; and

means for estimating said motion of said object from said minimised error function.

11. Apparatus according to

claim 10

, wherein said images are captured with a time difference between a start time of capture of said first blurred image and a start time of capture of said second blurred image.

12. Apparatus according to

claim 10

, wherein a start time of capture of the first blurred image and a start time of capture of the second blurred image are concurrent.

13. Apparatus according to

claim 11

, wherein said exposure duration of said first blurred image is substantially equal to said exposure duration of the second blurred image.

14. Apparatus according to

claim 12

or

13

, wherein said exposure duration of said second blurred image is a predetermined integer multiple of said exposure duration of said first blurred image.

15. Apparatus according to

claim 10

, wherein an exposure pattern (profile) of said exposure duration of at least one of said blurred images is non-uniform.

16. Apparatus according to

claim 15

, wherein said exposure pattern (profile) comprises a triangular profile.

19. Apparatus according to

claim 10

wherein said means for generating, said means for minimising and said means for estimation collectively comprise a computer system incorporating a sequence of program instructions for estimating said motion using said images output from said capture device.

17. Apparatus for estimating motion of a moving object, said apparatus comprising:

one or more capture devices for capturing at least first and second blurred images of said moving object, the blur in said blurred images arising from at least motion blur of said object, wherein exposure durations of said blurred images overlap at least partially;

means for generating an error function, said error function comprising a cross-correlation term being a cross-correlation between said first blurred image and said second blurred image;

means for minimising said generated error function; and

means for estimating said object motion from said minimised error function.

18. Apparatus according to

claim 17

, wherein said error function further comprises an auto-correlation term being an auto-correlation of said first blurred image.

20. Apparatus for estimating motion of a moving image, said apparatus comprising:

an image sensor for capturing pixels of at least first and second images in which one of said images is sampled during a formation of at least one other of said images;

means for comparing said images to determine at least one motion vector for association with one of said images.

21. Apparatus according to

claim 20

wherein said image sensor comprises, for each said pixel:

an pixel sensor; and

pixel sampling means for sampling a value of said pixel sensor at a time corresponding to each of said images.

22. Apparatus according to

claim 21

wherein said image sensor farther comprises, for each said pixel, reset means for resetting the corresponding said pixel sensor after sampling of each of said images.

23. Apparatus according to

claim 22

wherein said image sensor comprises a pixel bus configured for sequential coupling to a plurality of said pixel sampling means, and a switching arrangement associated with said pixel bus, said switching arrangement comprising a least first and second storage devices corresponding to each of said images and each being configured to store temporal value of said pixel sensor at said corresponding time, and switching means for outputting said temporal values to represent corresponding pixel values of said images of said images.

24. Apparatus according to

claim 23

wherein said pixel sensor comprises a cell capacitor, a voltage upon said cell capacitor being modified by exposure of said pixel sensor to an image to be captured, and said storage devices comprise storage capacitors.

25. Apparatus according to

claim 23

wherein said image sensor comprises plural said pixel buses and associated switching arrangements, each associated with plural of said pixel sensors and corresponding pixel sampling means thereby forming a matrix of said pixel sensors.

26. Apparatus according to

claim 20

, wherein at least said second image comprises an exposure duration greater than that of said first image, thereby providing at least said second iimage as blurred representation of said first image where a target of said images presents motion.

27. Apparatus according to

claim 26

wherein said second image is a predetermined integer multiple of an exposure duration of said first image.

28. Apparatus according to

claim 26

wherein exposure durations of said first and second images overlap.

29. Apparatus according to

claim 20

, wherein said means for comparing comprises:

an auto-correlator for calculating auto-correlation of said first image; and

a cross-correlator for calculating cross-correlation between said first image and said second image; and

error means for minimising an error function between said auto-correlation and said cross-correlation.

30. Apparatus according to

claim 29

wherein said error means comprises an error function calculator for determining an error function value at each pixel location between said auto-correlation and said cross-correlation, and an extreme locator for thresholding said error function value at each said pixel location to provide movement parameters corresponding to each said pixel location.

31. Apparatus according to

claim 30

wherein said movement parameters comprise direction and magnitude.

32. Apparatus according to

claim 30

wherein said movement parameters comprise magnitude.

36. Apparatus for capturing a moving image, said apparatus comprising:

at least one image sensor according to any one of

claims 20

to

35

;

means for controlling said image sensor to capture a sequence of said first and second images and for determining a motion vector relating to image blur between each pair of said first and second images in said sequence;

means for combining the corresponding said motion vector with at least one of said first and second images in said sequence to output a sequence of images each having an associated motion vector.

37. Apparatus according to

claim 36

wherein said sequence of images output from said apparatus comprise a sequence of said second images.

38. Apparatus for capturing a moving image, said apparatus comprising:

a first image sensor, said first image sensor being configured according to any one of

claims 20

to

35

;

means for controlling said first image sensor to capture a first sequence of said first and second images and for determining a motion vector relating to image blur between each pair of said first and second images in said sequence;

a second image sensor for capturing a second sequence of images to be output from said apparatus;

means for combining each said motion vector of said first sequence with a corresponding image in said second sequence to output from said apparatus a sequence of images each having an associated motion vector.

39. Apparatus according to

claim 38

wherein said images captured by said second sensor have a resolution exceeding that of those captured by said first sensor.

40. Apparatus according to

claim 38

, said apparatus comprising optical means for viewing a scene to be captured and presenting said scene to each of said sensors.

41. Apparatus according to

claim 36

further comprising synchronisation means for synchronising said motion vectors with corresponding images forming said output sequence.

33. An image sensor for capturing pixels of at least first and second images characterised in that at least one of said images is sampled during a formation of at least one other of said images.

34. An image sensor according to

claim 33

wherein said at least two captured images each have a corresponding start exposure time and different end exposure times.

35. An image sensor according to

claim 34

, said image sensor comprising:

an array of pixel cells, each said pixel cell for storing an electrical charge equivalent of an intensity of light to which said cell is exposed; and

at least two arrays of capacitors, each of said arrays of capacitors comprises capacitors electrically connected with corresponding pixel cells;

wherein at each of said end exposure times said array of capacitors is arranged to store tho charge of said array of pixel cells at said end exposure time.

42. A computer program product including a computer readable medium incorporating a computer program estimating motion of a moving object, said computer program product comprising:

code for capturing at least first and second blurred images of said moving object, the blur in said images arising from at least motion blur of said object, wherein exposure durations of said first and second blurred images overlap at least partially;

code for generating an error function, said error function being a function of said first blurred image and said second blurred image;

code for minimising said error function; and

code for estimating said motion of said object from said minimised error function.

43. A computer program product according to

claim 42

, wherein said images are captured with a time difference between a start time of capture of said first blurred image and a start time of capture of said second blurred image.

44. A computer program product according to

claim 42

, wherein a start time of capture of the first blurred image and a start time of capture of the second blurred image are concurrent.

45. A computer program product according to

claim 43

, wherein said exposure duration of said first blurred image is substantially equal to said exposure duration of the second blurred image.

46. A computer program product according to

claim 43

or

44

, wherein said exposure duration of said second blurred image is a predetermined integer multiple of said exposure duration of said first blurred image.

47. A computer program product according to

claim 42

, wherein an exposure pattern (profile) of said exposure duration of at least one of said blurred images is non-uniform.

48. Apparatus for estimating motion of an image from a first and a second blurred image, said apparatus being substantially as described herein with reference to any one of the embodiments as that embodiment is illustrated in the accompanying drawings.