Scanning apparatus and method

• US 20060146377A1
• Filed: 02/27/2004
• Published: 07/06/2006
• Est. Priority Date: 03/07/2003
• Status: Active Grant

• Current Assignee: Qinetiq Limited (QinetiQ Group PLC)
• Original Assignee: Qinetiq Limited (QinetiQ Group PLC)
• Inventors: Sinclair, Gordon Neil, Marshall, Gillian Fiona, Barrett, David John, Hughes, Janet, Kent, Philip John, Coward, Peter Russell, Savage, John Timothy, Hawkesley, Piers Thomas Winchcombe, Gillham, John Peter, Appleby, Roger
• Granted Patent: US 7,675,655 B2
• Time in Patent Office: Days
• US Class Current: 358/486
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

Apparatus for scanning a moving object includes a visible waveband sensor 12 oriented to collect a series of images of the object as it passes through a field of view 16. An image processor 14 uses the series of images to form a composite image. The image processor 14 stores image pixel data for a current image and a predecessor image in the series. It uses information in the current image and its predecessor to analyse images and derive likelihood measures indicating probabilities that current image pixels correspond to parts of the object. The image processor 14 estimates motion between the current image and its predecessor from likelihood weighted pixels. It generates the composite image from frames positioned according to respective estimates of object image motion. Image motion may alternatively be detected by a speed sensor such as a Doppler radar 200 sensing object motion directly and providing image timing signals.

Claims

1. Apparatus for scanning a moving object, the apparatus having a sensor oriented to collect a series of images of the object as it passes through a sensor field of view and image processing means arranged to input the series of images and to form a composite image therefrom, the image processing means includes:

a) motion estimating means for generating an estimate of object image motion between images in the series of images; and

b) display processing means for generating a composite image to which the series of images contributes at positions derived from estimated object image motion.

2. Apparatus according to claim 1 wherein in

a) the motion estimating means comprises an image analysing means for deriving, for pixels within a current image from the series of images, a likelihood measures indicating likelihoods of the respective pixels corresponding to a part of the object;

and

b) the series of images includes a predecessor image obtained prior to the current image and the motion estimating means is arranged to generate an estimate of object image motion between the current image and its predecessor from image pixels weighted in accordance with the likelihood measure.

3. Apparatus according to claim 2 wherein the sensor has an optic axis aligned to be substantially perpendicular to a direction of motion prearranged for a moving object to be scanned.

5. Apparatus according to claim 3 wherein the image analysing means is arranged to derive the likelihood measure for each pixel based on a comparison of that pixel'"'"'s it intensity in the current image with the intensity of a pixel located equivalently in the predecessor image.

6. Apparatus according to claim 5 wherein the likelihood measure for each pixel is proportional to the square of the difference between pixel intensity in the current image and the intensity of the equivalently located pixel.

7. Apparatus according to claim 5 wherein the image analysing means is arranged to derive the likelihood measure for each pixel additionally based on a comparison of intensity of the current image pixel with intensity of a pixel in the predecessor image estimated to be imaging an object region which is the same as that imaged by the current image pixel, the pixel in the predecessor having a location derived by motion compensation in accordance with an assumption that a current estimate of object image motion represents true image motion.

8. Apparatus according to claim 7 wherein the likelihood measure is derived from intensity of the current image pixel, intensity of a second pixel located in the predecessor equivalently to the current image pixel, and intensity of a third pixel which is in the predecessor image and is estimated to be imaging an object region which is the same as that imaged by the current image pixel, the third pixel having a location derived by compensation for object motion, and estimates of standard deviations of noise in the object image intensity and background image intensity σ

_b in accordance with equations as follows;

$\mathrm{likelihood}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{measure}=w_f<br><br>\left(x,y\right)=\frac{p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)}{p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)+p_b<br><br>\left(x,y\right)},<br><br>\mathrm{where}<br><br>\text{;<br><br>}$ $p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)=\frac{1}{{\mathrm{\sigma <br><br>}}_f<br><br>\sqrt{2<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\left(-\frac{{\left(I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i,y\right)\right)}^2}{2<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\sigma <br><br>}}_f^2}\right),<br><br>\mathrm{and}$ $p_b<br><br>\left(x,y\right)=\frac{1}{{\mathrm{\sigma <br><br>}}_b<br><br>\sqrt{2<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\text{\hspace{1em}<br><br>}<br><br>\left(-\frac{{\left(I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x,y\right)\right)}^2}{2<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\sigma <br><br>}}_b^2}\right)$

9. Apparatus according to claim 5 wherein the likelihood measure is derived from intensity of the current image pixel, intensity of a second pixel equivalently located in its predecessor and intensity of a third pixel which is in the predecessor image and which is estimated to be imaging an object region which is the same as that imaged by the current image pixel, the third pixel having a location derived by compensation for object motion, and the likelihood measure being given by:

$w_f<br><br>\left(x,y\right)=\frac{{\left(I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x,y\right)\right)}^n}{{\left(I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x,y\right)\right)}^n+{\left(I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i,y\right)\right)}^n+c<br><br>\text{\hspace{1em}<br><br>}}$

where w_f(x,y) is the likelihood measure, I^t_p(x,y), I^t-1_p(x,y) and I^t-1_p(x+Δ

x_i, y) are the intensities of the first, second and third pixels respectively, n is a positive number and c is a non-negative constant.

12. Apparatus according to claim 2 wherein the image analysing means is arranged to use a first estimate of object motion to calculate a second such estimate, and thereafter to calculate iteratively the likelihood measure based on an estimate of object motion generated by the motion estimating means in a previous iteration, and wherein the motion estimating means is arranged to generate iteratively an estimate of object motion for a current iteration based on likelihood measures updated using respective previous motion estimates

13. Apparatus according to claim 2 wherein the sensor is a framing sensor with a substantially rectangular field of view and the display processing means is arranged, for each of the series of images, to take image data from a plurality of strips within the sensor field of view and to generate a corresponding plurality of composite images, each composite image having a contribution from a respective strip in each image.

14. Apparatus according to claim 2 wherein the sensor is a first sensor and the apparatus includes at least one other sensor oriented to obtain a respective series of images of the object as the object passes through the at least one other sensor'"'"'s field of view, the at least one other sensor being at least one of the visible band imager, a millimetre wave imager and a thermal imager and the display processing means is further arranged to generate a respective composite image from each series of images in accordance with the estimate of image motion derived from the first sensor'"'"'s series of images.

17. Apparatus according to claim 1 wherein the motion estimating means is an object speed sensor.

18. Apparatus according to claim 17 wherein the object speed sensor is a Doppler radar providing an output signal with frequency proportional to object speed.

19. Apparatus according to claim 18 arranged to provide a common clock signal to time the Doppler radar output signal and collection of images in the series of images in terms of counts of clock signal pulses and to provide respective frame signals to time images and time/position data derived from the Doppler radar output signal to time object movement, to perform an interpolation to relate image timing to object position and form the composite image from images located in accordance with object position.

48. Apparatus according to claim 1 wherein the moving object is a vehicle controlled by a driver, and wherein the image processing means include image analysing means for deriving, for each of a number of pixels within a current image in the series of images, a respective likelihood measure indicating likelihood of pixel corresponding to a part of the vehicle, the image analysing means being arranged to use information contained within the current image and a predecessor thereof.

4. (canceled)

10-11. -11. (canceled)

15-16. -16. (canceled)

20. (canceled)

21. (canceled)

22. A method of constructing a composite image from a series of images taken at a sensor as an object moves through a field of view of the sensor, the method comprising the steps of:

a. inputting a current image for processing;

b. deriving an estimate of object image motion;

c. designating successive images in the series of images as the current image and iterating steps a) and b) for each current image so designated; and

d. constructing a composite image to which the series of image frames contributes at positions derived from the estimate of object image motion.

23. A method according to claim 22 wherein step b) comprises

b1) for pixels in the current image frame, deriving likelihood measures indicating likelihoods of respective pixels corresponding to parts of the object;

b2) deriving an estimate of object image motion from image pixel information weighted in accordance with the likelihood measures;

b3) iterating steps (b1) and (b2) until convergence of the estimate or a prearranged iteration limit is reached using the estimate of object image motion derived at step (b2) to update the likelihood measures derived at step (b1), thereby generating a final estimate of object image motion for the current image frame; and

step d) comprises constructing a composite image to which the series of image frames contributes at positions derived from the final estimate of object image motion.

24. A method according to claim 23 wherein the series of image frames includes a predecessor image obtained prior to the current image, and step (b) includes comparing information contained within the current image with that contained within a predecessor image in order to derive the likelihood measures.

25. A method according to claim 24 wherein step (b) includes for each pixel in the current image implementing a comparison of pixel intensity with that of a pixel at an equivalent location in the predecessor image and deriving the likelihood measure using such comparison.

26. A method according to claim 25 wherein the likelihood measure is proportional to a square of a difference between its intensity in the current image and that of an equivalently located pixel in the predecessor.

27. A method according to claim 25 wherein, for pixels in the current image step (b) additionally includes:

b4) compensating for relative object image motion between the current image and the predecessor on the basis of a current estimate of object image motion representing true image motion;

b5) implementing a comparison of intensities of current image pixels with intensities of respective equivalently imaging pixels in the predecessor, and

b6) deriving the likelihood measure from comparisons both with equivalently located pixels and with equivalently imaging pixels in the current image and the predecessor.

28. A method according to claim 27 wherein the likelihood measure for each pixel is derived from pixel intensity in the current image, the intensity of an equivalently located pixel in the predecessor image and the intensity of an equivalently imaging pixel in the predecessor image in accordance with an equation as follows:

$w_f<br><br>\left(x,y\right)=\frac{{<br><br>I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x,y\right)<br><br>}^n}{{<br><br>I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x,y\right)<br><br>}^n+{<br><br>I_p^t<br><br>\left(x,y\right)-I_p^{t-1}<br><br>\left(x+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i,y\right)<br><br>}^n+c}$

where w_f(x,y) is the likelihood measure, I^t_p(x,y), I^t-1_p(x,y) and I^t-1_p(x+Δ

x_i,y) are the intensities of the first, second and third pixels respectively, c is a non-negative constant and n>

0.

29. A method according to claim 27 wherein the measure of foreground likelihood for each pixel is also derived from estimates of standard deviations and of foreground and background noise intensities in accordance with equations as follows:

$\underset{\_}{w_f<br><br>\left(x,y\right)=\frac{p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)}{p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)+p_b<br><br>\left(x,y\right)}}<br><br>\underset{\_}{,\mathrm{where}<br><br>\text{\hspace{1em}<br><br>}<br><br>w_f<br><br>\left(x,y\right)<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{is}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{the}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{likelihood}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{measure},}$ $\underset{\_}{p_f<br><br>\left(x,y,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i\right)=\frac{1}{{\mathrm{\sigma <br><br>}}_f<br><br>\sqrt{2<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\left(-\frac{{\left(l_p^t<br><br>\left(x,y\right)-l_p^{t-1}<br><br>\left(x+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x_i,y\right)\right)}^2}{2<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\sigma <br><br>}}_f^2}\right)}<br><br>\underset{\_}{,\mathrm{and}}$ $\underset{\_}{p_b<br><br>\left(x,y\right)=\frac{1}{{\mathrm{\sigma <br><br>}}_b<br><br>\sqrt{2<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\left(-\frac{{\left(l_p^t<br><br>\left(x,y\right)-l_p^{t-1}<br><br>\left(x,y\right)\right)}^2}{2<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\sigma <br><br>}}_b^2}\right)}$

30. A method according to claim 24 wherein the predecessor image is that immediately precedes the current image in the image series.

31. A method according to claim 23 including an additional step of constructing respective composite images from at least one additional sensor having a field of view through which the object passes, each respective composite image being constructed by placing each current image from the at least one sensor at a respective position derived from the associated converged estimate of object image motion generated at step d).

32. A method according to claim 22 wherein step (b) of deriving an estimate of object image motion is implemented by means of an object speed sensor.

33. A method according to claim 32 wherein the object speed sensor is a Doppler radar providing an output signal with frequency proportional to object speed.

34. A method according to claim 33 employing a common clock signal and collection of images in the series of images in terms of counts of clock signal pulses and also employing respective frame signals to time images and time/position data derived from the Doppler radar output signal to time object movement, to perform an interpolation to relate image timing to object position and form the composite image from images located in accordance with object position.

36. A method according to claim 22 including the steps of:

a. associating a respective estimate of object image motion with each image when designated as a current image;

b. associating the current image with a respective position in a data accumulator derived from summation of estimates of object image motion for the current image and like estimates obtained for earlier captured images in the series; and

c. iterating steps a) and b) for the series of images and forming a composite image from image central portions located according to respective positions.

37. A method according to claim 36 wherein each image central portion is at least as wide as an associated estimate of object image motion.

38. A method according to claim 36 wherein step b) also includes the step of incrementing a count accumulator at pixel positions corresponding to those to which data is added to the data accumulator as appropriate to indicate the number of such additions for each pixel, and step c) comprises the step of dividing data accumulator pixel values with respective correspondingly positioned count accumulator pixel values.

39. A method according to claim 38 including weighting both the count accumulator increment and pixel values in a central portion of the current image by left and right weighting factors to provide weighted increments and weighted pixel values respectively, the weighted increments are added to first and second integral pixel positions in the count accumulator and the weighted pixel values added to like-located pixel positions in the data accumulator, the integral pixel positions being those adjacent an estimatedly true pixel position derived from summation of object image motion for the current image and predecessors, and the left and right weighting factors being determined having regard to differences between the true pixel position and its adjacent integral pixel positions.

40. A method according to claim 38 including weighting both the count accumulator increment and pixel values in a central portion of the current image by weighting factors to provide products which are added to corresponding integral pixel positions within respective accumulators, the integral pixel positions being those adjacent an estimatedly true pixel position derived from summation of object image motion for the current image and earlier captured images and the series of weighting factors being determined in accordance with a smooth interpolation function.

41. A method according to claim 38 wherein step c) comprises the steps of:

ci) applying a spatial smoothing filter to data in the data accumulator and count accumulator; and

cii) forming the composite image either by dividing data accumulator pixels value by respective correspondingly positioned count accumulator pixel values or by dividing a filtered data accumulator pixel values by respective correspondingly positioned filtered count accumulator pixel values, depending on whether or not respective count accumulator pixel values are non-zero in each case.

42. A method according to claim 38 wherein step c) comprises the steps of:

c1) setting an initial neighbourhood range of one pixel for each pixel in the count accumulator;

c2) implementing a count accumulator summation of pixel values within its respective neighbourhood range for each pixel in the count accumulator;

c3) implementing a check regarding whether or not the count accumulator summation is significantly non-zero;

c4) if the check outcome is untrue indicating that the count accumulator summation is not significantly non-zero, iterating steps c2) and c3) with increase in neighbourhood range until the check is true;

(c5) for a data accumulator pixel equivalently located to the count accumulator pixel referred to in step c2), implementing a data accumulator summation over pixel values within a neighbourhood range equal to that for which the check indicated a significantly non-zero summation;

c6) obtaining a division result by dividing the data accumulator summation by the significantly non-zero count accumulator summation; and

c7) adding the division result to a pixel in a composite image located correspondingly to the count accumulator pixel referred to in step c2).

43. A method according to claim 36 wherein the sensor is a first sensor and the method incorporates use of a plurality of sensors of which the first sensor is one, and the sensors include a second sensor obtaining data to generate the estimate of object image motion and the method includes rescaling or interpolating this estimate before associating it with a respective image.

44. A method according to claim 36 including the step of measuring object speed using a speed sensor and deriving from this measurement a respective estimate of object image motion associated with each of the series of images.

45. A method according to claim 36 wherein the sensor is a framing sensor with a substantially rectangular field of view and step b) includes the additional step of writing at least one co-located non-central portion of data contained within each of the series of images to respective further image data accumulating means and step c) includes forming a plurality of composite images from data contained in respective image data accumulator arrays.

35. (canceled)

46. A computer-readable medium embodying program code instructions for execution by computer processing apparatus, the instructions relating to processing image data obtained using a sensor having a field of view through which an object moves, wherein the instructions are executable to provide for:

a. inputting a current image in a series of images for processing;

b. deriving an estimate of object image motion;

c. iterating steps a) and b) for the series of images; and

d. constructing a composite image to which the series of images contributes at positions derived from respective estimates of object image motion.

47. A computer-readable medium according to claim 46 wherein the instructions embodied in the computer-readable medium provide for:

a. accessing pixel data relating to a current image in a series of images;

b. deriving likelihood measures indicating likelihoods that pixels in the current image are part of the object image, the likelihood measures being derived from information contained with the current image and a predecessor image in the series of images obtained prior to the current image;

c. deriving an estimate of object image motion from image pixel information weighted in accordance with corresponding likelihood measures;

d. iterating steps b) and c) until convergence or a prearranged iteration limit is reached, whichever occurs first,

e. using the estimate of foreground motion derived at step c) to update the likelihood measure derived at step b), thereby generating a final estimate of foreground image motion for the current image;

f. designating successive images in the series of images as current images and iterating steps a) to e) for each current image; and

g. constructing a composite image from the series of images, each image being positioned in the composite image in accordance with its respective final estimate of foreground image motion.