METHOD, SYSTEM, AND COMPUTER PROGRAM PRODUCT TO DETECT MICRON SCALE DISPLACEMENT OF OBJECTS FROM A LONG RANGE WITH AND OPTICAL DEVICE AND RELATED COMPUTER IMPLEMENTED METHODS

• US 20100328070A1
• Filed: 06/26/2009
• Published: 12/30/2010
• Est. Priority Date: 06/26/2009
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Murray, Robert Carl
• Granted Patent: US 8,270,678 B2
• Time in Patent Office: Days
• US Class Current: 340/568.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning

Abstract

Embodiments of systems, program products, and computer implemented methods to measure the displacement of an object located in a hazardous or otherwise inaccessible location at a long range from an optical device with micron-level accuracy are provided the object being. The objects can be machinery, valves, containers, or any other object whose displacement is to be measured. The object can be located in radioactive, chemically reactive, high voltage, or otherwise hazardous or inaccessible locations that are not accessible for conventional displacement measurement by personnel. A system can comprise an identifier on the object to be tracked, an optical device, an computer with at least processing, storage, and memory facilities, and a communications network.

Claims

1. A computer implemented method to detect displacement of objects from a long range, the computer implemented method comprising the steps of:

determining a cross-correlation between at least a portion of a first numerical function and a second numerical function by shifting a pixel at a time the portion of the first numerical function over the second numerical function to thereby determine the cross-correlation as a cross-correlation numerical value, the first numerical function derived from a first image of an identifier located on an object to be tracked when located at a first position, the second numerical function derived from a second image of the identifier located on the object to be tracked when located at a second position, the second image taken at a time temporally later than a time for the first image, the first image and the second image obtained by an optical device, the cross-correlation numerical value determined between the first image and the second image at a pixel produced by the optical device;

repeating the step of determining the cross-correlation between the portion of the first numerical function and the second numerical function by shifting the portion of the first numerical function over the second numerical function by a pixel offset to thereby generate a plurality of cross correlation numerical values where each cross-correlation numerical value corresponds to a distinct pixel;

comparing the plurality of cross-correlation numerical values to thereby determine a relative pixel location at which a peak cross-correlation value occurs;

fitting a curve through the peak cross-correlation value by selecting two cross-correlation values adjacent and to either side of the peak cross-correlation value to refine the determined relative pixel location;

determining a number of pixels the portion of the first numerical function moved that resulted in the peak cross-correlation value with the second numerical function, the peak cross-correlation corresponding to the refined relative pixel location; and

determining a displacement of the object to be tracked by dividing the number of pixels that the portion of the first numerical function moved by a pixel per measurement unit, the displacement being perpendicular to a line of sight with respect to the optical device.

2. A computer implemented method as defined in claim 1, further comprising the steps of:

converting the first image of the identifier located on the object to be tracked into the first numerical function, the object being at the first position;

selecting a portion of the first numerical function by using a window function to thereby extract the portion of the first numerical function;

removing noise from the portion of the first numerical function selected by the window function;

converting the second image of the identifier located on the object to be tracked into the second numerical function, the object being at the second position; and

removing noise from the second numerical function.

3. A computer implemented method as defined in claim 2, wherein the step of converting the first image of the identifier located on the object to be tracked comprises the steps of:

converting the first image of the identifier located on the object to be tracked into a first numerical array, and

converting the first numerical array into the first numerical function.

4. A computer implemented method as defined in claim 2, wherein the step of removing noise from the portion of the first numerical function selected by the window function comprises the steps of:

calculating at each pixel offset an average value for an image vector having a magnitude that exceeds the magnitude of a plurality of vectors found in the portion of the first numerical function extracted from the first numerical function; and

subtracting at each pixel offset the average value of each image vector from each original vector in the portion of the first numerical function extracted from the first numerical function.

5. A computer implemented method as defined in claim 2, wherein the step of removing noise from the second numerical function comprises the step of:

subtracting at each pixel offset the calculated average value of each image vector from each original vector in the second numerical function.

6. A computer implemented method as defined in claim 1, wherein the step of determining the cross-correlation between the portion of the first numerical function and the second numerical function by shifting the portion of the first numerical function over the second numerical function by a pixel offset to thereby generate a plurality of cross correlation numerical values where each cross-correlation numerical value corresponds to a distinct pixel, comprises the steps of:

a. overlapping the portion of the first numerical function over the second numerical function and offsetting the portion of the first numerical function by one pixel to define a pixel offset;

b. executing a cross-correlation algorithm between the portion of the first numerical function and the second numerical function at the pixel offset to calculate the first of a plurality of cross-correlation values;

c. storing the cross-correlation value at the pixel offset for comparison;

d. repeating steps a-c until the function portion extracted from the first numerical function is offset through the last pixel of the second numerical function;

e. normalizing the cross-correlation value results.

7. A computer implemented method as defined in claim 1,

wherein the curve fit through the peak cross-correlation value and two cross-correlation values adjacent and to either side of the peak correlation value is a 2^nd order polynomial;

wherein the displacement measurement is corrected for distortion by subtracting a peak-cross-correlation value for an identifier located on a fixed reference surface from the peak cross-correlation value for the identifier located on the object to be tracked, and

wherein the steps to determine the peak cross-correlation value for the fixed reference surface are substantially the same as the steps used to obtain the peak cross-correlation value for the identifier located on the object to be tracked.

8. A computer implemented method as defined in claim 1,

wherein determining the displacement of the object is for an object located in one or more of the following locations:

a radioactive location;

a chemically reactive location,

a high voltage location,

a hazardous location; and

inaccessible location;

wherein the object is located at least 100 yards from the optical device;

wherein the determined object displacement is in an axis perpendicular to the line of sight between the object and the optical device; and

wherein one or more of the following actions is taken in response to the calculated displacement;

the object is moved to a desired position;

a notice is issued by a warning system;

or

an alarm is issued by an alarm system.

9. A computer implemented method as defined in claim 8, wherein the determined object displacement is in an axis vertical to the object.

10. A computer implemented method as defined in claim 1,

wherein the determination of the number of pixels the portion of the first numerical function moved that resulted in the peak cross-correlation value with the second numerical function has an accuracy of at least 1/100^th of a pixel;

and

wherein the accuracy of the determined displacement of the object is at least 0.02 microns.

11. A computer implemented method as defined in claim 1, wherein the computer implemented method further includes the steps of:

storing the images and image data in a first database stored on a computer readable storage medium of the computer;

storing the numerical functions of the images in the first database stored on a computer readable storage medium of the computer;

storing the numerical functions of the images in the first database stored on a computer readable storage medium of the computer, the noise being removed from the numerical functions;

storing the cross-correlation values in a second database stored on a computer readable storage medium of the computer;

storing the object displacement data in pixels in the second database stored on a computer readable storage medium of the computer; and

storing the object displacement data in units of measurement in the second database stored on a computer readable storage medium of the computer.

12. A system to detect the displacement of objects from a long range, the system comprising:

an optical device for obtaining a first image and a second image of an identifier located on an object to be tracked;

a computer in communication with the optical device and having a processor and a memory element; and

a program product stored in the memory element of the computer, the program product comprising;

an image capturer configured to capture the first image and the second image of the identifier located on the object to be tracked to thereby convert the first image into a first numerical function and the second image into a second numerical function, the first image and the second image obtained by the optical device;

an image preprocessor configured to remove noise from the first numerical function and the second numerical function by removing a plurality of image vectors from each of the first and the second numerical functions;

an image cross-correlator configured to compare a portion of the first numerical function for the identifier located on the object to be tracked to the second numerical function of the same identifier to thereby determine a peak cross-correlation value between the portion of the first numerical function and the second numerical function at a pixel generated by the optical device, the portion of the first numerical function defining an extracted function portion of the first numerical function;

the image cross-correlator comparing by shifting the portion of the of the first numerical function pixel by pixel in an axis to thereby determine a cross-correlation value at each pixel and thereby generate a plurality of cross-correlation values, the peak cross-correlation peak value chosen from among the plurality of cross-correlation values to determine a relative pixel location,

the image cross-correlator further configured to fit a curve around the peak correlation value to determine a refined pixel location,

the image cross-correlator further configured to determine a displacement of the object to be tracked by utilizing the refined pixel location to determine a number of pixels the extracted portion of the first numerical function moved that resulted in the peak correlation value with the second numerical function; and

the image cross-correlator further configured to determine a displacement of the object to be tracked by dividing the number of pixels that the extracted portion of the first numerical function moved by a pixel per unit of measurement, the displacement being perpendicular to a line of sight with respect to the optical device.

13. A system as defined in claim 12,

wherein the optical device further obtains a first image and a second image of an identifier located on a fixed reference surface;

wherein the image capturer is further configured to capture the first image of the identifier located on the fixed reference surface to thereby convert the first image into a first numerical function, the first image obtained by the optical device;

wherein the image capturer is further configured to capture the second image of the identifier located on the fixed reference surface to thereby convert the second image into a second numerical function, the second image obtained by the optical device;

wherein the image cross-correlator is further configured to correct for distortion in the displacement measurement by;

determining a peak-cross-correlation value for the identifier located on the fixed reference surface; and

subtracting the peak-cross-correlation value for the identifier located on the fixed reference surface from the peak cross-correlation value for the identifier located on the object to be tracked;

wherein the optical device is one or more of the following;

a digital camera;

or

a charge coupled device camera;

wherein the digital camera has a field of view that obtains images of objects that move between 0.1 inches to 2 inches;

or

wherein the charge coupled device camera has a field of view that obtains images of objects that move between 0.1 inches to 2 inches;

wherein the identifier located on the object to be tracked includes a plurality of darkened stripes of random width and random spacing; and

wherein the identifier located on the fixed reference surface includes a plurality of darkened stripes of random width and random spacing;

wherein the stripes on the identifier located on the object to be tracked have a height of 1 inch and a maximum width of 1 inch;

wherein the stripes on the fixed reference surface have a height of 1 inch and a maximum width of 1 inch.

14. A system as defined in claim 12, further comprising:

a computer readable storage medium storing a plurality of images and image data for each of the object to be tracked and the fixed reference surface, a plurality of image numerical functions, and a plurality of cross-correlation data and displacement data.

15. A system as defined in claim 12, wherein the images and image data for the object to be tracked and the fixed reference surface stored on the computer readable storage medium is for a plurality of images obtained by the optical device.

16. A system as defined in claim 12,

wherein the computer further includes a graphical user interface device and a user interface input device;

wherein the image capturer is further configured to display via the graphical user interface of the computer the images taken by the optical device; and

wherein image cross-correlator is further configured to display via the graphical user interface device of the computer the numerical functions that represent images, cross-correlation values, peak cross-correlation value, refined peak-cross correlation value, displacement in pixels, and displacement of the object in units of measure.

17. A system as defined in claim 16, wherein the image cross-correlator is further configured to normalize the cross-correlation coefficient results.

18. A system as defined in claim 16, wherein the computer is a first computer the system further comprising a second computer having at least a processor and a memory element, wherein the graphical user interface and the user input device are located on the second computer, and wherein the second computer is in communication with the first computer.

19. A system as defined in claim 18, further comprising a third computer wherein the computer readable storage medium is stored on the third computer accessible by the first computer and the second computer.

20. A system as defined in claim 18, wherein the first computer and second computer are positioned to communicate via a communications network;

and

wherein the optical device is positioned to communicate with the first computer and the second computer via the communications network.

21. A system as defined in claim 12, wherein the image cross-correlator is further configured to determine the displacement of the object within one of the following:

at least 1/100^th of a pixel accuracy;

or

at least 0.02 micron accuracy.

22. Program product to detect the displacement of objects from a long range, the program product stored in a tangible computer readable medium and including instructions that when executed by a computer, cause the computer to perform the operations of:

converting a first image of an identifier into a first numerical function and a second image of the identifier into a second numerical function, the identifier located on an object to be tracked, the first numerical function derived from a first numerical array representing the first image of the identifier located on the object to be tracked when located at a first position, the second numerical function derived from a second numerical array representing the second image of the identifier located on the object to be tracked when located at a second position, the first and second numerical arrays each derived from a plurality of pixels occupied by a plurality of stripes on the identifier located on the object to be tracked, the pixels generated by an optical device, the stripes being of random width and having random spacing, the second image taken at a time temporally later than a time for the first image, the first image and the second image of the identifier located on the object to be tracked obtained by the optical device;

obtaining a portion of the first numerical function for comparison with the second numerical function, by extracting the portion from the first numerical function with a window function having a length and a height;

removing a plurality of image vectors from the first numerical function and a plurality of image vectors from the second numerical function of the identifier located on the object to be tracked to facilitate the identification of a peak cross-correlation value, the removed image vectors having a magnitude exceeding the height of the window function used to extract the portion of the first numerical function;

determining a relative pixel location at which a peak cross-correlation value occurs by comparing the portion extracted from the first numerical function to the second numerical function of the same identifier located on the object to be tracked, comparing including shifting portion extracted from the first numerical function pixel by pixel over the second numerical function in an axis perpendicular to a line of sight of the optical device to thereby generate a cross-correlation value at each pixel, the relative pixel location at which the peak cross-correlation value determined from among the cross-correlation values generated at each pixel;

fitting a polynomial around the determined peak cross-correlation value to refine the determined relative pixel location at which the peak cross-correlation value occurs;

determining a number of pixels the portion of the first numerical function moved that resulted in the peak cross-correlation value with the second numerical function, the peak cross-correlation corresponding to the refined relative pixel location; and

determining a displacement of the object to be tracked by dividing the number of pixels that the portion of the first numerical function moved by a pixel per measurement unit, the displacement being perpendicular to a line of sight with respect to the optical device.

23. A program product as defined in claim 22,

wherein the operation of converting the images is performed by one of the following optical devices:

a digital camera, or

a charge coupled device camera;

wherein the accuracy of the determined displacement of the object is one of the following;

at least 1/100^th of a pixel accuracy;

or

at least 0.02 micron accuracy; and

wherein the operation of determining displacement further includes the operations of;

determining a peak cross-correlation value for an identifier located on a fixed reference surface;

subtracting the determined peak cross-correlation value for the identifier on the fixed reference surface from the determined peak cross-correlation value for the identifier located on the object to be tracked to correct the displacement measurement of the object for distortion.

24. A computer readable program product as defined in claim 22,

wherein the operation of converting a first image into a first numerical function and a second image into a second numerical function of the identifier located on the object to be tracked is executed in response to an input via a user interface device initiating the displacement determination;

wherein the operations of the program product are performed by a first computer in communication with a second computer;

wherein the operation of converting a first image into a first numerical function and a second image into a second numerical function of the identifier located on the object to be tracked is performed by the optical device, the optical device in communication with the first computer and the second computer;

wherein the operation of removing a plurality of image vectors from the first numerical function and a plurality of image vectors from the second numerical function of the identifier located on the object to be tracked further comprises instructions that when executed by the first computer cause the first computer to perform the operations of;

calculating at each pixel offset the average value of each image vector having a magnitude that exceeds the magnitude of the vectors found in the portion extracted from the first numerical function by the window function; and

subtracting at each pixel offset the average value of each image vector from each original vector;

wherein the operation of determining a displacement of the object to be tracked is performed for an object located in one of the following locations;

a radioactive location;

a chemically reactive location,

a high voltage location,

a hazardous location;

or

inaccessible location.

25. A computer readable program product as defined in claim 22,

wherein determining a peak cross-correlation value between the function portion extracted from the first numerical function and a second numerical function to thereby determine a relative pixel location includes instructions that when executed by the computer cause the computer to perform the operations of:

applying an algorithm utilizing the following equation;