Image processing device and method

• US 7,860,276 B2
• Filed: 12/07/2006
• Issued: 12/28/2010
• Est. Priority Date: 12/08/2005
• Status: Active Grant

• Current Assignee: Topcon Corporation
• Original Assignee: Topcon Corporation
• Inventors: Sato, Hirokazu, Kochi, Nobuo, Anai, Tetsuji, Otani, Hitoshi
• Primary Examiner(s): Azarian; Seyed
• Application Number:
  US11/567,909
• Publication Number:
  US 20070140528A1
• Time in Patent Office: 1,482 Days
• Field of Search: 382/100, 382/103, 382/104, 382/106, 382/107, 382/154, 382/168, 382/181, 382190-194, 382/199, 382/209, 382/232, 382/254, 382/274, 382/276, 382/286, 382291-295, 382/305, 382/312, 396/55, 396/57, 396/58, 396/59, 348/218.1, 348/188, 348/207.99, 348/223.1, 348/187, 348/129, 340/435
• US Class Current: 382/107
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

A technique is provided that can precisely measure the photographing position and posture of a photographing device and the coordinates of an object from moving images or photographed images that sequentially change gradually, even in the case with the need to process the sway of the photographing device and the disappearance and reappearance of characteristic points. An image processing device 100 according to the present invention includes: a photographed image acquisition 2 section for sequentially acquiring photographed images, that change gradually, of a relatively moving object; a characteristic extraction section 3 for extracting characteristic points from a photographed image acquired with the photographed image acquisition section; a characteristic point tracking section 4 for searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and a processing section 5 for determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points.

Claims

1. An image processing device comprising:

a photographed image acquisition section for sequentially acquiring photographed images of a relatively moving object, the photographed images changing gradually;

a characteristic extraction section for extracting characteristic points from a photographed image acquired with the photographed image acquisition section;

a characteristic point tracking section for searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing section for determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the characteristic point tracking section uses the photographed image from which the characteristic points have been extracted and a plurality of the photographed images sequentially acquired prior or subsequent thereto to search for candidate corresponding points corresponding to the characteristic points;

forms the movement vectors based on positional relationship between candidate corresponding points found and the characteristic points;

selects candidate corresponding points judged to probably be corresponding points corresponding to the characteristic points based on the movement vectors and the screen relative movement amount;

searches a plurality of the photographed images sequentially acquired prior or subsequent thereto, for candidate corresponding points corresponding to the characteristic points and the corresponding points; and

sequentially repeats forming movement vectors, selecting candidate corresponding points and searching for candidate corresponding points.

2. An image processing device comprising:

a photographed image acquisition section for sequentially acquiring photographed images of a relatively moving object, the photographed images changing gradually;

a characteristic extraction section for extracting characteristic points from a photographed image acquired with the photographed image acquisition section;

a characteristic point tracking section for searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing section for determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the characteristic point tracking section uses tracking results of respective characteristic points between successive photographed images to obtain movement vectors of the respective characteristic points and a screen relative movement amount;

compares the movement vectors of the respective characteristic points with estimated positions of the respective characteristic points estimated from the screen relative movement amount; and

eliminates those with large differences as failing points.

3. The image processing device as recited in claim 2,

wherein the processing section makes an adjustment acquisition of successive images such that the screen relative movement amount between the successive images or between the first image and the second image is a predetermined amount.

4. The image processing device as recited in claim 2,

wherein the processing section makes an adjustment such that successive images or the first image and the second image are displaced from each other by a predetermined number of pixels.

5. An image processing device comprising:

a photographed image acquisition section for sequentially acquiring photographed images of a relatively moving object, the photographed images changing gradually;

a characteristic extraction section for extracting characteristic points from a photographed image acquired with the photographed image acquisition section;

a characteristic point tracking section for searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing section for determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the processing section detects disappearance of the characteristic points or the corresponding points; and

when a predetermined number of points have disappeared or a predetermined range of area or more has become vacant, commands the characteristic extraction section to create new characteristic points.

6. An image processing device comprising:

a photographed image acquisition section for sequentially acquiring photographed images of a relatively moving object, the photographed images changing gradually;

a characteristic extraction section for extracting characteristic points from a photographed image acquired with the photographed image acquisition section;

a characteristic point tracking section for searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing section for determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the processing section uses movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors of respective characteristic points of a first photographed image to create an estimated image related to a second photographed image acquired with a predetermined amount distanced therefrom (in time or space);

compares the second photographed image with the estimated image related thereto;

determines corresponding points from the candidate corresponding points; and

calculates coordinates of a photographing position or the object using the determined corresponding points.

7. The image processing device as recited in claim 6,

wherein the first photographed image is photographed prior or subsequent to the second photographed image temporally or spatially, and the movement tracks go forward or backward in time or space.

8. The image processing device as recited in claim 6,

wherein the characteristic point tracking section uses candidate corresponding points that were supposedly once excluded from a photographed image but reappear in a priorly or subsequently acquired photographed image, the priorly or subsequently acquired photographed image being defined as the second photographed image, to compare the second photographed image and an estimated image related thereto;

and restores candidate corresponding points judged to probably be the candidate corresponding points, as candidate corresponding points.

9. The image processing device as recited in claim 6, further comprising:

an image selection section for selecting the second image based on any one of an overlapping ratio, baseline length, and the movement vectors between photographed images acquired with the photographed image acquisition section.

10. An image processing method comprising:

a photographed image acquisition step of sequentially acquiring photographed images of a relatively moving object that change gradually;

a characteristic extraction step of extracting characteristic points from a photographed image acquired in the photographed image acquisition step;

a characteristic point tracking step of searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing step of determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the characteristic point tracking step comprises using the photographed image from which the characteristic points have been extracted and a plurality of the photographed images sequentially acquired prior or subsequent thereto to search for candidate corresponding points corresponding to the characteristic points;

forming the movement vectors based on positional relationship between candidate corresponding points found and the characteristic points;

selecting candidate corresponding points judged to probably be corresponding points corresponding to the characteristic points based on the movement vectors and the screen relative movement amount;

searching a plurality of the photographed images sequentially acquired prior or subsequent thereto, for candidate corresponding points corresponding to the characteristic points and the corresponding points; and

sequentially repeating forming movement vectors, selecting candidate corresponding points and searching for candidate corresponding points.

11. An image processing method comprising:

a photographed image acquisition step of sequentially acquiring photographed images of a relatively moving object that change gradually;

a characteristic extraction step of extracting characteristic points from a photographed image acquired in the photographed image acquisition step;

a characteristic point tracking step of searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing step of determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the characteristic point tracking step comprises using tracking results of respective characteristic points between successive photographed images to obtain movement vectors of the respective characteristic points and a screen relative movement amount;

comparing the movement vectors of the respective characteristic points with estimated positions of the respective characteristic points estimated from the screen relative movement amount; and

eliminating those with large differences as failing points.

12. An image processing method comprising:

a photographed image acquisition step of sequentially acquiring photographed images of a relatively moving object that change gradually;

a characteristic extraction step of extracting characteristic points from a photographed image acquired in the photographed image acquisition step;

a characteristic point tracking step of searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing step of determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the processing step comprises detecting disappearance of the characteristic points or the corresponding points; and

when a predetermined number of points have disappeared or a predetermined range of area or more has become vacant, creating new characteristic points.

13. An image processing method comprising:

a photographed image acquisition step of sequentially acquiring photographed images of a relatively moving object that change gradually;

a characteristic extraction step of extracting characteristic points from a photographed image acquired in the photographed image acquisition step;

a characteristic point tracking step of searching a plurality of the photographed images acquired before or after the photographed image from which the characteristic points have been extracted, for candidate corresponding points corresponding to the characteristic points, to keep track of the characteristic points; and

a processing step of determining corresponding points from the candidate corresponding points based on tracking results to calculate coordinates of a photographing position or the object using the determined corresponding points,

wherein the tracking results include a screen relative movement amount obtained through a projective transformation using two images out of the sequentially acquired photographed images, or movement vectors of characteristic points between the two photographed images, or movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors,

wherein the processing step comprises using movement tracks obtained by sequentially linking the screen relative movement amount or the movement vectors of respective characteristic points of a first photographed image to create an estimated image related to a second photographed image acquired with a predetermined amount distanced therefrom (in time or space);

comparing the second photographed image with the estimated image related thereto;

determining corresponding points from the candidate corresponding points; and

calculating coordinates of a photographing position or the object using the determined corresponding points.