Three-dimensional position and orientation sensing system

• US 6,724,930 B1
• Filed: 02/04/2000
• Issued: 04/20/2004
• Est. Priority Date: 02/04/1999
• Status: Expired due to Fees

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: Kosaka, Akio, Saito, Akito, Shibasaki, Takao, Furuhashi, Yukihito, Matsuzaki, Hiroshi, Asano, Takeo
• Primary Examiner(s): Mariam, Daniel
• Assistant Examiner(s): Patel, Shefali
• Application Number:
  US09/498,963
• Time in Patent Office: 1,537 Days
• Field of Search: 382/154, 382/287, 382/298, 382/291, 382/216, 356/12, 356/13, 356/14, 348/95, 345/95
• US Class Current: 382/154
• CPC Class Codes: G01B 11/002 for measuring two or more coordinates
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

A three-dimensional position and orientation sensing apparatus including: an image input section which inputs an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance; a region extracting section which extracts a region corresponding to each marker in the image; a marker identifying section which identifies the individual markers based on the color or geometric characteristics of the markers in the extracted regions; and a position and orientation calculating section which calculates the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using positions of the identified markers in the image input to the image input section, and the positional information of the markers with respect to the object to be measured.

Claims

1. A three-dimensional position and orientation sensing apparatus comprising:

image input means for inputting an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

region extracting means for extracting a region corresponding to each marker in the image;

marker identifying means for identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions; and

position and orientation calculating means for calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using positions of the identified markers in the image input to the image input means, and the three-dimensional positional information of the markers with respect to the object to be measured.

2. A three-dimensional position and orientation sensing apparatus comprising:

image input means for inputting an image acquired by an image acquisition apparatus and showing at least four markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

region extracting means for extracting an area corresponding to each marker in the image;

marker identifying means for identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

marker selecting means for selecting three markers from the markers identified by the marker identifying means;

parameter calculating means for calculating a plurality of sets of parameters for calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by utilizing positions of the three markers in the image input to the image input means which are selected by the marker selecting means and three-dimensional positional information of each marker with respect to the object to be measured; and

parameter selecting means for selecting one set of parameters by evaluating application of the parameters calculated by the parameter calculating means, to markers not selected by the marker selecting means.

3. A three-dimensional position and orientation sensing apparatus according to claim 2, further comprising parameter improving means for improving one set of parameters selected by the parameter selecting means, by evaluating the set of parameters through application thereto of three-dimensional positional information of each marker with respect to the object to be measured.

4. A three-dimensional position and orientation sensing apparatus comprising:

image input means for inputting an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristic as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

region extracting means for extracting a region corresponding to each marker in the image;

marker identifying means for identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

distance estimating means for estimating a distance from the image acquisition apparatus to each marker based on a size of the extracted region in the image including the identified marker; and

position and orientation calculating means for calculating the three-dimensional position and orientation of the object with respect to the image acquisition apparatus, by using the estimated distance to each marker, a position of the marker in the image and three-dimensional positional information of this marker with respect to the object.

5. A three-dimensional position and orientation sensing apparatus according to claim 4, further comprising:

feature extracting means for extracting features of the object to be measured, from the image; and

means for calibrating the position and orientation of the calculated object, from the position of the extracted features.

6. A three-dimensional position and orientation sensing apparatus comprising:

image input means for inputting an image acquired by an image acquisition apparatus and having markers, including color or geometric characteristics as one image, three-dimensional positional information of which with respect to an object to be measured is known in advance;

region extracting means for reducing the size of the input image and extracting a region corresponding to each marker in the reduced image; and

position and orientation calculating means for calculating the three-dimensional position and orientation of the object with respect to the image acquisition apparatus, by using the extracted position of each marker in the image and three-dimensional positional information of each marker with respect to the object.

7. A three-dimensional position and orientation sensing apparatus according to claim 6, further comprising:

feature extracting means for extracting features of the object to be measured, from the image; and

means for calibrating the position and orientation of the calculated object, from the position of the extracted features in the image.

8. A three-dimensional position and orientation sensing method for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing an image acquired by this image acquisition apparatus, the method comprising the steps of:

inputting an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

extracting a region corresponding to each marker in the image;

identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions; and

calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using positions of the identified markers in the image input to the image input in the inputting step, and the three-dimensional positional information of the markers with respect to the object to be measured.

9. A three-dimensional position and orientation sensing method for measuring a position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing an image acquired by this image acquisition apparatus, the method comprising the steps of:

inputting an image acquired by an image acquisition apparatus and showing at least four markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

extracting a region corresponding to each marker in the image;

identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

selecting three markers from the markers identified by the marker identifying means;

calculating a plurality of sets of parameters for calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by utilizing positions of the three markers in the image input in the inputting step which are selected by the marker selecting means and three-dimensional positional information of each marker with respect to the object; and

selecting one set of parameters by evaluating application of the parameters calculated by the parameter calculating means, to markers not selected by the marker selecting means.

10. A three-dimensional position and orientation sensing method according to claim 9, further comprising the step of improving one set of parameters selected by the parameter selecting means, by evaluating the set of parameters through application thereto of three-dimensional positional information of each marker with respect to the object.

11. A three-dimensional position and orientation sensing method for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing an image acquired by this image acquisition apparatus, the method comprising the steps of:

inputting an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

extracting a region corresponding to each marker in the image;

identifying the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

estimating a distance from the image acquisition apparatus to each marker based on a size of the extracted region in the image including the identified marker; and

calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using the estimated distance to each marker, a position of the marker in the image and three-dimensional positional information of this marker with respect to the object.

12. A three-dimensional position and orientation sensing method according to claim 11, further comprising the steps of:

extracting features of the object to be measured, from the image; and

calibrating the position and orientation of the calculated object, from a position of the extracted features in the image.

13. A three-dimensional position and orientation sensing method for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing an image acquired by this image acquisition apparatus, the method comprising the steps of:

inputting an image acquired by an image acquisition apparatus and having markers, including color or geometric characteristics as one image, three-dimensional positional information of which with respect to an object to be measured is known in advance;

reducing the input image and extracting a region corresponding to each marker on the reduced image; and

calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using the extracted position of each marker in the image and three-dimensional positional information of each marker with respect to the object to be measured.

14. A three-dimensional position and orientation sensing method according to claim 13, further comprising the steps of:

extracting features of the object to be measured, from the image; and

calibrating the position and orientation of the calculated object, from positions of the extracted features in the image.

15. An article of manufacture comprising a computer-readable recording medium having computer-readable program coding means as a processing program recorded for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing by computer an image acquired by this image acquisition apparatus, the computer-readable program coding means comprising:

computer-readable programming means for making an image to be input, the image having been acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

computer-readable programming means for making a region corresponding to each marker in the image to be extracted;

computer-readable programming means for making the individual markers to be identified based on the color or geometric characteristics of the markers in the extracted regions; and

computer-readable programming means for making the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus to be calculated, by using positions of the identified markers in the image input by the computer-readable programming means, and the three-dimensional positional information of the markers with respect to the object to be measured.

16. An article of manufacture comprising a computer-readable recording medium having computer-readable program coding means as a processing program recorded for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing by computer an image acquired by this image acquisition apparatus, the computer-readable program coding means comprising:

computer-readable programming means for making an image to be input, the image having been acquired by an image acquisition apparatus and showing at least four markers having color and geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

computer-readable programming means for making a region corresponding to each marker in the image to be extracted, making the individual markers to be identified based on the color or geometric characteristics of the markers in the extracted regions, and making three markers to be selected from the identified markers;

computer-readable programming means for making a plurality of sets of parameters for calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus to be calculated, by using positions of the selected three markers in the image input by the computer-readable programming means, and the three-dimensional positional information of the markers with respect to the object to be measured; and

computer-readable programming means for making one set of parameters selected, by evaluating the non-selected markers through application of the calculated parameters thereto.

17. An article of manufacture comprising a computer-readable recording medium storing a three-dimensional position and orientation sensing processing program according to claim 16, wherein

the computer-readable program coding means as a processing program further comprises:

computer-readable programming means for making one set of parameters selected by the parameter selecting means to be improved, by evaluating the set of parameters through application thereto of three-dimensional positional information of each marker with respect to the object.

18. An article of manufacture comprising a computer-readable recording medium having computer-readable program coding means as a processing program recorded for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing by computer an image acquired by this image acquisition apparatus, the computer-readable program coding means comprising:

computer-readable programming means for making an image to be input, the image having been acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

computer-readable programming means for making a region corresponding to each marker in the image to be extracted;

computer-readable programming means for making the individual markers to be identified based on the color or geometric characteristics of the markers in the extracted regions;

computer-readable programming means for making a distance from the image acquisition apparatus to each marker to be estimated based on a size of the identified marker in the image; and

computer-readable programming means for making the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus to be calculated, by using the estimated distance to each marker, a position of the marker in the image and three-dimensional positional information of this marker with respect to the object to be measured.

19. An article of manufacture comprising a computer-readable recording medium storing a three-dimensional position and orientation sensing processing program according to claim 18, wherein

the computer-readable program coding means as a processing program further comprises:

computer-readable programming means for making features of the object to be measured, to be extracted from the image, and making the position and orientation of the calculated object to be calibrated, from a position of the extracted features.

20. An article of manufacture comprising a computer-readable recording medium having computer-readable program coding means as a processing program recorded for measuring the position and orientation of an object to be measured with respect to an image acquisition apparatus, by analyzing by computer an image acquired by this image acquisition apparatus, the computer-readable program coding means comprising:

computer-readable programming means for making an image to be input, the image having been acquired by an image acquisition apparatus and having markers, including color or geometric characteristics as one image, three-dimensional positional information of which with respect to an object to be measured is known in advance;

computer-readable programming means for making the input image to be reduced and making a region corresponding to each marker to be extracted on the reduced image; and

computer-readable programming means for making the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus to be calculated, by using the extracted position of each marker in the image and three-dimensional positional information of each marker with respect to the object.

21. An article of manufacture comprising a computer-readable recording medium storing a three-dimensional position and orientation sensing processing program according to claim 20, wherein

the computer-readable program coding means as a processing program further comprises:

computer-readable programming means for making features of the object to be measured, to be extracted from the image, and making the position and orientation of the calculated object to be calibrated, from a position of the extracted features in the image.

22. An apparatus for measuring a relative three-dimensional position and orientation of an object with respect to an image acquisition apparatus by measuring image positions of at least three markers mounted on the object or near the object, based on an image showing the markers, acquired by the image acquisition apparatus, wherein the markers have color or geometric characteristics as one image, and three-dimensional positions of the markers are known in advance, and the measuring apparatus comprising:

means for extracting a region corresponding to each of the markers from within the image of the markers;

means for identifying the individual markers by calculating color or geometric characteristics of the region corresponding to each of the markers within the image; and

means for estimating the relative three-dimensional position and orientation of the object with respect to the image acquisition apparatus, based on the three-dimensional positions of the markers.

23. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein coded patterns are disposed within the individual markers.

24. A three-dimensional position and orientation sensing apparatus according to claim 23, wherein a circular marker is utilized as each marker, and small patterns of different colors are used as a code within the marker.

25. A three-dimensional position and orientation sensing apparatus according to claim 23, wherein patterns of different colors are generated on a concentric circle as the maker.

26. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein

the means for identifying the individual markers by calculating geometric characteristics of the region of each marker within the image measures the size of the region of each marker within the image, and

the means for estimating the relative three-dimensional position and orientation of the object with respect to the image acquisition apparatus calculates an initial estimate of the distance between the image acquisition apparatus and a marker based on the measured value of the size of the marker region, and calculates the relative three-dimensional position and orientation of the object with respect to the image acquisition apparatus based on the initial estimate.

27. A three-dimensional position and orientation sensing apparatus according to claim 26, wherein an estimate value is calculated by utilizing only a marker that takes a range which makes it possible to estimate the relative distance.

28. A three-dimensional position and orientation sensing apparatus according to claim 26, wherein an estimate error variance is calculated for the estimated relative distance, and the three-dimensional position and orientation of the object with respect to the image acquisition apparatus is estimated by taking both the estimate value and the error variance into consideration.

29. A three-dimensional position and orientation sensing apparatus according to claim 26, wherein the markers are circular markers, a projected image of each marker within the image is approximated to an ellipse, and the distance from the image acquisition apparatus to each marker is estimated by utilizing the length of the long axis of the ellipse.

30. A three-dimensional position and orientation sensing apparatus according to claim 29, wherein a circular marker is utilized as each marker.

31. A three-dimensional position and orientation sensing apparatus according to claim 29, wherein a three-dimensional positional sensing is carried out by adding direction information of the circular marker.

32. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein, for extracting a marker region from an original image, the apparatus generates a reduced image of the original image, extracts a candidate region that is considered to correspond to a marker from the reduced image, calculates the area of the candidate region within the original image, and then extracts and recognizes a region corresponding to the marker within the original image.

33. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein, after a marker having unique geometric characteristics has been recognized, the apparatus extracts from within the image a separate marker having no unique geometric characteristics, by utilizing a two-dimensional or three-dimensional positional relationship of the first marker, and updates the three-dimensional position and orientation parameter of the object with respect to the image acquisition apparatus, by utilizing the three-positional position of the marker with respect to the image acquisition apparatus.

34. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein the image acquisition apparatus acquires a plurality of images and utilizes the plurality of images.

35. A three-dimensional position and orientation sensing apparatus according to claim 22, wherein markers are mounted on the plane of a sensor probe, and the sensor probe is used as an object to be measured, whereby to estimate the relative position and orientation of the sensor probe and to measure the position of a probe tip of the sensor probe.

36. A three-dimensional position and orientation sensing apparatus comprising:

an image input section which inputs an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

a region extracting section which extracts a region corresponding to each marker in the image;

a marker identifying section which identifies the individual markers based on the color or geometric characteristics of the markers in the extracted regions; and

a position and orientation calculating section which calculates the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by using positions of the identified markers in the image input to the image input section, and the three-dimensional positional information of the markers with respect to the object to be measured.

37. A three-dimensional position and orientation sensing apparatus comprising:

an image input section which inputs an image acquired by an image acquisition apparatus and showing at least four markers having color or geometric characteristics as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

a region extracting section which extracts an area corresponding to each marker in the image;

a marker identifying section which identifies the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

a marker selecting section which selects three markers from the markers identified by the marker identifying section;

a parameter calculating section which calculates a plurality of sets of parameters for calculating the three-dimensional position and orientation of the object to be measured with respect to the image acquisition apparatus, by utilizing positions of the three markers in the image input to the image input section which are selected by the marker selecting section and three-dimensional positional information of each marker with respect to the object to be measured; and

a parameter selecting section which selects one set of parameters by evaluating application of the parameters calculated by the parameter calculating section, to markers not selected by the marker selecting section.

38. A three-dimensional position and orientation sensing apparatus comprising:

a image input section which inputs an image acquired by an image acquisition apparatus and showing at least three markers having color or geometric characteristic as one image, three-dimensional positional information of the markers with respect to an object to be measured being known in advance;

a region extracting section which extracts a region corresponding to each marker in the image;

a marker identifying section which identifies the individual markers based on the color or geometric characteristics of the markers in the extracted regions;

a distance estimating section which estimates a distance from the image acquisition apparatus to each marker based on a size of the extracted region in the image including the identified marker; and

a position and orientation calculating section which calculates the three-dimensional position and orientation of the object with respect to the image acquisition apparatus, by using the estimated distance to each marker, a position of the marker in the image and three-dimensional positional information of this marker with respect to the object.

39. A three-dimensional position and orientation sensing apparatus comprising:

an image input section which inputs an image acquired by an image acquisition apparatus and having markers, including color or geometric characteristics as one image, three-dimensional positional information of which with respect to an object to be measured is known in advance;

a region extracting section which reduces the size of the input image and extracting a region corresponding to each marker in the reduced image; and

a position and orientation calculating section which calculates the three-dimensional position and orientation of the object with respect to the image acquisition apparatus, by using the extracted position of each marker in the image and three-dimensional positional information of each marker with respect to the object.

40. An apparatus for measuring a relative three-dimensional position and orientation of an object with respect to an image acquisition apparatus by measuring image positions of at least three markers mounted on the object or near the object, based on an image showing the markers, acquired by the image acquisition apparatus, wherein the markers have color or geometric characteristics as one image and three-dimensional positions of the markers are known in advance, and the measuring apparatus comprising:

a section which extracts a region corresponding to each of the markers from within the image of the markers;

a section which identifies the individual markers by calculating color or geometric characteristics of the region corresponding to each of the markers within the image; and

a section which estimates the relative three-dimensional position and orientation of the object with respect to the image acquisition apparatus, based on the three-dimensional positions of the markers.