Methods, systems and computer program products for photogrammetric sensor position estimation

• US 6,574,494 B2
• Filed: 04/06/2001
• Issued: 06/03/2003
• Est. Priority Date: 11/10/1997
• Status: Expired due to Fees

• Current Assignee: MCG International Incorporated
• Original Assignee: MedAcoustics, Inc.
• Inventors: Van Horn, Mark Harold
• Primary Examiner(s): MANUEL, GEORGE C
• Application Number:
  US09/828,565
• Publication Number:
  US 20010034481A1
• Time in Patent Office: 788 Days
• Field of Search: 600/407, 600/420, 600/433, 600/434, 600/435, 600/462, 600/466, 600/467, 600/474, 600/528, 600/424, 600/589, 367/25, 367/33, 257/414, 257/415, 257/416
• US Class Current: 600/407
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2063 Acoustic tracking systems, e.g. using ultrasound
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 7/00 Instruments for auscultation
• CPC Class Codes: A61B 8/06 Measuring blood flow measuring volume flow in general G01F, e.g. G01F1/66, G01F1/72; measuring speed of fluids in general G01P5/00
• CPC Class Codes: A61B 8/4405 Device being mounted on a trolley
• CPC Class Codes: G01S 15/52 Discriminating between fixed and moving objects or between objects moving at different speeds
• CPC Class Codes: G01S 15/8906 Short-range imaging systems; Acoustic microscope systems using pulse-echo techniques
• CPC Class Codes: G01S 15/8979 Combined Doppler and pulse-echo imaging systems
• CPC Class Codes: G01S 15/899 Combination of imaging systems with ancillary equipment
• CPC Class Codes: G01S 15/8993 Three dimensional imaging systems
• CPC Class Codes: G01S 7/52036 using analysis of echo signal for target characterisation
• CPC Class Codes: G01S 7/5205 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/52073 Production of cursor lines, markers or indicia by electronic means
• CPC Class Codes: G01S 7/52077 with means for elimination of unwanted signals, e.g. noise or interference

Abstract

Methods, systems and computer program products are provided for determining an obscured contact point based on a visible portion of an acoustic sensor of a medical device contacting a patient by acquiring a first image containing an upper surface of the acoustic sensor from a first viewpoint and a second image containing the upper surface of the acoustic sensor from a second viewpoint different from the first viewpoint. The acoustic sensor is located in the first image and the second image and the centroid of the acoustic sensor is determined based on the location of the acoustic sensor in the first image and the corresponding location of the acoustic sensor in the second image. A plane of the visible portion of the acoustic sensor is also determined based on the position of the upper surface of the acoustic sensor in the first image and the corresponding position of the upper surface of the acoustic sensor in the second image. The contact point of the obscured portion of the acoustic sensor may then be determined by projecting a predetermined depth through the centroid of the acoustic sensor in a direction having a predefined relationship with the plane of the visible portion of the acoustic sensor.

Claims

1. A method of determining an obscured contact point of a sensor positioned on the skin of a person such that it has a visually obscured portion and a visible portion with an upper surface, based on the visible portion of the sensor from at least a first image containing the upper surface of the sensor from a first viewpoint and a second image containing the upper surface of the sensor from a second viewpoint different from the first viewpoint, the method comprising the steps of:

locating the sensor in the first image and the second image;

determining the location of a point on the upper surface of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image;

determining a plane of the upper surface of the sensor based on the sensor in the first image and the corresponding sensor in the second image; and

projecting a predetermined depth through the located point on the upper surface of the sensor in a direction having a predefined relationship with the plane of the visible portion of the sensor so as to determine the contact point of the obscured portion of the sensor on the skin of the person.

2. A method according to claim 1, wherein the step of determining the location of a point on the upper surface of the sensor comprises the step of determining the location of a centroid of the visible portion of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image.

3. A method according to claim 2, wherein the steps of determining the centroid, and determining a plane comprise the steps of:

determining the centroid of the sensor in each of the first and the second image; and

determining a location of the centroid in three-dimensional space from the centroid of the upper surface of the sensor in the first image and the centroid of the upper surface of the sensor in the second image.

4. A method according claim 3, wherein the step of projecting comprises the steps of:

determining a direction of a cross product of lines which pass through corners of the visible portion of the sensor; and

projecting the predetermined depth from the centroid of the upper surface of the sensor in the direction of the cross product.

5. A method according to claim 4, further comprising the steps of:

establishing calipers in a direction orthogonal to directions of detected edges of the visible portion of the sensor; and

moving the calipers until the calipers touch the located sensors so as to determine the locations of the corners of the visible portion of the sensor.

6. A method according to claim 5, wherein the visible portion of the sensor is a quadrangle.

7. A method according to claim 1, further comprising the steps of:

determining the contact points of a plurality of sensors on the person; and

determining the relative spatial relationship between the plurality of sensors based on the determination of the contact points of the plurality of sensors.

8. A method according to claim 1, wherein the sensor is substantially rectangular in shape such that the visible portion of the upper surface of the sensor is a quadrangle.

9. A method according to claim 1, wherein the first image is acquired by a first camera and wherein the second image is acquired by a second camera, the method further comprising the step of calibrating the first camera and the second camera so as to determine the relationship between the first viewpoint and the second viewpoint.

10. A method according to claim 9, wherein the step of calibrating comprises the steps of:

acquiring at the first camera a first reference image containing a predefined configuration of landmarks;

acquiring at the second camera a second reference image containing the predefined configuration of landmarks; and

determining the spatial relationship between an image plane of the first camera and an image plane of the second camera based on the first and second reference images.

11. A method according to claim 9, wherein the step of calibrating comprises the steps of:

computing camera parameters for each of the first and second cameras;

computing projection matrices for each of the first and second cameras; and

estimating object-space coordinates for each camera using corresponding points from at least two images of the same scene;

wherein calibration of the cameras is indicated by coincidence of the object space coordinates for the corresponding points.

12. A method according to claim 1, wherein the locating step comprises the steps of:

locating sensor positions in the first image;

locating sensor positions in the second image; and

determining a correspondence between the sensor positions in the first image and the sensor positions in the second image so that a sensor position in the first image is associated with the corresponding sensor position the second image.

13. A method according to claim 12, where the step of determining a correspondence comprises the steps of:

determining an epipolar line for a sensor position in the first image;

determining which of the sensor positions in the second image the epipolar line intersects; and

associating a sensor position in the second image which the epipolar line intersects with the sensor position in the first image.

14. A method according to claim 13, wherein the step of associating comprises the steps of:

determining if a plurality of sensor positions in the second image intersect the epipolar line; and

associating sensor positions in the first image with sensor positions in the second image which intersect the epipolar line in an order in which the sensor positions in the first image intersect the epipolar line.

15. A method according to claim 13, wherein the step of determining which of the sensor positions in the second image the epipolar line intersects comprises the steps of:

determining if the epipolar line intersects a bounding box around a sensor position;

determining the signed distance between the epipolar line and opposite corners of the bounding box; and

determining that the epipolar line intersects the sensor position if the signed distances between the epipolar line and the opposite corners of the bounding box are of opposite signs.

16. A system for determining an obscured contact point based on a visible portion of a sensor of a device contacting a person from at least a first image containing an upper surface of the sensor from a first viewpoint and a second image containing the upper surface of the sensor from a second viewpoint different from the first viewpoint, comprising:

means for locating the sensor in the first image and the second image;

means for determining the location of a point on the upper surface of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image;

means for determining a plane of the upper surface of the sensor based on the sensor in the first image and the corresponding sensor in the second image; and

means for projecting a predetermined depth through the located point on the upper surface of the sensor in a direction having a predefined relationship with the plane of the visible portion of the sensor so as to determine the contact point of the obscured portion of the sensor.

17. A system according to claim 16, wherein the means for determining the location of a point on the upper surface of the sensor comprises means for determining the location of a centroid of a visible portion of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image.

18. A system according to claim 17, wherein the means for determining the location of a centroid, and the means for determining a plane comprise:

means for determining the centroid of the sensor in each of the first and the second image; and

means for determining a location of the centroid in three-dimensional space from the centroid of the upper surface of the sensor in the first image and the centroid of the upper surface of the sensor in the second image.

19. A system according claim 18, wherein the means for projecting comprises:

means for determining a direction of a cross product of lines which pass through corners of the visible portion of the sensor; and

means for projecting the predetermined depth from the centroid of the upper surface of the sensor in the direction of the cross product.

20. A system according to claim 19, further comprising:

means for establishing calipers in a direction orthogonal to directions of detected edges of the visible portion of the sensor; and

means for moving the calipers until the calipers touch the located sensors so as to determine the locations of the corners of the visible portion of the sensor.

21. A system according to claim 19, wherein the visible portion of the sensor is a quadrangle.

22. A system according to claim 16, further comprising:

means for determining the contact points of a plurality of sensors;

means for determining the relative spatial relationship between the plurality of sensors based on the determination of the contact points of the plurality of sensors.

23. A system according to claim 16, wherein the sensor is substantially rectangular in shape such that the visible portion of the upper surface of the sensor is a quadrangle.

24. A system according to claim 16, wherein the first image is acquired by a first camera and wherein the second image is acquired by a second camera, the system further comprising means for calibrating the first camera and the second camera so as to determine the relationship between the first viewpoint and the second viewpoint.

25. A system according to claim 24, wherein the means for calibrating comprises:

means for acquiring at the first camera a first reference image containing a predefined configuration of landmarks;

means for acquiring at the second camera a second reference image containing the predefined configuration of landmarks; and

means for determining the spatial relationship between an image plane of the first camera and an image plane of the second camera based on the first and second reference images.

26. A system according to claim 23, wherein the means for calibrating comprises:

means for computing camera parameters for each of the first and second cameras;

means for computing projection matrices for each of the first and second cameras; and

means for estimating object-space coordinates for each camera using corresponding points from at least two images of the same scene;

wherein calibration of the cameras is indicated by coincidence of the object space coordinates for the corresponding points.

27. A system according to claim 16, wherein the means for locating comprises:

means for locating sensor positions in the first image;

means for locating sensor positions in the second image; and

means for determining a correspondence between the sensor positions in the first image and the sensor positions in the second image so that a sensor position in the first image is associated with the corresponding sensor position the second image.

28. A system according to claim 27, wherein the means for determining a correspondence comprises:

means for determining an epipolar line for a sensor position in the first image;

means for determining which of the sensor positions in the second image the epipolar line intersects; and

means for associating a sensor position in the second image which the epipolar line intersects with the sensor position in the first image.

29. A system according to claim 28, wherein the means for associating comprises:

means for determining if a plurality of sensor positions in the second image intersect the epipolar line; and

means for associating sensor positions in the first image with sensor positions in the second image which intersect the epipolar line in an order in which the sensor positions in the first image intersect the epipolar line.

30. A system according to claim 28, wherein the means for determining which of the sensor positions in the second image the epipolar line intersects comprises:

means for determining if the epipolar line intersects a bounding box around a sensor position;

means for determining the signed distance between the epipolar line and opposite corners of the bounding box; and

means for determining that the epipolar line intersects the sensor position if the signed distances between the epipolar line and the opposite corners of the bounding box are of opposite signs.

31. A computer program product for determining an obscured contact point of a sensor positioned on a person such that it has a visible portion and a visually obscured portion based on the visible portion of the sensor contacting a person from at least a first image containing an upper surface of the sensor from a first viewpoint and a second image containing the upper surface of the sensor from a second viewpoint different from the first viewpoint, comprising:

a computer readable storage medium having computer readable program code embodied in said medium, said computer readable program code comprising;

computer readable program code for locating the sensor in the first image and the second image;

computer readable program code for determining the location of a point on the upper surface of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image;

computer readable program code for determining a plane of the upper surface of the sensor based on the sensor in the first image and the corresponding sensor in the second image; and

computer readable program code for projecting a predetermined depth through the located point on the upper surface of the sensor in a direction having a predefined relationship with the plane of the visible portion of the sensor so as to determine the contact point of the obscured portion of the sensor.

32. A computer program product according to claim 31, wherein the computer readable program code for determining the location of a point on the upper surface of the sensor comprises computer readable program code for determining the location of a centroid of a visible portion of the sensor based on the location of the sensor in the first image and the corresponding location of the sensor in the second image.

33. A computer program product according to claim 32, wherein the computer readable program code for determining the location of a centroid, and the computer readable program code for determining a plane comprise:

computer readable program code for determining the centroid of the sensor in each of the first and the second image; and

computer readable program code for determining a location of the centroid in three-dimensional space from the centroid of the upper surface of the sensor in the first image and the centroid of the upper surface of the sensor in the second image.

34. A computer program product according claim 33, wherein the computer readable program code for projecting comprises:

computer readable program code for determining a direction of a cross product of lines which pass through corners of the visible portion of the sensor; and

computer readable program code for projecting the predetermined depth from the centroid of the upper surface of the sensor in the direction of the cross product.

35. A computer program product according to claim 34, further comprising:

computer readable program code for establishing calipers in a direction orthogonal to directions of detected edges of the visible portion of the sensor; and

computer readable program code for moving the calipers until the calipers touch the located sensors so as to determine the locations of the corners of the visible portion of the sensor.

36. A computer program product according to claim 35, wherein the visible portion of the sensor is a quadrangle.

37. A computer program product according to claim 31, further comprising:

computer readable program code for determining the contact points of a plurality of sensors positioned on a person;

computer readable program code for determining the relative spatial relationship between the plurality of sensors based on the determination of the contact points of the plurality of sensors.

38. A computer program product according to claim 31, wherein the sensor is substantially rectangular in shape such that the visible portion of the upper surface of the sensor is a quadrangle.

39. A computer program product according to claim 31, wherein the first image is acquired by a first camera and wherein the second image is acquired by a second camera, the computer program product further comprising computer readable program code for calibrating the first camera and the second camera so as to determine the relationship between the first viewpoint and the second viewpoint.

40. A computer program product according to claim 39, wherein the computer readable program code for calibrating comprises:

computer readable program code for acquiring at the first camera a first reference image containing a predefined configuration of landmarks;

computer readable program code for acquiring at the second camera a second reference image containing the predefined configuration of landmarks; and

computer readable program code for determining the spatial relationship between an image plane of the first camera and an image plane of the second camera based on the first and second reference images.

41. A computer program product according to claim 39, wherein the computer readable program code for calibrating comprises:

computer readable program code for computing camera parameters for each of the first and second cameras;

computer readable program code for computing projection matrices for each of the first and second cameras; and

computer readable program code for estimating object-space coordinates for each camera using corresponding points from at least two images of the same scene;

wherein calibration of the cameras is indicated by coincidence of the object space coordinates for the corresponding points.

42. A computer program product according to claim 31, wherein the computer readable program code for locating comprises:

computer readable program code for locating sensor positions in the first image;

computer readable program code for locating sensor positions in the second image; and

computer readable program code for determining a correspondence between the sensor positions in the first image and the sensor positions in the second image so that a sensor position in the first image is associated with the corresponding sensor position the second image.

43. A computer program product according to claim 42, where the computer readable program code for determining a correspondence comprises:

computer readable program code for determining an epipolar line for a sensor position in the first image;

computer readable program code for determining which of the sensor positions in the second image the epipolar line intersects; and

computer readable program code for associating a sensor position in the second image which the epipolar line intersects with the sensor position in the first image.

44. A computer program product according to claim 43, wherein the computer readable program code for associating comprises:

computer readable program code for determining if a plurality of sensor positions in the second image intersect the epipolar line; and

computer readable program code for associating sensor positions in the first image with sensor positions in the second image which intersect the epipolar line in an order in which the sensor positions in the first image intersect the epipolar line.

45. A computer program product according to claim 44, wherein the computer readable program code for determining which of the sensor positions in the second image the epipolar line intersects comprises:

computer readable program code for determining if the epipolar line intersects a bounding box around a sensor position;

computer readable program code for determining the signed distance between the epipolar line and opposite corners of the bounding box; and

computer readable program code for determining that the epipolar line intersects the sensor position if the signed distances between the epipolar line and the opposite corners of the bounding box are of opposite signs.