Robust sparse image matching for robotic surgery

• US 8,184,880 B2
• Filed: 05/13/2009
• Issued: 05/22/2012
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Wu, Chenyu, Hirvonen, David, Zhao, Tao, Hoffman, Brian D., Di Maio, Simon
• Primary Examiner(s): Chang, Joseph
• Application Number:
  US12/465,029
• Publication Number:
  US 20100166323A1
• Time in Patent Office: 1,105 Days
• Field of Search: 382/128, 345/419
• US Class Current: 382/128
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

Systems, methods, and devices are used to match images. Points of interest from a first image are identified for matching to a second image. In response to the identified points of interest, regions and features can be identified and used to match the points of interest to a corresponding second image or second series of images. Regions can be used to match the points of interest when regions of the first image are matched to the second image with high confidence scores, for example above a threshold. Features of the first image can be matched to the second image, and these matched features may be used to match the points of interest to the second image, for example when the confidence scores for the regions are below the threshold value. Constraint can be used to evaluate the matched points of interest, for example by excluding bad points.

Claims

1. A method of matching images, the method comprising:

identifying selected points of interest of a first image for matching to a second image; and

selectively matching the selected points of interest to the second image to determine matched points of interest of the second image by computing a plurality of image offsets using a plurality of image processing methods and using one of the plurality of image offsets according at least partially to a confidence score of one of the plurality of image offsets.

2. The method of claim 1 wherein one of the plurality of image offsets is a global offset which is computed by using normalized cross correlations to compare information of the first and second images.

3. The method of claim 1 wherein each of the selected points of interest is matched to the second image such that each of the matched points of interest of the second image corresponds to one of the selected points of interest of the first image.

4. The method of claim 1 wherein one of the plurality of image offsets is a region offset which is computed by matching a region of the first image to a region of the second image by using at least one of cross correlation, two-way matching, least squares regression, and non-linear regression.

5. The method of claim 4 wherein the region of the second image is determined using a global offset between the first and second images and a location of at least one of the selected points of interest in the first image.

6. The method of claim 5 wherein one of the plurality of image offsets is a feature offset which is computed by matching features of the first image to features of the second image by using at least one of Harris corner detection, scale-space extrema detection, local extrema detection, and scale invariant feature transform.

7. The method of claim 1, wherein a location of the second image corresponding to a first point of interest is determined before a second point of interest is identified.

8. The method of claim 7, further comprising sequentially highlighting the points of interest and the corresponding locations while selecting the points of interest.

9. The method of claim 1 wherein locations of the matched points of interest are determined by using at least one of the focus constraint or the soft epipolar constraint.

10. The method of claim 9 wherein at least one of the matched points of interest is excluded in response to the locations of the points of interest exceeding the at least one of the focus constraint or the soft epipolar constraint.

11. The method of claim 9 wherein regions and features of the first image are matched to the second image in response to the at least one of the focus constraint or the soft epipolar constraint.

12. The method of claim 9:

wherein the locations of the matched points of interest are determined in response to the focus constraint;

wherein each of the identified selected points of interest has a first horizontal location of the first image and each of the matched points of interest has a second horizontal location of the second image; and

wherein the focus constraint constrains the second horizontal location in response to a difference of the first horizontal location and the second horizontal location.

13. The method of claim 12 wherein at least one of the matched points of interest is excluded when the difference exceeds the focus constraint.

14. The method of claim 12 wherein the regions and features of the first image are matched to the second image in response to the focus constraint.

15. The method of claim 12 further comprising:

wherein one of the plurality of image offsets is a global offset comprising a horizontal global offset;

wherein the focus constraint constrains the second horizontal location in response to the horizontal global offset and the difference of the first horizontal location and the second horizontal location.

16. The method of claim 9:

wherein the second location is determined in response to the soft epipolar constraint;

wherein each of the points of interest comprises a first vertical location for the first image and a second vertical location for the second image; and

wherein the soft epipolar constraint constrains the second vertical location in response to the first vertical location.

17. The method of claim 16:

wherein the global offset comprises a vertical global offset; and

wherein the soft epipolar constraint constrains the second vertical location in response to the vertical global offset and the vertical location.

18. The method of claim 1 wherein the first image comprises a left image of a tissue structure and the second image comprises a right image of the tissue structure to display the tissue structure in a three-dimensional appearance.

19. The method of claim 1:

wherein the first image comprises a first series of real time images; and

wherein the second image comprises a second series of real time images.

20. A method of matching images, the method comprising:

identifying selected points of interest of a first image for matching to a second image;

determining a region of the first image for each of the selected points of interest;

matching each region of the first image to a second image to determine a confidence score for a matched region of the second image;

identifying a feature of the first image for each of the regions with a low confidence score;

matching each feature of the first image to the second image to determine a confidence score for a matched feature of the second image; and

determining a matched point of interest of the second image for each point of interest in response to;

a location of the matched region when the confidence score for the region is high, and

a location of the matched feature when the confidence score for the feature is high.

21. The method of claim 20 wherein the matched point of interest of the second image is determined in response to a global offset of the second image when the confidence score for the feature is low.

22. The method of claim 20:

wherein at least some of the selected points of interest are matched to the second image with interpolation when the confidence score for the feature is low; and

wherein the interpolation is determined based on the matched points of interest determined with at least one of the location of the matched region or the location of the matched feature.

23. The method of claim 20:

wherein at least some of the regions comprise high confidence regions having confidence scores above a region threshold value; and

wherein at least some of the locations of the points of interest are determined in response to the high confidence regions, the high confidence regions being sufficiently matched.

24. The method of claim 23:

wherein at least some of the regions comprise low confidence regions having confidence scores below the region threshold value such that the low confidence regions are insufficiently matched; and

wherein at least some of the locations of the points of interest are determined in response to matched features associated with low confidence regions.

25. The method of claim 20:

wherein the regions of the first image are matched to the second image with cross-correlation to determine a correlation surface for each region; and

wherein each confidence score is determined in response to the correlation surface.

26. The method of claim 20:

wherein at least some of the features comprise high confidence features having feature confidence scores above a feature threshold value such that the high confidence features are sufficiently matched; and

wherein at least some of the locations of the points of interest are determined in response to the high confidence features.

27. The method of claim 26:

wherein at least some of the features comprise low confidence features having feature confidence scores below the feature threshold value such that the low confidence features are insufficiently matched; and

wherein at least some of the features of the first image comprise additional features identified in the first image and matched to the second image in response to the low confidence features.

28. The method of claim 20 wherein the features of the first image are matched to the second image with a scale invariant feature transform.

29. The method of claim 20 wherein each region comprises a portion of the first image, the portion having an area of no more than about twenty percent of a total area of the first image.

30. A system for matching images, the system comprising:

a source of a first image and a second image; and

a processor system coupled to the source and comprising a tangible medium configured to identify selected points of interest of the first image and selectively match the selected points of interest to the second image by computing a plurality of image offsets using a plurality of image processing methods and using one of the plurality of image offsets according at least partially to a confidence score of one of the plurality of image offsets.

31. The system of claim 30 further comprising:

an input device configured to input locations of the first image;

wherein the processor system is configured to identify the selected points of interest in response to the input locations.