Robust sparse image matching for robotic surgery

• US 8,639,000 B2
• Filed: 04/24/2012
• Issued: 01/28/2014
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Wu, Chenyu, Hirvonen, David, Zhao, Tao, Hoffman, Brian D., Di Maio, Simon
• Primary Examiner(s): Chang, Joseph
• Application Number:
  US13/454,297
• Publication Number:
  US 20120237095A1
• Time in Patent Office: 644 Days
• Field of Search: 382/128, 354/419
• US Class Current: 382/128
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

Systems, methods, and devices are used to match images. Points of interest from a first image are identified for matching to a second image. In response to the identified points of interest, regions and features can be identified and used to match the points of interest to a corresponding second image or second series of images. Regions can be used to match the points of interest when regions of the first image are matched to the second image with high confidence scores, for example above a threshold. Features of the first image can be matched to the second image, and these matched features may be used to match the points of interest to the second image, for example when the confidence scores for the regions are below the threshold value. Constraint can be used to evaluate the matched points of interest, for example by excluding bad points.

Claims

1. A system comprising:

a source of a first image and a second image; and

a processor coupled to the source and configured to identify selected points of interest of the first image and selectively match the selected points of interest to the second image by computing a plurality of image offsets using a plurality of image processing methods and using one of the plurality of image offsets according at least partially to a confidence score of one of the plurality of image offsets.

2. The system of claim 1, wherein one of the plurality of image offsets is a global offset which is computed by using normalized cross correlations to compare information of the first and second images.

3. The system of claim 1, wherein each of the selected points of interest is matched to the second image such that each of the matched points of interest of the second image corresponds to one of the selected points of interest of the first image.

4. The system of claim 1, wherein one of the plurality of image offsets is a region offset which is computed by matching a region of the first image to a region of the second image by using at least one of cross correlation, two-way matching, least squares regression, and non-linear regression.

5. The system of claim 4, wherein the region of the second image is determined using a global offset between the first and second images and a location of at least one of the selected points of interest in the first image.

6. The system of claim 5, wherein one of the plurality of image offsets is a feature offset which is computed by matching features of the first image to features of the second image by using at least one of Harris corner detection, scale-space extrema detection, local extrema detection, and scale invariant feature transform.

7. The system of claim 1, wherein a location of the second image corresponding to a first point of interest is determined before a second point of interest is identified.

8. The system of claim 7, further comprising sequentially highlighting the points of interest and the corresponding locations while selecting the points of interest.

9. The system of claim 8, wherein each region comprises a portion of the first image, the portion having an area of no more than twenty percent of a total area of the first image.

10. The system of claim 1, wherein locations of the matched points of interest are determined by using at least one of the focus constraint or the soft epipolar constraint.

11. The system of claim 10, wherein at least one of the matched points of interest is excluded in response to the locations of the points of interest exceeding the at least one of the focus constraint or the soft epipolar constraint.

12. The system of claim 10, wherein regions and features of the first image are matched to the second image in response to the at least one of the focus constraint or the soft epipolar constraint.

13. The system of claim 10, wherein the locations of the matched points of interest are determined in response to the focus constraint, wherein each of the identified selected points of interest has a first horizontal location of the first image and each of the matched points of interest has a second horizontal location of the second image, and wherein the focus constraint constrains the second horizontal location in response to a difference of the first horizontal location and the second horizontal location.

14. The system of claim 13, wherein at least one of the matched points of interest is excluded when the difference exceeds the focus constraint.

15. The system of claim 13, wherein the regions and features of the first image are matched to the second image in response to the focus constraint.

16. The system of claim 13, wherein one of the plurality of image offsets is a global offset comprising a horizontal global offset, and wherein the focus constraint constrains the second horizontal location in response to the horizontal global offset and the difference of the first horizontal location and the second horizontal location.

17. The system of claim 10, wherein the second location is determined in response to the soft epipolar constraint, wherein each of the points of interest comprises a first vertical location for the first image and a second vertical location for the second image, and wherein the soft epipolar constraint constrains the second vertical location in response to the first vertical location.

18. The system of claim 17, wherein the global offset comprises a vertical global offset and wherein the soft epipolar constraint constrains the second vertical location in response to the vertical global offset and the vertical location.

19. The system of claim 1, wherein the first image comprises a left image of a tissue structure and the second image comprises a right image of the tissue structure to display the tissue structure in a three-dimensional appearance.

20. The system of claim 1, wherein the first image comprises a first series of real time images and wherein the second image comprises a second series of real time images.

21. A system comprising:

a source of a first image and a second image; and

a processor coupled to the source and configured to identify selected points of interest of the first image for matching to the second image, determine a region of the first image for each of the selected points of interest, match each region of the first image to a second image to determine a confidence score for a matched region of the second image, identify a feature of the first image for each of the regions with a low confidence score, match each feature of the first image to the second image to determine a confidence score for a matched feature of the second image, and determine a matched point of interest of the second image for each point of interest in response to a location of the matched region when the confidence score for the region is high and to a location of the matched feature when the confidence score for the feature is high.

22. The system of claim 21, wherein the matched point of interest of the second image is determined in response to a global offset of the second image when the confidence score for the feature is low.

23. The system of claim 21, wherein at least some of the selected points of interest are matched to the second image with interpolation when the confidence score for the feature is low and wherein the interpolation is determined based on the matched points of interest determined with at least one of the location of the matched region or the location of the matched feature.

24. The system of claim 21, wherein at least some of the regions comprise high confidence regions having confidence scores above a region threshold value and wherein at least some of the locations of the points of interest are determined in response to the high confidence regions, the high confidence regions being sufficiently matched.

25. The system of claim 24, wherein at least some of the regions comprise low confidence regions having confidence scores below the region threshold value such that the low confidence regions are insufficiently matched and wherein at least some of the locations of the points of interest are determined in response to matched features associated with low confidence regions.

26. The system of claim 21, wherein the regions of the first image are matched to the second image with cross-correlation to determine a correlation surface for each region and wherein each confidence score is determined in response to the correlation surface.

27. The system of claim 21, wherein at least some of the features comprise high confidence features having feature confidence scores above a feature threshold value such that the high confidence features are sufficiently matched and wherein at least some of the locations of the points of interest are determined in response to the high confidence features.

28. The system of claim 27, wherein at least some of the features comprise low confidence features having feature confidence scores below the feature threshold value such that the low confidence features are insufficiently matched and wherein at least some of the features of the first image comprise additional features identified in the first image and matched to the second image in response to the low confidence features.

29. The system of claim 21, wherein the features of the first image are matched to the second image with a scale invariant feature transform.