Method of real-time tracking of moving/flexible surfaces

• US 8,693,730 B2
• Filed: 11/15/2012
• Issued: 04/08/2014
• Est. Priority Date: 11/15/2011
• Status: Active Grant

• Current Assignee: Macdonald Dettwiler and Associates Limited
• Original Assignee: Macdonald Dettwiler and Associates Limited
• Inventors: Mukherji, Raja, Umasuthan, Manickam, Shang, Limin
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US13/678,162
• Publication Number:
  US 20130123801A1
• Time in Patent Office: 509 Days
• Field of Search: 382/103, 382/128, 382/130, 382/131, 382/132, 128/922, 348/169, 606/130, 606/139, 600/1, 600/11, 600/179, 600/178, 600/103, 600/129, 600/228, 600/414, 600/424, 600/407
• US Class Current: 382/103
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/06 with illuminating arrangements
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 90/10 for stereotaxic surgery, e.g. frame-based stereotaxis
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs

Abstract

The present invention provides a method for real-time tracking of moving flexible surfaces and an image guided surgical robotic system using this tracking method. A vision system acquires an image of the moving flexible surface and identifies and tracks visual features at different times. The method involves computing both rigid and stretching transformations based on the changing positions of the visual features which are then used to track any area of interest on the moving flexible surface as it evolves over time. A robotic surgical system using this real-time tracking is disclosed.

Claims

1. A method on a computer for real time tracking of a moving flexible surfaces, comprising:

a) synchronously acquiring left and right image frames from left and right camera members of a stereo camera, of a region of interest on a moving flexible at a time K;

b) computing, from said left and right image frames 3D point cloud data in a coordinate frame of said stereo camera at said time K;

c) fitting a parametric surface to said 3D point cloud at said time K;

d) extracting a location of at least three features of interest in said left image frame at said time K in 2D pixel coordinates, extracting a location of each at least three feature of interest in said right image frame at said time K in 2D pixel coordinates, storing said 2D pixel coordinates of said locations of said features of interest in said left and right image frames at said time K in a vision processor memory storage;

e) computing, using said stored 2D pixel coordinates of said locations of said features of interest in said left and right image frames at said time K, a 3D location of said features of interest at said time K in a coordinate frame of said camera and storing said 3D locations in said vision processor memory storage;

f) mapping said 3D location of said features of interest onto said parametric surface at time K and storing the parametric surface with mapped 3D locations in the vision processor memory storage;

g) retrieving from the vision processor memory storage 3D locations of said features of interest from previous left and right image frames synchronously taken at a time K−

1;

h) computing a best fit rigid transform using said retrieved 3D locations of the features of interest at time K−

1 and the 3D locations of the features at time K;

i) retrieving from the vision processor memory storage a parametric surface with mapped 3D locations at time K−

1, applying said rigid transformation to said retrieved parametric surface with mapped 3D locations at time K−

1 to obtain an intermediate parametric surface with mapped 3D locations;

j) matching the 3D locations of features of interest on the intermediate parametric surface at time K−

1 to the same corresponding 3D locations on the stored parametric surface at time K and from said correspondences computing a stretching transformation; and

k) using said rigid and stretching transformations to calculate at time K a location of any region on said moving flexible surface based on a location of the same region on said moving flexible surface at time K−

1 stored in said vision processor memory storage.

2. The method according to claim 1 wherein step b) includes computing dense stereo disparity data from said left and right image frames and converting said dense stereo disparity data for said left and right image frames into said 3D point cloud data.

3. The method according to claim 1 wherein step a) includes first illuminating the region of interest with a structured light pattern and acquiring calibrated color images of said left and right image frames, and wherein said step b) includes applying triangulation techniques to said left and right calibrated color images to compute said 3D point cloud data.

4. The method according to claim 1 wherein said at least three features of interest are scale invariant feature transform (SIFT) features.

5. The method according to claim 4 wherein each scale invariant feature transform (SIFT) feature includes a descriptor vector including 64 elements, and wherein said at least three features of interest are between 3 to about 100 features.

6. The method according to claim 5 wherein said at least three features of interest are between about 40 to about 60 features.

7. The method according to claim 1 wherein said at least three features of interest are any one of features from accelerated segment test (FAST) features, maximally stable extremal region (MSER) features and Harris-Laplace features.

8. The method according to claim 1 wherein step c) of fitting a parametric surface to said 3D point cloud at said time K includes fitting a B-spline surface to said 3D point cloud.

9. The method according to claim 1 wherein step c) of fitting a parametric surface to said 3D point cloud at said time K includes fitting a Bezier surface to said 3D point cloud.

10. The method according to claim 1 wherein step c) of fitting a parametric surface to said 3D point cloud at said time K includes fitting a nonuniform rational spline surface to said 3D point cloud.

11. The method according to claim 1 wherein step c) of fitting a parametric surface to said 3D point cloud at said time K includes fitting a surface defined by selected radial basis functions to said 3D point cloud.

12. The method according to claim 1 wherein the stretching transformation is a local, 2D nonlinear transformation applied to a parametric surface in 3D.

13. The method according to claim 12 wherein said one or more positions are designated using a software based cursor designation system configured such that an operator designates said one or more positions by overlaying the cursor on each designated position and activating the cursor to store each designated position in a computer memory storage.

14. The method according to claim 12 wherein said one or more positions are designated using a physical designation system configured such that an operator manually designates said one or more designated positions directly on a selected region of the moving flexible surface.

15. The method according to claim 14 wherein said physical designation system is a laser pointer operated by the operator who points a laser beam from said laser pointer to highlight said one or more designated positions.

16. The method according to claim 12 wherein said visual display is said stereo camera.

17. The method according to claim 12 wherein said visual display is a separate camera different from said stereo camera configured so that said moving flexible surface is contained in a field of view of said separate camera.

18. The method according to claim 1 including designating one or more positions on the moving flexible surface and displaying on a visual display said one or more positions for tracking the movement of the one or more locations in real time.

19. A method on a computer for real time tracking of a moving flexible surfaces, comprising:

a) acquiring at time K, using a sensor, a 3D point cloud by scanning a moving flexible surface;

b) fitting a parametric surface to said 3D point cloud at said time K;

c) computing and labeling at least three 3D features from said 3D point cloud at said time K in a coordinate frame of said sensor and storing said 3D locations of said at least three 3D features in a vision processor memory storage;

f) mapping said 3D location of said 3D features of interest onto said parametric surface at time K and storing the parametric surface with mapped 3D locations in the vision processor memory storage;

g) retrieving from the vision processor memory storage 3D locations of said features of interest from previous left and right image frames synchronously taken at a time K−

1;

h) computing a best fit rigid transform using said retrieved 3D locations of the features of interest at time K−

1 and the 3D locations of the features at time K;

i) retrieving from the vision processor memory storage a parametric surface with mapped 3D locations at time K−

1, applying said rigid transformation to said retrieved parametric surface with mapped 3D locations at time K−

1 to obtain an intermediate parametric surface with mapped 3D locations;

j) matching the 3D locations of features of interest on the intermediate parametric surface at time K−

1to the same corresponding 3D locations on the stored parametric surface at time K and from said correspondences computing a stretching transformation; and

k) using said rigid and stretching transformations to calculate at time K a location of any region on said moving flexible surface based on a location of the same region on said moving flexible surface at time K−

1 stored in said vision processor memory storage.

20. The method according to claim 19 wherein said sensor is any one of a Lidar and a flash Lidar.

21. An article of manufacture, which includes a non-transitory computer readable medium having stored thereon a computer program for real time tracking of a deformable surface, comprising:

a) a first code segment which, when executed on a computer, analyses left and right image frames from left and right camera members of a stereo camera of a region of interest on a deformable surface at a time K and computes, from said left and right image frames, 3D point cloud data in a coordinate frame of said stereo camera at said time K, and fits a parametric surface to said 3D point cloud at said time K;

b) a second code segment which, when executed on a computer, extracts a location of at least three features of interest in said left image frame at said time K in 2D pixel coordinates, extracts a location of each at least three feature of interest in said right image frame at said time K in 2D pixel coordinates, and stores said 2D pixel coordinates of said locations of said features of interest in said left and right image frames at said time K in a storage device;

c) a third code segment which, when executed on a computer, computes, using said stored 2D pixel coordinates of said locations of said features of interest in said left and right image frames at said time K, a 3D location of said features of interest at said time K in a coordinate frame of said camera and storing said 3D locations in said storage device;

d) a fourth code segment which, when executed on a computer, maps said 3D location of said features of interest onto said parametric surface at time K and stores the parametric surface with mapped 3D locations in the storage device;

e) a fifth code segment which, when executed on a computer, retrieves from the storage device 3D locations of said features of interest from previous left and right image frames synchronously taken at a time K−

1 and computes a best fit rigid transform using said retrieved 3D locations of the features of interest at time K−

1 and the 3D locations of the features at time K;

f) a sixth code segment which, when executed on a computer, retrieves from the storage device a parametric surface with mapped 3D locations at time K−

1, applies said rigid transformation to said retrieved parametric surface with mapped 3D locations at time K−

1 to obtain an intermediate parametric surface with mapped 3D locations and matches the 3D locations of features of interest on the intermediate parametric surface at time K−

1 to the same corresponding 3D locations on the stored parametric surface at time K and from said correspondences computes a stretching transformation; and

g) a seventh code segment which, when executed on a computer, uses said rigid and stretching transformations to calculate at time K a location of any region on said deformable surface based on a location of the same region on said deformable surface at time K−

1 stored in said storage device.