Apparatus and method for registering 2D radiographic images with images reconstructed from 3D scan data

• US 7,204,640 B2
• Filed: 08/29/2003
• Issued: 04/17/2007
• Est. Priority Date: 08/29/2003
• Status: Active Grant

• Current Assignee: Accuray Incorporated
• Original Assignee: Accuray Incorporated
• Inventors: Kuduvalli, Gopinath, Fu, Dongshan, Qureshi, Shehrzad
• Primary Examiner(s): Glick; Edward J.
• Assistant Examiner(s): KAO, CHIH CHENG G
• Application Number:
  US10/652,786
• Publication Number:
  US 20050047544A1
• Time in Patent Office: 1,327 Days
• Field of Search: 378/4, 378/62, 378/63, 378/65, 378/205, 382/132, 382/128, 382/131, 382/294
• US Class Current: 378/205
• CPC Class Codes: A61N 2005/1061 using an x-ray imaging system having a separate imaging source
• CPC Class Codes: A61N 2005/1062 using virtual X-ray images, e.g. digitally reconstructed radiographs [DRR]
• CPC Class Codes: A61N 5/1049 for verifying the position of the patient with respect to the radiation beam
• CPC Class Codes: A61N 5/1067 in real time, i.e. during treatment
• CPC Class Codes: A61N 5/107 in real time, i.e. during treatment
• CPC Class Codes: G01N 23/04 and forming images of the material
• CPC Class Codes: G06T 2207/10124 Digitally reconstructed radiograph [DRR]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/38 Registration of image sequences

Abstract

A method and system is provided for registering a 2D radiographic image of a target with previously generated 3D scan data of the target. A reconstructed 2D image is generated from the 3D scan data. The radiographic 2D image is registered with the reconstructed 2D images to determine the values of in-plane transformation parameters (x, y, θ) and out-of-plane rotational parameters (r, Φ), where the parameters represent the difference in the position of the target in the radiographic image, as compared to the 2D reconstructed image. An initial estimate for the in-plane transformation parameters is made by a 3D multi-level matching process, using the sum-of-square differences similarity measure. Based on these estimated parameters, an initial 1-D search is performed for the out-of-plane rotation parameters (r, Φ), using a pattern intensity similarity measure. The in-plane parameters (x, y, θ) and out-of-plane parameters (r, Φ) are iteratively refined, until a desired accuracy is reached.

Claims

1. A method of registering a 2D (two-dimensional) x-ray image of a target with previously generated 3D (three-dimensional) scan data of said target, said x-ray image being characterized by an image plane defined by mutually orthogonal x- and y-coordinates, the method comprising:

A. generating at least one reconstructed image from said 3D scan data, said reconstructed image also characterized by the same image plane;

B. determining the value of in-plane transformation parameters (x, y, θ

) and out-of-plane rotational parameters (r, Φ

) for registering said reconstructed image onto said x-ray image, said in-plane and out-of-plane parameters representing a difference in the position of the target as shown in said x-ray image as compared to the position of the target as shown by said image reconstructed from said 3D scan data;

wherein r and Φ

represent rotations of said target about first and second mutually orthogonal axes, said rotations being out-of-plane with respect to said image plane, said out-of-plane rotations representing a projection of said target onto said image plane;

wherein x and y represent an amount of translation of said target within said image plane in the directions of said x- and y-axes, respectively, and θ

represents an amount of rotation of said target within said image plane about an axis perpendicular to both said x- and said y-axes;

and wherein step B comprises;

a. obtaining an initial estimate for said in-plane transformation parameters (x, y, θ

) by multi-level matching in 3D (three dimensions), between said x-ray image and said reconstructed image;

b. based on said in-plane transformation parameters (x, y, θ

) estimated in step a, performing an initial search in one dimension (1D) for each pair of out-of-plane rotation parameters (r, Φ

); and

c. iteratively refining said in-plane transformation parameters (x, y, θ

) and said out-of-plane rotational parameters (r, Φ

), until said in-plane and out-of-plane parameters converge to a desired accuracy; and

C. aligning a radiation source to the target using the values of the in-plane transformation parameters and the out-of-plane rotational parameters.

2. A method in accordance with claim 1, wherein said 3D multi-level matching is performed sequentially in each of a succession of a plurality of image resolution levels, starting at the lowest resolution level and ending at the highest resolution level.

3. A method in accordance with claim 1, further wherein said 2D x-ray image of said target is obtained by transmitting through said target an imaging beam having a known position and angle relative to said target, and wherein said reconstructed image is a 2D synthesized DRR (digitally reconstructed radiograph) representing a radiographic image of said target that would be obtained with said imaging beam at said known position and angle, if said target were positioned in accordance with said 3D scan data.

4. A method in accordance with claim 1, wherein determining the value of in-plane transformation parameters comprises:

A. determining a plurality N_r and N_Φ

of out-of-plane rotation angles, respectively, for said rotational parameters (r, Φ

); and

B. generating a plurality N_r*N_Φ

of 2D reference images, one reference image for each of said plurality N_r and N_Φ

of said out-of-plane rotation angles.

5. A method in accordance with claim 1, further comprising the step of generating offline, before step a, a plurality of in-plane rotated 2D reference images, by performing a series of in-plane rotations on said reconstructed image.

6. A method in accordance with claim 5, wherein said 3D matching process in step a is performed upon said in-plane rotated 2D reference images.

7. A method in accordance with claim 1, wherein said 3D matching process in step a is performed using a similarity measure method.

8. A method in accordance with claim 7, wherein said similarity measure method is based on a sum of absolute differences.

9. A method in accordance with claim 1, wherein step c of iteratively refining said in-plane and out-of-plane parameters comprises:

d. refining the in-plane translation parameters (x, y), to increase the accuracy of said parameters;

e. refining the in-plane rotation parameter (θ

) based on said out-of-plane rotation parameters (r, Φ

) searched in step b, and on said refined in-plane transformation parameters (x, y) from step d;

f. separately refining each of the out-of-plane rotation parameters (r, Φ

), based on said refined in-plane translation parameters from step d, and said refined rotation parameter from step e;

g. iteratively and sequentially repeating steps d, e, and f, until a predetermined accuracy is reached; and

h. refining once more said out-of-plane rotation parameters (r, Φ

).

10. A method in accordance with claim 9, wherein step d of initially refining the in-plane translation parameters is performed by sub-pixel matching in two dimensions.

11. A method in accordance with claim 9, wherein step e of refining the in-plane rotation parameter is performed by 1D (one dimensional) interpolation.

12. A method in accordance with claim 9, wherein step f of separately refining said out-of-plane rotation parameters is performed through a 1D (one dimensional) search.

13. A method in accordance with claim 9, wherein step h of refining said out-of-plane rotation parameters (r, Φ

) is performed by 1D interpolation.

14. A method in accordance with claim 9, wherein steps d, e, and f are performed using a similarity measure based on pattern intensity.

15. A method in accordance with claim 1, wherein said accuracy is sufficient to achieve a resolution of less than about 1 mm.

16. A method in accordance with claim 1, wherein said 3D scan data comprise at least one of CT scan data, MRI scan data, and PET (positron emission tomography) data.

17. A method in accordance with claim 1, wherein said 1D search for said out-of-plane rotation parameters in step b is performed using a similarity measure.

18. A method in accordance with claim 17, wherein said similarity measure is based on pattern intensity.

19. A method in accordance with claim 1, wherein search space for said 1D search in step b is the full search range of out-of-plane rotation angles, and said full search range is sampled by one degree increments.

20. A method in accordance with claim 1, further comprising the step of processing said 2D x-ray image, after step A and before step B, so as to match the orientation, image size, and bit depth of said x-ray image with the orientation, image size, and bit depth of said reconstructed image.

21. A system for registering at least one 2D radiographic image of a target with at least one image reconstructed from previously generated 3D scan data of said target, said radiographic image being characterized by an image plane defined by mutually orthogonal x- and y-axes, the system comprising:

a. means for providing said 3D scan data of said target;

b. a radiation source for generating at least one radiographic imaging beam having a known intensity, and having a known location and angle relative to said target;

c. an imaging system for generating a 2D radiographic image of said target in near real time; and

d. a controller, including;

i) means for generating said at least one reconstructed 2D image of said target, using said 3D scan data, and using said known location, angle, and intensity of said imaging beam; and

ii) software for determining a set of in-plane transformation parameters (x, y, θ

) and out-of-plane rotational parameters (r, Φ

), said in-plane and out-of-plane parameters representing a difference in the position of the target as shown in said radiographic image as compared to the position of the target as shown by said 2D reconstructed image;

wherein said software comprises means fix performing a 3D multi-level matching to determine an initial estimate for said in-plane transformation parameters (x, y, θ

);

wherein r and Φ

represent rotations of said target about first and second mutually orthogonal axes, said rotations being out-of-plane with respect to said image plane, said out-of-plane rotations representing a projection of said target onto said image plane; and

wherein x and y represent an amount of translation of said target within said image plane in the directions of said x- and y-axes, respectively, and θ

represents an amount of rotation of said target within said image plane about an axis perpendicular to both said x- and said y-axes.

22. A system in accordance with claim 21, wherein said software for determining said in-plane and out-of-plane rotational parameters comprises:

means for performing a 1D search for each of the pair of out-of-plane rotation parameters (r, Φ

) based on said initially estimated in-plane parameters (x, y, θ

), and

means for iteratively refining said in-plane parameters (x, y, θ

) and said out-of-plane parameters (r, Φ

), until a desired accuracy is reached.

23. A system in accordance with claim 22, wherein said 3D multi-level matching means performs sequentially in each of a succession of a plurality of resolution levels, starting at the lowest resolution level and ending at the highest resolution level.

24. A system in accordance with claim 22, wherein said 3D multi-level matching means comprises similarity measure means based on a sum of absolute differences.

25. A system in accordance with claim 22, wherein said means for iteratively refining said in-plane and out-of-plane parameters comprises:

d. means for refining the in-plane translation parameters (x, y), to increase the accuracy of said parameters;

e. means for refining the in-plane rotation parameter (θ

) based on said out-of-plane rotation parameters (r, Φ

) searched in step b, and on said refined h-plane transformation parameters (x, y) from step d;

f. means for separately refining each of the out-of-plane rotation parameters (r, Φ

), based on said refined in-plane translation parameters from step d, and said refined rotation parameter from step e; and

g. means for iteratively and sequentially repeating steps d, e, and f, until a predetermined accuracy is reached, and for refining once more said out-of-plane rotation parameters (r, Φ

).

26. A system in accordance with claim 22, wherein said means for refining the in-plane translation parameters comprises 2D sub-pixel matching means.

27. A system in accordance with claim 22, wherein said means for refining the in-plane rotation parameters comprises 1D (one dimensional) interpolation means.

28. A system in accordance with claim 22, wherein said means for separately refining said out-of-plane rotation parameters comprises means for performing one or more 1D searches.

29. A system in accordance with claim 22, wherein said means for refining said out-of-plane rotation parameters (r, Φ

) comprises 1D interpolation means.

30. A system in accordance with claim 22, wherein said desired accuracy is sufficient to achieve a resolution of less than about 1 mm.

31. A system in accordance with claim 22, wherein means for performing a 1D search for said out-of-plane rotation parameters comprises means for performing a similarity measure based on pattern intensity.

32. A system in accordance with claim 22, wherein said means for refining the in-plane translation parameters (x, y), said means for refining the in-plane rotation parameter (θ

), and said means for separately refining said out-of-plane rotation parameters (r, Φ

) comprises means for performing one or more similarity measure.

33. A system in accordance with claim 22, further comprising means for processing said 2D x-ray image so as to match the orientation, image size, and bit depth of said x-ray image with the orientation, image size, and bit depth of said reconstructed image.

34. A system in accordance with claim 21, wherein said radiation source comprises an x-ray source, said 2D radiographic image comprises a 2D x-ray image, and said reconstructed image comprises a 2D DRR.

35. A system in accordance with claim 21, wherein said controller further comprises:

A. means for determining a plurality N_r and N_Φ

of out-of-plane rotation angles, respectively, for said rotational parameters (r, Φ

); and

B. means for generating a plurality N_r*N₁₀₁ of 2D reference images, one reference image for each of said plurality N_r and N_Φ

of said out-of-plane rotation angles.

36. A system in accordance with claim 21, wherein said controller further comprises means for generating offline a plurality of in-plane rotated 2D reference images by performing a series of in-plane rotations on said reconstructed image.

37. A system in accordance with claim 21, wherein said 3D scan data comprise at least one of CT scan data, MRI scan data, and PET (positron emission tomography) data.

38. A method, comprising:

acquiring x-ray images of a target volume in a first image plane and a second image plane, the x-ray images defining a present orientation of the target volume;

generating synthetic x-ray reference images of the target volume from 3-dimensional scan data representing a previous orientation of the target volume, the synthetic x-ray reference images corresponding to in-plane transformations and out-of-plane rotations of the target volume projected onto the first image plane and the second image plane;

determining a difference between the present orientation of the target volume and the previous orientation of the target volume in three translational coordinates and three rotational coordinates by comparing in-plane transformation parameters and out-of-plane rotation parameters of the x-ray images and the synthetic x-ray reference images in the first image plane and the second image plane, wherein determining said difference comprises searching the in-plane transformation parameters in the first image plane and the second image plane using a first similarity measure between the x-ray images and the synthetic x-ray reference images in a 3-dimensional multi-level search; and

aligning a radiation source to the target volume using the difference between the present orientation of the target volume and the previous orientation of the target volume.

39. The method of claim 38, wherein the in-plane transformation parameters comprise two in-plane translation parameters and one in-plane rotation parameter in each of the first image plane and the second image plane, and wherein the out-of-plane rotation parameters comprise two mutually orthogonal rotations with respect to each of the first image plane and the second image plane.

40. The method of claim 39, further comprising processing the x-ray images to match image properties of the synthetic x-ray reference images, wherein determining the difference between the present orientation of the target volume and the previous orientation of the target volume further comprises:

estimating the in-plane transformation parameters in the first image plane and the second image plane using a plurality of in-plane rotated synthetic x-ray reference images;

searching the out-of-plane rotation parameters in the first image plane and the second image plane in a 1-dimensional search using a second similarity measure between the x-ray images and the synthetic x-ray reference images;

refining the in-plane translation parameters in the first image plane and the second image plane using 2-dimensional sub-pixel matching between the x-ray images and the synthetic x-ray reference images;

refining the in-plane rotation parameters in the first image plane and the second image plane using 1-dimensional interpolation based on the in-plane translation parameters and the out-of-plane rotation parameters; and

refining the out-of-plane rotation parameters in the first image plane and the second image plane using a 1-dimensional search based on the refined in-plane translation and in-plane rotation parameters using the second similarity measure.

41. The method of claim 40, wherein the first similarity measure comprises a sum of absolute differences measure and the second similarity measure comprises an image pattern intensity measure.

42. The method of claim 40, wherein the 3-dimensional multi-level search comprises a four level search proceeding from a low resolution search at a first level, through progressively higher resolution searches at a second level and a third level, to a highest resolution search at a fourth level, a resolution parameter at each level defined by low pass filtering the x-ray images.

43. The method of claim 40, wherein processing the x-ray images to match image properties of the synthetic x-ray reference images comprises matching an orientation, an image size, and a bit depth of the x-ray images with an orientation, an image size, and a bit depth of the synthetic x-ray reference images.

44. A system, comprising:

a radiation source and an imaging system to generate 2D radiographic images of a target volume in a first image plane and a second image plane, the 2D radiographic images defining a present orientation of the target volume;

a 3D scan data generator to generate reconstructed 2D reference images of the target volume from 3D scan data representing a previous orientation of the target volume, the reconstructed 2D reference images corresponding to in-plane transformations and out-of-plane rotations of the target volume projected onto the first image plane and the second image plane; and

a controller coupled with the radiation source, the imaging system and the 3D scan data generator, the controller configured to;

determine a difference between the present orientation of the target volume and the previous orientation of the target volume in three translational coordinates and three rotational coordinates by comparing in-plane transformation parameters and out-of-plane rotation parameters of the 2D radiographic images and the reconstructed 2D reference images in the first image plane and the second image plane, wherein to determine said difference the controller is configured to search the in-plane transformation parameters in the first image plane and the second image plane using a first similarity measure between the 2D radiographic images and the reconstructed 2D reference images in a 3-dimensional multi-level search.

45. The system of claim 44, wherein the in-plane transformation parameters comprise two in-plane translation parameters and one in-plane rotation parameter in each of the first image plane and the second image plane, and wherein the out-of-plane rotation parameters comprise two mutually orthogonal rotations with respect to each of the first image plane and the second image plane.

46. The system of claim 45, the controller further to process the 2D radiographic images to match image properties of the reconstructed 2D reference images, wherein to determine the difference between the present orientation of the target volume and the previous orientation of the target volume, the controller is configured to:

estimate the in-plane transformation parameters in the first image plane and the second image plane using a plurality of in-plane rotated reconstructed 2D reference images;

to

search the out-of-plane rotation parameters in the first image plane and the second image plane in a 1-dimensional search using a second similarity measure between the 2D radiographic images and the reconstructed 2D reference images;

to

refine the in-plane translation parameters in the first image plane and the second image plane using 2-dimensional sub-pixel matching between the 2D radiographic images and the reconstructed 2D reference images;

to

refine the in-plane rotation parameters in the first image plane and the second image plane using 1-dimensional interpolation based on the in-plane translation parameters and the out-of-plane rotation parameters; and

to

refine the out-of-plane rotation parameters in the first image plane and the second image plane using a 1-dimensional search based on the refined in-plane translation and in-plane rotation parameters using the second similarity measure.

47. The system of claim 46, wherein yhe first similarity measure comprises a sum of absolute differences measure and the second similarity measure comprises an image pattern intensity measure.

48. The system of claim 46, wherein the 3-dimensional multi-level search comprises a four level search proceeding from a low resolution search at a first level, through progressively higher resolution searches at a second level and a third level, to a highest resolution search at a fourth level, a resolution parameter at each level defined by low pass filtering the 2D radiographic images.

49. The method of claim 46, the controller to process the 2D radiographic images to match an orientation, an image size, and a bit depth of the reconstructed 2D reference images.