Adversarial prediction of radiotherapy treatment plans

• US 11,077,320 B1
• Filed: 02/07/2020
• Issued: 08/03/2021
• Est. Priority Date: 02/07/2020
• Status: Active Grant

• Current Assignee: Elekta Incorporated (Elekta AB)
• Original Assignee: Elekta Incorporated (Elekta AB)
• Inventors: Hibbard, Lyndon Stanley
• Primary Examiner(s): Thomas, Courtney D
• Application Number:
  US16/784,919
• Publication Number:
  US 20210244971A1
• Time in Patent Office: 543 Days
• CPC Class Codes: A61N 5/103 Treatment planning systems
• CPC Class Codes: A61N 5/1031 using a specific method of dose optimization
• CPC Class Codes: A61N 5/1039 using functional images, e.g. PET or MRI
• CPC Class Codes: A61N 5/1047 with movement of the radiation head during application of radiation, e.g. for intensity modulated arc therapy or IMAT
• CPC Class Codes: A61N 5/1067 in real time, i.e. during treatment
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture

Abstract

Systems and methods are disclosed for generating radiotherapy treatment machine parameters based on projection images of a target anatomy. The systems and methods include operations including receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image; training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

Claims

1. A computer-implemented method for generating one or more radiotherapy treatment plans, the method comprising:

receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;

training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and

using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

2. The method of claim 1, wherein the GAN comprises a conditional adversarial network (cGAN).

3. The method of claim 1, wherein the GAN comprises a cycle-consistent generative adversarial network (CycleGAN).

4. The method of claim 1, wherein the given gantry angle of a first pair of the set of pairs differs from the given gantry angle of a second pair of the set of pairs by a predetermined amount.

5. The method of claim 1, wherein:

the GAN is configured to train the generative model using a discriminative model;

values applied by the generative model and the discriminative model are established using adversarial training between the discriminative model and the generative model; and

the generative model and the discriminative model comprise respective convolutional neural networks.

6. The method of claim 5, wherein:

the adversarial training comprises;

training the generative model to generate a first synthetic graphical aperture image representation of the MLC leaf positions at a first gantry angle from a projection image that represents a view of a training subject anatomy from the first gantry angle; and

training the discriminative model to classify the first synthetic graphical aperture image as a synthetic or a real training example graphical aperture image; and

an output of the generative model is used for training the discriminative model and an output of the discriminative model is used for training the generative model.

7. The method of claim 6, wherein the GAN is trained using a cycle-consistent generative adversarial network (CycleGAN) comprising the generative model and the discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, wherein the CycleGAN further comprises:

a second generative model trained to;

process, from a given pair of the set of pairs, a given graphical aperture image representation of the MLC leaf positions at a given gantry angle as an input; and

provide a synthetic projection image that represents a view of a training subject anatomy from the given gantry angle as an output; and

a second discriminative model trained to classify the synthetic projection image as a synthetic or a real projection image.

8. The method of claim 7, wherein the CycleGAN comprises a first portion to train the first generative model, the first portion being trained to:

obtain a set of training projection images representing different views of a patient anatomy from prior treatments that are paired with training graphical aperture images corresponding to each of the different views, each of the training graphical aperture images being aligned with a respective one of the training projection images;

transmit the set of training projection images to an input of the first generative model to output a first set of graphical aperture images;

receive the first set of graphical aperture images at an input of the first discriminative model to classify the first set of graphical aperture images as a synthetic or real training set of graphical aperture images; and

receive the first set of graphical aperture images at an input of the second generative model to generate a first set of cycle projection images, for calculating cycle-consistency losses.

9. The method of claim 8, wherein the CycleGAN comprises a second portion that is trained to:

transmit the training graphical aperture images corresponding to each of the different views to the input of the second generative model to output a first set of synthetic projection images;

receive the first set of synthetic projection images at an input of the second discriminative model to classify the first set of synthetic projection images as synthetic or real training projection images; and

receive the first set of synthetic projection images at the input of the first generative model to generate a first set of cycle graphical aperture images for calculating cycle-consistency losses.

10. The method of claim 9, wherein:

the cycle-consistency losses are generated based on a comparison of the first set of cycle projection images with the set of training projection images and a comparison of the first set of cycle graphical aperture images with the training graphical aperture images;

the first generative model is trained to minimize or reduce a first loss term that represents an expectation of difference between a plurality of synthetic graphical aperture images and respectively paired training graphical aperture images; and

the second generative model is trained to minimize or reduce a second loss term that represents an expectation of difference between a plurality of synthetic projection images and respectively paired training projection images.

11. The method of claim 1, further comprising:

obtaining radiotherapy treatment machine parameter information representing MLC leaf positions at gantry angles corresponding to each of the views represented by the projection images of the set of pairs of image data and radiotherapy beam intensities corresponding to each of the projection images;

generating training graphical aperture image representations based on the obtained radiotherapy treatment machine parameter information; and

aligning each of the generated training graphical aperture image representations with the corresponding projection images.

12. The method of claim 11, wherein the training graphical aperture image representations and corresponding projection images are two-dimensional images or three-dimensional images comprising stacks of two-dimensional projection image and graphical aperture image pairs corresponding to an entire treatment fraction.

13. The method of claim 1, further comprising generating one or more radiotherapy treatment machine parameters from the aperture image predicted by the trained GAN model.

14. The method of claim 13, further comprising computing a dose volume histogram or a three-dimensional dose distribution based on the generated one or more radiotherapy treatment machine parameters.

15. The method of claim 13, wherein the one or more parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiation therapy beam intensity.

16. The method of claim 13, further comprising computing dosimetric parameters based on the generated one or more radiotherapy treatment machine parameters.

17. The method of claim 1, wherein the given projection images in the set of pairs of image data is generated by ray tracing or Fourier reconstruction.

18. A system for generating one or more radiotherapy treatment plans, the system comprising:

one or more processors configured to perform operations comprising;

receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;

training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and

using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

19. The system of claim 18, wherein the GAN comprises a conditional adversarial network (cGAN) or a cycle-consistent generative adversarial network (CycleGAN).

20. The system of claim 18, further comprising operations for generating, as the projection images in the set of pairs of image data, a plurality of two-dimensional projection images that represent a plurality of two-dimensional views of the anatomy from a plurality of different gantry angles.

21. The system of claim 20, wherein the plurality of two-dimensional projection images represents views of the anatomy at a plurality of angles spanning the arc of motion of a gantry of the radiotherapy treatment machine during therapy.

22. A non-transitory computer-readable medium comprising non-transitory computer-readable instructions for performing operations comprising:

receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;

training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and

using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.