MODALITY-AGNOSTIC METHOD FOR MEDICAL IMAGE REPRESENTATION

• US 20190332900A1
• Filed: 05/22/2018
• Published: 10/31/2019
• Est. Priority Date: 04/30/2018
• Status: Active Grant

• Current Assignee: Elekta AB
• Original Assignee: Elekta AB
• Inventors: Sjolund, Jens Olof, Adler, Jonas Anders
• Granted Patent: US 10,762,398 B2
• Time in Patent Office: Days
• CPC Class Codes: A61N 5/1039 using functional images, e.g. PET or MRI
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 11/003 Reconstruction from projections, e.g. tomography
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2211/441 AI-based methods, deep learning or artificial neural networks
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 2201/03 : Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock

Abstract

Techniques for the operation and use of a model that learns the general representation of multimodal images is disclosed. In various examples, methods from representation learning are used to find a common basis for representation of medical images. These include aspects of encoding, fusion, and downstream tasks, with use of the general representation and model. In an example, a method for generating a modality-agnostic model includes receiving imaging data, encoding the imaging data by mapping data to a latent representation, fusing the encoded data to conserve latent variables corresponding to the latent representation, and training a model using the latent representation. In an example, a method for processing imaging data using a trained modality-agnostic model includes receiving imaging data, encoding the data to the defined encoding, processing the encoded data with a trained model, and performing imaging processing operations based on output of the trained model.

Claims

1. A computer implemented method for generating a modality-agnostic image processing model, the method comprising:

receiving imaging data from multiple imaging modalities;

encoding the imaging data based on a defined data encoding, by mapping the imaging data to respective latent representations;

fusing the encoded imaging data, by mapping the respective latent representations to a fused latent representation of the encoded imaging data, wherein the mapping conserves respective latent variables corresponding to a spatial representation of the respective latent representations;

training a model for medical imaging processing, using the fused latent representation of the encoded imaging data; and

outputting the trained model, the model adapted to produce an output from subsequent medical imaging data according to the training using the fused latent representation.

2. The method of claim 1, wherein the encoding further produces confidence values corresponding to the respective latent representations, and wherein the confidence values are used as weights in mapping the respective latent representations into the fused latent representation.

3. The method of claim 1, wherein the encoding is performed with a variational autoencoder model.

4. The method of claim 3, wherein the variational autoencoder model comprises a neural network.

5. The method of claim 3, wherein the variational autoencoder model is trained for encoding of imaging data, and wherein the respective latent variables used in the fusing are produced from spatial latent variables used in the variational autoencoder model.

6. The method of claim 1, wherein the imaging data is captured from among the multiple imaging modalities, wherein the respective sets of imaging data captured from among the multiple imaging modalities includes images produced from among a plurality of modality types or from a plurality of modes of at least one modality type.

7. The method of claim 1, wherein the fused latent representation is produced as a fused representation image, and wherein the model is trained based on the fused representation image.

8. The method of claim 7, wherein the fused representation image is produced using at least one of:

spectral representations, covariance tapering, low rank approximations, or Gaussian Markov Random Fields.

9. The method of claim 1, wherein the defined data encoding is provided from among a plurality of encoders corresponding to a respective type or mode of the imaging modalities.

10. The method of claim 9, wherein the fusing of the encoded imaging data applies respective weights corresponding to the plurality of encoders in producing the fused latent representation.

11. The method of claim 10, wherein the respective weights corresponding to the plurality of encoders are normalized exponential confidence weights.

12. The method of claim 1, wherein the fused latent representation includes a same number of image dimensions as included in the received imaging data.

13. The method of claim 1, further comprising,

receiving the subsequent medical imaging data, the subsequent medical imaging data provided from another modality type or mode that is not included in the received imaging data captured from multiple imaging modalities;

operating the trained model on the subsequent medical imaging data of the another modality type or mode.

14. The method of claim 1, wherein the output produced from operating the trained model is used for performing radiotherapy planning operations.

15. A computer-implemented method for processing imaging data using a trained modality-agnostic model, the method comprising:

receiving medical imaging data of a first imaging modality;

encoding the medical imaging data based on a defined data encoding, by mapping the medical imaging data to a latent representation;

processing the encoded medical imaging data with a trained model, wherein the trained model produces an output based on the latent representation, based on training of the model to a fused latent representation of the imaging data, wherein training of the model is performed based on a mapping of a latent representation from at least a second imaging modality; and

performing an image processing operation on the medical imaging data based on the output produced by the trained model.

16. The method of claim 15, wherein the image processing operation comprises at least one of:

segmentation, denoising, synthesis, classification, regression, or reconstruction.

17. The method of claim 15, the method further comprising:

decoding the encoded imaging data; and

performing image reconstruction based on the decoded imaging data.

18. The method of claim 17, wherein the first imaging modality comprises a first modality type or a first modality mode, wherein the trained model is a neural network, and wherein the trained model is trained using data from at least a second modality type or a second modality mode, and wherein the first modality type or first modality mode is unused in training the model.

19. The method of claim 17, wherein the first imaging modality and the second imaging modality are respective types of imaging procedures, comprising at least two of:

Magnetic resonance imaging (MRI), computed tomography (CT), Positron Emission Tomography (PET), PET-CT, Ultrasound, X-Ray, Fluoroscopy, Single-photon emission computed tomography (SPECT), Elastography, Photoacoustic, Magnetoencephalography (MEG), or Electroencephalography (EEG) imaging procedures, or combinations of such imaging procedures.

20. The method of claim 19, wherein the first imaging modality and the second imaging modality are respective types of MRI imaging modes, comprising at least two of:

T1, T1 with contrast, T2, PD, SSFP, STIR, FLAIR, DIR, DWI, PWI, or fMRI.

21. The method of claim 15, wherein the encoding is performed with a variational autoencoder model, wherein the variational autoencoder model comprises a neural network, wherein the variational autoencoder model is trained for encoding of imaging data, and wherein the latent representation is produced from spatial latent variables used in the variational autoencoder model.

22. A system for operation of a modality-agnostic imaging processing model, the system comprising:

processing circuitry comprising at least one processor; and

a storage medium comprising instructions, which when executed by the at least one processor, cause the processor to;

process imaging data produced from a medical imaging modality;

encode the imaging data to a latent representation based on a defined data encoding;

perform mapping of the encoded imaging data to a common latent representation, that conserves respective latent variables corresponding to a spatial representation of the latent representation;

train a model for medical imaging processing, using the common latent representation; and

utilize the trained model to produce an output from subsequent medical imaging data, in connection with an image processing operation.

23. The system of claim 22, wherein the encoding further produces at least one confidence value corresponding to the latent representation, and wherein the confidence values are used as weights in mapping the latent representation into the common latent representation.

24. The system of claim 22, wherein the defined data encoding is performed with a variational autoencoder model, wherein the variational autoencoder model comprises a neural network, wherein the variational autoencoder model is trained for encoding of imaging data, and wherein the respective latent variables are produced from spatial latent variables used in the variational autoencoder model.

25. The system of claim 22, wherein the common latent representation is produced as a fused representation image from among multiple latent representations, wherein the model is trained based on the fused representation image, and wherein the fused representation image is produced using at least one of:

spectral representations, covariance tapering, low rank approximations, or Gaussian Markov Random Fields.

26. The system of claim 22, wherein the imaging data is captured from one of multiple imaging modalities, wherein the common latent representation provides a fusion of latent representations from among the multiple imaging modalities, and wherein the multiple imaging modalities comprises modalities from among a plurality of modality types or from a plurality of modes of at least one modality type.

27. The system of claim 26, wherein the defined data encoding is provided from among a plurality of encoders corresponding to a respective type or mode of the multiple imaging modalities, and wherein the mapping of the encoded imaging data applies respective weights corresponding to the plurality of encoders in producing the common latent representation.

28. The system of claim 22, wherein utilizing the model occurs as part of performing radiotherapy planning operations based on the subsequent medical imaging data.

29. The system of claim 28, wherein the image processing operation comprises at least one of:

segmentation, denoising, synthesis, classification, regression, or reconstruction.

30. The system of claim 22, wherein the imaging data is produced from at least one:

Magnetic resonance imaging (MRI), computed tomography (CT), Positron Emission Tomography (PET), PET-CT, Ultrasound, X-Ray, Fluoroscopy, Single-photon emission computed tomography (SPECT), Elastography, Photoacoustic, Magnetoencephalography (MEG), or Electroencephalography (EEG) imaging procedure, or combinations of such imaging procedures.

31. The system of claim 30, wherein the imaging procedure includes a MRI imaging procedure that operates in at least one type of MRI imaging mode, wherein the MRI imaging mode includes at least one of:

T1, T1 with contrast, T2, PD, SSFP, STIR, FLAIR, DIR, DWI, PWI, or fMRI.