Phenotype analysis of cellular image data using a deep metric network

• US 10,134,131 B1
• Filed: 02/15/2017
• Issued: 11/20/2018
• Est. Priority Date: 02/15/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Ando, Dale M., Berndl, Marc
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US15/433,027
• Time in Patent Office: 643 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 7/0014 using an image reference approach

Abstract

The disclosure relates to phenotype analysis of cellular image data using a machine-learned, deep metric network model. An example method includes receiving, by a computing device, a target image of a target biological cell having a target phenotype. Further, the method includes obtaining, by the computing device, semantic embeddings associated with the target image and each of a plurality of candidate images of candidate biological cells each having a respective candidate phenotype. The semantic embeddings are generated using a machine-learned, deep metric network model. In addition, the method includes determining, by the computing device, a similarity score for each candidate image. Determining the similarity score for a respective candidate image includes computing a vector distance between the respective candidate image and the target image. The similarity score for each candidate image represents a degree of similarity between the target phenotype and the respective candidate phenotype.

Claims

1. A method, comprising:

receiving, by a computing device, a target image of a target biological cell having a target phenotype;

obtaining, by the computing device, a semantic embedding associated with the target image, wherein the semantic embedding associated with the target image is generated using a machine-learned, deep metric network model;

obtaining, by the computing device for each of a plurality of candidate images of candidate biological cells each having a respective candidate phenotype, a semantic embedding associated with the respective candidate image, wherein the semantic embedding associated with the respective candidate image is generated using the machine-learned, deep metric network model;

determining, by the computing device, a similarity score for each candidate image,

wherein determining the similarity score for a respective candidate image comprises computing, by the computing device, a vector distance in a multi-dimensional space described by the semantic embeddings between the respective candidate image and the target image, and

wherein the similarity score for each candidate image represents a degree of similarity between the target phenotype and the respective candidate phenotype;

determining, by the computing device, a threshold similarity score; and

determining, by the computing device, those candidate images having similarity scores that satisfy the threshold similarity score,

wherein the target phenotype is a healthy phenotype,

wherein the candidate images having similarity scores that satisfy the threshold similarity score have respective candidate phenotypes corresponding to the healthy phenotype, and

wherein those candidate images having similarity scores that do not satisfy the threshold similarity score have respective candidate phenotypes corresponding to an unhealthy phenotype.

2. The method of claim 1,

wherein the machine-learned, deep metric network model was trained using a plurality of photographic images as training data,

wherein the photographic images are query results ranked based on selections, and

wherein rankings of the photographic images are used in the training of the machine-learned, deep metric network model to determine image similarity between two images.

3. The method of claim 1, further comprising training the machine-learned, deep metric network model, wherein training the machine-learned, deep metric network model comprises:

receiving, by the computing device, a series of three-image sets as training data,

wherein each three-image set comprises a query image, a positive image, and a negative image,

wherein the query image, the positive image, and the negative image are query results ranked in comparison with one another based on selections, and

wherein the selections indicate that a similarity between the query image and the positive image is greater than a similarity between the query image and the negative image; and

refining, by the computing device, the machine-learned, deep metric network model based on each three-image set to account for image components of the query image, the positive image, and the negative image.

4. The method of claim 1, further comprising training the machine-learned, deep metric network model using unsupervised learning.

5. The method of claim 1, further comprising:

determining, by the computing device, a second threshold similarity score; and

determining, by the computing device, those candidate images having similarity scores that satisfy the second threshold similarity score and do not satisfy the threshold similarity score,

wherein the candidate images having similarity scores that satisfy the second threshold similarity score and do not satisfy the threshold similarity score have respective candidate phenotypes corresponding to a first disease state,

wherein the candidate images having similarity scores that do not satisfy the second threshold similarity score have respective candidate phenotypes corresponding to a second disease state, and

wherein the first disease state and the second disease state correspond to the unhealthy phenotype.

6. The method of claim 1, further comprising:

receiving, by the computing device, a second target image of a second target biological cell having a second target phenotype;

obtaining, by the computing device, a semantic embedding associated with the second target image, wherein the semantic embedding associated with the second target image is generated using the machine-learned, deep metric network model;

determining, by the computing device, a second similarity score for each candidate image,

wherein determining the second similarity score for a respective candidate image comprises computing, by the computing device, a vector distance in the multi-dimensional space described by the semantic embeddings between the respective candidate image and the second target image, and

wherein the second similarity score for each candidate image represents a degree of similarity between the second target phenotype and the respective candidate phenotype;

determining, by the computing device, a second threshold similarity score; and

determining, by the computing device, those candidate images having second similarity scores that satisfy the second threshold similarity score,

wherein the second target phenotype is a second unhealthy phenotype,

wherein the candidate images having second similarity scores that satisfy the second threshold similarity score have respective candidate phenotypes corresponding to the second unhealthy phenotype, and

wherein those candidate images having second similarity scores that do not satisfy the second threshold similarity score have respective candidate phenotypes that do not correspond to the second unhealthy phenotype.

7. The method of claim 1, further comprising:

determining, by the computing device, a maximum similarity score among all of the candidate images,

wherein the maximum similarity score corresponds to a candidate image of a candidate biological cell having a candidate phenotype that is most similar to the target phenotype.

8. The method of claim 7, further comprising:

determining, by the computing device, a treatment regimen for a patient based on the candidate biological cell having the candidate phenotype that is most similar to the target phenotype.

9. The method of claim 1, further comprising:

obtaining, by the computing device for each of a plurality of control group images of control group biological cells having control group phenotypes, a semantic embedding associated with the respective control group image, wherein the semantic embedding associated with the respective control group image is generated using the machine-learned, deep metric network model; and

normalizing, by the computing device, the semantic embeddings associated with the candidate images, wherein normalizing comprises;

computing, by the computing device, eigenvalues and eigenvectors of a covariance matrix defined by values of each dimension of the semantic embeddings associated with the control group images using principal component analysis;

scaling, by the computing device, the values of each dimension of the semantic embeddings associated with the control group images by a respective dimensional scaling factor such that each dimension is zero-centered and has unit variance; and

scaling, by the computing device, the values of each corresponding dimension of the semantic embeddings associated with the candidate images by the respective dimensional scaling factor.

10. The method of claim 9, wherein normalizing, by the computing device, the semantic embeddings associated with the candidate images negates an influence of common morphological variations among the candidate biological cells on the similarity scores.

11. The method of claim 10, wherein the common morphological variations among the candidate biological cells comprise variations in cellular size, variations in nuclear size, variations in cellular shape, variations in nuclear shape, variations in nuclear color, variations in nuclear size relative to cellular size, or variations in nuclear location within a respective cell.

12. The method of claim 1, further comprising:

obtaining the candidate biological cells, wherein the candidate biological cells initially exhibit unhealthy phenotypes;

treating the candidate biological cells with various concentrations of various candidate treatment compounds;

recording the candidate images of the candidate biological cells; and

transmitting the plurality candidate images to the computing device.

13. The method of claim 1,

wherein the target biological cell was acquired from a first anatomical region of a patient during a biopsy, and

wherein the candidate biological cells comprise biological cells acquired from anatomical regions of the patient during the biopsy other than the first anatomical region.

14. The method of claim 1, wherein the semantic embedding associated with the target image and the semantic embeddings associated with the candidate images comprise 64-dimensional embeddings for each channel of a corresponding image.

15. The method of claim 1, further comprising:

scaling, by the computing device, the target image and each of the candidate images such that an image size of the target image and images sizes of each of the candidate images match an image size interpretable using the machine-learned, deep metric network model.

16. The method of claim 15, wherein scaling the target image and each of the candidate images comprises, for each respective image:

determining, by the computing device, a location of a cellular nucleus within the respective image; and

cropping, by the computing device, the respective image based on a rectangular box centered on the cellular nucleus.

17. The method of claim 1, wherein obtaining the semantic embedding associated with the target image or one of the respective candidate images comprises:

retrieving, by the computing device, each channel of a respective image;

obtaining, by the computing device, a semantic embedding for each channel of the respective image, wherein the semantic embedding for each channel of the respective image is generated using the machine-learned, deep metric network model; and

concatenating, by the computing device, the semantic embeddings for each channel of the respective image into a single semantic embedding.

18. The method of claim 17,

wherein each channel in the target image and each of the candidate images corresponds to a predetermined range of wavelengths detectable by one or more cameras, each having one or more associated optical filters,

wherein each of the one or more cameras is configured to record the target image or at least one of the plurality of candidate images, and

wherein one or more investigated regions of the target biological cell or the candidate biological cells emit light within the predetermined ranges of wavelengths.

19. The method of claim 18, wherein the one or more investigated regions of the target biological cell or the candidate biological cells emit light within the predetermined ranges of wavelengths due to a fluorescent compound that targets the one or more investigated regions and is excited by an electromagnetic excitation source, a chemical dye that targets the one or more investigated regions, or a chemiluminescent compound that targets the one or more investigated regions.

20. The method of claim 18, wherein the one or more investigated regions correspond to a location of a nucleus, a nucleolus, a cytoskeleton, a mitochondrion, a cellular membrane, an endoplasmic reticulum, a golgi body, a ribosome, a vesicle, a vacuole, a lysosome, or a centrosome within the target biological cell or the candidate biological cells.

21. The method of claim 1, wherein the plurality of candidate cells each correspond to a candidate mechanism of action.

22. The method of claim 1,

wherein obtaining, by the computing device, the semantic embedding associated with the target image comprises generating, by an autoencoder, the semantic embedding associated with the target image, and

wherein obtaining, by the computing device for each candidate image, the semantic embedding associated with the respective candidate image comprises generating, by the autoencoder for each candidate image, the semantic embedding associated with the respective candidate image.

23. A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method, comprising:

receiving, by the processor, a target image of a target biological cell having a target phenotype;

obtaining, by the processor, a semantic embedding associated with the target image, wherein the semantic embedding associated with the target image is generated using a machine-learned, deep metric network model;

obtaining, by the processor for each of a plurality of candidate images of candidate biological cells each having a respective candidate phenotype, a semantic embedding associated with the respective candidate image, wherein the semantic embedding associated with the respective candidate image is generated using the machine-learned, deep metric network model;

determining, by the processor, a similarity score for each candidate image,

wherein determining the similarity score for a respective candidate image comprises computing, by the processor, a vector distance in a multi-dimensional space described by the semantic embeddings between the respective candidate image and the target image, and

wherein the similarity score for each candidate image represents a degree of similarity between the target phenotype and the respective candidate phenotype;

determining, by the processor, a threshold similarity score; and

determining, by the processor, those candidate images having similarity scores that satisfy the threshold similarity score,

wherein the target phenotype is a healthy phenotype,

wherein the candidate images having similarity scores that satisfy the threshold similarity score have respective candidate phenotypes corresponding to the healthy phenotype, and

wherein those candidate images having similarity scores that do not satisfy the threshold similarity score have respective candidate phenotypes corresponding to an unhealthy phenotype.

24. A method, comprising:

preparing a multi-well sample plate with a target biological cell having a target phenotype and candidate biological cells;

applying a variety of candidate treatment regimens to each of the candidate biological cells;

recording a target image of the target biological cell;

recording candidate images of each of the candidate biological cells, each having a respective candidate phenotype arising in response to the candidate treatment regimen being applied;

receiving, by a computing device, the target image and the candidate images;

obtaining, by the computing device, a semantic embedding associated with the target image, wherein the semantic embedding associated with the target image is generated using a machine-learned, deep metric network model;

obtaining, by the computing device for each candidate image, a semantic embedding associated with the respective candidate image, wherein the semantic embedding associated with the respective candidate image is generated using the machine-learned, deep metric network model;

determining, by the computing device, a similarity score for each candidate image,

wherein determining the similarity score for a respective candidate image comprises computing, by the computing device, a vector distance in a multi-dimensional space described by the semantic embeddings between the respective candidate image and the target image, and

wherein the similarity score for each candidate image represents a degree of similarity between the target phenotype and the respective candidate phenotype;

determining, by the computing device, a threshold similarity score; and

determining, by the computing device, those candidate images having similarity scores that satisfy the threshold similarity score,

wherein the target phenotype is a healthy phenotype,

wherein the candidate images having similarity scores that satisfy the threshold similarity score have respective candidate phenotypes corresponding to the healthy phenotype, and

wherein those candidate images having similarity scores that do not satisfy the threshold similarity score have respective candidate phenotypes corresponding to an unhealthy phenotype; and

selecting, by the computing device, a preferred treatment regimen among the variety of candidate treatment regimens based on the similarity scores.