Systems and methods for processing retinal images for screening of diseases or abnormalities

• US 9,008,391 B1
• Filed: 04/30/2014
• Issued: 04/14/2015
• Est. Priority Date: 10/22/2013
• Status: Active Grant

• Current Assignee: Eyenuk, Inc.
• Original Assignee: Eyenuk, Inc.
• Inventors: Solanki, Kaushal Mohanlal, Bhat Krupakar, Sandeep, Amai Ramachandra, Chaithanya, Bhaskaranand, Malavika
• Primary Examiner(s): Chen, Wenpeng
• Application Number:
  US14/266,753
• Publication Number:
  US 20150110368A1
• Time in Patent Office: 349 Days
• Field of Search: None
• US Class Current: 382/128
• CPC Class Codes: A61B 3/0025 characterised by electronic signal processing, e.g. eye models
• CPC Class Codes: A61B 3/12 for looking at the eye fundus, e.g. ophthalmoscopes A61B3/13 takes precedence
• CPC Class Codes: A61B 3/14 Arrangements specially adapted for eye photography apparatus or arrangements for taking photographs per se G03B
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 2207/30104 Vascular flow; Blood flow; Perfusion
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0014 : using an image reference approach
• CPC Class Codes: G06T 7/0016 : involving temporal comparison
• CPC Class Codes: G06V 10/267 : by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/50 : by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/758 : Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 2201/03 : Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/14 : Vascular patterns
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 : Preprocessing; Feature extraction
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass

Abstract

Embodiments disclose systems and methods that aid in screening, diagnosis and/or monitoring of medical conditions. The design includes a system and method for accessing retinal images related to a patient, each retinal image including a plurality of pixels. For each of the retinal images, the system designates a first set of the plurality of pixels as active pixels including interesting retinal image regions and computes a first vector of numbers comprising pixel-level descriptors for each of the active pixels. The system computes a second vector of numbers for each of the retinal images and provides a second classification using supervised learning.

Claims

1. A computing system comprising:

one or more hardware computer processors; and

one or more storage devices configured to store software instructions configured for execution by the one or more hardware computer processors in order to cause the computing system to;

access retinal images related to a patient, each of the retinal images comprising a plurality of pixels;

for each of the retinal images, designate a first set of the plurality of pixels as active pixels including interesting retinal image regions;

for each of the retinal images, compute a first vector of numbers comprising pixel-level descriptors for each of the active pixels;

for each of the retinal images, provide a first classification for each of the active pixels using supervised learning utilizing the corresponding first vector of numbers;

for each of the retinal images, compute a second vector of numbers including one or more of;

a first histogram of pixel-level classifier decision statistics computed during the first classification;

a third vector of numbers computed by obtaining a nearest codeword of pixel-level descriptors aggregated as a second histogram of codeword frequencies in each of the retinal images, wherein a set of codewords is computed from a representative set of images; and

a third histogram of active region-level decision statistic numbers computed by;

grouping neighboring pixels detected in the first set of the plurality of pixels into active regions, computing a number representing each of the active regions using one or more of;

mean, median, maximum, or minimum of pixel-level classifier decision statistics computed in the first classification for each pixel that is part of at least one active region, wherein the third histogram is computed over the active regions detected in each of the retinal images; and

provide a second classification for one or more of;

the plurality of pixels, an interesting region within one of the retinal images, one of the retinal images in its entirety, or a collection of the retinal images, using supervised learning.

2. The computing system of claim 1, wherein the computing system is further configured to, resize at least one of the retinal images to a standard size image such that dimensions of the standard size image are determined using field of view information provided or inferred from a structure in the at least one retinal image or using a diameter of a fundus mask.

3. The computing system of claim 1, wherein the computing system is further configured to process a plurality of images obtained from a single patient imaging session or encounter.

4. The computing system of claim 1, wherein the computing system is further configured to process a plurality of images from a single patient session or encounter such that the second vector of numbers for at least a subset of the plurality of images are combined with or without further processing to obtain a fourth vector of numbers.

5. The computing system of claim 1, wherein the computing system is further configured to train one or more classifiers such that each of the one or more classifiers can be used in classification of different abnormalities or diseases and to use the set of classifiers to ascertain presence, absence or severity of plurality of diseases, abnormalities, or lesions.

6. The computing system of claim 1, wherein the computing system is further configured to utilize one or more thresholds for different abnormalities applied on a computed decision statistic at each pixel, to infer an absence or presence of one of a plurality of diseases, lesions, or abnormalities.

7. The computing system of claim 1, wherein the computing system is further configured to run in a telemedicine architecture and to receive a request for an analysis from a remote computing system that is in a different geographic location than the computing system.

8. The computing system of claim 1, wherein the retinal image is one or more of:

a retinal fundus photograph, a widefield retinal fundus photograph, an ultra-widefield retinal fundus photograph, or an image obtained using fluorescein angiography, the plurality of diseases includes one or more of;

diabetic retinopathy, cytomegalovirus retinitis, retinopathy of prematurity, clinical myopia, hypertensive retinopathy, stroke, cardiovascular disease, glaucoma, macular degeneration, Alzheimer'"'"'s, or macular edema.

9. The computing system of claim 1, wherein the computing system is further configured to designate a neighboring group of pixels as being part of a lesion.

10. A computer implemented method comprising:

accessing retinal images related to a patient, each of the retinal images comprising a plurality of pixels;

for each of the retinal images, designating a first set of the plurality of pixels as active pixels including interesting retinal image regions;

for each of the retinal images, computing a first vector of numbers comprising pixel-level descriptors for each of the active pixels;

for each of the retinal images, providing a first classification for each of the active pixels using supervised learning utilizing the corresponding first vector of numbers;

for each of the retinal images, computing a second vector of numbers including one or more of;

a first histogram of pixel-level classifier decision statistics computed during the first classification;

a third vector of numbers computed by obtaining a nearest codeword of pixel-level descriptors aggregated as a second histogram of codeword frequencies in each of the retinal images, wherein a set of codewords is computed from a representative set of images; and

a third histogram of active region-level decision statistic numbers computed by;

grouping neighboring pixels detected in the first set of the plurality of pixels into active regions, computing a number representing each of the active regions using one or more of;

mean, median, maximum, or minimum of pixel-level classifier decision statistics computed in the first classification for each pixel that is part of at least one active region, wherein the third histogram is computed over the active regions detected in each of the retinal images; and

providing a second classification for one or more of;

the plurality of pixels, an interesting region within one of the retinal images, one of the retinal images in its entirety, or a collection of the retinal images, using supervised learning.

11. The computer implemented method of claim 10, further comprising resizing at least one of the retinal images to a standard size image such that dimensions of the standard size image are determined using field of view information provided or inferred from a structure in the at least one retinal image or using a diameter of a fundus mask.

12. The computer implemented method of claim 10, further comprising:

processing a plurality of images obtained from a single patient imaging session or encounter.

13. The computer implemented method of claim 10, further comprising processing a plurality of images from a single patient session or encounter such that the second vector of numbers for at least a subset of the plurality of images are combined with or without further processing to obtain a fourth vector of numbers.

14. The computer implemented method of claim 10, further comprising:

training one or more classifiers such that each of the one or more classifiers can be used in classification of different abnormalities or diseases and to use the set of classifiers to ascertain presence, absence or severity of plurality of diseases, abnormalities, or lesions.

15. The computer implemented method of claim 10, further comprising:

utilizing one or more thresholds for different abnormalities applied on a computed decision statistic at each pixel, to infer an absence or presence of one of a plurality of diseases, lesions, or abnormalities.

16. The computer implemented method of claim 10, wherein the computer implemented method is performed in a telemedicine architecture and the computer implemented method further comprises:

receiving a request for an analysis from a remote computing system that is in a different geographic location than where the computer implemented method is being performed.

17. The computer implemented method of claim 10, wherein the retinal image is one or more of:

a retinal fundus photograph, a widefield retinal fundus photograph, an ultra-widefield retinal fundus photograph, or an image obtained using fluorescein angiography, the plurality of diseases includes one or more of;

diabetic retinopathy, cytomegalovirus retinitis, retinopathy of prematurity, clinical myopia, hypertensive retinopathy, stroke, cardiovascular disease, glaucoma, macular degeneration, Alzheimer'"'"'s, or macular edema.

18. The computer implemented method of claim 10, further comprising:

designating a neighboring group of pixels as being part of a lesion.

19. Non-transitory computer storage that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

accessing retinal images related to a patient, each of the retinal images comprising a plurality of pixels;

for each of the retinal images, designating a first set of the plurality of pixels as active pixels including interesting retinal image regions;

for each of the retinal images, computing a first vector of numbers comprising pixel-level descriptors for each of the active pixels;

for each of the retinal images, computing a second vector of numbers including one or more of;

a first histogram of pixel-level classifier decision statistics computed during the first classification;

a third vector of numbers computed by obtaining a nearest codeword of pixel-level descriptors aggregated as a second histogram of codeword frequencies in each of the retinal images, wherein a set of codewords is computed from a representative set of images; and

a third histogram of active region-level decision statistic numbers computed by;

grouping neighboring pixels detected in the first set of the plurality of pixels into active regions, computing a number representing each of the active regions using one or more of;

mean, median, maximum, or minimum of pixel-level classifier decision statistics computed in the first classification for each pixel that is part of at least one active region, wherein the third histogram is computed over the active regions detected in each of the retinal images; and

providing a second classification for one or more of;

the plurality of pixels, an interesting region within one of the retinal images, one of the retinal images in its entirety, or a collection of the retinal images using supervised learning.

20. The non-transitory computer storage of claim 19, further comprising resizing at least one of the retinal images to a standard size image such that dimensions of the standard size image are determined using field of view information provided or inferred from a structure in the at least one retinal image or using a diameter of a fundus mask.

21. The non-transitory computer storage of claim 19, further comprising:

processing a plurality of images obtained from a single patient imaging session or encounter.

22. The non-transitory computer storage of claim 19, further comprising:

processing a plurality of images from a single patient session or encounter such that the second vector of numbers for at least a subset of the plurality of images are combined with or without further processing to obtain a fourth vector of numbers.

23. The non-transitory computer storage of claim 19, further comprising:

training one or more classifiers such that each of the one or more classifiers can be used in classification of different abnormalities or diseases and to use the set of classifiers to ascertain presence, absence or severity of plurality of diseases, abnormalities, or lesions.

24. The non-transitory computer storage of claim 19, further comprising:

utilizing one or more thresholds for different abnormalities applied on a computed decision statistic at each pixel, to infer an absence or presence of one of a plurality of diseases, lesions, or abnormalities.

25. The non-transitory computer storage of claim 19, wherein the one or more computing devices perform operations in a telemedicine architecture, and the one or more computing devices further perform operations comprising:

receiving a request for an analysis from a remote computing system that is in a different geographic location than the one or more computing devices.

26. The non-transitory computer storage of claim 19, wherein the retinal image is one or more of:

a retinal fundus photograph, a widefield retinal fundus photograph, an ultra-widefield retinal fundus photograph, or an image obtained using fluorescein angiography, the plurality of diseases includes one or more of;

diabetic retinopathy, cytomegalovirus retinitis, retinopathy of prematurity, clinical myopia, hypertensive retinopathy, stroke, cardiovascular disease, glaucoma, macular degeneration, Alzheimer'"'"'s, or macular edema.

27. The non-transitory computer storage of claim 19, further comprising:

designating a neighboring group of pixels as being part of a lesion.