Systems and methods for feature detection in retinal images

• US 10,115,194 B2
• Filed: 04/06/2016
• Issued: 10/30/2018
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Digital Diagnostics, Inc.
• Original Assignee: Idx, Llc (Elm Street Technology LLC)
• Inventors: Niemeijer, Meindert, Amelon, Ryan, Claride, Warrent, Abramoff, Michael D.
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US15/092,421
• Publication Number:
  US 20160292856A1
• Time in Patent Office: 937 Days
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Provided is a method for training a neural network to detect features in a retinal image. The method may include the steps of: combining and randomizing feature images into a Training data set; combining and randomizing the feature images into a testing dataset; training a plurality of neural networks having different architectures using a subset of the training dataset while testing on a subset of the testing dataset; identifying the best neural network based on each of the plurality of neural networks performance on the testing data set; inputting images to the best neural network and identifying a limited number of false positives and false negative and adding the false positives and false negatives to the training dataset and testing dataset; and repeating the foregoing steps until an objective performance threshold is reached.

Claims

1. A method for training a neural network to detect features in a retinal image comprising:

a) extracting one or more Features Images from a Train_0 set, a Test_0 set, a Train_1 set and a Test_1 set;

b) combining and randomizing the Feature Images from Train_0 and Train_1 into a training data set;

c) combining and randomizing the Feature Images from Test_0 and Test_1 into a testing dataset;

d) training a plurality of neural networks having different architectures using a subset of the training dataset while testing on a subset of the testing dataset;

e) identifying the best neural network based on each of the plurality of neural networks performance on the testing dataset;

f) inputting images from Test_0, Train_1, Train_0 and Test_1 to the best neural network and identifying a limited number of false positives and false negative and adding the false positives and false negatives to the training dataset and testing dataset; and

g) repeating steps d)-g) until an objective performance threshold is reached.

2. The method of claim 1, wherein the step (a) further comprises sampling a region of interest (ROI).

3. The method of claim 2, wherein sampling is performed by random sampling.

4. The method of claim 2, wherein sampling is performed by grid sampling.

5. The method of claim 2, further comprising resampling the images of Train_0, Test_0, Train_1 and Test_1 after the completion of step (e) with a sampling method different from the sampling method performed in the initial sampling.

6. The method of claim 1, wherein the images undergo preprocessing prior to the step of extracting one or more Feature Images.

7. The method of claim 1, wherein the objective performance threshold is softmax_loss or accuracy.

8. The method of claim 1, further comprising:

a) creating a spatial distribution of features in an image; and

b) inputting the spatial distribution of features into a neural network trained according to steps (a)-(f), wherein the output of the neural network is disease diagnosis.

9. The method of claim 8, wherein the spatial distribution of features is a heat map.

10. The method of claim 8, the spatial distribution of features is a point-wise output from feature detection.

11. The method of claim 8, wherein the step of creating a spatial distributions of features in an image further comprises creating scalar features.

12. The method of claim 11, wherein the scalar features are created as histograms.

13. The method of claim 11, further comprising training a machine learning algorithm, wherein the input is scalar features and the output is disease diagnosis.

14. A system for detecting lesions in a retinal image comprising:

a) a feature extractor, configured to receive retinal image input from a user, and further configured to extract image features from the retinal image;

b) a multilevel neural network, trained according to the steps of;

i) extracting one or more Features Images from a Train_0 set, a Test_0 set, a Train_1 set and a Test_1 set;

ii) combining and randomizing the Feature Images from Train_0 and Train_1 into a training data set;

iii) combining and randomizing the Feature Images from Test_0 and Test_1 into a testing dataset;

iv) training a plurality of neural networks having different architectures using a subset of the training dataset while testing on a subset of the testing dataset;

v.) identifying the best neural network based on each of the plurality of neural networks performance on the testing dataset;

vi) inputting images from Test_0, Train_1, Train_0 and Test_1 to the best neural network and identifying a limited number of false positives and false negative and adding the false positives and false negatives to the training dataset and testing dataset; and

vii) repeating steps iv)-vii) until an objective performance threshold is reached;

and configured to receive image features from one or more feature extractor, and further configured to output scalar image features; and

c) a machine learning program, configured to receive scalar image features from the multilevel neural network b) and further configured to output disease diagnosis.

15. The system of claim 14, wherein the image features are spatial distributions of image features.

16. The system of claim 14, where the feature extractor a) is itself a multilevel neural network.

17. The system of claim 14, wherein the feature extractor a) or the multilevel neural network b) are Convolutional Neural Networks.

18. The system of claim 15, wherein the spatial distributions are heat maps or point-wise outputs.