APPARATUSES AND METHODS FOR MOBILE IMAGING AND ANALYSIS

• US 20150065803A1
• Filed: 09/05/2014
• Published: 03/05/2015
• Est. Priority Date: 09/05/2013
• Status: Active Grant

• Current Assignee: Johnson & Johnson Consumer Inc. (Johnson & Johnson)
• Original Assignee: Janakiramanan Ramachandran
• Inventors: DOUGLAS, Erik Scott, GOLDEN, Daniel Irving, FOX, Christopher Todd, REEVE, Thomas Burnell, RAMACHANDRAN, Janakiramanan
• Granted Patent: US 9,445,713 B2
• Time in Patent Office: Days
• US Class Current: 600/200
• CPC Class Codes: A61B 1/000096 using artificial intelligence
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/227 for ears, i.e. otoscopes
• CPC Class Codes: A61B 3/12 for looking at the eye fundus, e.g. ophthalmoscopes A61B3/13 takes precedence
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

Apparatuses and methods for mobile imaging and image analysis. In particular, described herein are methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media). These apparatuses may guide or direct a subject in taking an image of a tympanic membrane, including automatically detecting which direction to adjust the position of the apparatus to capture an image of the tympanic membrane and automatically indicating when the tympanic membrane has been imaged.

Claims

1. An automated method of guiding a subject to take an image of a tympanic membrane using an otoscope apparatus, the method comprising:

displaying, on a display, an image from the otoscope apparatus while the otoscope apparatus is inserted into a patient'"'"'s ear;

using the otoscope apparatus to automatically detect from the image either or both of;

one or more deeper regions in the image, and at least a portion of a tympanic membrane; and

indicating a direction to orient the otoscope apparatus in the patient'"'"'s ear based on either or both of the detected one or more deeper regions and the detected at least the portion of the tympanic membrane.

2. The method of claim 1, further comprising using the otoscope to detect an occlusion of an ear canal in the image, and indicating when an occlusion is detected.

3. The method of claim 1, wherein displaying on the display comprises displaying on a screen of a mobile telecommunications device.

4. The method of claim 1, further comprising using the otoscope device to detect if an ear canal in the image is not straight, and indicating that the ear canal should be straightened.

5. The method of claim 1, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a field of view for the image, correcting for uneven illumination in the field of view, identifying from the field of view one or more regions of brightness below a threshold in the image, extracting features for each identified region, and determining if an identified region is deeper in the ear canal based on the extracted features.

6. The method of claim 1, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a field of view for the image, converting the field of view of the image to greyscale, filtering the field of view of the image to remove small objects, and dividing the image by an average illumination value.

7. The method of claim 1, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a distribution of pixel values from the image from the otoscope and identifying regions having the pixel values below a threshold percentile of the distribution.

8. The method of claim 1, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises using a trained model to determine if the one or more regions are deeper regions in an ear canal.

9. The method of claim 1, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises extracting features from one or more regions of the image that are not as bright as other regions and using a trained model to determine if the one or more regions are deeper regions in an ear canal, wherein the extracted features include one or more of:

region area, region eccentricity, region solidity, mean intensity of the region, and mean intensity of the region in an illumination-corrected image.

10. The method of claim 1, wherein indicating a direction to orient the otoscope comprises displaying a direction on the display device.

11. The method of claim 1, further comprising using the otoscope apparatus to automatically indicate when the at least a portion of the tympanic membrane is detected.

12. The method of claim 1, wherein using the otoscope apparatus to automatically detect at least a portion of a tympanic membrane comprises:

extracting a set of feature values from a plurality of subregions from the image, and estimating, for each subregion, a probability that the subregion is part of a tympanic membrane based on the extracted sets of feature values.

13. An automated method of guiding a subject to take an image of a tympanic membrane using an otoscope apparatus, the method comprising:

displaying, on a display, an image from the otoscope apparatus while the otoscope apparatus is inserted into a patient'"'"'s ear;

using the otoscope apparatus to automatically detect one or more deeper regions in the image;

indicating a direction to orient the otoscope apparatus on the display based on the detected one or more deeper regions; and

using the otoscope to indicate when the image includes at least a portion of a tympanic membrane.

14. The method of claim 13, wherein using the otoscope to indicate when the image includes at least a portion of a tympanic membrane comprises:

extracting a set of feature values from a plurality of subregions from the image, estimating, for each subregion, a probability that the subregion is part of a tympanic membrane based on the extracted sets of feature values.

15. The method of claim 13, wherein displaying on the display comprises displaying on a screen of a mobile telecommunications device.

16. The method of claim 13, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a field of view for the image, correcting for uneven illumination in the field of view, identifying from the field of view one or more regions of brightness below a threshold in the image, extracting features for each identified region, and determining if an identified region is deeper in the ear canal based on the extracted features.

17. The method of claim 13, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a field of view for the image, converting the field of view of the image to greyscale, filtering the field of view of the image to remove small objects, and dividing the image by an average illumination value.

18. The method of claim 13, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises:

determining a distribution of pixel values from the image from the otoscope and identifying regions having the pixel values below a threshold percentile of the distribution.

19. The method of claim 13, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises using a trained model to determine if the one or more regions are deeper regions in an ear canal.

20. The method of claim 13, wherein using the otoscope apparatus to automatically detect one or more deeper regions in the image comprises extracting features from one or more regions of the image that are not as bright as other regions and using a trained model to determine if the one or more regions are deeper regions in an ear canal, wherein the extracted features include one or more of:

region area, region eccentricity, region solidity, mean intensity of the region, and mean intensity of the region in an illumination-corrected image.

21. An automated method of guiding a subject to take an image of a tympanic membrane using an otoscope apparatus, the method comprising:

displaying, on a display, an image from the otoscope apparatus;

using the otoscope apparatus to detect one or more deeper regions in the image by;

correcting for uneven illumination in the image, identifying one or more regions of brightness below a threshold in the image, extracting features for each identified region, and determining if an identified region is deeper in the ear canal based on the extracted features;

indicating a direction to orient the otoscope apparatus based on the detected one or more deeper regions; and

determining if the image includes a tympanic membrane by;

extracting a set of feature values from a plurality of subregions from the image, estimating, for each subregion, a probability that the subregion is part of a tympanic membrane based on the extracted sets of feature values.

22. The method of claim 21, further comprising indicating that the image includes at least a portion of the tympanic membrane.

23. An otoscope apparatus adapted to guide a subject to take an image of a tympanic membrane, the apparatus comprising:

an otoscope imaging portion comprising one or more lenses and a speculum configured to insert into a subject'"'"'s ear canal;

a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor, that when executed by the processor causes the processor to;

display on a display screen an image from the otoscope apparatus while the otoscope apparatus is inserted into a patient'"'"'s ear;

automatically detect from the image either or both of;

one or more deeper regions in the image, and at least a portion of a tympanic membrane; and

indicate a direction to orient the otoscope apparatus in the patient'"'"'s ear based on either or both of the detected one or more deeper regions and the detected at least the portion of the tympanic membrane.

24. The apparatus of claim 23, wherein the otoscope imaging portion further comprises a lightpipe configured to connect to an LED and transfer light from the LED to an illumination ring near a distal end of the speculum.

25. The apparatus of claim 23, wherein the otoscope imaging portion is adapted to couple with a smartphone or case for a smartphone so that the one or more lenses are in optical communication with a lens of a camera on the smartphone.

26. The apparatus of claim 23, wherein the instructions, when executed by the processor, causes the processor to display the image from the otoscope apparatus on a display screen of a smartphone to which the otoscope is attached.

27. The apparatus of claim 23, wherein the instructions, when executed by the processor, causes the processor to automatically detect from the image one or more deeper regions in the image by:

correcting for uneven illumination in the image, identifying one or more regions of brightness below a threshold in the image, extracting features for each identified region, and determining if an identified region is deeper in the ear canal based on the extracted features.

28. The apparatus of claim 23, wherein the instructions, when executed by the processor, causes the processor to automatically detect from the image at least a portion of a tympanic membrane by:

extracting a set of feature values from a plurality of subregions from the image, estimating, for each subregion, a probability that the subregion is part of a tympanic membrane based on the extracted sets of feature values.

29. The apparatus of claim 23, wherein the instructions, when executed by the processor, further causes the processor to automatically indicate when the at least a portion of the tympanic membrane is detected.

30. The apparatus of claim 23, wherein the instructions, when executed by the processor, further causes the processor to automatically detect at least a portion of a tympanic membrane comprises:

extracting a set of feature values from a plurality of subregions from the image, and estimating, for each subregion, a probability that the subregion is part of a tympanic membrane based on the extracted sets of feature values.