System and method for lesion analysis and recommendation of screening checkpoints for reduced risk of skin cancer

• US 10,282,843 B2
• Filed: 10/27/2016
• Issued: 05/07/2019
• Est. Priority Date: 10/27/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation, MoleMap NZ, Ltd.
• Original Assignee: International Business Machines Corporation
• Inventors: Abedini, Mani, Bowling, Adrian, Chakravorty, Rajib, Demyanov, Sergey, Garnavi, Rahil
• Primary Examiner(s): Conner, Sean M
• Application Number:
  US15/336,347
• Publication Number:
  US 20180122076A1
• Time in Patent Office: 922 Days
• Field of Search: None
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/444 Evaluating skin marks, e.g. mole, nevi, tumour, scar
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/00 ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A method for image analysis comprises receiving one or more current images of a lesion from a body of a person, wherein the one or more current images are electronically captured by and transmitted from a capture device, and analyzing the one or more current images, wherein the analyzing comprises performing image processing to compare the one or more current images captured at a first time to one or more previous images of the lesion captured at a second time prior to the first time, and determining at least one difference between the one or more current images and the one or more previous images based on the comparing. The method further comprises determining a probability that the lesion will become diseased based on the analysis, and recommending a time for a future image capture of the lesion and/or a consultation with a practitioner based on the determined probability.

Claims

1. A method for image analysis, comprising:

receiving one or more current images of a lesion from a body of a person, wherein the one or more current images are electronically captured by and transmitted from a capture device;

analyzing the one or more current images of the lesion, wherein the analyzing comprises;

performing image processing to compare the one or more current images of the lesion captured at a first time to one or more previous images of the lesion captured at a second time prior to the first time; and

determining at least one difference between the one or more current images and the one or more previous images of the lesion based on the comparing;

determining a probability that the lesion will become diseased based on the analysis; and

recommending at least one of a time for a future image capture of the lesion and a consultation with a practitioner based on the determined probability;

wherein the comparing is performed using a plurality of machine learning based engines;

wherein a first machine learning based engine of the plurality of machine learning based engines processes an arithmetical difference between the one or more current images and the one or more previous images of the lesion;

wherein a second machine learning based engine of the plurality of machine learning based engines processes a qualitative difference between the one or more current images and the one or more previous images of the lesion;

wherein a third machine learning based engine of the plurality of machine learning based engines processes the outputs of first and second machine learning based engines to yield a combined result; and

wherein the method is performed by at least one computer system comprising at least one memory and at least one processor coupled to the memory.

2. The method according to claim 1, further comprising training at least one of the plurality of machine learning based engines using lesion images each having a known label.

3. The method according to claim 1, wherein at least one of the plurality of machine learning based engines comprises a convolutional neural network (CNN).

4. The method according to claim 3, further comprising considering an output of the CNN for a pair of consecutive lesion images as an observed variable for a statistical model.

5. The method according to claim 1, further comprising analyzing metadata of the person, wherein the analyzing of the metadata comprises:

comparing current metadata corresponding to a later time to previous metadata corresponding to an earlier time than the later time; and

determining at least one difference between the current metadata and the previous metadata based on the comparing;

wherein the determining of the probability that the lesion will become diseased is further based on the analysis of the metadata.

6. The method according to claim 1, further comprising generating an approximation of the appearance of the lesion at any point in time between the first time and the second time.

7. The method according to claim 1, wherein performing the image processing comprises preprocessing the one or more current images and the one or more previous images of the lesion to have at least one of a same size, a same resolution and a same brightness.

8. The method according to claim 1, wherein performing the image processing comprises preprocessing the one or more current images and the one or more previous images of the lesion to be linearly aligned to each other.

9. The method according to claim 1, wherein determining the probability comprises:

building a time series of scores for probabilities that the lesion will become diseased; and

analyzing the time series of scores using a statistical model.

10. The method according to claim 1, wherein:

each of the plurality of machine learning based engines is trained with a different set of lesion images from each other, each lesion image having a known label.

11. The method according to claim 1, wherein recommending the time for the future image capture of the lesion comprises:

identifying a time when the probability that the lesion will become diseased exceeds a predefined threshold; and

assigning the identified time as the time for the future image capture of the lesion.

12. The method according to claim 1, wherein recommending the time for the future image capture of the lesion comprises:

defining upper and lower limits for the future image capture of the lesion according to different confidence thresholds for determining the probability that the lesion will become diseased.

13. The method according to claim 1, further comprising:

automatically transmitting the one or more current images to a device of a specialist if the probability that the lesion will become diseased exceeds a threshold.

14. A system for image analysis, comprising:

a memory and at least one processor coupled to the memory, wherein the at least one processor is configured to;

receive one or more current images of a lesion from a body of a person, wherein the one or more current images are electronically captured by and transmitted from a capture device;

analyze the one or more current images of the lesion, wherein in analyzing, the at least one processor is further configured to;

perform image processing to compare the one or more current images of the lesion captured at a first time to one or more previous images of the lesion captured at a second time prior to the first time; and

determine at least one difference between the one or more current images and the one or more previous images of the lesion based on the comparing;

determine a probability that the lesion will become diseased based on the analysis; and

recommend at least one of a time for a future image capture of the lesion and a consultation with a practitioner based on the determined probability;

wherein the comparing is performed using a plurality of machine learning based engines;

wherein a first machine learning based engine of the plurality of machine learning based engines processes an arithmetical difference between the one or more current images and the one or more previous images of the lesion;

wherein a second machine learning based engine of the plurality of machine learning based engines processes a qualitative difference between the one or more current images and the one or more previous images of the lesion; and

wherein a third machine learning based engine of the plurality of machine learning based engines processes the outputs of first and second machine learning based engines to yield a combined result.

15. The system according to claim 14, wherein in determining the probability, the at least one processor is further configured to:

build a time series of scores for probabilities that the lesion will become diseased; and

analyze the time series of scores using a statistical model.

16. The system according to claim 14, wherein:

each of the plurality of machine learning based engines is trained with a different set of lesion images from each other, each lesion image having a known label.

17. A computer program product for image analysis, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

receiving one or more current images of a lesion from a body of a person, wherein the one or more current images are electronically captured by and transmitted from a capture device;

analyzing the one or more current images of the lesion, wherein the analyzing comprises;

performing image processing to compare the one or more current images of the lesion captured at a first time to one or more previous images of the lesion captured at a second time prior to the first time; and

determining at least one difference between the one or more current images and the one or more previous images of the lesion based on the comparing;

determining a probability that the lesion will become diseased based on the analysis; and

recommending at least one of a time for a future image capture of the lesion and a consultation with a practitioner based on the determined probability;

wherein the comparing is performed using a plurality of machine learning based engines;

wherein a first machine learning based engine of the plurality of machine learning based engines processes an arithmetical difference between the one or more current images and the one or more previous images of the lesion;

wherein a second machine learning based engine of the plurality of machine learning based engines processes a qualitative difference between the one or more current images and the one or more previous images of the lesion; and

wherein a third machine learning based engine of the plurality of machine learning based engines processes the outputs of first and second machine learning based engines to yield a combined result.