Dermoscopic data acquisition employing display illumination

• US 9,414,780 B2
• Filed: 05/20/2014
• Issued: 08/16/2016
• Est. Priority Date: 04/18/2013
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B.
• Primary Examiner(s): Peng, Bo J
• Application Number:
  US14/283,060
• Publication Number:
  US 20150005644A1
• Time in Patent Office: 819 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1032 Determining colour for diagnostic purposes measuring colour in general G01J3/46
• CPC Class Codes: A61B 5/1034 by means of colour cards
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: A61B 5/444 Evaluating skin marks, e.g. mole, nevi, tumour, scar
• CPC Class Codes: A61B 5/445 Evaluating skin irritation or skin trauma, e.g. rash, eczema, wound, bed sore
• CPC Class Codes: A61B 5/4806 Sleep evaluation A61B5/4821 takes precedence; devices for inducing sleep A61M21/02
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7246 using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/7425 Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image geometric transforms for image registration G06T3/14; enhancement using image fusion G06T5/50; algorithms for registration of images G06T7/30
• CPC Class Codes: A61B 5/743 Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
• CPC Class Codes: A61B 5/7485 Automatic selection of region of interest
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G10L 19/018 : Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass
• CPC Class Codes: Y02A 50/30 : Against vector-borne diseases, e.g. mosquito-borne, fly-borne, tick-borne or waterborne diseases whose impact is exacerbated by climate change
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

In one arrangement, a smartphone camera gathers skin imagery while controlled spectral illumination is emitted from the smartphone display. In a particular embodiment, differently colored bands move across a darkened screen to illuminate the skin from different angles as plural images are being captured. The collection of imagery from different viewing angles, with different spectral illumination, and different illumination angles, can provide imagery (including composite imagery) useful in manual or automated dermoscopic analysis. A great variety of other features and arrangements are also detailed.

Claims

1. A method of acquiring dermoscopic imagery, employing a portable device that includes both a camera and a display screen, the method comprising the acts:

presenting a first illumination from the device display screen to a subject at a first time;

capturing a first image of said subject with the device camera while presenting said first illumination;

presenting a second, different illumination from the device display screen to the subject at a second time; and

capturing a second image of said subject with the device camera while presenting said second illumination;

wherein neither said first nor said second illumination comprises a viewfinder rendering of the captured first and second images;

wherein the subject comprises a dermal mole on a person'"'"'s skin;

wherein the first and the second images are multiple frames of imagery depicting said mole and are captured under varying spectral illumination from the display screen; and

diagnosing a pathology of said mole by using the multiple frames of imagery.

2. The method of claim 1 in which:

said presenting actions comprise controlling the display to present a color band of a first uniform color that moves from one side of the device display screen to a second, opposite side of the device display screen; and

said capturing actions comprise capturing the first image with said color band at a first location, and capturing the second image with said color band at a second location.

3. The method of claim 1 in which said first illumination consists of light from display pixels of only a first color.

4. The method of claim 1 in which said first illumination consists of light from display pixels of only first and second colors.

5. The method of claim 1 in which the portable device includes both said camera and display screen on a common side thereof, wherein illumination from the device display screen is directed at a subject depicted in the first and second images.

6. The method of claim 1 in which the first illumination comprises one portion of the device display screen presenting a first area of uniform color, with a second portion of the device display screen being dark.

7. The method of claim 6 in which the first illumination comprises a third portion of the device display screen presenting a second color different than the first.

8. The method of claim 1 that further includes providing the first and second images to a healthcare professional for use in medical diagnosis concerning the mole.

9. The method of claim 1 wherein the portable device comprises a smartphone with front- and rear-facing cameras, wherein said acts of presenting the first and second illumination comprise presenting said illumination from the display screen of said smartphone, and said acts of capturing comprise capturing the first and second images with the front-facing camera of the smartphone.

10. The method of claim 1 wherein the portable device comprises a smartphone with front- and rear-facing cameras, and said display screen comprises an OLED display screen, wherein said acts of presenting the first and second illumination comprise presenting said illumination from the OLED display screen of said smartphone, and said acts of capturing comprise capturing the first and second images with the front-facing camera of the smartphone.

11. The method of claim 10 wherein a spectral profile of blue light emitted from said smartphone screen has a spectral profile that does not match a spectral profile of blue light passed by blue filters in a color filter array of said front-facing camera.

12. The method of claim 1 in which the portable device comprises a smartphone with the display screen which is a flexible screen and both front- and rear-facing cameras, wherein said acts of presenting the first and second illumination comprise presenting said illumination from the flexible display screen of said smartphone, and said acts of capturing comprise capturing the first and second images with said front-facing camera.

13. The method of claim 1 in which the first and second illumination are of different colors, and the method includes automatically switching between presentation of the first and second illumination at a rate sufficiently high that human perception of vision causes the illumination to seem uniform to human viewers, rather than with different colors on the screen.

14. The method of claim 1 in which the device comprises a smartphone and the method includes:

presenting the first illumination which is a red illumination from the smartphone display screen to said mole at the first time;

capturing the first image of said mole with the smartphone camera while presenting said red illumination;

presenting the second illumination which is a green illumination from the smartphone display screen to said mole at the second time;

capturing the second image of said mole with the smartphone camera while presenting said green illumination;

presenting a third illumination which is a blue illumination from the smartphone display screen to said mole at a third time; and

capturing a third image of said mole with the smartphone camera while presenting said third illumination;

wherein said third illumination does not comprise a viewfinder rendering of the captured third image.

15. The method of claim 14 in which said act of presenting green illumination at the second time comprises presenting green illumination alone, and not in combination with red or blue illumination.

16. The method of claim 14 in which said act of presenting green illumination comprises presenting green illumination in combination with red illumination but not blue illumination for one interval of time, and presenting green illumination in combination with blue illumination but not red illumination for another interval of time.

17. The method of claim 1 in which the device is a smartphone with the display screen having first and second edges, and the method includes:

presenting said first illumination from a band adjacent the first edge of the display screen, while other portions of the display screen are dark; and

presenting said second illumination from a band adjacent the second edge of the display screen, while other portions of the display screen are dark;

wherein the first and second illumination originates from different locations on the display screen, giving rise to different oblique illuminations of the mole, helping reveal topological features of the mole in the captured first and second images.

18. The method of claim 1 in which the device is a smartphone with the display screen having first and second edges, wherein the method includes:

(a) presenting the first illumination with a first color from a band adjacent the first edge of the display screen, while other portions of the display screen are dark, and capturing the first image of said subject while presenting said illumination of the first color;

(b) presenting the second illumination with a second color from a band adjacent the second edge of the display screen, while other portions of the display screen are dark, and capturing the second image of said subject while presenting said illumination of the second color;

(c) presenting a third illumination with a third color from a band at an intermediate region of the display screen between the first and second edges, while the portions of the display screen adjacent the first and second edges are dark, and capturing an image of said subject while presenting said illumination of the third color; and

moving the smartphone relative to the mole and repeating acts (a), (b) and (c);

wherein images of the mole are captured having diversity in illumination angle, illumination spectrum, and viewpoint.

19. The method of claim 1 in which a distance from the device display screen to the person'"'"'s skin is less than six inches when capturing the first image.

20. The method of claim 1 in which the first and second illumination are presented under control of a software program executing on a processor of the portable device, wherein said software program causes the device to present—

at the first time—

only one non-black color on said display screen, and causes the device to present—

at the second time—

only one non-black color on said display screen, wherein said non-black colors are different at said first and second times.

21. A dermal imaging method employing a portable device, the method comprising the acts:

moving the portable device to plural different viewpoint positions relative to a skin location;

at a first of said plural viewpoint positions, illuminating the skin location with light of a first spectrum from the portable device, and capturing a first image of the skin location illuminated with said light of the first spectrum using the portable device;

at a second of said plural viewpoint positions, illuminating the skin location with light of a second spectrum from the portable device, and capturing a second image of the skin location illuminated with said light of the second spectrum using the portable device;

wherein said skin location is imaged by the portable device from plural different viewpoint positions and with plural different illumination spectra;

wherein said first and second viewpoint positions are at distances of less than six inches from said skin location;

wherein neither said illumination with light of the first spectrum, nor said illumination with light of the second spectrum, comprises illumination with a viewfinder rendering of the captured first and second images;

wherein the skin location comprises a dermal mole;

wherein said first and said second images are multiple frames of imagery depicting said mole and are captured under varying spectral illumination; and

diagnosing a pathology of said mole by using the multiple frames of imagery.

22. The method of claim 21 that further includes processing said captured first and second images, and based on said processing, advising a user whether to seek a professional evaluation of the mole.

23. The method of claim 21 that further includes:

discerning movement of the device, and performing said acts in response to said discerned movement;

analyzing said first and second images of the skin captured from said different viewpoint positions and with said different illumination spectra;

discerning a static pose of the device after said discerned movement; and

presenting data resulting from said analyzing in response to said discerned static pose.

24. A method of imaging a skin location using a portable device, the method comprising the acts:

(a) illuminating the skin location with light of a first spectrum emitted from a first region of a display of the portable device, and capturing a first image of the skin location illuminated with said light of the first spectrum using the portable device; and

(b) illuminating the skin location with light of a second spectrum emitted from a second region of the display of the portable device that is different than said first region, and capturing a second image of the skin location illuminated with said light of the second spectrum using the portable device;

wherein;

the second region of the display is dark during act (a), and the first region of the display is dark during act (b);

the second spectrum of light is different than the first spectrum of light;

the illumination switches from the first region to the second region, and from the first spectrum to the second spectrum, in accordance with a software-defined sequence;

the capturing of imagery is synchronized with said software-defined sequence;

imagery of the skin location is captured by the portable device with plural different illumination angles and plural different illumination spectra;

neither said illumination with light of the first spectrum, nor said illumination with light of the second spectrum, comprises illumination with a viewfinder rendering of the captured first and second images;

wherein the skin location comprises a dermal mole;

wherein said first and the second images are multiple frames of imagery depicting said mole and are captured under varying spectral illumination; and

diagnosing a pathology of said mole by using the multiple frames of imagery.