Sensor-synchronized spectrally-structured-light imaging

• US 10,113,910 B2
• Filed: 08/26/2015
• Issued: 10/30/2018
• Est. Priority Date: 08/26/2014
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Brunk, Hugh L., Rhoads, Geoffrey B., Archer, Cynthia, Conner, Arlie
• Primary Examiner(s): Vo, Tung
• Assistant Examiner(s): Cattungal, Rowina
• Application Number:
  US14/836,878
• Publication Number:
  US 20160187199A1
• Time in Patent Office: 1,161 Days
• CPC Class Codes: G01J 2003/2826 Multispectral imaging, e.g. filter imaging
• CPC Class Codes: G01J 3/0208 using focussing or collimating elements, e.g. lenses or mirrors; performing aberration correction
• CPC Class Codes: G01J 3/0256 Compact construction
• CPC Class Codes: G01J 3/10 Arrangements of light sources specially adapted for spectrometry or colorimetry
• CPC Class Codes: G01J 3/2803 using photoelectric array detector
• CPC Class Codes: G01J 3/2823 Imaging spectrometer
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/58 relating to hyperspectral data
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

An image capture device, such as a smartphone or point of sale scanner, is adapted for use as an imaging spectrometer, by synchronized pulsing of different LED light sources as different image frames are captured by the image sensor. A particular implementation employs the CIE color matching functions, and/or their orthogonally transformed functions, to enable direct chromaticity capture. These and various other configurations of spectral capture devices are employed to capture spectral images comprised of spectral vectors having multi-dimensions per pixel. These spectral images are processed for use in object identification, classification, and a variety of other applications. Particular applications include produce (e.g., fruit or vegetable) identification. A great variety of other features and arrangements are also detailed.

Claims

1. An apparatus comprising:

an imaging apparatus for obtaining pixels sampled from a produce item in a scene, the pixels each comprising an N-dimensional spectral vector, where N is an integer greater or equal to four, and corresponding to a number of spectral values obtained per pixel, the spectral values obtained from capture of the pixels by an image sensor in response to pulsed illumination of the object by 4 or more different LED light sources;

a digital signal processing device in communication with the imaging apparatus for computing texture feature values of the pixels at plural spatial scales, sampled from different locations of the produce item;

a classifier for classifying the produce item based on the N-dimensional spectral vector and texture feature values of the pixels at plural spatial scales to identify a variety of the produce item in the scene, wherein the classifier is trained on N-dimensional spectral vector and texture feature values of varieties of each of different produce items to discriminate between the varieties of each of the different produce items,

the classifier configured to identify the variety of the produce item with a plural stage classifier, the plural stage classifier having a first stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector of the produce item for each of plural categories of produce, and a second stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector and texture feature values of the produce item for variety of produce within each of the plural categories; and

a display device configured to display identifying information of the variety of the produce item.

2. The apparatus of claim 1 including a programmed processor configured with instructions, which when executed by the programmed processor, compute spectral distribution values from N-dimensional spectral vectors of pixels, and input the spectral distribution values to the classifier, which is trained to identify the variety of the produce item based on the spectral distribution values.

3. The apparatus of claim 1 including a programmed processor configured with instructions, which when executed by the programmed processor, compute texture feature distribution values from an image captured of the scene, and input the texture feature distribution values to the classifier, which is trained to identify the variety of the produce item based on the texture feature distribution values.

4. The apparatus of claim 3 including a programmed processor configured with instructions, which when executed by the programmed processor, compute spectral distribution values from N-D spectral vectors of pixels, and input the spectral distribution values to the classifier, which is trained to identify the variety of the produce item based on the spectral distribution values.

5. The apparatus of claim 1 wherein the imaging apparatus comprises:

LED arrays;

a camera for capturing images of the scene in a field of view;

an LED controller coupled to the LED arrays for sending LED drive signals to LEDs of the LED arrays;

a camera and light source controller coupled to the camera to issue control signals for image capture synchronized with strobe control signals to the LED controller, which, responsive to the strobe control signals, issues the LED drive signals to the LEDs; and

diffusers positioned relative to the LED arrays to diffuse light from the LED arrays prior to the light entering the field of view.

6. The apparatus of claim 5 wherein the LED arrays are arranged in rows and columns, wherein a column comprises at least 4 different LED light sources, each different LED light source corresponding to a different wavelength.

7. The apparatus of claim 1 wherein the imaging apparatus comprises:

LEDs arranged in a cluster and positioned to project light from the LEDs into an aperture of an optical system comprising a reflector shaped to direct the light into a beam and through a lens;

a camera for capturing images of the scene in a field of view;

an LED controller coupled to the LEDs for sending LED drive signals to the LEDs;

a camera and light source controller coupled to the camera to issue control signals for image capture synchronized with strobe control signals to the LED controller, which, responsive to the strobe control signals, issues the LED drive signals to the LEDs.

8. A method of recognizing a produce item comprising:

obtaining pixels sampled from a produce item in a scene, the pixels each comprising an N-dimensional spectral vector, the pixels being sampled under different illumination to obtain the N-dimensional spectral vector, comprising N different spectral samples per pixel, where N is an integer greater or equal to four, and corresponding to a number of spectral values obtained per pixel, the spectral values obtained from capture of the pixels by an image sensor in response to pulsed illumination of the object by 4 or more different LED light sources;

computing texture feature values of the pixels at plural spatial scales, sampled from different locations of the produce item in the scene;

with a trained classifier, classifying the produce item based on the N-dimensional spectral vector and the texture feature values of the pixels at plural spatial scales, wherein the classifier is trained on N-dimensional spectral vector and texture feature values of varieties of each of different produce items to discriminate between the varieties of each of the different produce items, the trained classifier configured to identify the variety of the produce item with a plural stage classifier, the plural stage classifier having a first stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector of the produce item for each of plural categories of produce, and a second stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector and texture feature values of the produce item for variety of produce within each of the plural categories; and

displaying identifying information of the variety of the produce item.

9. The method of claim 8 including:

executing a programmed processor to compute spectral distribution values from N-dimensional spectral vectors of pixels, and with the trained classifier, classifying the produce item based on the spectral distribution values and the texture feature values of the pixels at plural spatial scales.

10. The method of claim 8 including:

executing a programmed processor to compute texture feature distribution values from an image captured of the produce item in the scene, and with the trained classifier, classifying the produce item based on the texture feature distribution values.

11. The method claim 10 including:

executing a programmed processor to compute spectral distribution values from N-dimensional spectral vectors of pixels, and classifying the produce item based on the spectral distribution values.

12. A non-transitory processor-readable medium comprising instructions that, when executed by a processor, cause the processor to:

determine aggregate spectral values for a spectral image of a produce item, the spectral image comprising a two-dimensional array of pixels, with each pixel having a N-dimensional vector of spectral values captured of an object, where N is at least 4 and corresponds to different spectral bands obtained from capture of the pixels by an image sensor in response to pulsed illumination of the object by 4 or more different LED light sources, the aggregate spectral values being derived from a distribution of spectral values within each spectral band;

determine aggregate texture feature values for the spectral image; and

with a trained classifier, identify the variety of the produce item by submitting the aggregate spectral values and aggregate texture feature values to the trained classifier, wherein the trained classifier is trained on N-dimensional spectral vector and texture feature values of varieties of each of different produce items to discriminate between the varieties of each of the different produce items;

the instructions further configured to cause the processor to identify the variety of the produce item with a plural stage classifier, the plural stage classifier having a first stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector of the produce item for each of plural categories of produce, and a second stage classifier comprising one-versus-one classifiers to discriminate the N-dimensional spectral vector and texture feature values of the produce item for variety of produce within each of the plural categories.

13. The non-transitory processor-readable medium of claim 12 comprising instructions that, when executed by a processor, cause the processor to:

determine the aggregate texture feature values for the spectral image by determining texture feature values from differences between pixel values of the spectral image at pixel locations at plural spatial distances, and deriving the aggregate texture feature values from a distribution of the texture feature values.

14. The non-transitory processor-readable medium of claim 12 comprising instructions that, when executed by a processor, cause the processor to:

derive the aggregate spectral values from the distribution of spectral values by determining spectral values at pre-determined points in the distribution of the spectral values.

15. The non-transitory processor-readable medium of claim 12 comprising instructions that, when executed by a processor, cause the processor to:

identify the produce item with a tree structured classifier in which the first stage comprises a top level of the tree structured classifier that divides classes into categories of produce, and the second stage comprises a set of one vs. one classifiers for each of the categories.

16. The non-transitory processor-readable medium of claim 13 comprising instructions that, when executed by a processor, cause the processor to:

derive the aggregate spectral values from the distribution of spectral values by determining spectral values at pre-determined points in the distribution of the spectral values.