Imaging system for object recognition and assessment

• US 9,959,628 B2
• Filed: 11/20/2015
• Issued: 05/01/2018
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: Spectrapixel LLC
• Original Assignee: Christopher M. Mutti
• Inventors: Mutti, Christopher M., Lau, Daniel L., Coates, John P.
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US14/947,334
• Publication Number:
  US 20160150213A1
• Time in Patent Office: 893 Days
• Field of Search: 348143, 348142, 348144
• CPC Class Codes: G01N 33/02 Food
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/58 relating to hyperspectral data
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/142 : using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: H04N 13/254 : in combination with electromagnetic radiation sources for illuminating objects

Abstract

A method and system for using one or more sensors configured to capture two-dimensional and/or three dimensional image data of one or more objects. In particular, the method and system combine one or more digital sensors with visible and near infrared illumination to capture visible and non-visible range spectral image data for one or more objects. The captured spectral image data can be used to separate and identify the one or more objects. Additionally, the three-dimensional image data can be used to determine a volume for each of the one or more objects. The identification and volumetric data for one or more objects can be used individually or in combination to obtain characteristics about the objects. The method and system provide the user with the ability to capture images of one or more objects and obtain related characteristics or information about each of the one or more objects.

Claims

1. A system for automated identification of one or more food items and determination of nutritional content for the one or more food items of plated food utilizing different wavelengths of emitted light, system comprising:

a three-dimensional scanner, a digital RGB camera, a digital NIR camera and an array of Near Infrared (NIR) light emitting diodes (LEDs), the array containing LEDS emitting at least two different wavelengths of emitted light from each other;

an image processor configured to control;

the three-dimensional scanner configured to capture three-dimensional image data of the one or more food items;

the digital RGB camera configured to capture at least two-dimensional visible image data of the one or more food items;

the array of NIR LEDs sequentially producing at least two different wavelengths of emitted light and

the digital NIR camera configured to sequentially capture a series of NIR image data sets of the one or more food items contemporaneous with the array of NIR LEDs sequentially producing each of the at least two different wavelengths of emitted light; and

the image processor transforms the at least two-dimensional image data of the one or more food items, the three-dimensional visible image data of the one or more food items, and the series of NIR data sets of the one or more food items into a characteristic correlated definition in terms of composition of the one or more food items and volume of the one or more food items to separate and identify individual food items from the one or more food items based on pattern matching against known unique sizes, textures, shapes and colors stored in a database accessible by the system.

2. The system of claim 1, wherein:

the digital RGB camera and the digital NIR camera are configured to transform captured visible light in a spectrum range and Near Infrared (NIR) light in a spectrum range into captured image data; and

the digital RGB camera and the digital NIR camera capture at least two different and non-overlapping subsets of light spectrum ranges within the NIR spectrum range and do not capture light in gaps of light spectrum ranges between the non-overlapping subsets of light spectrum ranges.

3. The system of claim 2, wherein the digital NIR camera further comprises a visible light blocking filter configured to capture two-dimensional and/or three-dimensional NIR image data of the one or more food items.

4. The system of claim 3, further comprising at least one image processor configured to transform image data from the three-dimensional scanner sensor, the digital RGB camera, and the digital NIR camera into a representative set of image data originating from a single sensor.

5. The system of claim 2, further comprising at least one image processor configured to:

capture an N set of image data, the N set of image data comprising;

image data for a visible light image capture; and

image data from N−

1 NIR image captures;

capture ambient image data with all of the at least two light source devices set to off; and

subtract the ambient image data from the N set of image data.

6. The system of claim 2, further comprising a lens, a sensor, and the three-dimensional scanner that captures the three-dimensional image data using at least one of stereovision, time of flight, and structured light.

7. The system of claim 2, further comprising at least a micro electrical mechanical spectrometry chipset, an infrared (IR) source, a plurality of condenser lenses, a slit, an IR bandpass filter, a diffraction grating, a digital micro-mirror device, a detector, and microprocessor.

8. The system of claim 7, further comprising wired and wireless connection devices configured to communicate data over a network.

9. The system of claim 2, wherein:

the two-dimensional and/or three-dimensional visible image data is captured in the spectrum range of 400 nm to 700 nm; and

the NIR image data is captured in the spectrum range of 700 nm to 1050 nm.

10. A method for automated detection and identification of one or more food items on a plate of food and the determination of nutritional values for the one or more food items on the plate of food utilizing different wavelengths of emitted light, the method comprising:

capturing, by a three-dimensional image scanner, a three-dimensional image of the one or more food items;

determining, by a processor, a range between the three-dimensional image scanner and one or more food items based on data from the three-dimensional image;

determining, by an image processor, a volume of each of the one or more food items based on the range and the data from the three-dimensional image of the one or more food items;

capturing, by an RGB camera, a visible light image of the one or more food items;

capturing, by an near infrared (NIR) camera, a sequence of NIR images of the one or more food items contemporaneous with an array of Near Infrared (NIR) light emitting diodes (LEDs) simultaneously triggering emitted light of a unique LED wavelength, wherein at least two LEDs of the array of NIR LEDs emit light of different wavelengths from each other for each captured NIR image;

analyzing, utilizing artificial intelligence, the captured visible light image and the sequence of NIR images to identify each unique food item of the one or more food items by pattern matching against known unique sizes, textures, shapes and colors stored in a database accessible by the processor; and

determining, by a processor, additional characteristics about the one or more food items based on the identified unique food items of the one or more food items and the volume of each of the one or more food items, the additional characteristics comprising determining nutritional values for the plate of food.