Produce recognition method

• US 9,412,050 B2
• Filed: 10/12/2010
• Issued: 08/09/2016
• Est. Priority Date: 10/12/2010
• Status: Active Grant

• Current Assignee: NCR Voyix Corporation
• Original Assignee: NCR Corporation (NCR Voyix Corporation)
• Inventors: He, Chao, Ross, Gary
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US12/902,304
• Publication Number:
  US 20120087547A1
• Time in Patent Office: 2,128 Days
• Field of Search: 382/110, 382/141, 382155-161, 382224-228, 235440-473
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables

Abstract

A produce recognition method which uses hierarchical Bayesian learning and kernel combination, and which offers classification-oriented synergistic data integration from diverse sources. An example method includes providing a classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality; mapping the produce data to the respective input of the classifier by a computer; for each input, independently operating on the data relating to that input to create a feature set by the computer; comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set; combining all similarity description sets using a dedicated weighting function to produce a composite similarity description by the computer; and deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by the computer.

Claims

1. An automatic method of recognizing at a checkout system a produce item different from a number of different produce items to be sold, the method comprising:

pre-training a multinomial regression based model with variations in sizes, shapes, colors, types, thermal data, and aroma data for a training set of produce items, and pre-training the multinomial regression based model with environmental factors present during the pre-training including with the environmental factors at least one factor for background lighting and at least another factor for humidity;

providing a single classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality, wherein at least one modality relevant to thermal information captured for the produce item, the thermal information providing data relevant to an internal structure and composition of the produce item, and wherein at least another modality relevant to aroma information captured as chemicals given off by the produce items and the aroma information provided by an olfactory sensor, the aroma information relevant to chemical signatures for the produce items;

mapping the produce data to the respective input of the classifier by a computer executing produce recognition software;

for each input, independently operating on the data relating to that input to create a feature set by the computer;

comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set by the computer;

combining all similarity description sets using a dedicated weighting function to produce a composite similarity description by the computer; and

deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by the computer by processing the multinomial regression based model and producing an identity for the produce item.

2. The method according to claim 1, wherein the providing step comprises providing a Bayesian model as the classifier.

3. The method according to claim 1, wherein the providing step comprises providing a classifier with parameters derived from training involving capturing features of multiple other produce items of similar type as the one produce item.

4. The method according to claim 1, wherein the mapping step comprises utilizing kernel techniques to create the similarity description set.

5. The method according to claim 1, wherein the combining step comprises accessing an optimized weighting parameter for each of the feature similarity description sets to create the composite similarity description by the computer.

6. The method according to claim 1, wherein the deriving step comprises using regression techniques to derive the plurality of class values by the computer.

13. The method according to claim 1, wherein the checkout system is in a point of sale environment, and the method further comprises:

obtaining purchase frequency information associated with each of the number of different produce items; and

utilizing the purchase frequency information as a predictor of produce identity.

14. The method according to claim 1, wherein the checkout system is in a point of sale environment, and the produce recognition software works in combination with point of sale transaction software;

and the method further comprises;

displaying a candidate identity for the produce item for operator verification.

15. The method of claim 1, further comprising:

operating the produce recognition software in a training mode; and

capturing data from multiple items of produce of the same type to provide training data.

16. The method of claim 1, wherein the recognition result includes both a type of produce and a variety of produce.

17. The method of claim 1, wherein the feature set comprises:

color, size, shape and weight.

18. The method of claim 17, wherein the feature set further comprises:

non-visible electromagnetic data, aroma, and density.

7. A method of automatically recognizing at a checkout system a produce item from a number of produce items to be sold comprising:

pre-training a multinomial regression based model with variations in sizes, shapes, colors, types, thermal data, and aroma data for a training set of produce items, and pre-training the multinomial regression based model with environmental factors present during the pre-training including with the environmental factors at least one factor for background lighting and at least another factor for humidity;

capturing features of multiple other produce items of similar type as the one produce item by a single produce data collector and capturing at least one feature as thermal information for the one produce item, the thermal information providing data relevant to an internal structure and a composition of the one produce item and capturing at least one other feature as aroma information captured as chemicals given off by the produce items and the aroma information provided by an olfactory sensor, the aroma information relevant to chemical signatures for the produce items;

determining parameters based upon the features of the multiple other produce items by a computer executing produce recognition software;

providing a decision engine including a Bayesian classifier configured with the parameters and having a plurality of inputs, each input being adapted to receive produce data of a different modality captured from the produce item by a produce data collector comprising a camera and a scale;

mapping the produce data to the respective input of the classifier by the computer;

for each input, independently operating on the data relating to that input to create a feature set by the computer;

comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set by the computer;

combining all similarity description sets using a dedicated weighting function to produce a composite similarity description by the computer; and

deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by the computer by processing the multinomial regression based model and producing an identity for the one produce item.

8. The method according to claim 7, further comprising providing an indication of the identity of the produce item to an operator by the computer.

9. The method according to claim 8, further comprising providing multiple identity predictions and probabilities and ranking them according to the probabilities for the operator by the computer.

10. An automatic produce recognition system at a checkout system for recognizing a produce item from a number of different produce items to be sold, the produce recognition system comprising:

a single produce data collector; and

a computer for controlling the produce data collector, the computer including a decision engine including a classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality, and wherein at least a portion of the produce data includes a chemical signature and wherein at least another portion of the data includes thermal information captured for the produce item, the thermal information providing data relevant to an internal structure and composition of the produce item, and wherein still another portion of the data includes aroma information captured as chemicals given off by the produce item and the aroma information provided by an olfactory sensor, the aroma information relevant to a chemical signature for the produce item;

wherein the computer also for pre-training a multinomial regression based model with variations in sizes, shapes, colors, types, thermal data, and aroma data for a training set of produce items, and pre-training the multinomial regression based model with environmental factors present during the pre-training including with the environmental factors at least one factor for background lighting and at least another factor for humidity;

wherein the computer is additionally for mapping the produce data to the respective input of the classifier;

for each input, independently operating on the data relating to that input to create a feature set;

comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set;

combining all similarity description sets using a dedicated weighting function to produce a composite similarity description; and

deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by processing the multinomial regression based model and producing an identity for the produce item.

11. A transaction system employing an automatic produce recognition system to recognize a product item from a number of different produce items to be sold, comprising:

a single produce data collector comprising a camera and a scale; and

a computer executing produce recognition software and controlling the produce data collector, the computer including a decision engine including a classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality, at least one modality relevant to thermal information captured for the produce item, the thermal information providing data relevant to an internal structure and composition of the produce item, and wherein at least another modality relevant to aroma information captured as chemicals given off by the produce item and the aroma information provided by an olfactory sensor, the aroma information relevant to a chemical signature for the produce item;

wherein the computer also for pre-training a multinomial regression based model with variations in sizes, shapes, colors, types, thermal data, and aroma data for a training set of produce items, and pre-training the multinomial regression based model with environmental factors present during the pre-training including with the environmental factors at least one factor for background lighting and at least another factor for humidity;

wherein the computer is additionally for mapping the produce data to the respective input of the classifier;

for each input, independently operating on the data relating to that input to create a feature set;

comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set;

combining all similarity description sets using a dedicated weighting function to produce a composite similarity description;

deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by processing the multinomial regression based model and producing an identity for the produce item;

determining a price of the produce item; and

for recording payment for the produce item.

12. The transaction system according to claim 11, wherein the computer comprises a self-service computer.