System for food recognition method using portable devices having digital cameras

• US 8,625,889 B2
• Filed: 12/30/2010
• Issued: 01/07/2014
• Est. Priority Date: 12/30/2010
• Status: Active Grant

• Current Assignee: Samsung Eletrônica da Amazônia Ltda (Samsung Electronics Co. Ltd.)
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: De Azevedo Filho, Alberto Almeida, De Oliveira, Luciano Reboucas, De Freitas Jorge, Eduardo Manuel, Franco Costa, Victor, De Almeida Neves, Gustavo, De Oliveira, Talmai Brando, Anunciaco, Geovane Dos Santos, Fonseca Santos, Eduardo Telmo
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Niu, Feng
• Application Number:
  US12/981,634
• Publication Number:
  US 20120170801A1
• Time in Patent Office: 1,104 Days
• Field of Search: None
• US Class Current: 382/165
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

A method for automatic food recognition by using portable devices equipped with digital cameras. With this system, it is possible to identify a previously established food menu. To this purpose, a semi-automated method of segmentation is applied to delineate the regions in which each type of food in an image of a plate of food, captured by a user. Pattern recognition techniques are used in images, integrated into a system whose goal is to label each type of food contained in the photo of a plate of food. No type of preprocessing is performed to correct deficiencies in capturing the image, just using the auto-focus component present in the portable device to capture a clear image.

Claims

1. A method for food recognition using a portable device equipped with a digital camera to record food and calories, the method comprising:

segmenting items of food of a top-view photo of a plate of food into regions using the centers, selected by a user, of each item of food;

broadening the regions based on each center and on different color spaces (HSV, RGB, and LAB);

providing three hypotheses of segments chosen for each center to obtain robust results from the segmentation;

training, for each hypothesis, a Support Vector Machine (SVM) having a radial based kernel-type function;

obtaining, from a predetermined menu of food images, a list of probabilities for each segment of food;

choosing the segment with the highest probability; and

showing the results of segmented food and their possible identifications.

2. The method of claim 1, wherein coefficients of spatial variation (CV) are computed for each color space the segmentation method is applied to, and the CV defines a texture of each color.

3. The method of claim 1, wherein, for each segment obtained by the broadening of regions, a set of square regions of the image having 30×

30 pixels in size are obtained throughout the area found within the segment.

4. The method of claim 1, wherein, for each feature are extracted features of color (RGB and CIE L*a*b*), form (difference of Gaussians [DOGs]) and texture (CV on doGs).

5. The method of claim 1, wherein the feature vector is composed as follows for each cut-off:

a DoG is applied to each RGB channel, and a CV is obtained for each space in the DoG, generating three elements;

the average of the R, G and B channels of each cut-off is computed, generating three more elements;

the average across the channel L*a*b* is computed, obtaining three more elements;

a CV is computed for each color channel L*a*b*, obtaining three more elements, being the vector composed of twelve elements sorted by an SVM using a radial based kernel-type function, and the SVMs will be responsible for transforming the input space into a higher dimensional space, however, linearly separable.

6. The method of claim 1, wherein at the stage of prediction, the SVM classifies each cut-off present in one segment, providing a “

score”

obtained by the distance between the vector to be classified and the separation hyperplane.

7. The method of claim 1, wherein the probability of a final segment is given by the average of the probabilities of all cut-offs of a given food, and at the end, the probabilities of various foods are obtained.

8. The method of claim 1, wherein, since some cut-offs can be classified incorrectly, the food with the highest probability will be used for the segment.

9. The method of claim 1, wherein instead of providing only a hypothesis of the final food, the food identification system provides a list of foods in decreasing order by probability.

10. The method of claim 9, wherein, after obtaining this list of SVMs, a reordering of the list based on context is applied.

11. The method of claim 9, wherein in that ordered list of possible food, there is still the possibility of errors in classification.

12. The method of claim 1, wherein to deal with uncertainties, minimizing the classification errors, a MMR method, which is responsible for reordering the final list of SVMs from requirements and contexts listed previously is used.

13. The method of claim 1, wherein, after qualifying with SVMs, using contextual information in order to increase the robustness of naming food.

14. The method of claim 1, wherein the decision on the food recognition is made from the best selection of alternatives based on partial uncertainty, SVMs outputs, and these uncertainties are solved, in the final stage, from the MinMax method Regret (MMR), under the approach of reclassifying the list provided by SVMs.

15. The method of claim 14, wherein the MMR method works from the construction of an regret matrix whose elements show the “

regret”

of the decision maker in solving an alternative Ai, for a given attribute Cj associated with a weight wij, as follows;

16. The method of claim 14, wherein the results are chosen to show the minimal remorse, from the following:

calculating Wj=Max[wij] for each Cj and all Ai'"'"'s;

calculating rij=Wj−

wij for each combination of Ai'"'"'s and CJ'"'"'s;

calculating Ri=Max[rij] for each Ai'"'"'s and all Cj'"'"'s; and

selecting A*I*R such that i=min[Ri], with the maximal regret given by;

Ri=Maxi[rij].

17. The method of claim 14, wherein instead of using a food on each element rij, a class of foods grouped by similarity, color, and texture is used.

18. The method of claim 17, wherein, after reclassification, not only particular foods will be reordered, but also groups of objects with similar visual characteristics.

19. The method of claim 18, wherein the user manually distinguishes possible recognition errors.

20. The method of claim 1, wherein the user can further correct identifications of food through a manual note tool, if any incorrect result is displayed.