System for Food Recognition Method Using Portable Devices Having Digital Cameras

• US 20120170801A1
• Filed: 12/30/2010
• Published: 07/05/2012
• Est. Priority Date: 12/30/2010
• Status: Active Grant

• Current Assignee: Samsung Eletrônica da Amazônia Ltda (Samsung Electronics Co. Ltd.)
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: DE OLIVEIRA, Luciano REBOUCAS, DE FREITAS JORGE, Eduardo MANUEL, DE AZEVEDO FILHO, Alberto ALMEIDA, FRANCO COSTA, Victor, DE ALMEIDA NEVES, Gustavo, DE OLIVEIRA, Talmai BRANDÃO, ANUNCIACÃO, Geovane DOS SANTOS, FONSECA SANTOS, Eduardo TELMO
• Granted Patent: US 8,625,889 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

The present invention relates to a method for automatic food recognition by means of portable devices equipped with digital cameras.

With this system, it is possible to identify a previously established food menu. To this purpose, a semi-automated method of segmentation is applied to delineate the regions in which each type of food in an image of a plate of food, captured by a user.

Pattern recognition techniques are used in images, integrated into a system whose goal is to label each type of food contained in the photo of a plate of food. No type of preprocessing is performed to correct deficiencies in capturing the image, just using the auto-focus component present in the portable device to capture a clear image.

Claims

1. ) Automatic method for food recognition by means of portable devices equipped with digital cameras that segments each food on a image of a plate of food, identifying these foods segmented from a pre-set menu, identifying and recording food and calories to their over a period, said method comprising the steps of:

Semi-automatic segmentation, where the objects of the photo of the plate are segmented from the choice of a user image of the centers of each food, and being applied, then, a method based on broadening of region to each center and on different color spaces (HSV, RGB and LAB) (212), providing three hypotheses of segments chosen for each center (214), making possible to obtain robust results from the segmentation under the user control;

From each segment obtained at (214), training an MVS having a radial based kernel-type function (215), and from a predetermined menu of food images, obtaining a list of probabilities for each segment of food, being the segment with the highest probability chosen (216) and the results of segmented food and their possible identifications (217), shown.

2. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein three hypothesis (310, 311, 312) are generated by the segmentator based on broadening the region, from three color spaces:

RGB, HSV and CIE L*a*b*, and for each color space are computed coefficients of spatial variation (CVE), where effectively the segmentation method is applied to, and the CVE is responsible for defining the texture of each color.

3. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein, for each segment obtained by the method of automatic broadening the region, a set of square regions of the image having 30×

30 pixels in size, are obtained throughout the area found within the segment.

4. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein, for each feature are extracted features of color (RGB and CIE L*a*b*), form (difference of Gaussians [DOGs]) and texture (CVE on doGs).

5. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein the feature vector is composed as follows for each cut-off:

it is applied a DoG is to each RGB channel, and obtained a CVE for each space in the DoG, generating three elements;

it is computed the average of the R, G and B channels of each cut-off, generating more three elements;

it is computed the averaged across the channel L*a*b*, obtaining more three elements;

for each color channel L*a*b*, it is computed CVE, obtaining more three elements, being the vector composed of twelve elements (313) sorted by an MVS (314) using a radial based kernel-type function, and the MVSs will be responsible for transforming the input space into a higher dimensional space, however, linearly separable.

6. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein at the stage of prediction, the MVS classifies each cut-off present in one segment, providing a “

score”

obtained by the distance between the vector to be classified and the separation hyperplane.

7. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein the probability of a final segment is a given element is given by the average of the probabilities of all cut-offs of a given food, and at the end, it is obtained the probabilities of various foods.

8. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein, since some cut-offs can be classified wrongly, the food with the highest probability will set the segment.

9. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein instead of providing only a hypothesis of the final food, the food identification system provides a list of foods ordered decreasingly by food with higher hypothesis.

10. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 9, wherein, after obtaining this list of MVSs, a reordering of the list based on context is applied.

11. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 9, wherein in that ordered list of possible food, there is still the possibility of errors in classification.

12. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein to deal with uncertainties, minimizing the classification errors, a MMR method, which is responsible for reordering the final list of MVSs from requirements and contexts listed previously is used.

13. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein, after qualifying with MVSs, using contextual information in order to increase the robustness of naming food.

14. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein the decision on the food recognition is made from the best selection of alternatives based on partial uncertainty, MVSs ouputs, and these uncertainties are solved, in the final stage, from the MinMax method Regret (MMR), under the approach of reclassifying the list provided by MVSs.

15. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 14, wherein said MMR method works from the construction of an regret matrix whose elements show the “

regret”

of the decision maker in solving an alternative Ai, for a given attribute Cj associated with a weight wij, as follows;

16. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 14, wherein the results are chosen to show the minimal remorse, from the following steps:

calculating Wj=Max[w ij] for each Cj and all Ai'"'"'s;

calculating rij=Wj−

wij for each combination of Ai'"'"'s and CJ'"'"'s;

calculating Ri=Max[rij] for each Ai'"'"'s and all Cj'"'"'s;

selecting A*I*R such that i=min[Ri], with the maximal regret given by;

Ri=Maxi[rij].

17. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 14, wherein instead of using a food on each element rij, a class of foods grouped by similarity color and texture is used.

18. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 17, wherein, after reclassification, not only particular foods will be reordered, but also groups of objects with similar visual characteristics.

19. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 18, wherein the user manually distinguish possible recognition errors.

20. ) Automatic method for food recognition by means of portable devices equipped with digital cameras, as in claim 1, wherein the user can further correct identifications of food through a manual note tool, if any incorrect result is displayed.