Food recognition using visual analysis and speech recognition

• US 8,439,683 B2
• Filed: 01/06/2010
• Issued: 05/14/2013
• Est. Priority Date: 01/07/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Puri, Manika, Zhu, Zhiwei, Lubin, Jeffrey, Pschar, Tom, Divakaran, Ajay, Sawhney, Harpreet S.
• Primary Examiner(s): Thai, Xuan
• Assistant Examiner(s): PAGE, EVAN RANDALL
• Application Number:
  US12/683,124
• Publication Number:
  US 20100173269A1
• Time in Patent Office: 1,224 Days
• Field of Search: 434/127, 382/110
• US Class Current: 434/127
• CPC Class Codes: G09B 19/0092 Nutrition

Abstract

A method and system for analyzing at least one food item on a food plate is disclosed. A plurality of images of the food plate is received by an image capturing device. A description of the at least one food item on the food plate is received by a recognition device. The description is at least one of a voice description and a text description. At least one processor extracts a list of food items from the description; classifies and segments the at least one food item from the list using color and texture features derived from the plurality of images; and estimates the volume of the classified and segmented at least one food item. The processor is also configured to estimate the caloric content of the at least one food item.

Claims

1. A computer-implemented method for analyzing at least one food item on a food plate, the method being executed by at least one processor, comprising the steps of:

receiving a plurality of images of the food plate;

receiving a description of the at least one food item on the food plate;

extracting a list of food items from the description;

classifying and segmenting the at least one food item from the list using color and texture features derived from the plurality of images by;

applying an offline feature-based learning method of different food types to train a plurality of classifiers to recognize individual food items; and

applying an online feature-based segmentation and classification method using at least a subset of the food type recognition classifiers trained during offline feature-based learning; and

estimating the volume of the classified and segmented at least one food item based on determining correspondences between at least two images of the plurality of images containing the at least one food item and performing a 3D reconstruction on the correspondences to generate 3D point clouds of the at least one food item.

2. The method of claim 1, further comprising the step of estimating the caloric content of the at least one food item.

3. The method of claim 1, wherein the description is at least one of a voice description and a text description.

4. The method of claim 1, wherein the step of applying an offline feature-based learning method further comprises the steps of:

selecting at least three images of the plurality of images wherein the at least three images capture the same scene;

color normalizing at least one of the three images;

employing an annotation tool to identify each food type; and

processing the color normalized image to extract color and texture features.

5. The method of claim 4, wherein the step of color normalizing comprises detecting a color pattern in the scene.

6. The method of claim 1, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

selecting at least three images of the plurality of images, the at least three images capturing the same scene;

color normalizing at least one of the three images;

locating the food plate using a contour based circle detection method; and

processing the color normalized image to extract color and texture features.

7. The method of claim 6, wherein the step of color normalizing comprises detecting a color pattern in the scene.

8. The method of claim 4, wherein the step of processing the color normalized image to extract color and texture features further comprises the steps of:

transforming color features to a CIE L*A*B color space;

determining 2D texture features by applying a histogram of orientation gradient (HOG) method; and

placing the color features and 2D texture features into bins of histograms in a higher dimensional space.

9. The method of claim 6, further comprising the steps of:

representing at least one food type by a cluster of color and texture features in a high-dimensional space using an incremental K-means clustering method;

representing at least one food type by Texton histograms; and

classifying the one food type using an ensemble of boosted SVM classifiers.

10. The method of claim 6, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

applying a k-nearest neighbors (k-NN) classification method to the extracted color and texture features to each pixel of the color normalized image and assigning at least one label to each pixel;

applying a dynamic assembled multi-class classifier to an extracted color and texture feature for each patch of the color normalized image and assigning one label to each patch; and

applying an image segmentation technique to obtain a final segmentation of the plate into its constituent food labels.

11. The method of claim 4, wherein the step of processing the color normalized image to extract color and texture features further comprises the steps of:

extracting color and texture features using Texton histograms;

training a set of one-versus-one classifiers between each pair of foods; and

combining color and texture information from the Texton histograms using an Adaboost-based feature selection classifier.

12. The method of claim 6, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

applying a multi-class classifier to every patch of the three input images to generate a segmentation map; and

dynamically assembling a multi-class classifier from a subset of the offline trained pair-wise classifiers to assign a small set of labels to each pixel of the three images.

13. The method of claim 12, wherein features are selected for applying a multi-class classifier to every patch of the three input images by employing a bootstrap procedure to sample training data and select features simultaneously.

14. The method of claim 13, wherein the bootstrap procedure comprises the steps of:

randomly sampling a set of training data and computing all features in feature pool;

training individual SVM classifiers;

applying a 2-fold validation process to evaluate the expected normalized margin for each feature to update the strong classifier;

applying a current strong classifier to densely sampled patches in the annotated images, wherein wrongly classified patches are added as new samples, and weights of all training samples are updated; and

stopping the training if the number of wrongly classified patches in the training images falls below a predetermined threshold.

15. The method of claim 1, wherein the step of estimating volume of the classified and segmented at least one food item further comprises the steps of:

capturing a set of at least three 2D images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating camera poses among the at least three images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between the at least two images selected from at least the at least three images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate 3D point cloud;

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

16. The method of claim 1, further including the step of profiling at least one of the user and meal to include at least one food item not input during said step of receiving a description of the at least one food item on the food plate.

17. A system for analyzing at least one food item on a food plate, comprising:

an image capturing device for receiving a plurality of images of the food plate;

a description recognition device for receiving a description of the at least one food item on the food plate; and

a processor for;

classifying and segmenting the at least one food item using one or more features of the at least one food item derived from the plurality of images, by applying an online feature-based segmentation and classification method using one or more food type recognition classifiers trained during offline feature-based learning; and

estimating the volume of the classified and segmented at least one food item based on determining correspondences between at least two images of the plurality of images containing the at least one food item.

18. The system of claim 17, wherein the description device is at least one of a voice recognition device and a text recognition device.

19. The system of claim 18, wherein the voice recognition device is equipped with automatic speech recognition software.

20. The system of claim 19, wherein the image capturing device and the voice recognition device are integrated in a single device.

21. The system of claim 17, wherein the image capturing device is one of a cell phone or smart phone equipped with a camera, a laptop or desktop computer or workstation equipped with a webcam, or a camera operating in conjunction with a computing platform.

22. The system of claim 17, wherein the processor is integrated into a voice processing server/computer one of directly connected to the image capturing device and remotely over a cell network and/or the Internet.

23. The system of claim 17, wherein the processor estimates the caloric content of the at least one food item.

32. The system of claim 17, wherein the estimating volume of the classified and segmented at least one food item further comprises:

capturing a set of at least three images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating camera poses among the at least three images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between at least two image selected from at least the at least three 2D images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate a 3D point cloud; and

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

24. A non-transitory computer-readable medium storing computer code for analyzing at least one food item on a food plate, the code being executed by at least one processor, wherein the computer code comprises code for:

receiving a plurality of images of the food plate;

receiving a description of the at least one food item on the food plate;

extracting a list of food items from the description;

classifying and segmenting the at least one food item from the list using color and texture features derived from the plurality of images by;

applying an offline feature-based learning method of different food types to train a plurality of classifiers to recognize individual food items; and

applying an online feature-based segmentation and classification method using at least a subset of the food type recognition classifiers trained during offline feature-based learning; and

estimating the volume of the classified and segmented at least one food based on determining correspondences between at least two images of the plurality of images containing the at least one food item and performing a 3D reconstruction on the correspondences to generate 3D point clouds of the at least one food item.

25. The computer-readable medium of claim 24, further comprising code for estimating the caloric content of the at least one food item.

26. The computer-readable medium of claim 24, wherein the description is at least one of a voice description and a text description.

27. The computer readable medium of claim 24, wherein code for applying an offline feature-based learning method further comprises code for:

selecting at least three images of the plurality of images, the at least three images capturing the same scene;

color normalizing one of the three images;

employing an annotation tool to identify each food type; and

processing the color normalized image to extract color and texture features.

28. The computer readable medium of claim 24, wherein code for applying an online feature-based segmentation and classification method further comprises code for:

selecting at least three images of the plurality of images, wherein the at least three images capture the same scene;

color normalizing one of the three images;

locating the food plate using a contour based circle detection method; and

processing the color normalized image to extract color and texture features.

29. The computer readable medium of claim 28, wherein code for processing the color normalized image to extract color and texture features further comprises:

extracting color and texture features using Texton histograms;

training a set of one-versus-one classifiers between each pair of foods; and

combining color and texture information from the Texton histograms using an Adaboost-based feature selection classifier.

30. The computer readable medium of claim 24, wherein code for applying an online feature-based segmentation and classification method further comprises code for:

applying a multi-class classifier to every patch of the three input images to generate a segmentation map; and

dynamically assembling a multi-class classifier from a subset of the offline trained pair wise classifiers to assign a small set of labels to each pixel of the three input images.

31. The computer readable medium of claim 24, wherein code for estimating volume of the classified and segmented at least one food item further comprises code for:

capturing a set of at least three 2D images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating relative camera poses among the at least three m images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between the at least two images selected from at least the at least three images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate 3D point cloud;

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.