FOOD RECOGNITION USING VISUAL ANALYSIS AND SPEECH RECOGNITION

• US 20100173269A1
• Filed: 01/06/2010
• Published: 07/08/2010
• Est. Priority Date: 01/07/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Pschar, Tom, Sawhney, Harpreet S., Puri, Manika, Divakaran, Ajay, Zhu, Zhiwei, Lubin, Jeffrey
• Granted Patent: US 8,439,683 B2
• Time in Patent Office: Days
• US Class Current: 434/127
• CPC Class Codes: G09B 19/0092 Nutrition

Abstract

A method and system for analyzing at least one food item on a food plate is disclosed. A plurality of images of the food plate is received by an image capturing device. A description of the at least one food item on the food plate is received by a recognition device. The description is at least one of a voice description and a text description. At least one processor extracts a list of food items from the description; classifies and segments the at least one food item from the list using color and texture features derived from the plurality of images; and estimates the volume of the classified and segmented at least one food item. The processor is also configured to estimate the caloric content of the at least one food item.

Claims

1. A computer-implemented method for analyzing at least one food item on a food plate, the method being executed by at least one processor, comprising the steps of:

receiving a plurality of images of the food plate;

receiving a description of the at least one food item on the food plate;

extracting a list of food items from the description;

classifying and segmenting the at least one food item from the list using color and texture features derived from the plurality of images; and

estimating the volume of the classified and segmented at least one food item based on determining correspondences between at least two images of the plurality of images containing the at least one food item and performing a 3D reconstruction on the correspondences to generate 3D point clouds of the at least one food item.

2. The method of claim 1, further comprising the step of estimating the caloric content of the at least one food item.

3. The method of claim 1, wherein the description is at least one of a voice description and a text description.

4. The method of claim 1, wherein the step of classifying and segmenting the at least one food item further comprises the steps of:

applying an offline feature-based learning method of different food types to train a plurality of classifiers to recognize individual food items; and

applying an online feature-based segmentation and classification method using at least a subset of the food type recognition classifiers trained during offline feature-based learning.

5. The method of claim 4, wherein the step of applying an offline feature-based learning method further comprises the steps of:

selecting at least three images of the plurality of images, wherein the at least three images capture the same scene;

color normalizing at least one of the three images;

employing an annotation tool is used to identify each food type; and

processing the color normalized image to extract color and texture features.

6. The method of claim 5, wherein the step of color normalizing comprises detecting a color pattern in the scene.

7. The method of claim 4, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

selecting at least three images of the plurality of images, the at least three images capturing the same scene;

color normalizing at least one of the three images;

locating the food plate using a contour based circle detection method; and

processing the color normalized image to extract color and texture features.

8. The method of claim 7, wherein the step of color normalizing comprises detecting a color pattern in the scene.

9. The method of claim 5, wherein the step of processing the color normalized image to extract color and texture features further comprises the steps of:

transforming color features to a CIE L*A*B color space;

determining 2D texture features by applying a histogram of orientation gradient (HOG) method; and

placing the color features and 2D texture features into bins of histograms in a higher dimensional space.

10. The method of claim 7, further comprising the steps of:

representing at least one food type by a cluster of color and texture features in a high-dimensional space using an incremental K-means clustering method;

representing at least one food type by Texton histograms; and

classifying the one food type using an ensemble of boosted SVM classifiers.

11. The method of claim 7, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

applying a k-nearest neighbors (k-NN) classification method to the extracted color and texture features to each pixel of the color normalized image and assigning at least one label to each pixel;

applying a dynamic assembled multi-class classifier to an extracted color and texture feature for each patch of the color normalized image and assigning one label to each patch; and

applying an image segmentation technique to obtain a final segmentation of the plate into its constituent food labels.

12. The method of claim 5, wherein the step of processing the color normalized image to extract color and texture features further comprises the steps of:

extracting color and texture features using Texton histograms;

training a set of one-versus-one classifiers between each pair of foods; and

combining color and texture information from the Texton histograms using an Adaboost-based feature selection classifier.

13. The method of claim 7, wherein the step of applying an online feature-based segmentation and classification method further comprises the steps of:

applying a multi-class classifier to every patch of the three input images to generate a segmentation map; and

dynamically assembling a multi-class classifier from a subset of the offline trained pair-wise classifiers to assign a small set of labels to each pixel of the three images.

14. The method of claim 13, wherein features are selected for applying a multi-class classifier to every patch of the three input images by employing a bootstrap procedure to sample training data and select features simultaneously.

15. The method of claim 14, wherein the bootstrap procedure comprises the steps of:

randomly sampling a set of training data and computing all features in feature pool;

training individual SVM classifiers;

applying a 2-fold validation process to evaluate the expected normalized margin for each feature to update the strong classifier,

applying a current strong classifier to densely sampled patches in the annotated images, wherein wrongly classified patches are added as new samples, and weights of all training samples are updated; and

stopping the training if the number of wrongly classified patches in the training images falls below a predetermined threshold.

16. The method of claim 1, wherein the step of estimating volume of the classified and segmented at least one food item further comprises the steps of:

capturing a set of at least three images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating camera poses among the at least three images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between the at least two images selected from at least the at least three images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate 3D point cloud

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

17. The method of claim 1, further including the step of profiling at least one of the user and meal to include at least one food item not input during said step of receiving a description of the at least one food item on the food plate.

18. A system for analyzing at least one food item on a food plate, comprising:

an image capturing device for receiving a plurality of images of the food plate;

a description recognition device for receiving a description of the at least one food item on the food plate; and

a processor for;

extracting a list of food items from the description;

classifying and segmenting the at least one food item from the list using color and texture features derived from the plurality of images; and

estimating the volume of the classified and segmented at least one food item based on determining correspondences between at least two images of the plurality of images containing the at least one food item and performing a 3D reconstruction on the correspondences to generate 3D point clouds of the at least one food item.

19. The system of claim 18, wherein the description device is at least one of a voice recognition device and a text recognition device.

20. The system of claim 19, wherein the voice recognition device is equipped with automatic speech recognition software.

21. The system of claim 20, wherein the image capturing device and the voice recognition device are integrated in a single device.

22. The system of claim 18, wherein the image capturing device is one of a cell phone or smart phone equipped with a camera, a laptop or desktop computer or workstation equipped with a webcam, or a camera operating in conjunction with a computing platform.

23. The system of claim 18, wherein the processor is integrated into a voice processing server/computer one of directly connected to the image capturing device and remotely over a cell network and/or the Internet.

24. The system of claim 18, wherein the processor estimates the caloric content of the at least one food item.

34. The system of claim 18, wherein the estimating volume of the classified and segmented at least one food item further comprises:

capturing a set of at least three images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating camera poses among the at least three images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between at least two image selected from at least the at least three 2D images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate a 3D point cloud; and

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

25. A computer-readable medium storing computer code for analyzing at least one food item on a food plate, the code being executed by at least one processor, wherein the computer code comprises code for:

receiving a plurality of images of the food plate;

receiving a description of the at least one food item on the food plate;

extracting a list of food items from the description;

classifying and segmenting the at least one food item from the list using color and texture features derived from the plurality of images; and

estimating the volume of the classified and segmented at least one food based on determining correspondences between at least two images of the plurality of images containing the at least one food item and performing a 3D reconstruction on the correspondences to generate 3D point clouds of the at least one food item.

26. The computer-readable medium of claim 25, further comprising code for estimating the caloric content of the at least one food item.

27. The computer-readable medium of claim 25, wherein the description is at least one of a voice description and a text description.

28. The computer readable medium of claim 25, wherein code for classifying and segmenting the at least one food item further comprises code for:

applying an offline feature-based learning method of different food types to train a plurality of classifiers to recognize individual food items; and

applying an online feature-based segmentation and classification method using at least a subset of the food type recognition classifiers trained during offline feature-based learning.

29. The computer readable medium of claim 28, wherein code for applying an offline feature-based learning method further comprises code for:

selecting at least three images of the plurality of images, the at least three images capturing the same scene;

color normalizing one of the three images;

employing an annotation tool is used to identify each food type; and

processing the color normalized image to extract color and texture features.

30. The computer readable medium of claim 28, wherein code for applying an online feature-based segmentation and classification method further comprises code for:

selecting at least three images of the plurality of images, wherein the at least three images capture the same scene;

color normalizing one of the three images;

locating the food plate using a contour based circle detection method; and

processing the color normalized image to extract color and texture features.

31. The computer readable medium of claim 30, wherein code for processing the color normalized image to extract color and texture features further comprises:

extracting color and texture features using Texton histograms;

training a set of one-versus-one classifiers between each pair of foods; and

combining color and texture information from the Texton histograms using an Adaboost-based feature selection classifier.

32. The computer readable medium of claim 28, wherein code for applying an online feature-based segmentation and classification method further comprises code for:

applying a multi-class classifier to every patch of the three input images to generate a segmentation map; and

dynamically assembling a multi-class classifier from a subset of the offline trained pair-wise classifiers to assign a small set of labels to each pixel of the three input images.

33. The computer readable medium of claim 25, wherein code for estimating volume of the classified and segmented at least one food item further comprises code for:

capturing a set of at least three images from the plurality of images taken at different positions above the food plate with a calibrated image capturing device using an object of known size for 3D scale determination;

extracting and matching multiple feature points in each image frame;

estimating camera poses among the at least three images using the matched feature points;

selecting at least two images from the at least three images;

determining correspondences between the at least two images selected from at least the at least three images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate 3D point cloud

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

35. A method for estimating volume of at least one food item on a food plate, the method being executed by at least one processor, comprising the steps of:

capturing a set of at least three images taken at different positions above the food plate with a calibrated image capturing device using an object of known size;

estimating relative camera poses among the at least three images;

determining correspondences between at least two image selected from at least the at least three images;

performing a 3D reconstruction on the correspondences and determining a 3D scale based on the object of known size to generate a 3D point cloud; and

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

36. The method of claim 35, wherein estimating a pose further comprises the steps of:

(a) establishing a plurality of feature tracks from image patch correspondences;

(b) applying a preemptive RANSAC-based method to the feature tracks to produce a best pose for a first camera view; and

(c) refining the best pose using an iterative minimization of a robust cost function of re-projection errors through a Levenberg-Marquardt method to obtain a final pose.

37. The method of claim 36, wherein the step of estimating at least one surface based on at least the reconstructed 3D point cloud and the object of known size further comprises the step of estimating a table plane associated with the food plate.

38. The method of claim 37, wherein the step of estimating the table plane further comprises the steps of employing RANSAC to fit a 3D plane equation to feature points used for pose estimation;

and

removing points falling on the plate for the purpose of plane fitting by using the boundaries obtained from a plate detection step.

39. The method of claim 38, further comprising the step of using the estimated table plane to slice the reconstructed 3D point cloud into an upper and lower portion such that only 3D points above the table plane are considered for the purpose of volume estimation.

40. The method of claim 39, further comprising the step of employing at least one segmentation mask produced by a classification engine to partition the 3D points above the table plane into at least one surface belonging to the at least one food item.

41. The method of claim 35, wherein computing the volume of the at least one food item further comprises the steps of:

(a) performing Delaunay triangulation to fit the at least one surface of the at least one of food item to obtain a plurality of Delaunay triangles; and

(b) calculating a volume of the at least one food item as a sum of individual volumes for each Delaunay triangle obtained from step (a).

42. The method of claim 35, wherein the 3D scale is based on the size of the object as measured by ground truth and its size as obtained by estimation in 3D from the set of at least three 2D images.