Method for computing food volume in a method for analyzing food

• US 8,345,930 B2
• Filed: 04/12/2010
• Issued: 01/01/2013
• Est. Priority Date: 01/22/2010
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Tamrakar, Amir, Sawhney, Harpreet Singh, Yu, Qian, Divakaran, Ajay
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US12/758,208
• Publication Number:
  US 20110182477A1
• Time in Patent Office: 995 Days
• Field of Search: 382/110, 382/141, 382/145, 434/127, 177/25.16
• US Class Current: 382/110
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/44 using image operators, e.g. filters, edge density metrics or local histograms
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/77 using statistical methods

Abstract

A computer-implemented method for estimating a volume of at least one food item on a food plate is disclosed. A first and second plurality of images are received from different positions above a food plate, wherein angular spacing between the positions of the first plurality of images is greater than angular spacing between the positions of the second plurality of images. A first set of poses of each of the first plurality of images is estimated. A second set of poses of each of the second plurality of images is estimated based on at least the first set of poses. A pair of images taken from each of the first and second plurality of images is rectified based on at least the first and second set of poses. A 3D point cloud is reconstructed based on at least the rectified pair of images. At least one surface of the at least one food item above the food plate is estimated based on at least the reconstructed 3D point cloud. The volume of the at least one food item is estimated based on the at least one surface.

Claims

1. A computer-implemented method for estimating a volume of at least one food item on a food plate, the method being executed by at least one processor, comprising the steps of:

receiving a first plurality of images and a second plurality of images from different positions above a food plate, wherein angular spacing between the positions of the first plurality of images are greater than angular spacing between the positions of the second plurality of images;

estimating a first set of poses of each of the first plurality of images;

estimating a second set of poses of each of the second plurality of images based on at least the first set of poses;

rectifying a pair of images taken from each of the first and second plurality of images based on at least the first and second set of poses;

reconstructing a 3D point cloud based on at least the rectified pair of images;

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

2. The method of claim 1, further comprising extracting and matching a plurality of scale invariant feature transform (SIFT feature) points among each of the first and second plurality of images to produce feature correspondences.

3. The method of claim 2, further comprising the step of producing a sparse 3D point cloud of matched features corresponding to the first plurality of images.

4. The method of claim 3, wherein the step of estimating a second set of poses of each of the second plurality of images is further based on the sparse 3D point cloud.

5. The method of claim 4, wherein focal lengths corresponding to the first plurality of images are optimized based on at least a subset of the feature correspondences, and wherein focal lengths corresponding to the second plurality of images are optimized based on at least the sparse 3D point cloud.

6. The method of claim 2, wherein the step of estimating at least one surface based on at least the reconstructed 3D point cloud further comprises the step of estimating a table plane associated with the food plate.

7. The method of claim 6, wherein the step of estimating the table plane further comprises the steps of employing RANSAC to fit a 3D plane equation to feature points used for pose estimation;

and

removing points falling on the plate for the purpose of plane fitting by using the boundaries obtained from a plate detection step.

8. The method of claim 6, further comprising the step of using the estimated table plane to slice the reconstructed 3D point cloud into an upper and lower portion such that only 3D points above the table plane are considered for the purpose of volume estimation.

9. The method of claim 8, further comprising the step of employing at least one segmentation mask produced by a classification engine to partition the 3D points above the table plane into at least one surface belonging to the at least one food item.

10. The method of claim 1, wherein the step of reconstructing the 3D point cloud further comprises the step of:

(a) decomposing the rectified pair of images using an image pyramid to estimate a disparity image;

(b) establishing image patch correspondences between the rectified pair of images and the disparity image over the entire rectified stereo pair and the disparity image using correlation to produce a correlated disparity image;

(c) converting the correlated disparity image to a depth image for a selected image of the rectified pair of images;

(d) employing a depth value for a selected pixel in the depth image along with pixel coordinates of the corresponding pixel in the depth image and pose information for the selected image to locate the selected pixel in 3D space coordinates; and

(e) repeating step (d) for all of the remaining pixels in the depth image to produce the reconstructed 3D point cloud.

11. The method of claim 10, wherein the step of reconstructing the 3D point cloud further comprises the step of estimating a 3D scale factor by employing an object with known dimensions placed and captured along with the at least one food item on a food plate in the plurality of images.

12. The method of claim 1, wherein estimating a pose further comprises the steps of:

(a) establishing a plurality of feature tracks from image patch correspondences;

(b) applying a preemptive random sample consensus (RANSAC)-based method to the feature tracks to produce a best pose for a first camera view; and

(c) refining the best pose using an iterative minimization of a robust cost function of re-projection errors through a Levenberg-Marquardt method to obtain a final pose.

13. The method of claim 1, wherein computing the volume of the at least one food item further comprises the steps of:

(a) performing Delaunay triangulation to fit the at least one surface of the at least one of food item to obtain a plurality of Delaunay triangles; and

(b) calculating a volume of the at least one food item as a sum of individual volumes for each Delaunay triangle obtained from step (a).

14. A system for estimating a volume of at least one food item on a food plate, comprising:

a processor for;

receiving a first plurality of images and a second plurality of images from different positions above a food plate, wherein angular spacing between the positions of the first plurality of images are greater than angular spacing between the positions of the second plurality of images;

estimating a first set of poses of each of the first plurality of images;

estimating a second set of poses of each of the second plurality of images based on at least the first set of poses;

rectifying a pair of images taken from each of the first and second plurality of images based on at least the first and second set of poses;

reconstructing a 3D point cloud based on at least the rectified pair of images;

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

15. The system of claim 14, further comprising at least one image capturing device for capturing the first plurality of images and the second plurality of images and for providing the first plurality of images and the second plurality of images to the processor.

16. The system of claim 14, wherein reconstructing the 3D point cloud based on at least the rectified pair of images further comprises:

(a) decomposing the rectified pair of images using an image pyramid to estimate a disparity image;

(b) establishing image patch correspondences between the rectified pair of images and the disparity image over the entire rectified stereo pair and the disparity image using correlation to produce a correlated disparity image;

(c) converting the correlated disparity image to a depth image for a selected image of the rectified pair of images;

(d) employing a depth value for a selected pixel in the depth image along with pixel coordinates of the corresponding pixel in the depth image and pose information for the selected image to locate the selected pixel in 3D space coordinates; and

(e) repeating step (d) for all of the remaining pixels in the depth image to produce the reconstructed 3D point cloud.

17. A non-transitory computer-readable medium storing computer code for estimating a volume of at least one food item on a food plate, the code being executed by at least one processor, wherein the computer code comprises code for:

receiving a first plurality of images and a second plurality of images from different positions above a food plate, wherein angular spacing between the positions of the first plurality of images are greater than angular spacing between the positions of the second plurality of images;

estimating a first set of poses of each of the first plurality of images;

estimating a second set of poses of each of the second plurality of images based on at least the first set of poses;

rectifying a pair of images taken from each of the first and second plurality of images based on at least the first and second set of poses;

reconstructing a 3D point cloud based on at least the rectified pair of images;

estimating at least one surface of the at least one food item above the food plate based on at least the reconstructed 3D point cloud; and

estimating the volume of the at least one food item based on the at least one surface.

18. The computer readable medium of claim 17, further comprising code for extracting and matching a plurality of scale invariant feature transform (SIFT) feature points among each of the first and second plurality of images to produce feature correspondences.

19. The computer readable medium of claim 18, further comprising code for producing a sparse 3D point cloud of matched features corresponding to the first plurality of images.

20. The computer readable medium of claim 19, wherein code for estimating a second set of poses of each of the second plurality of images is further based on the sparse 3D point cloud.

21. The computer readable medium of claim 20, wherein focal lengths corresponding to the first plurality of images are optimized based on at least a subset of the feature correspondences, and wherein focal lengths corresponding to the second plurality of images are optimized based on at least the sparse 3D point cloud.

22. The computer readable medium of claim 17, wherein code for reconstructing the 3D point cloud further comprises code for:

(a) decomposing the rectified pair of images using an image pyramid to estimate a disparity image;

(b) establishing image patch correspondences between the rectified pair of images and the disparity image over the entire rectified stereo pair and the disparity image using correlation to produce a correlated disparity image;

(c) converting the correlated disparity image to a depth image for a selected image of the rectified pair of images;

(d) employing a depth value for a selected pixel in the depth image along with pixel coordinates of the corresponding pixel in the depth image and pose information for the selected image to locate the selected pixel in 3D space coordinates; and

(e) repeating step (d) for all of the remaining pixels in the depth image to produce the reconstructed 3D point cloud.