System and method for nutrition analysis using food image recognition

• US 9,349,297 B1
• Filed: 09/09/2015
• Issued: 05/24/2016
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Fitly Inc.
• Original Assignee: Fitly Inc.
• Inventors: Ortiz, Caonabo Anthony, Mery, Domingo
• Primary Examiner(s): Grant, Michael
• Application Number:
  US14/848,992
• Time in Patent Office: 258 Days
• Field of Search: 434/127
• US Class Current: 1/1
• CPC Class Codes: A47G 19/025 with means for amusing or giving information to the user
• CPC Class Codes: B65D 25/04 Partitions
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

The present disclosure provides a system and method for determining a nutritional value of a food item. The system and method utilizes a food container as a model to adjust various types of distortions that exists in an instant image of the food container that retains the food item. The instant image may be compared to the model image of the food container to correct any distortions. The food container includes a boundary which has a predetermined color. The predetermined color of the boundary can be used to adjust the color configuration of the instant image, thereby increasing the accuracy of the food identification.

Claims

1. A system using a food recognition device in communication with a database via a network, that determines a nutritional value of a food item, comprising:

a food container with one or more partitions, the one or more partitions being defined by a boundary outlined by a divider and a peripheral wall formed by the food container, wherein the food container receives the food item, the food container being characterized by a predetermined structural properties;

a weight sensor being placed within the food container, positioned to measure a weight of the food item being received by each of the one or more partitions;

an image capturing device capturing an image of the food container; and

the food recognition device, with a processor and a memory, in communication with the weight sensor and the image capturing device, configured to;

obtain the image of the food container from the image capturing device;

correct orientation of the image by aligning the boundary in the image with the boundary in a reference food container image stored in the database;

segment the image based on the boundary to isolate each of the one or more partitions; and

identify a type of the food item by comparing each segmented image to a reference segmented food image stored in the database.

2. The system of claim 1 wherein the step of correcting orientation of the image comprises correcting the image to a canonical view, wherein the canonical view is obtained by applying a transform to compensate for an image distortion of the image, the image distortion being identified based on the predetermined structural properties of the food container.

3. The system of claim 2 wherein the transform is determined by:

identifying at least two markings in the image of the food container;

determining two-dimensional coordinates of each of the at least two markings with respect to the reference food container image; and

determining the transform to compensate for the image distortion in the image by comparing the two-dimensional coordinates between the image and the reference food container image.

4. The system of claim 1 wherein the food recognition device is further configured to adjust color of the image based on a predetermined color of the boundary, the boundary having the predetermined color dissimilar from other surfaces of the food container.

5. The system of claim 1 wherein the food recognition device is integrated with the image capturing device.

6. The system of claim 1 wherein the food recognition device is further configured to:

remove background in the image based on the boundary of the food container; and

isolate the food item, within the image, based on difference in color between the food item and other surfaces of the food container.

7. The system of claim 6 wherein the step of isolating the food item comprises analyzing the image, pixel by pixel, to eliminate pixels of the image that do not represent the food item.

8. The system of claim 1 wherein the step of identifying the type of the food item comprises comparing features of the food item from the image to features of the reference segmented food image, the features comprising color and texture that indicates types of food items.

9. The system of claim 1 wherein the food recognition device is further configured to determine the nutritional value of the food item based on the weight and the type of the food item.

10. The system of claim 9 wherein the food recognition device determines the nutritional value of the food item further based on a spatial ratio among ingredients of the food item in the image, the nutritional value of each of the ingredients being determined separately, and the spatial ratio being identified based on at least one of:

a spatial ratio-to-type stored by the database; and

individually identifying each of the ingredients and their composition in the food item.

11. The system of claim 1 wherein the food recognition device is further configured to determine the nutritional value of the food item based on a volume of the food item, the volume being determined by identifying a density of the food item from a type-to-density list stored in the database and the weight.

12. A method to use a system using a food recognition device in communication with a database via a network, that determines a nutritional value of a food item, comprising:

capturing, with an image capturing device, an image of a food container, wherein the food container has one or more partitions, the one or more partitions being defined by a boundary outlined by a divider and a peripheral wall formed by the food container, wherein the food container receives the food item, the food container being characterized by a predetermined structural properties;

obtaining, with a food recognition device, the image of the food container from the image capturing device, the image capturing device being in communication with the food recognition device;

obtaining, with the food recognition device, a weight of the food item from a weight sensor, the weight sensor being positioned to measure the weight of the food item being received by each of the one or more partitions, wherein the weight sensor is in communication with the food recognition device;

correcting, with the food recognition device, orientation of the image by aligning the boundary in the image with the boundary in a reference food container image stored in the database;

segmenting, with the food recognition device, the image based on the boundary to isolate each of the one or more partitions; and

identifying, with the food recognition device, a type of the food item by comparing each segmented image to a reference segmented food image stored in the database.

13. The method of claim 12 wherein the step of correcting orientation of the image comprises correcting the image to a canonical view, wherein the canonical view is obtained by applying a transform to compensate for an image distortion of the image, the image distortion being identified based on the predetermined structural properties of the food container.

14. The method of claim 13 wherein the transform is determined by:

identifying at least two markings in the image of the food container;

determining two-dimensional coordinates of each of the at least two markings with respect to the reference food container image; and

determining the transform to compensate for the image distortion in the image by comparing the two-dimensional coordinates between the image and the reference food container image.

15. The method of claim 12 further comprising the step of adjusting, with the food recognition device, color of the image based on a predetermined color of the boundary, the boundary having the predetermined color dissimilar from other surfaces of the food container.

16. The method of claim 12 further comprising the steps of:

removing, with the food recognition device, background in the image based on the boundary of the food container; and

isolating, with the food recognition device, the food item within the image, based on difference in color between the food item and other surfaces of the food container.

17. The method of claim 12 wherein the step of identifying the type of the food item comprises comparing features of the food item from the image to features of the reference segmented food image, the features comprising color and texture that indicates types of food items.

18. The method of claim 12 further comprising the step of determining the nutritional value of the food item, with the food recognition device, based on the weight and the type of the food item.

19. The method of claim 18 wherein the nutritional value of the food item is determined further based on a spatial ratio among ingredients of the food item in the image, the nutritional value of each of the ingredients being determined separately, and the spatial ratio being identified based on at least one of:

a spatial ratio-to-type stored by the database; and

individually identifying each of the ingredients and their composition in the food item.

20. The method of claim 12 further comprising the step of determining the nutritional value of the food item, with the food recognition device, based on a volume of the food item, the volume being determined by identifying a density of the food item from a type-to-density list stored in the database and the weight.