System and method for nutrition analysis using food image recognition

• US 9,892,656 B2
• Filed: 04/25/2016
• Issued: 02/13/2018
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Fitly Inc.
• Original Assignee: Fitly Inc.
• Inventors: Ortiz, Caonabo Anthony, Mery, Domingo
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/138,083
• Publication Number:
  US 20170069225A1
• Time in Patent Office: 659 Days
• CPC Class Codes: A47G 19/025 with means for amusing or giving information to the user
• CPC Class Codes: B65D 25/04 Partitions
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

The present disclosure provides a system and method for determining a nutritional value of a food item. The system and method utilizes a food container as a model to adjust various types of distortions that exists in an instant image of the food container that retains the food item. The instant image may be compared to the model image of the food container to correct any distortions. The food container includes a boundary which has a predetermined color. The predetermined color of the boundary can be used to adjust the color configuration of the instant image, thereby increasing the accuracy of the food identification.

Claims

1. A system using a food recognition device in communication with a database, that determines a nutritional value of a food item, comprising:

a food container with one or more partitions, the one or more partitions being defined by a boundary outlined by a periphery formed by the food container, wherein the food container receives the food item;

an image capturing device capturing an image of the food container including the received food item; and

the food recognition device, with a processor and a memory, in communication with the image capturing device, configured to;

obtain the image of the food container from the image capturing device, wherein the image includes the received food item;

correct orientation of the obtained image by aligning the boundary in the image with the boundary in a reference food container image stored in the database;

identify a type of the food item by comparing the image of the food item to a reference food image stored in the database; and

determine the nutritional value of the food item based on the type of the food item.

2. The system of claim 1 wherein the food recognition device is further configured to correct the image to a canonical view, the canonical view being obtained by applying a transform to compensate for an image distortion of the image, the image distortion being identified by comparing the image with the reference food container image.

3. The system of claim 2 wherein the transform is determined by:

identifying at least two markings in the image of the food container;

determining two-dimensional coordinates of each of the at least two markings with respect to the reference food container image; and

determining the transform to compensate for the image distortion in the image by comparing the two-dimensional coordinates between the image and the reference food container image.

4. The system of claim 1 wherein the food recognition device is further configured to adjust color of the image based on a predetermined color of at least a portion of the food container, the predetermined color being stored in the database.

5. The system of claim 1 wherein the food recognition device is further configured to adjust size of the image, by comparing the size of the image to size of the reference food container image stored in the database, the size of the image being identified based on resolution of the image.

6. The system of claim 1 wherein the food recognition device is further configured to:

remove background in the image based on the boundary of the food container; and

isolate the food item, within the image, based on difference in color between the food item and other surfaces of the food container.

7. The system of claim 6 wherein the food recognition device is further configured to estimate a volume of the food item by identifying a surface area of the food item in the image, the surface area being identified based on the number of pixels the food item occupies within the image.

8. The system of claim 1 wherein the food recognition device is further configured to determine a volume of the food item, the volume being determined based on two or more images of the food container with the received food item captured at varying angles.

9. The system of claim 1 wherein the food recognition device is further configured to identify the type of the food item based on a spatial ratio among ingredients of the food item in the image, the spatial ratio being identified based on at least one of:

a spatial ratio-to-type stored by the database; and

individually identifying each of the ingredients and their composition in the food item.

10. The system of claim 1 wherein the food recognition device is further configured to determine a volume of the food item, the volume being determined by identifying a density of the food item from a type-to-density list stored in the database and a weight of the food item.

11. A method to use a system using a food recognition device in communication with a database, that determines a nutritional value of a food item, comprising:

capturing, with an image capturing device, an image of a food container, wherein the food container has one or more partitions, the one or more partitions being defined by a boundary outlined by a periphery formed by the food container, wherein the food container receives the food item, the image including the received food item;

obtaining, with a food recognition device, the image of the food container from the image capturing device, the image capturing device being in communication with the food recognition device, wherein the image includes the received food item;

correcting, with the food recognition device, orientation of the obtained image by aligning the boundary in the image with the boundary in a reference food container image stored in the database;

identifying, with the food recognition device, a type of the food item by comparing the image of the food item to a reference food image stored in the database; and

determining, with the food recognition device, the nutritional value of the food item based on the type of the food item.

12. The method of claim 11 wherein the step of correcting orientation of the image comprises at least one of:

correcting the image to a canonical view, wherein the canonical view is obtained by applying a transform to compensate for an image distortion of the image, the image distortion being identified by comparing the image with the reference food container image; and

adjusting size of the image, by comparing the size of the image to size of the reference food container image stored in the database, the size of the image being identified based on resolution of the image.

13. The method of claim 12 wherein the transform is determined by:

identifying at least two markings in the image of the food container;

determining two-dimensional coordinates of each of the at least two markings with respect to the reference food container image; and

determining the transform to compensate for the image distortion in the image by comparing the two-dimensional coordinates between the image and the reference food container image.

14. The method of claim 11 further comprising the step of adjusting, with the food recognition device, color of the image based on a predetermined color of at least a portion of the food container, the predetermined color being stored in the database.

15. The method of claim 11 further comprising the steps of:

removing, with the food recognition device, background in the image based on the boundary of the food container; and

isolating, with the food recognition device, the food item within the image, based on difference in color between the food item and other surfaces of the food container.

16. The method of claim 15 further comprising the step of estimating, with the food recognition device, a volume of the food item by identifying a surface area of the food item in the image, the surface area being identified based on the number of pixels the food item occupies within the image.

17. The method of claim 11 further comprising the step of determining, with the food recognition device, a volume of the food item, the volume being determined based on two or more images of the food container with the received food item captured at varying angles.

18. The method of claim 11 further comprising the steps of:

segmenting, with the food recognition device, the image based on the boundary to isolate each of the one or more partitions; and

identifying, with the food recognition device, the type of the food item by comparing features of each segmented image to features of a reference segmented food image stored in the database.

19. The method of claim 11 further comprising the step of identifying, with the food recognition device, the type of the food item based on a spatial ratio among ingredients of the food item in the image, the spatial ratio being identified based on at least one of:

a spatial ratio-to-type stored by the database; and

individually identifying each of the ingredients and their composition in the food item.

20. The method of claim 11 further comprising the step of determining, with the food recognition device, a volume of the food item, the volume being determined by identifying a density of the food item from a type-to-density list stored in the database and a weight of the food item.