Food logging from images

• US 9,977,980 B2
• Filed: 04/17/2017
• Issued: 05/22/2018
• Est. Priority Date: 02/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Joshi, Neel Suresh, Khullar, Siddharth, Saponas, T Scott, Morris, Daniel, Beijbom, Oscar
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US15/489,267
• Publication Number:
  US 20170323174A1
• Time in Patent Office: 400 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06Q 50/12 Hotels or restaurants
• CPC Class Codes: G06V 10/58 relating to hyperspectral data
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/153 using recognition of characters or words

Abstract

A “Food Logger” provides various approaches for learning or training one or more image-based models (referred to herein as “meal models”) of nutritional content of meals. This training is based on one or more datasets of images of meals in combination with “meal features” that describe various parameters of the meal. Examples of meal features include, but are not limited to, food type, meal contents, portion size, nutritional content (e.g., calories, vitamins, minerals, carbohydrates, protein, salt, etc.), food source (e.g., specific restaurants or restaurant chains, grocery stores, particular pre-packaged foods, school meals, meals prepared at home, etc.). Given the trained models, the Food Logger automatically provides estimates of nutritional information based on automated recognition of new images of meals provided by (or for) the user. This nutritional information is then used to enable a wide range of user-centric interactions relating to food consumed by individual users.

Claims

1. A computer-implemented process for evaluating images of meals, comprising:

using a computer to perform process actions for;

receiving a meal image of a single meal consisting of multiple food items;

wherein one or more food items are visible and one or more of the food items are occluded in the meal image;

receiving information indicating a source of the meal;

recognizing one or more of the visible food items and one or more of the occluded food items in the meal by evaluating the meal image using a machine-learned meal model that is constrained by the source of the meal; and

presenting one or more interactive messages automatically generated based on the recognized food items.

2. The computer-implemented process of claim 1 wherein the source of the meal is a particular restaurant.

3. The computer-implemented process of claim 1 further comprising process actions for using the meal model to estimate nutritional content of the recognized food items.

4. The computer-implemented process of claim 3 further comprising process actions for presenting one or more interactive messages automatically generated based on the nutritional content of the recognized food items.

5. The computer-implemented process of claim 3 further comprising process actions for storing the estimated nutritional content of the recognized food items to a database.

6. The computer-implemented process of claim 1 wherein one or more machine-learned meal model are trained using source-specific sets of meal images and corresponding nutritional information.

7. The computer-implemented process of claim 1 wherein one or more machine-learned meal model are trained using one or more meal images retrieved in response to an automated search for images at least partially matching text-based menu descriptions of corresponding meals.

8. The computer-implemented process of claim 1 wherein one or more machine-learned meal model are trained using nutritional information retrieved in response to an automated search for nutritional information corresponding to text-based menu descriptions of corresponding meals.

9. The computer-implemented process of claim 1 wherein a remote service is provided for recognizing the meal from one or more received meal images.

10. A system for recognizing meals, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

provide one or more machine-learned meal models trained on combinations of image features extracted from one or more sets of training images of representative meals and nutritional information corresponding to the representative meals;

acquire a single meal image in which one or more food items are visible and one or more other food items are occluded;

extract a plurality of image features from the meal image;

recognize the visible food items and one or more of the occluded food items by applying one or more of the machine-learned meal models to the image features extracted from the meal image;

generate one or more interactive messages in response to one or more of the recognized food items; and

present one or more of the interactive messages via a user interface.

11. The system of claim 10 wherein one or more of the machine-learned meal models are further trained on one or more particular meal sources associated with one or more of the representative meals.

12. The system of claim 11 further comprising:

constraining one or more of the machine-learned meal models based on a source of the corresponding meals; and

applying one or more of the constrained meal models to the meal image to recognize the overall meal.

13. The system of claim 10 further comprising generating an estimate of nutritional information of the meal in response to recognition of the visible and occluded food items.

14. The system of claim 13 further comprising generating an automated alert in response to the estimated nutritional information, the automated alert including food ingredients to which a user may have an allergy.

15. A computer storage device having computer executable instructions stored therein for recognizing meals from images of those meals, said instructions causing a computing device to execute a method comprising:

receive a machine-learned meal model trained on combinations of one or more sets of training images of representative meals and sources associated with one or more of the representative meals;

acquire a single current meal image of a meal comprising a plurality of food items;

wherein one or more of the food items are visible in the current meal image and one or more other food items are fully occluded in the current meal image;

receive information representing a source of a meal corresponding to the current meal image;

recognize the visible food items and at least one of the occluded food items from the current meal image by evaluating the current meal image using the machine-learned meal model as constrained based on the source of the meal; and

present one or more interactive messages automatically generated based on the recognized visible food items and the recognized occluded food items.

16. The computer storage device of claim 15 further comprising instructions for applying the machine-learned meal model to estimate nutritional information associated with the recognized food items.

17. The computer storage device of claim 16 further comprising instructions for presenting one or more interactive messages automatically generated based on the estimated nutritional information.

18. The computer storage device of claim 16 further comprising generating an automated alert in response to estimated nutritional information including food ingredients to which a user may have an allergy.

19. The computer storage device of claim 15 wherein one or more of the training images of representative meals are obtained response to an automated search for images at least partially matching text-based descriptions of corresponding meals.

20. The computer storage device of claim 15 wherein a remote service is applied to recognize the visible and occluded food items in the current meal image.