Automated Food Recognition and Nutritional Estimation With a Personal Mobile Electronic Device

• US 20160063734A1
• Filed: 12/11/2014
• Published: 03/03/2016
• Est. Priority Date: 09/03/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Divakaran, Ajay, Zhang, Weiyu, Yu, Qian, Sawhney, Harpreet S.
• Granted Patent: US 9,916,520 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis

Abstract

A food recognition assistant system includes technologies to recognize foods and combinations of foods depicted in a digital picture of food. Some embodiments include technologies to estimate portion size and calories, and to estimate nutritional value of the foods. In some embodiments, data identifying recognized foods and related information are generated in an automated fashion without relying on human assistance to identify the foods. In some embodiments, the system includes technologies for achieving automatic food detection and recognition in a real-life setting with a cluttered background, without the images being taken in a controlled lab setting, and without requiring additional user input (such as user-defined bounding boxes). Some embodiments of the system include technologies for personalizing the food classification based on user-specific habits, location and/or other criteria.

Claims

1. A food recognition assistant embodied in one or more non-transitory machine accessible storage media of a computing system, the food recognition assistant comprising instructions executable by one or more processors to cause the computing system to:

receive a digital image;

obtain context data indicative of the context in which the digital image was captured;

detect a salient region of the image by algorithmically recognizing a plurality of different features present in the image;

identify a plurality of characteristics of the salient region, including at least two of;

color, texture, size and shape;

compare the context data to data indicative of known relationships between foods and contextual cues; and

identify a food depicted in the salient region based on the salient region characteristics and the context data.

2. The food recognition assistant of claim 1, comprising instructions executable to algorithmically weight the salient region characteristics and the context data, and identify the food based on the weight of the salient region characteristics and the weight of the context data.

3. The food recognition assistant of claim 1, wherein the context data comprises data indicative of a location at which the image was taken cross-referenced with a map to determine if the image was taken in a food dispensary.

4. The food recognition assistant of claim 1, wherein the context data includes a selection of foods available at a food dispensary at which the image was taken, and the food recognition assistant further comprises instructions executable to compare the characteristics of the salient region to the selection of foods available.

5. The food recognition assistant of claim 1, wherein the context data includes the time of day at which the image was taken, and the food recognition assistant comprises instructions executable to compare the characteristics of the salient region of the image to stored data indicative of foods that are commonly consumed at the time the image was taken.

6. The food recognition assistant of claim 1, comprising instructions executable to utilize the digital image, the results of the food identification, and the context data to build a user-specific model that includes user-specific data indicative of commonly eaten foods.

7. The food recognition assistant of claim 6, comprising instructions executable to compare the characteristics of the salient region to stored information in the user-specific model to identify a food in the digital image.

8. The food recognition assistant of claim 6, wherein the user-specific model includes one or more of:

user dietary history, user dietary preferences, and user restaurant preferences.

9. The food recognition assistant of claim 1, comprising instructions executable to algorithmically recognize the plurality of different features of the salient region using at least two different scales of resolution.

10. A personal mobile electronic device comprising:

one or more processors;

an input/output subsystem coupled to the one or more processors;

a camera coupled to the input/output subsystem;

a context sensor coupled to the input/output subsystem;

a display device coupled to the input/output subsystem; and

memory comprising instructions executable by the one or more processors to cause the personal mobile electronic device to;

with the camera, take a picture of a plate of food;

with the context sensor, obtain context data indicative of the context in which the picture of the plate of food was taken;

with the one or more processors;

detect a salient region of the picture of the plate of food by executing one or more feature detection algorithms;

identify a plurality of characteristics of the salient region, including at least two of;

color, texture, size and shape;

compare the context data to data indicative of known relationships between foods and contextual cues; and

identify a food depicted in the salient region based on the salient region characteristics and the context data; and

with the display device, display a label indicative of the identified food.

11. The personal mobile electronic device of claim 10, comprising instructions executable to, with the input/output subsystem, transmit the picture of the plate of food and the context data to a server computing device and receive from the server computing device, data identifying a food present in the picture of the plate of food and an algorithmically determined confidence level associated with the identified food.

12. The personal mobile electronic device of claim 10, comprising instructions executable to, with the input/output subsystem, transmit the picture of the plate of food and the context data to a server computing device and receive from the server computing device, data identifying a food present in the picture of the plate of food and an algorithmically determined estimate of a portion size of the identified food depicted in the picture of the plate of food.

13. The personal mobile electronic device of claim 12, comprising instructions executable to determine a nutritional fact associated with the identified food based on the estimated portion size, and, with an output device of the personal mobile electronic device, output the nutritional fact associated with the identified food.

14. The personal mobile electronic device of claim 12, comprising instructions executable to, with the input/output subsystem, transmit data indicative of the identified food and the estimated portion size to a food-related computer application.

15. The personal mobile electronic device of claim 10, wherein the context data includes a selection of foods available at a food dispensary at which the picture of the plate of food was taken, and the personal mobile electronic device is to execute instructions to compare the characteristics of the salient region to the selection of foods available at the food dispensary.

16. The personal mobile electronic device of claim 10, wherein the context data includes the time of day at which the picture of the plate of food was taken, and the personal mobile electronic device is to execute instructions to compare the characteristics of the salient region of the image to stored data indicative of foods that are commonly consumed at the time the picture of the plate of food was taken.

17. The personal mobile electronic device of claim 10, comprising instructions executable to utilize the picture of the plate of food, the results of the food identification, and the context data to build a user-specific model that includes user-specific, context-specific data indicative of previously consumed foods.

18. The personal mobile electronic device of claim 17, comprising instructions executable to compare the characteristics of the salient region to stored information in the user-specific model to identify a food in the picture of the plate of food.

19. The personal mobile electronic device of claim 17, comprising a microphone, a speaker, and instructions executable to utilize the user-specific model to, by the microphone and the speaker, engage in a spoken natural language dialog relating to food.

20. The personal mobile electronic device of claim 10, comprising instructions executable to store data indicative of the identified food and the estimated portion size in an electronic food journal.