WEARABLE FOOD NUTRITION FEEDBACK SYSTEM

• US 20140147829A1
• Filed: 11/29/2012
• Published: 05/29/2014
• Est. Priority Date: 11/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jerauld, Robert
• Granted Patent: US 9,189,021 B2
• Time in Patent Office: Days
• US Class Current: 434/430
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets

Abstract

A see-through, head mounted display and sensing devices cooperating to provide feedback on food items detected in the device field of view. Feedback can include warnings based on personal wearer needs, general nutrition information, food consumption tracking and social interactions. The system includes one or more processing devices in communication with display and the sensors which identify food items proximate to the apparatus, determine feedback information relevant to a wearer of the apparatus; and render feedback information in the display.

Claims

1. A see through head mounted display apparatus, comprising:

a see-through, head mounted display;

a plurality of sensors detecting visual information in a field of view of the apparatus;

one or more processing devices in communication with display and the sensors, the one or more processing devices automatically

identify food items proximate to the apparatus;

determine feedback information relevant to a wearer of the apparatus; and

render feedback information in the display.

2. The apparatus of claim 1 wherein the one or more processing devices determine a location for the apparatus, and wherein the one or more processing devices identify food items proximate to the apparatus using the location of the apparatus.

3. The apparatus of claim 1 wherein the one or more processing devices access data to match sensor data to food types and wherein the one or more processing devices identify food items proximate to the apparatus using sensor data matching sensor data.

4. The apparatus of claim 3 wherein the one or more processing devices access wearer profile information, and wherein the one or more processing device identify food items based on wearer consumption history.

5. The apparatus of claim 1 wherein the one or more processors access recipe information provided by a third party, and wherein the one or more processors provide nutritional feedback in the display based on said recipe information.

6. The apparatus of claim 1 wherein the one or more processing devices access wearer profile information, and wherein the one or more processing devices provide wearer feedback incorporating wearer health information relative to food items identified.

7. The apparatus of claim 1 wherein the food items comprise one of an item identified on a menu, an item identified on a product package, or an edible food item identified proximate to the apparatus.

8. The apparatus of claim 1 wherein feedback to the display is a visual indication of nutritional content of at least one food item identified.

9. The apparatus of claim 1 wherein the one or more processing devices identify food items by providing a wearer interface and receiving identification of food items from wearer of the apparatus.

10. A computer readable medium including instructions operable on a processing device to perform a method comprising the steps of:

receiving input from a plurality of sensors mounted on a see through head mounted display device, the sensors mounted to detect image data in a field of view of the see through head mounted display device and wearer gaze of the see through head mounted display device;

identifying food items proximate to the device based on the image data and the wearer gaze;

accessing feedback information for the food items, the feedback information; and

providing the feedback in the display device, the feedback information identified with one or more of the food items.

11. The computer readable medium of claim 10 wherein the feedback information comprises one of:

nutritional information for the one or more food items;

wearer health information for the one or more food items; and

social friend recommendations concerning the one or more food items.

12. The computer readable medium of claim 10 wherein identifying comprises:

determining a location for the display device;

accessing food item data associated with the location;

evaluating the image data to determine food items based on the location and the food item data.

13. The computer readable medium of claim 12 wherein identifying comprises evaluating the image data and the wearer gaze to parse items on a menu associated with the location, matching items on the menu with the food item data associated with the location, and wherein providing feedback includes providing location specific feedback for the food items.

14. The computer readable medium of claim 10 wherein identifying comprises evaluating the image data and the wearer gaze matching items in the image data with data associated with known food on a menu with food item data associated with a location, and wherein providing feedback includes providing location specific feedback for the food items.

15. A method for providing nutritional feedback for food items a field of view of a see through head mounted display system, comprising:

determining a location, orientation and field of view for the system;

determining whether third party data is available for the location and if so, associating third party data with the location, the third party data including food identification data and nutritional information;

evaluating sensor data from the see through head mounted display to identify food items proximate to the system, said evaluating using the third party data if available;

determining nutritional feedback information matching the food items and relevant to a wearer of the system; and

providing nutritional feedback for at least one of the food items to the wearer in the display.

16. The method of claim 15 wherein determining includes accessing user profile information indicating wearer specific nutritional data, such data determining nutritional feedback information relevant to the wearer.

17. The method of claim 16 wherein the step of evaluating includes matching food item data with sensor data to identify food items in the field of view.

18. The method of claim 17 wherein the step of determining includes matching third party nutritional data with identified food items.

19. The method of claim 18 further including detecting a wearer gaze and providing feedback based on a food item identified as a focus of wearer gaze.

20. The method of claim 19 wherein providing feedback includes rendering a visual indication of at least a nutritional warning indicating a possible hazard to the wearer.