Personal Audio/Visual System Providing Allergy Awareness

• US 20130085345A1
• Filed: 03/30/2012
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Mount, Brian J., McCulloch, Daniel, Hirzel, Holly A., Brown, Cameron G.
• Granted Patent: US 9,053,483 B2
• Time in Patent Office: Days
• US Class Current: 600/300
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06Q 30/00 Commerce

Abstract

A system provides a recommendation of food items to a user based on nutritional preferences of the user, using a head-mounted display device (HMDD) worn by the user. In a store, a forward-facing camera of the HMDD captures an image of a food item. The food item can be identified by the image, such as based on packaging of the food item. Nutritional parameters of the food item are compared to nutritional preferences of the user to determine whether the food item is recommended. The HMDD displays an augmented reality image to the user indicating whether the food item is recommended. If the food item is not recommended, a substitute food item can be identified. The nutritional preferences can indicate food allergies, preferences for low calorie foods and so forth. In a restaurant, the HMDD can recommend menu selections for a user.

Claims

1. A method for recommending food to a user, the method comprising:

receiving image data of a scene from at least one forward-facing camera of a head-mounted display device worn by the user, the scene comprises a food item;

responsive to the image data, accessing a data source using an identifier of the food item and an identifier associated with the user;

based on the accessing of the data source, obtaining a determination as to whether or not the food item is recommended for the user; and

controlling an augmented reality projection system of the head-mounted display device to project an augmented reality image to the user, based on the determination.

2. The method of claim 1, wherein:

the identifier associated with the user is linked to at least one nutritional preference associated with the user;

the identifier of the food item is linked to at least one nutritional parameter of the food item; and

the determination is based on a comparison of the at least one nutritional preference associated with the user to the at least one nutritional parameter of the food item.

3. The method of claim 2, wherein:

the comparison indicates whether the food item contains an ingredient which is an allergen to the user.

4. The method of claim 1, wherein:

the food item comprises a package; and

the identifier of the food item comprises an image of the package.

5. The method of claim 1, wherein:

the identifier associated with the user is linked to at least one nutritional preference of another user;

the identifier of the food item is linked to at least one nutritional parameter of the food item; and

the determination is based on a comparison of the at least one nutritional preference of the another user to the at least one nutritional parameter of the food item.

6. The method of claim 1, wherein:

the identifier of the food item comprises a logo of the food item, the logo indicating that the food item is at least one of;

heart healthy, kosher, low sodium, low calorie, whole grain, vegan, vegetarian, fat-free, sugar-free, nut-free and wheat-free.

7. The method of claim 1, wherein:

the determination indicates that the food item is not recommended for the user based on a violation of a dietary restriction associated with the user; and

the at least one augmented reality image informs the user of the violation of the dietary restriction.

8. The method of claim 1, wherein:

the determination indicates that the food item is not recommended for the user; and

the augmented reality image identifies a substitute food item which is recommended for the user.

9. The method of claim 8, further comprising:

determining that the user is in a store having a known arrangement of food items in different areas of the store, the augmented reality image indicates one of the areas of the store which has the substitute food item, based on the known arrangement.

10. The method of claim 1, wherein:

the augmented reality image is projected to the user at a defined location relative to the food item in a field of view of the user.

11. The method of claim 10, wherein:

the determination indicates that the food item is not recommended for the user; and

the augmented reality image at least partially blocks or erases a view by the user of the food item.

12. The method of claim 10, wherein:

the determination indicates that the food item is recommended for the user; and

the augmented reality image highlights the food item to the user.

13. The method of claim 1, further comprising:

determining that the user is in a store, the accessing the data source uses an identifier of the store, and an inventory of the store is associated with the identifier of the store.

14. The method of claim 1, further comprising:

determining that the user is in a particular area of a store, the accessing the data source uses an identifier of the store and an identifier of the particular area, an inventory of food items of the store is associated with the identifier of the store, and a subset of food items of the inventory of food items is associated with the identifier of the particular area.

15. The method of claim 1, wherein:

the determination indicates that the food item is recommended for the user is based on a determination that the food item does not contain an ingredient which is harmful to the user, the determination that the food item does not contain the ingredient is made with an associated degree of confidence; and

the at least one augmented reality image provides a message to the user instructing the user to further investigate the food item, when the associated degree of confidence is below a confidence threshold.

16. A method at a server for recommending food to a user, the method comprising:

receiving a request from a head-mounted display device worn by the user, the request comprises;

(a) an identifier associated with the user and (b) image data of a scene, the scene comprises a food item;

detecting an identifier of the food item from the image data;

responsive to the detecting, accessing a data source using the identifier of the food item and the identifier associated with the user;

based on the accessing of the data source, providing a determination as to whether or not the food item is recommended for the user; and

communicating data indicating the determination to the head-mounted display device.

17. The method of claim 16, wherein:

the accessing the data store comprises identifying at least one nutritional preference associated with the user based on the identifier associated with the user, and identifying at least one nutritional parameter of the food item based on the identifier of the food item; and

the providing the determination comprises comparing the at least one nutritional preference associated with the user to the at least one nutritional parameter of the food item.

18. A head-mounted display device worn by a user, the head-mounted display device comprising:

an augmented reality projection system; and

at least one control circuit in communication with the augmented reality projection system, the at least one control circuit;

determines that the user is in a restaurant which provides menu selections;

accesses a data source associated with the restaurant using an identifier associated with the user, to obtain a recommendation of at least one of the menu selections for the user; and

to provide the recommendation to the user, at least one of;

(a) controls the augmented reality projection system to project an augmented reality image, and (b) provides an audio output.

19. The head-mounted display device of claim 18, further comprising:

a forward-facing camera, the at least one control circuit receives image data of a scene from the forward-facing camera, the scene includes a menu listing the menu selections, the at least one control circuit identifies the at least one of the selections from the image data; and

the augmented reality image is projected to the user at a defined location in the field of view of the user, relative to the location of at least one of the menu selections in the field of view of the user.

20. The head-mounted display device of claim 18, wherein:

the at least one control circuit determines that the user is in the restaurant based on location data or based on receiving a wireless signal of the restaurant.