Personal audio/visual system providing allergy awareness

• US 9,053,483 B2
• Filed: 03/30/2012
• Issued: 06/09/2015
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A, Latta, Stephen G, Kipman, Alex Aben-Athar, McCulloch, Daniel, Perez, Kathryn Stonw, Sugden, Ben J, Vaught, Benjamin I, Brown, Cameron G, Hirzel, Holly A, Mount, Brian J
• Primary Examiner(s): Hoang, Phi
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US13/436,526
• Publication Number:
  US 20130085345A1
• Time in Patent Office: 1,166 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06Q 30/00 Commerce

Abstract

A system provides a recommendation of food items to a user based on nutritional preferences of the user, using a head-mounted display device (HMDD) worn by the user. In a store, a forward-facing camera of the HMDD captures an image of a food item. The food item can be identified by the image, such as based on packaging of the food item. Nutritional parameters of the food item are compared to nutritional preferences of the user to determine whether the food item is recommended. The HMDD displays an augmented reality image to the user indicating whether the food item is recommended. If the food item is not recommended, a substitute food item can be identified. The nutritional preferences can indicate food allergies, preferences for low calorie foods and so forth. In a restaurant, the HMDD can recommend menu selections for a user.

Claims

1. A method for recommending an item to a user, the method comprising:

receiving image data of a scene from a camera, the scene comprises a particular item on a shelf, wherein the particular item is arranged in a known relative position to another item on the shelf, the particular item has a visual characteristic and is arranged in a certain location among a plurality of locations in a store, and the another item has a visual characteristic;

identifying the certain location based on a location of the user;

accessing a data source comprising an inventory of items in the store, each item in the inventory of items is cross-referenced to a location among the plurality of locations;

identifying candidate items in the inventory of items, the candidate items comprise items in the inventory which are cross-referenced to the certain location and excludes items in the inventory of items which are not cross-referenced to the certain location, and the candidate items are cross-referenced to image data;

performing a comparison which compares the visual characteristic of the particular item and the visual characteristic of the another item to the image data of the candidate items;

identifying one of the candidate items as a match to the particular item based on the comparison and the known relative position to the another item on the shelf, the one of the candidate items is cross-referenced to a parameter in the inventory;

obtaining a determination as to whether or not the particular item is recommended for the user based on a comparison of the parameter to a preference associated with the user; and

informing the user as to whether or not the particular item is recommended for the user based on the determination.

2. The method of claim 1, wherein:

the plurality of locations comprises a plurality of aisles in the store;

the certain location of the user comprises a certain aisle in the store; and

each item in the inventory of items is cross-referenced to an aisle among the plurality of aisles.

3. The method of claim 1, wherein:

the camera comprises a front-facing camera of a head-mounted display device.

4. The method of claim 1, wherein:

the informing the user comprises causing an augmented reality projection system to project an augmented reality image to the user, based on the determination.

5. The method of claim 1, wherein:

the particular item comprises a food item; and

the parameter of the one of the candidate items comprises a nutritional parameter and the preference associated with the user comprises a nutritional preference.

6. A display device, comprising:

an augmented reality projection system;

a camera which receives image data of a scene, the scene comprises a particular item which is arranged in a known relative position to another item in a store, the particular item has a visual characteristic and is arranged in a certain location among a plurality of locations in the store, and the another item has a visual characteristic;

a location-detection system which determines a location of the display device; and

a control circuit in communication with the augmented reality projection system and the camera, the control circuit is configured to;

communicate the image data and the location of the display device to a server and, in response, receive from the server a determination as to whether or not the particular item is recommended for a user, the determination is based on a correspondence between a preference associated with the user and a parameter of a candidate item in an inventory of items in the store which is a best match to the visual characteristic of the particular item, wherein the best match is based on;

(a) a comparison which compares the visual characteristic of the particular item and the visual characteristic of the another item to image data of the inventory of items, and (b) the known relative position to the another item in the store; and

cause the augmented reality projection system of the display device to project an augmented reality image to the user, based on the determination.

7. ) The display device of claim 6, wherein:

in the inventory, the candidate item is cross-referenced to the certain location.

8. The display device of claim 6, wherein:

the plurality of locations comprises a plurality of aisles in the store;

the certain location comprises a certain aisle in the store; and

each item in the inventory of items is cross-referenced to an aisle among the plurality of aisles.

9. The display device of claim 6, wherein:

the candidate items are cross-referenced to the image data.

10. The display device of claim 6, wherein:

the particular item in the scene is a food item; and

the parameter of the candidate item comprises a nutritional parameter and the preference associated with the user comprises a nutritional preference.

11. The display device of claim 6, wherein:

the known relative position of the particular item to the another item comprises a known relative position of the particular item to the another item on a shelf in the store.

12. The display device of claim 6, wherein:

the another item is identified before the identifying the candidate item in the inventory as the match to the particular item.

13. A server, comprising:

a communication interface;

a database comprising an inventory of candidate items in a store, each candidate item in the inventory of candidate items is cross-referenced to an aisle among a plurality of aisles in the store and to image data of the candidate item;

a memory which stores program code; and

a control circuit associated with the communication interface, the database and the memory, the control circuit is configured to execute the program code to;

receive, via the communication interface, from a device of a user, an identification of an aisle in which the device is located and image data of a scene, the image data is from a camera of the device and identifies a visual characteristic of a particular item which is arranged in a certain aisle among a plurality of aisles in a store and a visual characteristic of another item which is arranged in the certain aisle,

identify the certain aisle based on the aisle of the device,

perform a comparison between the visual characteristic of the particular item and the visual characteristic of the another item to the image data of candidate items in the inventory which are cross-referenced to the certain aisle but not to candidate items in the inventory which are not cross-referenced to the certain aisle,

identify a candidate item in the inventory as a match to the another item,

based on the identifying of the candidate item and a known relative position of the another item to the particular item, identify a candidate item in the inventory as a match to the particular item;

obtain a determination as to whether or not the particular item is recommended for the user based on a comparison of a parameter and a preference associated with the user, the parameter is cross-referenced to the candidate item which is the match to the particular item, and

inform the user as to whether or not the particular item is recommended for the user based on the determination.

14. The server of claim 13, wherein the control circuit executes the program code to:

receive, via the communication interface, from the device of a user, an identification of an area in which the user is looking, and

narrow a number of the candidate items used in the comparison by excluding candidate items in the inventory which are outside the area.

15. The server of claim 13, wherein:

the informing the user comprises causing an augmented reality projection system of a head-mounted display device to project an augmented reality image to the user.

16. The server of claim 13, wherein:

the particular item comprises a food item; and

the parameter of the one of the candidate items comprises a nutritional parameter; and

the preference associated with the user comprises a nutritional preference.