Personal audio/visual apparatus providing resource management

• US 9,606,992 B2
• Filed: 06/27/2012
• Issued: 03/28/2017
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Kohler, Jeffrey A., McCulloch, Daniel J.
• Primary Examiner(s): Faulk, Devona
• Assistant Examiner(s): Beard, Charles L
• Application Number:
  US13/535,074
• Publication Number:
  US 20130083064A1
• Time in Patent Office: 1,735 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players

Abstract

Technology is described for resource management based on data including image data of a resource captured by at least one capture device of at least one personal audiovisual (A/V) apparatus including a near-eye, augmented reality (AR) display. A resource is automatically identified from image data captured by at least one capture device of at least one personal A/V apparatus and object reference data. A location in which the resource is situated and a 3D space position or volume of the resource in the location is tracked. A property of the resource is also determined from the image data and tracked. A function of a resource may also be stored for determining whether the resource is usable for a task. Responsive to notification criteria for the resource being satisfied, image data related to the resource is displayed on the near-eye AR display.

Claims

1. A method for providing resource management using one or more personal audiovisual (A/V) apparatus including a near-eye, augmented reality (AR) display comprising:

automatically identifying a resource based on image data of the resource captured by at least one capture device of at least one personal A/V apparatus and object reference data;

automatically tracking a three dimensional (3D) space position of the resource in a location identified based on location data detected by the at least one personal A/V apparatus;

automatically determining a property of the resource based on the image data of the resource;

automatically tracking the property of the resource;

automatically causing display of image data related to the resource in the near-eye, augmented reality display based on a notification criteria for the property associated with the resource;

causing the near-eye, AR display to display discrepancy data of item descriptions and pricing information of a purchase list compared to store purchase data; and

performing a task, wherein the task is an eating monitoring application, such that at least one user physical action of at least one body part with respect to the resource is automatically identified within an approximation to a mouth of a wearer of the near-eye, AR display, and the identified resource is determined to be moving to and from the mouth so as to recognize at least a reduction in an amount of the resource.

2. The method of claim 1 wherein the property associated with the resource comprises at least one of the following:

a quantity;

an expiration date;

a physical damage indicator;

a quality control indicator; and

a nutritional value.

3. The method of claim 1 further comprising:

generating and storing a three dimensional (3D) virtual object version of the resource in a memory based on the image data of the resource.

4. The method of claim 1 wherein automatically tracking the 3D space position of the resource in the location identified based on location data detected by the at least one personal A/V apparatus further comprises:

automatically identifying the location in which the resource is situated;

identifying a 3D space position of the resource in a three dimensional (3D) mapping for the location; and

storing the 3D space position and the location in a resource data record linked with the resource in a memory.

5. The method of claim 4 further comprising:

receiving user input indicating a request for finding the resource;

identifying the personal A/V apparatus is in the location of the resource;

automatically determining directions from a reference point in a display field of view to the 3D space position of the resource based on the 3D mapping for the location; and

electronically outputting the directions to the 3D space position of the resource by the personal A/V apparatus using at least one of visual data or audio data.

6. The method of claim 4 further comprising:

receiving user input indicating a request for finding the resource;

identifying the personal A/V apparatus is not in the location;

displaying image data representing the resource and any object around the resource as virtual 3D objects in the near-eye, AR display of the personal A/V apparatus; and

responsive to any object blocking the resource in the display field of view, displaying an animation of any 3D virtual object representing any object blocking the resource being removed from its respective 3D space position blocking the virtual object representing the resource.

7. The method of claim 1 further comprising performing a task which further comprises:

automatically identifying natural user interface (NUI) input indicating at least one user physical action of at least one body part with respect to the resource based on the image data captured by the at least one capture device of the at least one personal A/V apparatus;

updating the task data based on the at least one user physical action.

8. The method of claim 7 wherein the task is a budgeting application and automatically identifying the NUI input indicating at least one user physical action of at least one body part with respect to the resource further comprises automatically identifying a gesture of placing the resource in a shopping cart;

and

causing the near-eye, augmented reality display to display the task data further comprises causing the near-eye, augmented reality display to display data identifying a cost of the resource and a budget amount.

9. The method of claim 7 wherein the resource is a food resource and automatically identifying the NUI input indicating at least one user physical action of at least one body part with respect to the resource further comprises:

automatically identifying a first gesture of directing a holder of the food resource into a predetermined space relative to a front of the near-eye, AR display in a direction approximating an intersection with an approximated position of the mouth of the wearer wearing the near-eye, AR display and exiting the space while maintaining the direction;

automatically identifying a second gesture of the holder re-appearing in image data in the space moving in a direction away from the approximated position of the mouth of the wearer; and

causing the near-eye, augmented reality display to display the task data further comprises causing the near-eye, augmented reality display to display data identifying a nutritional value associated with the food resource.

10. A personal audiovisual (A/V) apparatus including a near-eye, augmented reality display for providing resource management comprising:

the near-eye augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors being communicatively coupled to the near-eye, augmented reality display for controlling the display;

the one or more processors being communicatively coupled to at least one capture device on the near-eye support structure for receiving image data captured by the at least one capture device of a location in which the support structure is situated;

the one or more processors automatically identifying a real object in the image data based on object reference data stored in an accessible memory;

the one or more processors identifying the real object as a resource falling within a category of a resource profile based on a type of object determined for the real object and associating a shared resource data record for the resource with the resource profile in data stored in a memory; and

the one or more processors identifying and storing in the shared resource data record a three dimensional (3D) space position of the resource in the location based on the image data and a 3D mapping of the location, and automatically tracking a change in the 3D space position of the resource and position of a volume of the resource in the location and updating the shared resource data record to identify the location in which the resource is located and the position of a volume occupied by the resource;

the one or more processors identifying a sought after identified resource selected by a user; and

the one or more processors being responsive to any object blocking the selected sought after resource in the display field of view, cause an automatic displaying of visual data of an indicator to identify a location of the selected sought after resource being blocked by the object.

11. The apparatus of claim 10 wherein the one or more processors identifying and storing in the resource data record a three dimensional (3D) space position of the resource in the location based on the image data and a 3D mapping of the location further comprises:

the one or more processors identifying and storing in the resource data record a three dimensional (3D) space position of the resource in the location based on depth data captured by the at least one capture device on the near-eye support structure.

12. The apparatus of claim 10 further comprising:

the one or more processors of the personal A/V apparatus associating one or more authorized users with the resource data record and storing notification criteria for notifying a user when at least one of the following occurs;

an unauthorized use of the resource, and

an unauthorized relocation of the resource.

13. The apparatus of claim 10 further comprising:

the one or more processors being communicatively coupled to a second personal A/V apparatus and updating the location and 3D space position of the resource in the location based on image data of the resource received from the second personal A/V apparatus.

14. The apparatus of claim 10 further comprising:

the one or more processors determining a function of the resource based on a function associated with a type of object of the resource in the accessible object reference data.

15. The apparatus of claim 14 further comprising:

the one or more processors being communicatively coupled to a network accessible computer system for accessing a network accessible copy of the resource data record; and

the one or more processors identifying a state of a property for the resource based on the network accessible copy of the resource data record.

16. The apparatus of claim 10 further comprising:

a natural user interface comprising an eye tracking system being positioned by the near-eye support structure, and eye tracking software being executed by the one or more processors for identifying a user physical action of an eye based action based on eye data from the eye tracking system indicating selection of an object in the display field of view.

17. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing resource management using one or more personal audiovisual (A/V) apparatus including a near-eye, augmented reality display, the method comprising:

automatically identifying a food resource based on image data of the food resource captured by at least one capture device of at least one personal A/V apparatus and object reference data;

automatically tracking a 3D space position of the food resource in a location identified based on location data detected by the at least one personal A/V apparatus;

automatically determining a property of the food resource based on the image data of the food resource;

automatically identifying a function associated with the food resource based on the object reference data;

storing data identifying the location of the food resource, the property of the food resource and the function associated with the food resource in a resource data record associated with the food resource, the location, property and function forming the basis upon which one or more tasks related to the food resource determine actions to perform and task data to output, the actions to perform based on a user of the near-eye augmented reality display identifying a set of food resources;

identifying one or more recipes to be made using the identified set of food resources;

automatically outputting the one or more recipes to the augmented reality display;

determining actions to perform based on the user of the near-eye augmented reality display identifying at least one of the one or more recipes;

storing the one or more selected recipes in the resource data record; and

associating the resource data record with a resource profile associated with a user of the personal A/V apparatus.

18. The one or more processor readable storage devices of claim 17 wherein the method further comprises:

storing image data of the resource in the resource data record associated with the resource.

19. The one or more processor readable storage devices of claim 17 further comprising:

determining a resource check has been triggered for a task using resources;

identifying resource criteria for the task;

identifying any resource associated with the user satisfying the resource criteria for the task based on one or more resource data records associated with the user;

identifying any resource criteria for the task which is not satisfied by any resource associated with the user based on the one or more resource data records associated with the user; and

causing to be displayed a notification identifying any resource associated with the user satisfying the resource criteria and any resource criteria for the task which is not satisfied.

20. The one or more processor readable storage devices of claim 19 wherein the resource criteria for the task comprises at least one of the following:

one or more functions to be performed by one or more resources;

one or more properties of one or more resources; and

one or more identifiers of one or more resources.