PERSONAL AUDIO/VISUAL APPARATUS PROVIDING RESOURCE MANAGEMENT

• US 20130083064A1
• Filed: 06/27/2012
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, McCulloch, Daniel J., Kohler, Jeffrey A.
• Granted Patent: US 9,606,992 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players

Abstract

Technology is described for resource management based on data including image data of a resource captured by at least one capture device of at least one personal audiovisual (A/V) apparatus including a near-eye, augmented reality (AR) display. A resource is automatically identified from image data captured by at least one capture device of at least one personal A/V apparatus and object reference data. A location in which the resource is situated and a 3D space position or volume of the resource in the location is tracked. A property of the resource is also determined from the image data and tracked. A function of a resource may also be stored for determining whether the resource is usable for a task. Responsive to notification criteria for the resource being satisfied, image data related to the resource is displayed on the near-eye AR display.

Claims

1. A method for providing resource management using one or more personal audiovisual (A/V) apparatus including a near-eye, augmented reality (AR) display comprising:

automatically identifying a resource based on image data of the resource captured by at least one capture device of at least one personal A/V apparatus and object reference data;

automatically tracking a three dimensional (3D) space position of the resource in a location identified based on location data detected by the at least one personal A/V apparatus;

automatically determining a property of the resource based on the image data of the resource;

automatically tracking the property of the resource; and

automatically causing display of image data related to the resource in the near-eye, augmented reality display based on a notification criteria for the property associated with the resource.

2. The method of claim 1 wherein the property associated with the resource comprises at least one of the following:

a quantity;

an expiration date;

a physical damage indicator;

a quality control indicator; and

a nutritional value.

3. The method of claim 1 further comprising:

generating and storing a three dimensional (3D) virtual object version of the resource in a memory based on the image data of the resource.

4. The method of claim 1 wherein automatically tracking the 3D space position of the resource in the location identified based on location data detected by the at least one personal A/V apparatus further comprises:

automatically identifying the location in which the resource is situated;

identifying a 3D space position of the resource in a three dimensional (3D) mapping for the location; and

storing the 3D space position and the location in a resource data record linked with the resource in a memory.

5. The method of claim 4 further comprising:

receiving user input indicating a request for finding the resource;

identifying the personal A/V apparatus is in the location of the resource;

automatically determining directions from a reference point in a display field of view to the 3D space position of the resource based on the 3D mapping for the location; and

electronically outputting the directions to the 3D space position of the resource by the personal A/V apparatus using at least one of visual data or audio data.

6. The method of claim 4 further comprising:

receiving user input indicating a request for finding the resource;

identifying the personal A/V apparatus is not in the location;

displaying image data representing the resource and any object around the resource as virtual 3D objects in the near-eye, AR display of the personal A/V apparatus; and

responsive to any object blocking the resource in the display field of view, displaying an animation of any 3D virtual object representing any object blocking the resource being removed from its respective 3D space position blocking the virtual object representing the resource.

7. The method of claim 1 further comprising performing a task which further comprises:

automatically identifying natural user interface (NUI) input indicating at least one user physical action of at least one body part with respect to the resource based on the image data captured by the at least one capture device of the at least one personal A/V apparatus;

updating task data based on the at least one user physical action; and

causing the near-eye, augmented reality display to display the task data.

8. The method of claim 7 wherein the task is a budgeting application and automatically identifying the NUI input indicating at least one user physical action of at least one body part with respect to the resource further comprises automatically identifying a gesture of placing the resource in a shopping cart;

and

causing the near-eye, augmented reality display to display the task data further comprises causing the near-eye, augmented reality display to display data identifying a cost of the resource and a budget amount.

9. The method of claim 7 wherein the task is an eating monitoring application and automatically identifying the NUI input indicating at least one user physical action of at least one body part with respect to the resource further comprises

automatically identifying a first gesture of directing a holder of a food resource into a predetermined space relative to a front of the near-eye, AR display in a direction approximating an intersection with an approximated position of a mouth of a user wearing the near-eye, AR display and exiting the space while maintaining the direction;

automatically identifying a second gesture of the holder re-appearing in image data in the space moving in a direction away from the approximated position of the mouth of the user with at least one of a reduction in size of the resource or a disappearance of the resource; and

causing the near-eye, augmented reality display to display the task data further comprises causing the near-eye, augmented reality display to display data identifying a nutritional value associated with the food resource.

10. A personal audiovisual (A/V) apparatus including a near-eye, augmented reality display for providing resource management comprising:

the near-eye augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors being communicatively coupled to the near-eye, augmented reality display for controlling the display;

the one or more processors being communicatively coupled to at least one capture device on the near-eye support structure for receiving image data captured by the at least one capture device of a location in which the support structure is situated;

the one or more processors automatically identifying a real object in the image data based on object reference data stored in an accessible memory;

the one or more processors identifying the real object as a resource falling within a category of a resource profile based on a type of object determined for the real object and associating a resource data record for the resource with the resource profile in data stored in a memory; and

the one or more processors identifying and storing in the resource data record a three dimensional (3D) space position of the resource in the location based on the image data and a 3D mapping of the location.

11. The apparatus of claim 10 wherein the one or more processors identifying and storing in the resource data record a three dimensional (3D) space position of the resource in the location based on the image data and a 3D mapping of the location further comprises

the one or more processors identifying and storing in the resource data record a three dimensional (3D) space position of the resource in the location based on depth data captured by the at least one capture device on the near-eye support structure.

12. The apparatus of claim 10 further comprising:

the one or more processors of the personal A/V apparatus associating one or more authorized users with the resource data record and storing notification criteria for notifying a user when at least one of the following occurs;

an unauthorized use of the resource, and

an unauthorized relocation of the resource.

13. The apparatus of claim 10 further comprising:

the one or more processors being communicatively coupled to a second personal A/V apparatus 8 and updating the location and 3D space position of the resource in the location based on image data of the resource received from the second personal A/V apparatus 8.

14. The apparatus of claim 10 further comprising

the one or more processors determining a function of the resource based on a function associated with a type of object of the resource in the accessible object reference data.

15. The apparatus of claim 14 further comprising:

the one or more processors being communicatively coupled to a network accessible computer system for accessing a network accessible copy of the resource data record; and

the one or more processors identifying a state of a property for the resource based on the network accessible copy of the resource data record.

16. The apparatus of claim 10 further comprising:

a natural user interface comprising

an eye tracking system being positioned by the near-eye support structure, and eye tracking software being executed by the one or more processors for identifying a user physical action of an eye based action based on eye data from the eye tracking system indicating selection of an object in the display field of view.

19. The one or more processor readable storage devices of claim 16 further comprising:

determining a resource check for a task using resources has been triggered;

identifying resource criteria for the task;

identifying any resource associated with the user satisfying the resource criteria for the task based on one or more resource data records associated with the user;

identifying any resource criteria for the task which is not satisfied by any resource associated with the user based on the one or more resource data records associated with the user; and

causing to be displayed a notification identifying any resource associated with the user satisfying the resource criteria and any resource criteria for the task which is not satisfied.

20. The one or more processor readable storage devices of claim 19 wherein the resource criteria for the task comprises at least one of the following:

one or more functions to be performed by one or more resources;

one or more properties of one or more resources; and

one or more identifiers of one or more resources.

17. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing resource management using one or more personal audiovisual (A/V) apparatus including a near-eye, augmented reality display, the method comprising:

automatically identifying a resource based on image data of the resource captured by at least one capture device of at least one personal A/V apparatus and object reference data;

automatically tracking a 3D space position of the resource in a location identified based on location data detected by the at least one personal A/V apparatus;

automatically determining a property of the resource based on the image data of the resource;

automatically identifying a function associated with the resource based on the object reference data;

storing data identifying the location of the resource, the property of the resource and the function associated with the resource in a resource data record associated with the resource; and

associating the resource data record with a resource profile associated with a user of the personal A/V apparatus.

18. The one or more processor readable storage devices of claim 17 wherein the method further comprises:

storing image data of the resource in the resource data record associated with the resource.