Location based object tracking

• US 8,903,430 B2
• Filed: 02/21/2008
• Issued: 12/02/2014
• Est. Priority Date: 02/21/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sands, Ian M., Russ, V. Kevin
• Primary Examiner(s): Brandt, Christopher M
• Assistant Examiner(s): MANOHARAN, MUTHUSWAMY GANAPATHY
• Application Number:
  US12/035,277
• Publication Number:
  US 20090215471A1
• Time in Patent Office: 2,476 Days
• Field of Search: 455/457, 348/14.02
• US Class Current: 455/457
• CPC Class Codes: G01S 13/876 Combination of several spaced transponders or reflectors of known location for determining the position of a receiver G01S13/874 takes precedence
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A user of a mobile device is able to display information about objects in the surrounding environment and to optionally interact with those objects. The information may be displayed as a graphical overlay on top of a real-time display of imagery from a camera in the mobile device with the overlay indexed to the real-time display. The graphical overlay may include positional information about an external object and may include navigational information intended to assist the user in moving to the object'"'"'s location. There may also be a graphical user interface which allows the user to utilize the mobile device to interact with an external object.

Claims

1. A method of locating and interacting with an external object comprising:

obtaining real-time imagery from a mobile camera;

retrieving stored image data and metadata regarding a known environment, the known environment comprising a plurality of known objects;

receiving a signal from a first object, the first object corresponding to a first known object of the plurality of known objects, the signal identifying a position of the first object relative to the mobile camera;

identifying a first stored position of the first known object relative to the plurality of known objects based on the stored image data and metadata;

comparing the real-time imagery to the stored image data and metadata to determine a position of the mobile camera relative to the plurality of known objects;

presenting the real-time imagery on a mobile display associated with the mobile camera;

determining information related to the first object based on the stored image data and metadata regarding the first known object;

presenting information related to the first object as a graphical overlay on the real-time imagery; and

accessing an operational interface of the first object via a user interface comprised in the graphical overlay, wherein accessing the operational interface of the first object comprises at least one of;

sending data, receiving data, and sending commands.

2. The method of claim 1 further comprising:

determining a location of the mobile camera;

determining a heading of the mobile camera; and

using the location and the heading of the mobile camera for indexing the graphical overlay to the real-time imagery on the mobile display.

3. The method of claim 2 further comprising analyzing the real-time imagery to identify a second known object.

4. The method of claim 3 wherein determining the heading comprises calculating a relative bearing to the second known object having a second stored position.

5. The method of claim 3 further comprising:

accessing an operational interface associated with a second object corresponding to the second known object via the user interface comprised in the graphical overlay of the mobile display.

6. The method of claim 5 wherein the user interface is presented to the user in response to the mobile camera location moving to within a predetermined distance from the second object.

7. The method of claim 6 wherein the user interface comprises information about the second known object which is indexed to the image of the second object in the graphical overlay.

8. The method of claim 1, wherein the graphical overlay comprises a link to the user interface for accessing the operational interface of the object.

9. A method of locating and interacting with an object comprising:

obtaining real-time imagery from a mobile camera;

retrieving stored image data and stored metadata regarding a known environment, the known environment comprising a plurality of known objects;

receiving a signal from an object, the object corresponding to a known object of the plurality of known objects, the signal identifying a position of the object relative to the mobile camera;

identifying a stored position of the known object relative to the plurality of known objects based on at least one of;

the stored image data and the stored metadata;

comparing the real-time imagery to the stored image data to determine a position of the mobile camera relative to the plurality of known objects;

presenting the real-time imagery on a mobile display associated with the mobile camera;

determining information related to the object based on at least one of;

the stored image data and the stored metadata regarding the known object;

presenting the information as a graphical overlay on the real-time imagery; and

accessing an operational interface of the object via a user interface comprised in the graphical overlay, wherein accessing the operational interface of the object comprises at least one of;

sending data, receiving data, and sending commands.

10. The method of claim 9 further comprising determining a heading of the mobile camera and using the position and the heading of the mobile camera to index the graphical overlay to the real-time imagery on the mobile display.

11. The method of claim 10 wherein the information comprises navigational information to the object.

12. The method of claim 11 wherein the navigational information comprises a highlighted path indexed to the real-time imagery.

13. The method of claim 10 wherein the graphical data overlay indicates a position of the object.

14. The method of claim 13 wherein the position of the object is indicated by highlighting a container within which the object is located.

15. A system for locating and interacting with external objects, the system comprising:

a mobile camera, having a known position, adapted to;

capture real-time imagery; and

supply the real-time imagery in a digital format;

a processor communicatively coupled to the mobile camera adapted to;

retrieve the real-time imagery;

retrieve stored image data and stored metadata regarding a known environment, the known environment comprising a plurality of known objects;

receive a signal from an object, the object corresponding to a known object of the plurality of known objects, the signal identifying a position of the object relative to the known position of the mobile camera;

identify a stored position of the known object relative to the plurality of known objects based on at least one of;

the stored image data and the stored metadata;

compare the real-time imagery to the stored image data to determine a position of the mobile camera relative to the plurality of known objects; and

determine information related to the object based on the stored image data and the stored metadata regarding the known object;

a video display, physically proximate to the mobile camera and communicatively coupled to the processor, adapted to;

display the real-time imagery in a viewfinder mode; and

display the information related to the object as graphical data overlaid on and indexed to the real-time imagery, wherein the graphical data comprises a link to a user interface for accessing an operational interface of the object, wherein accessing the operational interface of the object comprises at least one of;

sending data, receiving data, and sending commands.

16. The system of claim 15 wherein the graphical data indicates the position of the object.

17. The system of claim 16 wherein the position of the object is indicated by highlighting a container within which the object is located.

18. The system of claim 16 wherein the position of the object is indicated by highlighting a navigable path from the known position of the mobile camera to the position of the object.

19. The system of claim 16 wherein the object is portable and the processor tracks the position of the object as it is moved.

20. The system of claim 15 wherein the user interface is presented to the user in response to the mobile camera location moving to within a predetermined distance from the object.