Change invariant scene recognition by an agent

• US 8,942,917 B2
• Filed: 02/14/2011
• Issued: 01/27/2015
• Est. Priority Date: 02/14/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chrysanthakopoulos, Georgios
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US13/026,300
• Publication Number:
  US 20120209514A1
• Time in Patent Office: 1,443 Days
• Field of Search: 701/209
• US Class Current: 701/410
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A technology can be provided for change invariant scene recognition of an environment by an agent that uses image sensors to capture images. An example method can include grouping the images into situation image sets to form an image database. A navigation image can be captured by the agent at the agent'"'"'s current location. Another operation can be searching through images by separately loaded situation image set to identify images that are a probable match for the navigation image. Error values can be computed between the navigation image and situation image sets. A probability can be calculated that situation image sets are to be used as a relevant situation image set based on the error values. Then the relevant situation image set can be searched for matching purposes with the navigation image while the error value is below a defined threshold.

Claims

1. A system for change invariant scene recognition of an environment by an agent, the system comprising:

an image camera configured to use image sensors to capture images for the agent;

a grouping module configured to group the images into situation image sets that are identified as being at locations and edges of a navigation graph;

an image database configured to contain the situation image sets;

a query module configured to query the image database by comparing a first image signature of a navigation image with second image signatures of the situation image sets, wherein the situation image sets are loaded as separate sets from the image database;

a probability module configured to identify a selected situation image set based on a probability that the selected situation image set has an individual image that matches the navigation image;

a feature matching module configured to identify one or more matching images in the selected situation image set that match the navigation image and to provide a navigation location for the agent based on the one or more matching images; and

at least one processing device configured to execute the grouping module, the query module, the probability module, and the feature matching module.

2. The system as in claim 1, wherein the images are grayscale depth images or visual images.

3. The system as in claim 1, wherein the first image signature and the second image signatures are global image signatures.

4. The system as in claim 1, wherein the agent is a robotic agent having a navigation module configured to navigate from the navigation location to another navigation location.

5. The system as in claim 1, further comprising a pruning module configured to remove some situation image sets from the image database when the some situation image sets have not been used in navigation for a period of time, wherein the at least one processing device is also configured to execute the pruning module.

6. The system as in claim 1, wherein the probability module is further configured to:

determine the probability that the selected situation image set has the individual image that matches the navigation image; and

determine other probabilities that other situation image sets have other individual images that match the navigation image,

wherein the probability for the selected situation image set is higher than the other probabilities for the other situation image sets.

7. A method implemented by one or more computer processing devices, the method comprising:

obtaining images for an agent, the images being captured with a camera;

grouping the images into situation image sets and storing the situation image sets in an image database;

querying the image database for first image signatures of the situation image sets and comparing the first image signatures to a second image signature of a navigation image;

based on the comparing, determining probabilities that individual situation image sets of the situation image sets have individual images that match the navigation image;

identifying a selected situation image set based on the probabilities;

identifying one or more matching images in the selected situation image set that match the navigation image; and

providing a navigation location for the agent based on the one or more matching images.

8. The method as in claim 7, wherein the situation image sets are associated with corresponding locations and edges of a navigation graph.

9. The method as in claim 7, wherein the situation image sets are associated with corresponding locations.

10. The method as in claim 7, wherein:

the grouping further comprises grouping the images within the situation image sets into subsets of the situation image sets based on a time of day or based on weather conditions; and

the determining the probabilities further comprises determining the probabilities for the subsets.

11. The method as in claim 7, wherein the image database is a relational database or an object-oriented database.

12. The method as in claim 7, wherein the determining the probabilities further comprises computing error values between the first image signatures of the situation image sets and the second image signature of the navigation image and basing the probabilities on the error values.

13. The method as in claim 12, the method further comprising excluding some situation image sets having corresponding error values above a predetermined threshold when identifying the selected situation image set.

14. The method as in claim 7, the method further comprising iteratively performing the querying, the determining, and the identifying the selected situation image set until the one or more matching images are found.

15. One or more computer readable memory devices or storage devices storing computer readable instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform acts comprising:

obtaining images for an agent, wherein the images were captured with an image camera using one or more image sensors;

grouping the images into situation image sets corresponding to locations of a navigation graph and storing the situation image sets in an image database;

comparing situation image signatures of the situation image sets to one or more navigation image signatures of a navigation image; and

identifying a selected situation image set based on a corresponding probability that the selected situation image set has an individual image that matches the navigation image.

16. The one or more computer readable memory devices or storage devices as in claim 15, wherein the comparing further comprises comparing a global navigation image signature associated with the navigation image to global situation image signatures associated with the situation image sets.

17. The one or more computer readable memory devices or storage devices as in claim 16, wherein the comparing the global navigation image signature and the global situation image signatures produces a subset of the situation image sets for further comparison.

18. The one or more computer readable memory devices or storage devices as in claim 17, the acts further comprising performing the further comparison by comparing higher granularity features associated with the navigation image to corresponding higher granularity features associated with other images in the subset of the situation image sets.

19. The one or more computer readable memory devices or storage devices as in claim 15, the acts further comprising:

identifying one or more matching images in the selected situation image set that match the navigation image; and

providing a navigation location for the agent based on the one or more matching images.

20. The one or more computer readable memory devices or storage devices as in claim 15, the acts further comprising:

computing error values between the situation image signatures of the situation image sets and the one or more navigation image signatures of the navigation image; and

determining a set of probabilities based on the error values, the set of probabilities including the corresponding probability,

wherein the identifying the selected situation image set further comprises comparing the corresponding probability to other probabilities from the set of probabilities.