Method and system for directed area search using cognitive swarm vision and cognitive Bayesian reasoning

• US 8,213,709 B1
• Filed: 11/03/2009
• Issued: 07/03/2012
• Est. Priority Date: 11/03/2009
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Medasani, Swarup, Owechko, Yuri, Khosla, Deepak, Allen, David L., Lu, Tsai-Ching
• Primary Examiner(s): Liew, Alex
• Application Number:
  US12/590,110
• Time in Patent Office: 973 Days
• Field of Search: 382/100, 382155-161, 382181-231
• US Class Current: 382/155
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A method and system for a directed area search using cognitive swarm vision and cognitive Bayesian reasoning is disclosed. The system comprises a domain knowledge database, a top-down reasoning module, and a bottom-up module. The domain knowledge database is configured to store Bayesian network models comprising visual features and observables associated with various sets of entities. The top-down module is configured to receive a search goal, generate a plan of action using Bayesian network models, and partition the plan into a set of tasks/observables to be located in the imagery. The bottom-up module is configured to select relevant feature/attention models for the observables, and search the visual imagery using a cognitive swarm for the at least one observable. The system further provides for operator feedback and updating of the domain knowledge database to perform better future searches.

Claims

1. A system for directed area search in visual imagery, comprising:

a domain knowledge database configured to store Bayesian network models comprising visual features and observables associated with various sets of entities, where the observables are associated with feature/attention models for locating the observables in visual imagery;

a top-down module configured to;

receive a search goal comprising a set of entities to be located in an image;

generate a plan of action for locating the set of entities by combining the Bayesian network models relating to the set of entities to be located according to joint probability distributions to yield a combined Bayesian network model, the combined Bayesian network model describing visual features associated with the set of entities, where the Bayesian network models and associated visual features are obtained from the domain knowledge database; and

partition the plan into a set of tasks, each task comprising at least one observable to be located in the visual imagery; and

a bottom-up module configured to;

select a relevant feature/attention model for each observable from the domain knowledge database;

search the visual imagery for the at least one observable using the selected feature/attention model; and

output search results as a set of candidate entities.

2. The system of claim 1, further comprising a user interface configured to present the set of candidate entities to an operator for validation of the set of entities.

3. The system of claim 2, wherein the top-down module is further configured to receive feedback from the operator via the user interface regarding the search results and to update the domain knowledge database based on the feedback from the operator.

4. The system of claim 3, further configured to generate a new plan of action using the updated domain knowledge database, and to iteratively repeat the search until the search goal is accomplished.

5. The system of claim 4, wherein the bottom-up module is further configured to send a report of the search results to the top-down module, the top-down module is further configured to assign a validation score to the search results based on performance accuracy of the bottom-up module in executing the set of tasks, and the validation score is then used to refine the classifier and feature/attention model used for that particular task.

6. The system of claim 5, further configured to iteratively repeat the search using the refined feature/attention model until the search goal is accomplished.

7. The system of claim 6, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, the search is performed using unbiased, center-surround type saliency algorithms.

8. The system of claim 7, wherein the bottom up module is configured to search the visual imagery using a cognitive swarm.

9. The system of claim 1, wherein the bottom-up module is further configured to send a report of the search results to the top-down module, the top-down module is further configured to assign a validation score to the search results based on performance accuracy of the bottom-up module in executing the set of tasks, and the validation score is then used to refine the feature/attention model used for that particular task.

10. The system of claim 9, further configured to iteratively repeat the search using the refined feature/attention model until the search goal is accomplished.

11. The method of claim 1, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, the search is performed using unbiased center surround type saliency algorithms.

12. A method for directed area search in visual imagery comprising acts of:

receiving a search goal comprising a set of entities to be located in an image;

generating a plan of action for locating the set of entities by combining Bayesian network models relating to the set of entities to be located according to joint probability distributions to yield a combined Bayesian network model, the combined Bayesian network model describing visual features associated with the set of entities, where the Bayesian network models and associated visual features are obtained from the domain knowledge database; and

partitioning the plan into a set of tasks, each task comprising at least one observable to be located in the visual imagery;

selecting a relevant feature/attention model for each observable from the domain knowledge database;

searching the visual imagery for the at least one observable using the selected feature/attention model; and

outputting search results as a set of candidate entities.

13. The method of claim 12, further comprising an act of presenting the set of candidate entities to an operator for validation of the set of entities.

14. The method of claim 13, further comprising an act of receiving feedback from the operator regarding the search results and updating the domain knowledge database based on the feedback from the operator.

15. The method of claim 14, further comprising an act of generating a new plan of action using the updated domain knowledge database, and iteratively repeating the search until the search goal is accomplished.

16. The method of claim 15, further comprising an act of assigning a validation score to the search results based on accuracy of the results of the executed set of tasks, and then using the validation score to refine the feature/attention models used for that particular set of tasks.

17. The method of claim 16, further comprising an act of iteratively repeating the search using the refined classifier and feature/attention model until the search goal is accomplished.

18. The method of claim 17, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, the act of searching is performed using unbiased, center-surround type saliency algorithms.

19. The method of claim 18, wherein in the act of searching the visual imagery, the search is done using a cognitive swarm.

20. The method of claim 12, further comprising an act of assigning a validation score to the search results based on accuracy of the results of the executed set of tasks, and then using the validation score to refine the classifier and feature/attention models used for that particular set of tasks.

21. The method of claim 20, further comprising an act of iteratively repeating the search using the refined classifier and feature/attention model until the search goal is accomplished.

22. The method of claim 12, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, the act of searching is performed using unbiased center surround type saliency algorithms.

23. A computer program product for directed area search in visual imagery, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving a search goal comprising a set of entities to be located in an image;

generating a plan of action for locating the set of entities by combining Bayesian network models relating to the set of entities to be located according to joint probability distributions to yield a combined Bayesian network model, the combined Bayesian network model describing visual features associated with the set of entities, where the Bayesian network models and associated visual features are obtained from the domain knowledge database; and

partitioning the plan into a set of tasks, each task comprising at least one observable to be located in the visual imagery;

selecting a relevant feature/attention model for each observable from the domain knowledge database;

searching the visual imagery for the at least one observable using the selected feature/attention model; and

outputting search results as a set of candidate entities.

24. The computer program product of claim 23, further comprising computer-readable instruction means for presenting the set of candidate entities to an operator for validation of the set of entities.

25. The computer program product of claim 24, further comprising computer-readable instruction means for receiving feedback from the operator regarding the search results and updating the domain knowledge database based on the feedback from the operator.

26. The computer program product of claim 25, further comprising computer-readable instruction means for generating a pew plan of action using the updated domain knowledge database, and iteratively repeating the search until the search goal is accomplished.

27. The computer program product of claim 26, further comprising computer-readable instruction means for assigning a validation score to the search results based on accuracy of the results of the executed set of tasks, and then using the validation score to refine the classifier and feature/attention models used for that particular set of tasks.

28. The computer program product of claim 27, further comprising computer-readable instruction means for iteratively repeating the search using the refined classifier and feature/attention model until the search goal is accomplished.

29. The computer program product of claim 28, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, searching is performed using unbiased, center-surround type saliency algorithms.

30. The computer program product of claim 29, further comprising computer-readable instruction means to search the visual imagery using a cognitive swarm.

31. The computer program product of claim 23, further comprising computer-readable instruction means for assigning a validation score to the search results based on accuracy of the results of the executed set of tasks, and then using the validation score to refine the classifier and feature/attention models used for that particular set of tasks.

32. The computer program product of claim 31, further comprising computer-readable instruction means for iteratively repeating the search using the refined classifier and feature/attention model until the search goal is accomplished.

33. The computer program product of claim 23, wherein in a case where the domain knowledge database contains no feature/attention models relating to the search goal, searching is performed using unbiased center surround type saliency algorithms.