Predicate Logic based Image Grammars for Complex Visual Pattern Recognition

• US 20100278420A1
• Filed: 03/16/2010
• Published: 11/04/2010
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Siemens Corp. (Siemens AG)
• Original Assignee: Siemens Corp. (Siemens AG)
• Inventors: Singh, Maneesh Kumar, Neumann, Jan, Suarez, Roberto Antonio, Masticola, Stephen P., Gall, Michael A., Parag, Toufiq, Ramesh, Visvanathan, Bahlmann, Claus, Shet, Vinay Damodar
• Granted Patent: US 8,548,231 B2
• Time in Patent Office: Days
• US Class Current: 382/156
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

First order predicate logics are provided, extended with a bilattice based uncertainty handling formalism, as a means of formally encoding pattern grmmars, to parse a set of image features, and detect the presence of different patterns of interest implemented on a processor. Information from different sources and uncertainties from detections, are integrated within the bilattice framework. Automated logical rule weight learning in the computer vision domain applies a rule weight optimization method which casts the instantiated inference tree as a knowledge-based neural network, to converge upon a set of rule weights that give optimal performance within the bilattice framework. Applications are in (a) detecting the presence of humans under partial occlusions and (b) detecting large complex man made structures in satellite imagery (c) detection of spatio-temporal human and vehicular activities in video and (c) parsing of Graphical User Interfaces.

Claims

1-8. -8. (canceled)

9. A method for a detection of a visual pattern from image data of a scene, comprising:

obtaining image data of the scene by a processor;

processing image data with one or more detectors implemented on the processor to detect one or more features;

defining a pattern grammar to describe the visual pattern by encoding knowledge about contextual cues, scene geometry and visual pattern constraints;

encoding the pattern grammar using a plurality of first order predicate logic based rules;

executing by the processor the plurality of first order predicate logic based rules to detect the visual pattern based on a presence or an absence of the one or more features; and

generating by the processor data related to;

a justification that the plurality of first order predicate logic based rules detected the visual pattern;

a location and time stamp in the scene where the visual pattern occurs, and

a measure of uncertainty related to the detection of the visual pattern.

10. The method as claimed in claim 9, wherein the image data is video data.

11. The method as claimed in claim 9, wherein the measure of uncertainty is associated with each one of the plurality of first order predicate logic rules.

12. The method as claimed in claim 9, further comprising a set of composition rules which encodes the visual pattern as a hierarchy of parts.

13. The method as claimed in claim 9, further comprising encoding a geometric layout of the visual pattern with one or more embodiment rules.

14. The method as claimed in claim 9, further comprising encoding a surrounding of the pattern with one or more context rules.

15. The method as claimed in claim 9, wherein a formal meta grammar describes the pattern grammar.

16. The method as claimed in claim 9, further comprising:

the processor optimizing the measure of uncertainty associated with the plurality of first order predicate logic based rules, wherein an instantiated proof tree generated by the logic program is applied to implement a knowledge based artificial neural network.

17. The method as claimed in claim 16, wherein a rule weight in the plurality of first order predicate logic based rules is represented as a link weight in the knowledge based artificial neural network.

18. The method as claimed in claim 17, further comprising the processor using a modified version of a standard back propagation algorithm to optimize link weights in the knowledge based artificial neural network.

19. The method as claimed in claim 9, wherein the visual pattern represents a hum

20. The method as claimed in claim 9, wherein the visual pattern represents a man made structure.

21. The method as claimed in claim 20, wherein the man made structure is a surface to air missile site.

22. The method as claimed in claim 9, wherein the visual pattern represents at least a part of a graphical user interface (GUI).

23. The method as claimed in claim 9, wherein the visual pattern represents a human activity.

24. The method as claimed in claim 9, wherein the visual pattern represents a vehicular activity in video data.

25. The method as claimed in claim 9, wherein the visual pattern represents a human and vehicular activity as observed in video data.

26. A system for a detection of a visual pattern from image data of scene, comprising:

a memory to store and retrieve data and instructions;

a processor that accesses the memory to execute instructions to;

obtain image data of the scene;

process image data with one or more detectors to detect one or more features;

define a pattern grammar to describe the visual pattern in a language that applies the detected one or more features;

encode a pattern grammar that describes the visual pattern related to the detected one or more features in a plurality of first order predicate logic based rules;

execute the plurality of first order predicate logic based rules to detect the visual pattern based on a presence or an absence of the one or more features; and

generate data related to;

a justification that the plurality of first order predicate logic based rules detected the visual pattern;

a location and time stamp in the scene where the visual pattern occurs, and

a measure of uncertainty related to the detection of the visual pattern.

27. The system as claimed in claim 26, wherein the visual pattern represents a human.

28. The system as claimed in claim 26, wherein the visual pattern represents a man made structure.