Representation of activity in images using geospatial temporal graphs

• US 9,959,647 B1
• Filed: 09/08/2015
• Issued: 05/01/2018
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Inventors: Brost, Randolph, McLendon, III, William C., Parekh, Ojas D., Rintoul, Mark Daniel, Watson, Jean-Paul, Strip, David R., Diegert, Carl
• Primary Examiner(s): Lee, Jonathan S
• Application Number:
  US14/848,165
• Time in Patent Office: 966 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10036 Multispectral image; Hyperspectral image
• CPC Class Codes: G06T 2207/10044 Radar image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

Various technologies pertaining to modeling patterns of activity observed in remote sensing images using geospatial-temporal graphs are described herein. Graphs are constructed by representing objects in remote sensing images as nodes, and connecting nodes with undirected edges representing either distance or adjacency relationships between objects and directed edges representing changes in time. Activity patterns may be discerned from the graphs by coding nodes representing persistent objects like buildings differently from nodes representing ephemeral objects like vehicles, and examining the geospatial-temporal relationships of ephemeral nodes within the graph.

Claims

1. A computing device, comprising:

at least one processor; and

memory that comprises instructions that, when executed by the processor, cause the processor to perform acts comprising;

assigning classifications to respective portions of a first remote sensing image of a scene captured at a first time and a second remote sensing image of the scene captured at a second time that is subsequent to the first time, the portions representative of objects in the scene captured in the first remote sensing image and the second remote sensing image;

based upon the classifications assigned to the respective portions, constructing in the memory a geospatial temporal (GST) graph that is representative of the objects in the scene, the GST graph comprises;

nodes that represent objects captured in at least one of the first remote sensing image or the second remote sensing image, wherein a first node in the nodes is assigned data that indicates that the first node is representative of an object that is expected to change position over time; and

undirected edges, wherein each undirected edge in the undirected edges couples a respective pair of nodes in the nodes, the undirected edge is assigned data that is indicative of a symmetric relationship between objects in the scene represented by nodes in the pair of nodes; and

providing the GST to a search system, the GST searchable in response to receipt of a query.

2. The computing device of claim 1, wherein assigning the classifications to the respective portions of the first remote sensing image and the second remote sensing image comprises:

constructing a first classification image of the scene based upon the first remote sensing image, the first classification image comprises a first plurality of segments; and

constructing a second classification image of the scene based upon the second remote sensing image, the second classification image comprises a second plurality of segments, wherein each node in the GST graph corresponds to a segment in at least one of the first classification image or the second classification image.

3. The computing device of claim 2, wherein constructing the first classification image comprises segmenting a portion of the first classification image into a plurality of segments based upon the portion of the first classification image being identified as corresponding to a path network object.

4. The computing device of claim 2, further comprising:

segmenting the first classification image into a plurality of segments; and

assigning classification values to respective segments in the plurality of segments, wherein assigning a first classification value comprises;

performing morphological erosion on a segment to eliminate boundary noise or pixel filament connections to a second segment; and

performing morphological dilation to the segment to recover an original size of the segment.

5. The computing device of claim 1, wherein the GST graph further comprises a plurality of directed edges, each directed edge in the plurality of directed edges connects a respective pair of nodes in the nodes, each directed edge assigned second data that is indicative of an asymmetric relationship between an object or objects represented by the pair of nodes.

6. The computing device of claim 1, wherein each node in the nodes is assigned data comprising:

a value that is indicative of a size of an object represented by the node;

a value that is indicative of a type of the object represented by the node; and

chronology data that indicates when the object was observed in at least one of the first remote sensing image or the second remote sensing image.

7. The computing device of claim 6, wherein:

the chronology data assigned to the first node comprises;

a first time attribute, the first time attribute being an earliest time at which the object represented by the first node potentially existed in the scene; and

a second time attribute, the second time attribute being a latest time at which the object represented by the first node potentially existed in the scene.

8. The computing device of claim 6, wherein the chronology data assigned to a second node in the nodes comprises values for:

a first time attribute, the first time attribute being an earliest point in time in which a second object represented by the second node was observed as failing to exist in the scene;

a second time attribute, the second time attribute being an earliest point in time in which the second object was observed in the scene;

a third time attribute, the third time attribute being a latest point in time which the second object was observed in the scene; and

a fourth time attribute, the fourth time attribute being an earliest point in time in which the second object was observed to change in the scene.

9. The computing device of claim 1, the remote sensing image of the scene is based upon at least one of:

a LiDAR image;

digital surface model data;

an RGB+infrared image;

a synthetic aperture radar image;

or

a GIS image.

10. The computing device of claim 1, wherein the first node is representative of a vehicle or vehicle track captured in at least one of the first remote sensing image or the second remote sensing image.

11. The computing device of claim 1, wherein the nodes of a GST graph are encoded with pointers back to the remote sensing image in which the object was observed.

12. A method, comprising:

constructing a geospatial temporal (GST) graph in memory of a computing device based upon a remote sensing image of a scene, the remote sensing image of the scene based upon a radar image of the scene, the GST graph comprises;

a node that is representative of an object captured in the remote sensing image, the node assigned first data that indicates that an object in the scene represented by the node is expected to change position over time, the first data comprises chronology data that indicates an earliest point in time that the object represented by the node potentially first existed in the scene; and

an edge that couples the node to a second node in the GST graph, the edge assigned second data that is indicative of at least one of;

a distance between the object represented by the node and a second object represented by the second node;

or

a common overlap area between objects represented by the node and a second node; and

executing a search over the GST graph in response to receipt of a query to generate an output, the output based upon the data assigned to the node.

13. The method of claim 12, wherein the node is representative of a vehicle or a vehicle track.

14. The method of claim 12, wherein constructing the GST graph further comprises constructing the GST graph based upon a second remote sensing image of the scene, the second remote sensing image of the scene captured subsequent in time to when the remote sensing image of the scene was captured.

15. The method of claim 12, wherein constructing the GST graph further comprises assigning the node a label that is indicative of a type of the object that is represented by the node.

16. The method of claim 12, wherein constructing the GST graph further comprises:

computing the second data in response to receipt of the query.

17. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

segmenting a remote sensing image of a scene into a plurality of segments based upon elevation data aligned with the remote sensing image, the remote sensing image of the scene based upon an optical image of the scene;

constructing a geospatial temporal (GST) graph in memory of a computing device based upon the plurality of segments, the GST graph comprises;

a node that is representative of an object captured in the remote sensing image, the node assigned first data that indicates that an object in the scene represented by the node is expected to change position over time; and

an edge that couples the node to a second node in the GST graph, the edge assigned second data that is indicative of a distance between the object represented by the node and a second object represented by the second node; and

executing a search over the GST graph in response to receipt of a query to generate an output, the output based upon the data assigned to the node.