System and method for multi resolution and multi temporal image search

• US 9,529,824 B2
• Filed: 11/27/2013
• Issued: 12/27/2016
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Maxar Intelligence Incorporated (Galileo Parent, Inc.)
• Original Assignee: DigitalGlobe, Inc. (Galileo Parent, Inc.)
• Inventors: Tusk, Carsten
• Primary Examiner(s): SYED, FARHAN M
• Application Number:
  US14/091,412
• Publication Number:
  US 20140365463A1
• Time in Patent Office: 1,126 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/11 Region-based segmentation

Abstract

A system for modular image mining and search, comprising a plurality of image capture devices, a search engine, an indexing engine, a database, and user interface software. A plurality of image capture devices capture images and store them to a database. The indexing engine indexes images to create image data for each image with associated metadata and stores the information to the database. The interface software is used to request search queries via a search engine. The search engine accesses the image data to find images or parts of images that satisfy the search query.

Claims

1. A system for modular image mining and search, the system comprising:

a plurality of image capture devices stored and operating on a plurality of network-attached computer;

an indexing engine software module operating on a first network-attached server computer and configured to;

index image data captured by the plurality of image capture devices by performing the following steps;

for each of a plurality of spatial scales, dividing each image of the captured image data into a plurality of tiles of the corresponding spatial scale, thereby generating a set of tiles of varying spatial scales for each image;

using a plurality of feature extractors, automatically extracting a plurality of features for each tile;

generating a vector for each tile, the vector comprising a numeric value for each of the plurality of features corresponding to a degree of presence of each corresponding feature; and

composing the set of generated vectors into an inverted feature presence index;

wherein the indexing and categorization of image data is performed using automated feature extraction at multiple spatial scales, including at least pixel level features, scene level features including object identities, locations and counts, and region level features, and at multiple times;

an image information database operating on the first network-attached server computer and adapted to receive and store image data and indices from the indexing engine software module;

an interface software module operating on a second network-attached computer; and

a search engine software module operating on a second network-attached server computer and configured to receive text-based search requests from users via the interface software module for searches of elements and features within an image.

2. A method for searching mined image data, the method comprising the steps of:

(a) indexing image data using an indexing engine software module stored and operating on a first network-attached server computer and based on images captured from a plurality of image capture devices, and storing the image data and a plurality of indices generated by the indexing in a database; and

(b) searching the image data and the image indices using a search engine software module stored and operating on a second network-attached server computer based on input by a user through an interface software module stored and operating on a network-attached client computer;

wherein the indexing of image data is performed using automated feature extraction at multiple spatial scales, including at least pixel level features, scene level features including object identities, locations and counts, and region level features, and at multiple times;

wherein the indexing engine software module indexes image data captured by the plurality of image capture devices by performing the following steps;

for each of a plurality of spatial scales, dividing each image of the captured image data into a plurality of tiles of the corresponding spatial scale, thereby generating a set of tiles of varying spatial scales for each image;

using a plurality of feature extractors, automatically extracting a plurality of features for each tile;

generating a vector for each tile, the vector comprising a numeric value for each of the plurality of features corresponding to a degree of presence of each corresponding feature; and

composing the set of generated vectors into an inverted feature presence index.

3. A system for searching over multiple spatial resolutions on a multi-temporal grid of images, the system comprising:

an image storage repository operating on a network-attached server computer and storing a plurality of images each stored at a plurality of spatial resolutions the plurality of spatial resolutions of each of the plurality of images decomposed from the respective image as captured by a plurality of image capture devices that capture the plurality of images over a period of time and store them in the image storage repository with timestamps;

an indexing engine software module operating on a first network-attached server computer and configured to analyze the plurality of images is analyzed by;

performing automated feature extraction at multiple spatial scales, including at least pixel level features, scene level features including object identities, locations and counts, and region level features, and at multiple times; and

indexing image data captured by the plurality of image capture devices by performing the following steps;

for each of a plurality of spatial scales, dividing each image of the captured image data into a plurality of tiles of the corresponding spatial scale, thereby generating a set of tiles of varying spatial scales for each image;

using a plurality of feature extractors, automatically extracting a plurality of features for each tile;

generating a vector for each tile, the vector comprising a numeric value for each of the plurality of features corresponding to a degree of presence of each corresponding feature; and

composing the set of generated vectors into an inverted feature presence index; and

a search engine software module stored and operating on a network-attached server computer and configured to;

select an appropriate image resolution from a set of resolutions to conduct an automated search of features and elements within the image data by using indices and image data from the image storage repository; and

conduct a search of images or changes within a scene over a period of time in a set of images captured by a plurality of image capture devices.