System and method for multiresolution and multitemporal image search

• US 10,482,122 B2
• Filed: 12/27/2016
• Issued: 11/19/2019
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Maxar Intelligence Incorporated (Galileo Parent, Inc.)
• Original Assignee: DigitalGlobe, Inc. (Galileo Parent, Inc.)
• Inventors: Tusk, Carsten
• Primary Examiner(s): Syed, Farhan M
• Application Number:
  US15/391,786
• Publication Number:
  US 20170235767A1
• Time in Patent Office: 1,057 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/11 Region-based segmentation

Abstract

A system for modular image mining and search, comprising a plurality of image capture devices, a search engine, an indexing engine, a database, and user interface software. A plurality of image capture devices capture images and store them to a database. The indexing engine indexes images to create image data for each image with associated metadata and stores the information to the database. The interface software is used to request search queries via a search engine. The search engine accesses the image data to find images or parts of images that satisfy the search query.

Claims

1. A system for multiresolution and multitemporal image mining and search, the system comprising:

an indexing engine comprising at least a first processor, a first memory, and a first plurality of programming instructions stored in the first memory and operating on the first processor, wherein the first plurality of programming instructions, when operating on the first processor, cause the first processor to;

index image data captured by a plurality of image capture devices, the indexing comprising the following steps;

for each of a plurality of spatial scales, dividing each image of the captured image data into a plurality of tiles of the corresponding spatial scale, thereby generating a set of tiles of varying spatial scales for each image;

using a plurality of geographical feature extractors operating on the processor of the computing device, automatically extracting a plurality of geographical features for each tile;

generating a vector for each tile, the vector comprising a numeric value for each of the plurality of geographical features corresponding to a degree of presence of each corresponding feature; and

composing the set of generated vectors into an inverted feature presence index;

an image information database operating on the first processor and adapted to receive and store image data and indices from the indexing engine; and

a search engine comprising at least a second processor, a second memory, and a second plurality of programming instructions stored in the second memory and operating on the second processor, wherein the second plurality of programming instructions, when operating on the second processor, cause the second processor to operate an interface configured to receive text-based search requests from users for searches of elements and features within an image;

wherein the indexing and categorization of image data is performed using automated geographical feature extraction at multiple spatial scales, including at least pixel level features, scene level features including object identities, locations and counts, and region level features, and at multiple times; and

wherein the extraction of a plurality of geographical features for each tile occurs without the need for user interaction.

2. A method for multiresolution and multitemporal searching of image data, the method comprising the steps of:

(a) receiving, at an indexing engine comprising at least a first processor, a first memory, and a first plurality of programming instructions stored in the first memory and operating on the first processor, image data captured by a plurality of image capture devices;

(b) for each of a plurality of spatial scales, dividing each image of the received image data into a plurality of tiles of the corresponding spatial scale;

(c) using a plurality of geographical feature extractors, automatically extracting a plurality of geographical features for each tile of each of the plurality of spatial scales, wherein the extraction of a plurality of geographical features for each tile occurs without the need for user interaction;

(d) generating a vector for each tile, the vector comprising a numeric value for each of the plurality of geographical features corresponding to a degree of presence of each corresponding feature; and

(e) composing the set of generated vectors into an inverted feature presence index;

(f) storing, in an image information database operating on the first processor, the image data and the inverted feature presence index in a database; and

(g) searching, using a search engine comprising at least a second processor, a second memory, and a second plurality of programming instructions stored in the second memory and operating on the second processor, the image data and the image indices using input received from a user via a network.

3. A system for searching over multiple spatial resolutions on a multi-temporal grid of images, the system comprising:

an image storage repository comprising at least a first processor, a first memory, and a first plurality of programming instructions stored in the first memory and operating on the first processor, wherein the first plurality of programming instructions, when operating on the first processor, cause the first processor to store a plurality of images each stored at a plurality of spatial resolutions, the plurality of spatial resolutions of each of the plurality of images decomposed from the respective image as captured by a plurality of image capture devices that capture the plurality of images over a period of time and store them in the image storage repository with timestamps;

an indexing engine operating on the first processor and configured to analyze the plurality of images is analyzed by;

performing automated geographical feature extraction at multiple spatial scales, including at least pixel level features, scene level features including object identities, locations and counts, and region level features, and at multiple times; and

indexing image data captured by the plurality of image capture devices by performing the following steps;

for each of a plurality of spatial scales, dividing each image of the captured image data into a plurality of tiles of the corresponding spatial scale, thereby generating a set of tiles of varying spatial scales for each image;

using a plurality of feature extractors, automatically extracting a plurality of geographical features for each tile, wherein the extraction of a plurality of geographical features for each tile occurs without the need for user interaction;

generating a vector for each tile, the vector comprising a numeric value for each of the plurality of geographical features corresponding to a degree of presence of each corresponding feature; and

composing the set of generated vectors into an inverted feature presence index; and

a search engine comprising at least a second processor, a second memory, and a second plurality of programming instructions stored in the second memory and operating on the second processor, wherein the second plurality of programming instructions, when operating on the second processor, cause the second processor to;

select an appropriate image resolution from a set of resolutions; and

conduct a search of a set of images at the selected image resolution to identify changes within a scene over a period of time in a set of images captured by a plurality of image capture devices, using at least the inverted feature presence index.