Information searching system

• US 10,331,732 B1
• Filed: 12/16/2016
• Issued: 06/25/2019
• Est. Priority Date: 12/16/2016
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Inventors: McClain, Jonathan T., Rothganger, Fredrick
• Primary Examiner(s): Strege, John B
• Application Number:
  US15/381,632
• Time in Patent Office: 921 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

A method and computer system for searching a collection of documents. The method comprises first identifying text in the document and then identifying an image in the document. Then the features of the image are extracted from the document and image terms are identified from the features. Finally, the collection of documents is searched using the image terms in conjunction with text terms.

Claims

1. A computer-implement method for searching a collection of documents comprising:

identifying, by one or more processors, text in a document and text terms from the text;

identifying, by one or more processors, an image in the document;

extracting, by one or more processors, features in the image in the document, wherein the features are described using descriptor vectors;

identifying, by one or more processors, image terms for the features by comparing descriptor vectors for the features to descriptor vectors representing features in a dictionary;

transforming, by one or more processors, the image into image terms;

combining, by one or more processors, the text terms and the image terms to form search terms; and

searching, by one or more processors, the collection of documents for images in conjunction with text using text-only searching processes.

2. The method of claim 1, wherein the image has a spatial structure and wherein extracting the features in the image in the document comprises:

dividing, by one or more processors, the image into regions, wherein pixels in the regions form the features.

3. The method of claim 2 further comprising:

repeatedly dividing, by one or more processors, the image into the regions for different region sizes; and

repeatedly identifying, by one or more processors, the image terms using different region sizes.

4. The method of claim 1 further comprising:

repeatedly dividing, by one or more processors, the text terms and image terms into sections.

5. The method of claim 3, wherein the different region sizes form a spatial structure in which a term count for the image is based on the image terms located in each of the regions.

6. The method of claim 2, wherein identifying the image terms from the features comprises:

performing, by one or more processors, vector quantization for the features to identify the image terms.

7. The method of claim 3, wherein extracting the features in the image in the document comprises:

creating, by one or more processors, a group of descriptors for each region in the different region sizes.

8. The method of claim 7, wherein the group of descriptors for a region form an initial image term for the region.

9. The method of claim 8, wherein the initial image term for the region is identified from the dictionary of image terms using the initial image term.

10. The method of claim 1 further comprising:

repeatedly dividing, by one or more processors, the document into sections; and

repeatedly identify, by one or more processors, the image terms and text terms in the sections to form image term structures used to search the collection of documents.

22. The method of claim 1, wherein features in the image are described by scale-invariant feature transform (SIFT) descriptors, and wherein the image terms are identified using centroids of SIFT descriptor clusters.

11. A computer system for searching a collection of documents, computer system comprising:

a bus system;

a storage device connected to the bus system, wherein the storage device stores program instructions;

a processor unit; and

a search engine, running on the processor unit, wherein the search engine;

identifies text in a document and text terms from the text;

identifies an image in the document;

extracts features in an image in the document, wherein the features are described using descriptor vectors;

identifies image terms for the features by comparing descriptor vectors for the features to descriptor vectors representing features in a dictionary;

transforms the image into image terms;

combines the text terms and the image terms to form search terms; and

searches a collection of documents utilizing the search terms to search for images in conjunction with text using text-only searching processes.

12. The computer system of claim 11, wherein in extracting the features in the image in the document, the search engine divides the image into regions, wherein pixels in the regions form the features.

13. The computer system of claim 12, wherein the search engine repeatedly dividing the image into the regions for different region sizes and repeatedly identifying the image terms for the features using the different region sizes.

14. The computer system of claim 13, wherein the different region sizes form a spatial structure in which a term count for the image is based on the text terms located in each of the regions.

15. The computer system of claim 12, wherein in identifying the image terms for the features, the search engine performs vector quantization for the features to identify the image terms.

16. The computer system of claim 13, wherein in extracting the features in the image in the document, the search engine creates a group of descriptors for each region in the different region sizes.

17. The computer system of claim 16, wherein the group of descriptors for each region form an initial image term for the regions.

18. The computer system of claim 17, wherein the initial image term for the regions is identified from the dictionary of image terms using the initial image term.

23. The computer system of claim 11, wherein features in the image are described by scale-invariant feature transform (SIFT) descriptors, and wherein the image terms are identified using centroids of SIFT descriptor clusters.

19. A computer program product for searching a collection of documents, the computer program product comprising:

a non-transitory computer-readable storage media;

a first program code, stored on the computer-readable storage media, for identifying text in a document and text terms from the text;

a second program code, stored on the computer-readable storage media, for identifying an image in the document;

a third program code, stored on the computer-readable storage media, for extracting features in the image in the document, wherein the features are described using descriptor vectors;

a fourth program code, stored on the computer-readable storage media, for identifying image terms for the features by comparing descriptor vectors for the features to descriptor vectors representing features in a dictionary;

a fifth program code, stored on the computer-readable storage media, for transforming the image into image terms;

a sixth program code, stored on the computer-readable storage media, for combining, by one or more processors, the text terms and the image terms to form search terms; and

a seventh program code, stored on the computer-readable storage media, for searching the collection of documents utilizing the search terms to search for images in conjunction with text using text-only searching processes.

20. The computer program product of claim 19, wherein the third program code comprises:

a program code, stored on the computer-readable storage media, for dividing the image into regions, wherein pixels in the regions form the features.

21. The computer program product of claim 20 further comprising:

an eighth program code, stored on the computer-readable storage media, for repeatedly dividing the image into the regions for different regions sizes; and

a ninth program code, stored on the computer-readable storage media, for repeatedly identifying the image terms using the different region sizes.

24. The computer program product of claim 19, wherein features in the image are described by scale-invariant feature transform (SIFT) descriptors, and wherein the image terms are identified using centroids of SIFT descriptor clusters.