Method and system for unified information representation and applications thereof

• US 8,548,951 B2
• Filed: 03/10/2011
• Issued: 10/01/2013
• Est. Priority Date: 03/10/2011
• Status: Active Grant

• Current Assignee: TextWise Company, LLC
• Original Assignee: TextWise Company, LLC
• Inventors: Solmer, Robert, Ruan, Wen
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US13/044,763
• Publication Number:
  US 20120233127A1
• Time in Patent Office: 936 Days
• Field of Search: 707/661, 707/737, 707/738, 707/758, 707/780, 704/9
• US Class Current: 707/661
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 2212/454 Vector or matrix data

Abstract

Method, system, and programs for information search and retrieval. A query is received and is processed to generate a feature-based vector that characterizes the query. A unified representation is then created based on the feature-based vector, that integrates semantic and feature based characterizations of the query. Information relevant to the query is then retrieved from an information archive based on the unified representation of the query. A query response is generated based on the retrieved information relevant to the query and is then transmitted to respond to the query.

Claims

1. A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for archiving a document, comprising the steps of:

receiving a document via the communication platform;

analyzing, by a feature extractor, the received document in accordance with at least one model to form a feature-based vector characterizing the document;

generating, by a semantic extractor, a semantic-based representation of the document based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing, by a reconstruction unit, a reconstructed feature-based vector based on the semantic-based representation of the document, by mapping the semantic-based representation to a feature space of the feature-based vector;

comparing, by a discrepancy analyzer, the feature-based vector with the reconstructed feature-based vector to identify a difference between the feature-based vector and the reconstructed feature-based vector;

forming a residual feature-based representation of the document based on the difference between the feature-based vector and the reconstructed feature-based vector;

generating, by a unified representation construction unit, a unified representation for the document based on the semantic-based representation and the residual feature-based representation; and

archiving the document in an information archive based on the unified representation of the document.

2. The method of claim 1, wherein the at least one model includes an information model and a language model, wherein the feature-based vector according to the information model has a plurality of attributes, each of which represents a portion of information contained in the document allocated to an associated feature.

3. The method of claim 1, wherein the semantic extractor and the reconstruction unit are implemented based on an auto-encoder.

4. The method of claim 1, wherein the step of archiving comprises:

computing, by an indexing system, at least one index value based on the unified representation for the document;

establishing a link between the index value and the document archived in accordance with the unified representation of the document in the information archive.

5. The method of claim 1, further comprising forming a blurred feature-based representation of the document by modifying the feature-based vector based on the reconstructed feature-based vector.

6. The method of claim 5, wherein the step of modifying the feature-based vector comprises:

obtaining a first attribute value from the feature-based vector;

obtaining a second attribute value corresponding to the first attribute from the reconstructed feature-based vector; and

computing a third attribute value as the corresponding attribute value of the blurred feature-based representation of the document based on the first and second attribute values.

7. The method of claim 5, wherein the unified representation for the document is constructed further based on the blurred feature-based representation of the document.

8. A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for archiving a document, comprising the steps of:

receiving a document via the communication platform;

analyzing, by a feature extractor, the received document in accordance with at least one model to form a feature-based vector characterizing the document;

generating, by a semantic extractor, a semantic-based representation of the document based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing, by a reconstruction unit, a reconstructed feature-based vector based on the semantic-based representation of the document, by mapping the semantic-based representation to a feature space of the feature-based vector;

forming a blurred feature-based representation of the document based on a difference between the feature-based vector and the reconstructed feature-based vector;

generating, by a unified representation construction unit, a unified representation for the document based on the blurred feature-based representation and the semantic-based representation; and

archiving the document in an information archive based on the unified representation of the document.

9. The method of claim 8, further comprising:

forming a residual feature-based representation of the document based on one or more features identified in accordance with discrepancy between the feature-based vector and the reconstructed feature-based vector; and

incorporating, in the unified representation for the document, the semantic-based representation and the residual feature-based representation.

10. The method of claim 8, wherein the at least one model includes an information model and a language model, wherein the feature-based vector according to the information model has a plurality of attributes, each of which represents a portion of information contained in the document allocated to an associated feature.

11. The method of claim 8, wherein the semantic extractor and the reconstruction unit are implemented based on an auto-encoder.

12. A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for search and retrieval of information archived based on a unified representation, comprising the steps of:

obtaining a query via the communication platform;

processing, by a query processor, the query to generate a feature-based vector characterizing the query;

generating, by a semantic extractor, a semantic-based representation of the query based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing, by a reconstruction unit, a reconstructed feature-based vector based on the semantic-based representation of the query, by mapping the semantic-based representation to a feature space of the feature-based vector;

comparing, by a discrepancy analyzer, the feature-based vector with the reconstructed feature-based vector to identify a difference between the feature-based vector and the reconstructed feature-based vector;

forming a residual feature-based representation of the query based on the difference between the feature-based vector and the reconstructed feature-based vector;

generating, by a unified representation construction unit, a unified representation of the query based on the semantic-based representation and the residual feature-based representation;

retrieving, by a candidate search unit, information relevant to the query from an information archive based on the unified representation of the query;

generating, by a query response generator, a query response based on the information relevant to the query retrieved from the information archive; and

transmitting the query response to respond to the query.

13. The method of claim 12, wherein the unified representation of the query further includes a blurred feature-based representation generated by modifying the feature-based vector based on the reconstructed feature-based vector.

14. The method of claim 12, wherein the step of retrieving comprises:

generating a first index value based on the unified representation of the query;

identifying a second index value stored in an indexing system of the information archive;

obtaining a group of information items in the information archive that have similar index values; and

selecting the information relevant to the query from the obtained group of information items.

15. A system having at least one processor, storage, and a communication platform for generating a unified representation for a document, comprising:

a communication platform through which a document can be received;

a feature extractor configured for analyzing the received document in accordance with at least one model to form a feature-based vector characterizing the document;

a semantic extractor configured for generating a semantic-based representation of the document based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

a reconstruction unit configured for producing a reconstructed feature-based vector based on the semantic-based representation of the document by mapping the semantic-based representation to a feature space of the feature-based vector;

a residual feature identifier configured for forming a residual feature-based representation of the document based on the difference between the feature-based vector and the reconstructed feature-based vector; and

a unified representation construction unit configured for generating a unified representation for the document based on the semantic-based representation and the residual feature-based representation.

16. The system of claim 15, wherein the at least one model includes an information model and a language model, wherein the feature-based vector built according to the information model has a plurality of attributes, each of which represents a portion of information contained in the document allocated to an associated feature.

17. The system of claim 15, wherein the semantic extractor and the reconstruction unit are implemented based on an auto-encoder.

18. The system of claim 15, further comprising a feature vector blurring unit configured for forming a blurred feature-based representation of the document by modifying the feature-based vector based on the reconstructed feature-based vector.

19. A system having at least one processor, storage, and a communication platform for search and retrieval of information archived based on a unified representation, comprising:

a communication platform for obtaining a query and transmitting a query response;

a query processor configured for processing the query to generate a feature-based vector characterizing the query;

a semantic extractor configured for generating a semantic-based representation of the query based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

a reconstruction unit configured to construct a reconstructed feature-based vector based on the semantic-based representation of the query by mapping the semantic-based representation to a feature space of the feature-based vector;

a residual feature identifier configured for forming a residual feature-based representation of the query based on the difference between the feature-based vector and the reconstructed feature-based vector;

a query representation generator configured for generating a unified representation for the query based on the semantic-based representation and the residual feature-based representation, wherein the unified representation integrates semantic and residual feature based characterizations of the query;

a candidate search unit configured for retrieving information relevant to the query from an information archive based on the unified representation for the query; and

a query response generator configured for generating the query response based on the information relevant to the query retrieved from the information archive and transmitting the query response to respond to the query.

20. A system having at least one processor, storage, and a communication platform for search and retrieval of information archived based on a unified representation, comprising:

a communication platform for obtaining a query and transmitting a query response;

a query processor configured for processing the query to generate a feature-based vector characterizing the query;

a semantic extractor configured for generating a semantic-based representation of the query based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

a reconstruction unit configured to construct a reconstructed feature-based vector based on the semantic-based representation of the query by mapping the semantic-based representation to a feature space of the feature-based vector;

a feature vector blurring unit configured for generating a blurred feature-based representation of the query based on a difference between the feature-based vector and the reconstructed feature-based vector;

a query representation generator configured for generating a unified representation for the query based on the semantic-based representation and the blurred feature-based representation;

a candidate search unit configured for retrieving information relevant to the query from an information archive based on the unified representation for the query; and

a query response generator configured for generating the query response based on the information relevant to the query retrieved from the information archive and transmitting the query response to respond to the query.

21. A machine-readable non-transitory medium having information recorded thereon related to document archiving, the information, when read by the machine, causes the machine to perform the following:

receiving a document via a communication platform;

analyzing the received document in accordance with at least one model to form a feature-based vector characterizing the document;

generating a semantic-based representation of the document based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing a reconstructed feature-based vector based on the semantic-based representation of the document, by mapping the semantic-based representation to a feature space of the feature-based vector;

comparing the feature-based vector with the reconstructed feature-based vector to identify a difference between the feature-based vector and the reconstructed feature-based vector;

forming a residual feature-based representation of the document based on the difference between the feature-based vector and the reconstructed feature-based vector;

generating a unified representation for the document based on the semantic-based representation and the residual feature-based representation; and

archiving the document in an information archive based on the unified representation of the document.

22. The medium of claim 21, wherein the at least one model includes an information model and a language model, wherein the feature-based vector according to the information model has a plurality of attributes, each of which represents a portion of information contained in the document allocated to an associated feature.

23. The medium of claim 21, wherein the semantic extractor and the reconstruction unit are implemented based on an auto-encoder.

24. The medium of claim 21, wherein the information, when read by the machine, further causes the machine to perform the following:

forming a blurred feature-based representation of the document by modifying the feature-based vector based on the reconstructed feature-based vector; and

incorporating the blurred feature-based representation of the document as part of the unified representation for the document.

25. A machine-readable non-transitory medium having information recorded thereon for document archiving, the information, when read by the machine, causes the machine to perform the following:

receiving a document via a communication platform;

analyzing the received document in accordance with at least one model to form a feature-based vector characterizing the document;

generating a semantic-based representation of the document based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing a reconstructed feature-based vector based on the semantic-based representation of the document, by mapping the semantic-based representation to a feature space of the feature-based vector;

forming a blurred feature-based representation of the document based on a difference between the feature-based vector and the reconstructed feature-based vector;

generating a unified representation for the document based on the blurred feature-based representation and the semantic-based representation; and

archiving the document in an information archive based on the unified representation of the document.

26. The medium of claim 25, wherein the information, when read by the machine, further causes the machine to perform the following:

forming a residual feature-based representation of the document based on one or more features identified in accordance with discrepancy between the feature-based vector and the reconstructed feature-based vector; and

incorporating, in the unified representation for the document, the semantic-based representation and the residual feature-based representation.

27. The medium of claim 25, wherein the at least one model includes an information model and a language model, wherein the feature-based vector according to the information model has a plurality of attributes, each of which represents a portion of information contained in the document allocated to an associated feature.

28. The medium of claim 25, wherein the semantic extractor and the reconstruction unit are implemented based on an auto-encoder.

29. A machine-readable non-transitory medium having information recorded thereon for information search and retrieval, when read by the machine, causes the machine to perform the following:

obtaining a query via a communication platform;

processing the query to generate a feature-based vector characterizing the query;

generating a semantic-based representation of the query based on the feature-based vector, wherein the semantic-based representation has a reduced dimension;

constructing a reconstructed feature-based vector based on the semantic-based representation of the query, by mapping the semantic-based representation to a feature space of the feature-based vector;

comparing the feature-based vector with the reconstructed feature-based vector to identify a difference between the feature-based vector and the reconstructed feature-based vector;

forming a residual feature-based representation of the query based on the difference between the feature-based vector and the reconstructed feature-based vector;

generating a unified representation of the query based on the semantic-based representation and the residual feature-based representation, wherein the unified representation integrates semantic and residual feature based characterizations of the query;

retrieving information relevant to the query from an information archive based on the unified representation of the query;

generating a query response based on the information relevant to the query retrieved from the information archive; and

transmitting the query response to respond to the query.

30. The medium of claim 29, wherein the unified representation of the query further includes a blurred feature-based representation generated by modifying the feature-based vector based on the reconstructed feature-based vector.

31. The medium of claim 29, wherein the step of retrieving comprises:

generating a first index value based on the unified representation of the query;

identifying a second index value stored in an indexing system of the information archive;

obtaining a group of information items in the information archive that have similar index values; and

selecting the information relevant to the query from the obtained group of information items.