SYSTEM, METHOD, AND APPARATUS FOR MULTIDIMENSIONAL EXPLORATION OF CONTENT ITEMS IN A CONTENT STORE

• US 20110047159A1
• Filed: 08/19/2009
• Published: 02/24/2011
• Est. Priority Date: 08/19/2009
• Status: Active Grant

• Current Assignee: Beijing Zitiao Network Technology Company Limited (ByteDance Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Sismanis, John, Reinwald, Berthold, Simitsis, Alkis, Baid, Akanksha
• Granted Patent: US 8,560,548 B2
• Time in Patent Office: Days
• US Class Current: 707/738
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

A computer-implemented method for accessing content items in a content store are described. In one embodiment, the computer-implemented method includes maintaining a text index of content items in a content store to enable a keyword search on the content items, receiving a query having a keyword and generating a hit list from the text index using the keyword, and extracting frequent phrases from text within content items of the hit list. The computer-implemented method also includes assigning a relative relevance to the frequent phrases and grouping content items into topics based on presence of relevant phrases within the content items of the hit list. The hit list includes one or more content items of the content store. The frequent phrases having a relatively high relevance are relevant phrases.

Claims

1. A computer-implemented method for accessing content items in a content store comprising:

maintaining a text index of content items in a content store to enable a keyword search on the content items;

receiving a query having a keyword and generating a hit list from the text index using the keyword, the hit list comprising one or more content items of the content store;

extracting frequent phrases from text within content items of the hit list;

assigning a relative relevance to the frequent phrases wherein frequent phrases having a relatively high relevance are relevant phrases; and

grouping content items into topics based on presence of relevant phrases within the content items of the hit list.

2. The computer-implemented method of claim 1, further comprising ranking content items of the hit list by importance.

3. The computer-implemented method of claim 2 wherein importance is based on associations with other content items in the content store.

4. The computer-implemented method of claim 3, wherein the associations are citations to one or more different content items.

5. The computer-implemented method of claim 2 wherein grouping content items into topics is further based on importance.

6. The computer-implemented method of claim 1, further comprising:

maintaining a content model for metadata of the content items, wherein the metadata includes links between two or more content items;

using metadata of the content items in the hit list to populate static dimensions of a multidimensional schema;

analyzing content items of the hit list to determine dynamic dimensions; and

providing, based on multidimensional schema, OLAP-style analysis of the content items of the hit list, including at least one analysis selected from the group consisting of aggregation, navigation and reporting.

7. The computer-implemented method of claim 1, wherein relative relevance a frequent phrase is based on a ratio of frequency of the frequent phrase in content items of the hit list to frequency of the frequent phrase in a corpus comprising the content items in the content store.

8. The computer-implemented method of claim 1, further comprising:

preprocessing the content items to enrich the text index, wherein the preprocessing comprises generating a list of globally frequent phrases, wherein the globally frequent phrases are phrases that consist of at least two words that appear in more than a first threshold number of different content items in the content store; and

adding the globally frequent phrases to the text index.

9. The computer-implemented method of claim 1, further comprising:

maintaining a priority queue of the top-k most frequent phrases while intersecting the hit list with globally frequent phrases in descending order based upon length of the globally frequent phrases;

terminating the intersecting in response to identifying a globally frequent phrase with a length less than a current minimum intersection size in the priority queue; and

identifying phrases in the queue as the top-k dynamically frequent phrases.

10. The computer-implemented method of claim 1, further comprising:

maintaining a priority queue of the top-k most frequent phrases while intersecting the hit list with globally frequent phrases, wherein the intersecting includes estimating an intersection size, wherein the estimating comprises;

randomizing the globally frequent phrases in the text index to produce a first randomized posting list;

randomizing the hit list to produce a second randomized posting list; and

executing a modified zipper algorithm to intersect the first randomized posting list and the second randomized posting list, wherein the modified zipper algorithm terminates the executing in response to the earlier of identifying a predetermined M maximum number of comparisons or a predetermined I maximum number of common points.

11. The computer-implemented method of claim 10, wherein the estimating further comprises multiplying a Jaccard Distance estimator with a union estimator to produce an estimate for the intersection of the first randomized posting list and the second randomized posting list.

12. The computer-implemented method of claim 1, further comprising extracting labels for topics based on relevant phrases within the content items of the hit list.

13. The computer-implemented method of claim 1, further comprising:

receiving an input selecting a topic;

extracting frequent phrases from text within content items of the topic;

assigning a relative relevance to the frequent phrases wherein frequent phrases having a relatively high relevance are relevant phrases;

grouping content items into subtopics based on presence of relevant phrases within the content items of the topic.

14. A computer program product comprising a computer useable storage medium to store a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to perform operations for finding top-k sets in a collection of static sets P, wherein the top-k sets have maximal overlap with an input dynamic set H, the operations comprising:

during a preprocessing state independent of the input dynamic set H, randomizing items L in each static set P[i] in a collection of static sets P, wherein the randomizing uses a technique comprising;

hashing each item L[i] into a random number in a [0,1] domain to obtain hashed values; and

sorting the hashed values in increasing order;

at query time for a given dynamic set H, using the technique to randomize an input dynamic set H;

for each static set P[i], estimating an intersection size of an intersection of the static set P[i] and the input dynamic set H; and

maintaining a priority queue of top-k static sets having largest intersection sizes.

15. The computer program product of claim 14, further comprising:

ordering static sets P[i] in a collection of static sets P in descending order of cardinality;

receiving an input dynamic set H, wherein the input dynamic set H is a result of a keyword query;

initiating an intersecting process, wherein the process comprises;

for each static set P[i], in descending order of cardinality, estimating an intersection size of an intersection of the static set P[i] and the input dynamic set H; and

maintaining a priority queue of top-k static sets having largest intersection sizes; and

ceasing the intersecting process when a static set P[j] in the collection P is reached, wherein the static set P[j] has a cardinality less than a minimum intersection size in the priority queue.

16. The computer program product of claim 14, further comprising:

maintaining a priority queue of top-k′

static sets having largest intersection sizes, wherein the intersection sizes are estimated intersection sizes and k′

is greater than or equal to k;

for each set in the top-k′

static sets, intersecting the static set and the input dynamic set H to determine exact intersection sizes; and

determining the top-k static sets having largest intersection sizes based on the exact intersection sizes.

17. The computer program product of claim 16, wherein k′

=2k.

18. The computer program product of claim 14, wherein estimating an intersection size of an intersection of the static set P[i] and the input dynamic set H comprises:

zipping the hashed values of the static set P[i] with the hashed values of the input dynamic set H, wherein the zipping comprises;

skipping uniformly over certain comparisons between points in the hashed values of the static set P[i] and points in the hashed values of the input dynamic set H; and

ceasing the zipping after earlier of reaching a maximum number M of comparisons is reached or reaching a maximum number I of common points.

19. The computer program product of claim 14, wherein each set represents a document, and the collection of static sets represents a collection of documents to be text analyzed.

20. A system comprising:

a content management system (CMS) comprising;

a plurality of content items;

a content store to store content items;

a text index of text within the content items; and

an exploration server coupled to the content store and the text index and configured to;

generate a hit list from the text index using the keyword, the hit list comprising one or more content items of the content store;

extract frequent phrases from text within content items of the hit list;

assign a relative relevance to the frequent phrases wherein frequent phrases having a relatively high relevance are relevant phrases; and

group content items into topics based on presence of relevant phrases within the content items of the hit list; and

a multidimensional schema manager to manage a multidimensional schema comprising a schema of a fact table and schemata of static dimensions, wherein dynamic dimensions of the multidimensional schema are populated in response to the exploration server returning the hit list, and wherein dynamic dimensions of the multidimensional schema are identified in response to the exploration server returning the hit list based upon content dynamically extracted from the subset of content items identified in the hit list.