Systems and methods for attribute analysis of one or more databases

• US 10,114,884 B1
• Filed: 03/22/2016
• Issued: 10/30/2018
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Valensi, Boris, Ong, James, Carter, Howard, Tao, Joy, Schafer, Mark, McCardel, Barry, Falter, Kelsey, Hu, Roger
• Primary Examiner(s): Andersen, Kristopher
• Application Number:
  US15/077,650
• Time in Patent Office: 952 Days
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range

Abstract

Systems and techniques for indexing and/or querying a database are described herein. Multiple, large disparate data sources may be processed to cleanse and/or combine item data and/or item metadata. Further, attributes may be extracted from the item data sources. The interactive user interfaces allow a user to select one or more attributes and/or other parameters to present visualizations based on the processed data.

Claims

1. A computer-implemented method for identifying trends of one or more items from an item database, the computer-implemented method comprising:

accessing item data from a first data source;

accessing item metadata from a second data source, the item metadata comprising a mapping of item identifiers to respective categories, and one or more metadata values for each item of the item metadata;

analyzing the item data by at least;

determining cleansed item data;

determining a set of attributes from the cleansed item data; and

determining a category for each respective item using the mapping of item identifiers to respective categories;

receiving input comprising data indicative of a first attribute from the set of attributes and a first category;

determining a set of items where each item of the set of items are associated with the first category;

determining combinations of pairs of attributes from each item of the set of items;

determining, from the combinations of pairs of attributes, filtered combinations of pairs of attributes that include the first attribute and at least one other attribute, the at least one other attribute different from the first attribute;

determining, from the set of items, a plurality of collections of items, wherein each collection of items from the plurality of collections of items comprises one or more items associated with at least a pair of attributes from the filtered combinations of pairs of attributes;

determining, from the item metadata, a set of metadata values associated with one or more items from the plurality of collections of items;

aggregating the set of metadata values to determine a plurality of summary indicators, wherein each summary indicator is representative of each collection of items from the plurality of collections of items;

generating a ranking from the plurality of summary indicators;

selecting, from the plurality of collections of items, a subset of the plurality of collections of items according to the ranking;

generating, from the plurality of summary indicators, a plurality of graphs for the subset of the plurality of collections of items;

presenting, in a user interface, the plurality of graphs associated with a respective indicator for a particular pair of attributes from the filtered combinations of pairs of attributes;

receiving second user input via the user interface, the second user input comprising data indicative of the first attribute, the first category, and a second exclusion attribute from the set of attributes;

determining a second set of items associated with the first category;

determining a first subset of items from the second set of items, wherein each item of the first subset of items is further associated with the first attribute, and wherein the first subset of items comprises a first item and a second item, the first item associated with the first attribute and the second exclusion attribute;

determining a second subset of items by removing the first item from the first subset of items;

determining, from the item metadata, a second set of metadata values associated with the second subset of items; and

presenting a visualization, in the user interface, associated with the second set of metadata values.

2. The computer-implemented method of claim 1, further comprising:

determining at least one binary column within the item data;

replacing a column value of the at least one binary column with a predefined attribute value based at least on the column value being non-empty; and

adding the predefined attribute value to the set of attributes.

3. The computer-implemented method of claim 1, wherein determining cleansed item data comprises at least one of:

removing a number from the item data, removing punctuation from the item data, or removing text data from the item data wherein the text data does not correspond to a set of valid words.

4. The computer-implemented method of claim 1, wherein the set of attributes comprises a multi-word attribute.

5. A non-transitory computer storage medium storing computer executable instructions that when executed by a computer hardware processor perform operations comprising:

accessing item data from a first data source;

analyzing the item data by at least;

determining cleansed item data by at least one of;

removing a number from the item data, removing punctuation from the item data, or removing text data from the item data wherein the text data does not correspond to a set of valid words;

determining that two or more words from the item data are synonyms;

selecting a first word as a representative attribute; and

determining a set of attributes from the cleansed item data, the set of attributes comprising the representative attribute;

receiving user input via a user interface, the user input comprising data indicative of a first attribute from the set of attributes and a first category;

determining a set of items where each item of the set of items are associated with the first category;

determining combinations of pairs of attributes from each item of the set of items;

determining, from the combinations of pairs of attributes, filtered combinations of pairs of attributes that include the first attribute and at least one other attribute, the at least one other attribute different from the first attribute;

determining, from the set of items, a plurality of collections of items, wherein each collection of items from the plurality of collections of items comprises one or more items associated with at least a pair of attributes from the filtered combinations of pairs of attributes;

determining, from item metadata, a set of metadata values associated with one or more items from the plurality of collections of items;

aggregating the set of metadata values to determine a plurality of summary indicators, wherein each summary indicator is representative of each collection of items from the plurality of collections of items;

generating a ranking from the plurality of summary indicators;

selecting, from the plurality of collections of items, a subset of the plurality of collections of items according to the ranking;

generating, from the plurality of summary indicators, a plurality of visualizations for the subset of the plurality of collections of items;

presenting, in the user interface, the plurality of visualizations associated with a respective indicator for a particular pair of attributes from the filtered combinations of pairs of attributes;

receiving second user input via the user interface, the second user input comprising data indicative of the first attribute, the first category, and a second exclusion attribute from the set of attributes;

determining a second set of items associated with the first category;

determining a first subset of items from the second set of items, wherein each item of the first subset of items is further associated with the first attribute, and wherein the first subset of items comprises a first item and a second item, the first item associated with the first attribute and the second exclusion attribute;

determining a second subset of items by removing the first item from the first subset of items;

determining, from the item metadata, a second set of metadata values associated with the second subset of items; and

presenting a second visualization, in the user interface, associated with the second set of metadata values.

6. The non-transitory computer storage medium of claim 5, wherein the operations further comprise:

determining at least one binary column within the item data;

replacing a column value of the at least one binary column with a predefined attribute value based at least on the column value being non-empty; and

adding the predefined attribute value to the set of attributes.

7. The non-transitory computer storage medium of claim 5, wherein determining the cleansed item data further comprises removing a stopword from the item data based at least on a predefined stopword list.

8. The non-transitory computer storage medium of claim 5, wherein the operations further comprise:

determining one or more multi-word attributes from the cleansed item data using an n-gram identification algorithm, wherein the set of attributes comprises the one or more multi-word attributes.

9. The non-transitory computer storage medium of claim 8, wherein the item data comprises three or more columns of data, and wherein determining the one or more multi-word attributes from the cleansed item data further comprises pseudo-randomly reordering the three or more columns of data.

10. The non-transitory computer storage medium of claim 5, wherein a visualization of the plurality of visualizations comprises at least one of:

a graph visualization, a time series visualization, or a summary table visualization.

11. A computing system comprising:

one or more hardware computer processors that execute code instructions to;

access item data from a first data source;

analyze the item data by at least;

determining cleansed item data; and

determining the set of attributes from the cleansed item data;

receive input comprising data indicative of a first attribute from the set of attributes and a first category;

determine a set of items where each item of the set of items are associated with the first category;

determine combinations of pairs of attributes from each item of the set of items;

determine, from the combinations of pairs of attributes, filtered combinations of pairs of attributes that include the first attribute and at least one other attribute, the at least one other attribute different from the first attribute;

determine, from the set of items, a plurality of collections of items, wherein each collection of items from the plurality of collections of items comprises one or more items associated with at least a pair of attributes from the filtered combinations of pairs of attributes;

access item metadata from a second data source;

determine, from the item metadata, a set of metadata values associated with one or more items from the plurality of collections of items;

aggregate the set of metadata values to determine a plurality of summary indicators, wherein each summary indicator is representative of each collection of items from the plurality of collections of items;

generate a ranking from the plurality of summary indicators;

select, from the plurality of collections of items, a subset of the plurality of collections of items according to the ranking;

generate, from the plurality of summary indicators, a plurality of visualizations for the subset of the plurality of collections of items;

present, in a user interface, the plurality of visualizations associated with a respective indicator for a particular pair of attributes from the filtered combinations of pairs of attributes

receive second user input via the user interface, the second user input comprising data indicative of the first attribute, the first category, and a second exclusion attribute from the set of attributes;

determine a second set of items associated with the first category;

determine a first subset of items from the second set of items, wherein each item of the first subset of items is further associated with the first attribute, and wherein the first subset of items comprises a first item and a second item, the first item associated with the first attribute and the second exclusion attribute;

determine a second subset of items by removing the first item from the first subset of items;

determine, from the item metadata, a second set of metadata values associated with the second subset of items; and

present a visualization, in the user interface, associated with the second set of metadata values.

12. The computing system of claim 11, wherein the one or more hardware computer processors further execute the code instructions to:

determine at least one binary column within the item data;

replace a column value of the at least one binary column with a predefined attribute value based at least on the column value being non-empty; and

add the predefined attribute value to the set of attributes.

13. The computing system of claim 11, wherein determining the cleansed item data further comprises removing a stopword from the item data based at least on a predefined stopword list.

14. The computing system of claim 11, wherein the one or more hardware computer processors further execute the code instructions to:

determine one or more multi-word attributes from the cleansed item data using an n-gram identification algorithm, wherein the set of attributes comprises the one or more multi-word attributes.

15. The computing system of claim 11, wherein the one or more hardware computer processors further execute the code instructions to:

receive second user input via the user interface, the second user input comprising data indicative of the first attribute and a filter parameter value;

determine a first set of items associated with the first attribute and a second set of items associated with the first attribute;

determine, from the item metadata, a first metadata value associated with the first set of items, and a second metadata value associated with the second set of items;

select the first set of items based at least on;

the first metadata value being within the first parameter value, and the second metadata value being outside of the first parameter value; and

present a graph associated with the first set of items.

16. The computing system of claim 11, wherein a visualization of the plurality of visualizations comprises a graph visualization.

17. The computing system of claim 16, wherein the graph visualization comprises at least one of:

growth rate data, annual sales data, volume sales data, market share data, or sample size data.