Aggregating data from a plurality of data sources

• US 9,105,000 B1
• Filed: 06/13/2014
• Issued: 08/11/2015
• Est. Priority Date: 12/10/2013
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Bingham, Eli, White, Nicholas, Ural, Engin, Grewal, Jasjit
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US14/304,741
• Time in Patent Office: 424 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function

Abstract

According to certain aspects, a computer system may be configured to aggregate and analyze data from a plurality of data sources. The system may obtain data from a plurality of data sources, each of which can include various types of data, including email data, system logon data, system logoff data, badge swipe data, employee data, job processing data, etc. associated with a plurality of individuals. The system may also transform data from each of the plurality of data sources into a format that is compatible for combining the data from the plurality of data sources. The system can resolve the data from each of the plurality of data sources to unique individuals of the plurality of individuals. The system can also determine an efficiency indicator based at least in part on a comparison of individuals of the unique individuals that have at least one common characteristic.

Claims

1. A computer system configured to aggregate and analyze data from a plurality of data sources, the computer system comprising:

one or more hardware computer processors configured to execute code in order to cause the system to;

obtain first and second data from a plurality of data sources, the plurality of data sources comprising a first data source and a second data source, wherein the first data source and the second data source are different from one another, each of the first data source and the second data source comprising one or more of;

email data, system logon data, system logoff data, badge swipe data, employee data, software version data, software license data, remote access data, phone call data, work production data, VPN login data, remote access data, or job processing data, each data source of the plurality of data sources associated with a plurality of employees, wherein for each data source, each employee of the plurality of employees is associated with a respective identifier;

detect a discrepancy associated with data from the first data;

in response to the detection of the discrepancy, reobtain first data from the first data source;

detect inconsistencies in formatting of data from each of the reobtained first data and the second data;

transform data from each of the reobtained first data and the second data into a format that is compatible for combining data from respective data sources of the plurality of data sources, and wherein the reobtained first data and the second data comprise different identifiers that identify the same employee;

generate a mapping of unique employees from the plurality of employees to data from each of the reobtained first data and the second data, wherein the mapping is based at least in part on correlating each employee'"'"'s respective identifier to the different identifiers from the reobtained first data and the second data that identify the same employee;

generate combined data by combining data from the reobtained first data and the second data;

generate an intermediate output based at least in part on the combined data associated with respective employees and the mapping, wherein the intermediate output associated with a particular employee comprises a reduced version of the combined data associated with the particular employee from the plurality of data sources, and wherein the intermediate output associated with the particular employee summarizes at least some of data items associated with efficiency indicators of the particular employee;

determine a plurality of efficiency indicators for the respective employees based at least in part on a comparison of data from the intermediate output associated with the respective employees and other employees that have at least one common characteristic, wherein the at least one common characteristic comprises at least a common job title or job description, wherein the plurality of efficiency indicators provides information relating to efficiency of the respective employees having the at least one common characteristic, and wherein a first efficiency indicator of the plurality of efficiency indicators is based at least in part on intermediate output associated with email summary data of the respective employees, and wherein a second indicator of the plurality of efficiency indicators is based at least in part on intermediate output associated with at least an average work product statistic over a period of time for the respective employees, and wherein the average work product statistic comprises a correlation between the email summary data and the common job title or job description for the respective employees; and

display, in a user interface, summary data relating to at least some of the respective employees, wherein the at least some of the respective employees share at least one common characteristic, and wherein the summary data comprises the first efficiency indicator and the second efficiency indicator for the at least some of the respective individuals.

2. The system of claim 1, wherein the code is further configured to cause the computer system to:

access expected format information indicating an expected format of data from the first data source; and

determine whether data from the first data source is consistent with the expected format.

3. The system of claim 1, wherein said generating of the combined data comprises:

mapping a first column in the first data source to a second column in the second data source.

4. The system of claim 1, wherein a first data item of the first data source comprises a first identifier that refers to a first employee, and a second data item of the second data source comprises a second identifier that refers to the first employee, wherein the first identifier and the second identifier are different;

and

said transforming of data items comprises mapping the first identifier and the second identifier.

5. The system of claim 1, wherein the code is further configured to cause the system to:

in response to the detection of inconsistencies in formatting of data from the second data, reobtain second data from the second data source.

6. The system of claim 1, wherein data of the first data source has a different format from data of the second data source.

7. The system of claim 1, wherein data from a data source of the plurality of data sources is in one or more of the following formats:

database, web service, flat file, and log file.

8. The system of claim 1, wherein the at least one common characteristic further comprises at least one of a location, a department, or a supervisor.

9. The system of claim 1, wherein:

the first data source comprises employee data and the second data source comprises email data, wherein the employee data comprises one or more of an employee identifier, an employee name, and an employee email address for each of a plurality of employees; and

the code is further configured to cause the system to associate data from the respective data sources of the plurality of data sources at least partly by associating the employee data and the email data to unique employees.

10. A computer system configured to aggregate and analyze data from a plurality of data sources, the computer system comprising:

one or more hardware computer processors configured to execute code in order to cause the system to;

access first and second data from a plurality of data sources, the plurality of data sources comprising a first data source and a second data source, wherein the first data source and the second data source are different from one another, each of the first data source and the second data source comprising one two or more of;

email data, system logon data, system logoff data, badge swipe data, employee data, software version data, software license data, remote access data, phone call data, work production data, VPN login data, remote access data, or job processing data, each data source of the plurality of data sources associated with a plurality of workers, wherein for each data source, each worker of the plurality of workers is associated by a respective identifier;

detect a discrepancy associated with data from the first data;

in response to the detection of the discrepancy, reaccess first data from the first data source;

detect inconsistencies in formatting of data from each of the reaccessed first data and the second data;

transform data from each of the reaccessed first data and the second data into a format that is compatible for combining data from respective data sources of the plurality of data sources, and wherein the reaccessed first data and the second data comprise different identifiers that identify the same employee;

generate a mapping of data items from each of the reaccessed first data and the second data to respective workers of the plurality of workers, wherein the mapping is based at least in part on correlating each worker'"'"'s respective identifier to the different identifiers from the reaccessed first data and the second data that identify the same employee;

generate combined data by combining data from the reaccessed first data, the second data, and the mapping;

generate an intermediate output based at least in part on generating summary data associated with some data items of the combined data; and

display, in a user interface, statistics associated with the respective workers based on the intermediate output, wherein the statistics indicate efficiency of the respective workers who have at least one common characteristic and wherein the at least one common characteristic comprises at least a common job description or job title, and wherein the statistics are associated with work productivity data over a period of time for the respective workers, and wherein the work productivity data comprises a correlation between email summary data and the common job description or job title of the respective workers.

11. The system of claim 10, wherein the code is further configured to cause the computer system to:

access expected format information indicating an expected format of data from the second data source; and

determine whether the second data from the second data source is consistent with the expected format.

12. The system of claim 10, wherein said transforming of the data items comprises converting a time in a first time zone to a time in a second time zone.

13. The system of claim 10, wherein:

a first data item of the first data source comprises a first identifier that refers to a first worker, and a second data item of the second data source comprises a second identifier that refers to the first worker, wherein the first identifier and the second identifier are different; and

said transforming of data items comprises mapping the first identifier and the second identifier.

14. The system of claim 10, wherein:

the first data source comprises employee data and the second data source comprises email data, wherein the employee data comprises one or more of an employee identifier, an employee name, and an employee email address for each of a plurality of employees;

the employee data and the email data are associated to respective employees of the plurality of employees; and

the code is further configured to cause the computer system to generate a report relating to employee email activity based at least in part on data associated with the respective employees, the report comprising a list of top email senders, a list of top email recipients, a list of attachment count, and a list of top email domains.

15. The system of claim 10, wherein the intermediate output comprises a list of top emails senders and a list of top email recipients for each respective worker.

16. The system of claim 10, wherein an inconsistency in formatting of at least one data item comprises an inconsistency from an expected format of a column of the at least one data item.

17. The system of claim 10, wherein an inconsistency in formatting of at least one data item comprises an inconsistency from an expected size of the at least one data item.

18. The system of claim 10, wherein:

the first data source comprises employee data, the employee data comprising one or more of an employee identifier, an employee name, and an employee email address for a first plurality of employees;

the second data source comprises job processing data, the job processing data comprising information relating to jobs processed by a second plurality of employees;

the employee data and the job processing data are associated to respective employees of the first plurality of employees and the second plurality of employees; and

the code is further configured to cause the computer system to generate a report relating to employee job processing activity based at least in part on data associated with the respective employees, the report comprising an efficiency level of the respective employees.

19. A non-transitory computer readable storage medium comprising instructions for aggregating and analyzing data from a plurality of data sources that cause a computer processor to:

access first and second data from a plurality of data sources, the plurality of data sources comprising a first data source and a second data source, wherein the first data source and the second data source are different from one another, each of the first data source and the second data source comprising one or more of;

email data, system logon data, system logoff data, badge swipe data, employee data, software version data, software license data, remote access data, phone call data, work production data, VPN login data, remote access data, or job processing data, each data source of the plurality of data sources associated with a plurality of employees, wherein for each data source, each employee of the plurality of employees is associated by a respective identifier;

detect a discrepancy associated with data from the first data;

in response to the detection of the discrepancy, reaccess first data from the first data source;

detect inconsistencies in formatting of data from each of the reaccessed first data and the second data;

transform data from each of the reaccessed first data and the second data into a format that is compatible for combining data from respective data sources of the plurality of data sources, and wherein the reaccessed first data and the second data comprise different identifiers that identify the same employee;

associate data items from each of the reaccessed first data and the second data to respective employees of the plurality of employees, wherein the association is based at least in part on correlating each employee'"'"'s respective identifier to the different identifiers from the reaccessed first data and the second data that identify the same employee;

generate combined data by combining data from two or more data sources from the reaccessed first data, the second data, and the associated data items; and

generate an intermediate output based at least in part on generating summary data associated with some data items of the combined data; and

generate output data comprising statistics associated with respective employees based at least in part on the the intermediate output, wherein the statistics indicate efficiency of respective employees who have at least one common characteristic and wherein the at least one common characteristic comprises at least a common job description or job title, and wherein the statistics are associated with work productivity data over a period of time for the respective workers, and wherein the work productivity data comprises a correlation between email summary data and the common job description or job title of the respective employees.

20. The computer readable storage medium of claim 19, wherein the intermediate output comprises a list of top emails senders and a list of top email recipients for each respective worker.