SYSTEMS AND METHODS FOR ADAPTIVELY IDENTIFYING AND MITIGATING STATISTICAL OUTLIERS IN AGGREGATED DATA

• US 20160048552A1
• Filed: 08/13/2015
• Published: 02/18/2016
• Est. Priority Date: 08/13/2014
• Status: Active Grant

• Current Assignee: FINCA International, Inc.
• Original Assignee: FINCA International, Inc.
• Inventors: GRAHAM, Scott, TEVOSYAN, Anahit
• Granted Patent: US 10,417,201 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

The disclosed embodiments include computerized methods and systems that facilitate automated detection and precision correction of aggregated data collected by multiple, geographically dispersed mobile communications devices. In one embodiment, an apparatus detect a data outlier within portions of the aggregated data having numerical and/or categorical values. The apparatus may transmit information identifying the data outliner and a portion of the aggregated data that includes the data outlier to an additional communications device, which may present the aggregated data portion to a user in a manner that visually distinguishes the data outlined from other elements of aggregated data. In response to a request from the additional communications device, the apparatus may modify portions of the aggregated data in an effort to mitigate the data outlier.

Claims

1. An apparatus, comprising:

a storage device; and

at least one processor coupled to the storage device, the storage device storing software instructions for controlling the at least one processor when executed by the at least one processor, and the at least one processor being operative with the software instructions and configured to;

obtain aggregated data collected by a plurality of first communications devices;

detect at least one data outlier within the aggregated data, the data outlier corresponding to at least one element of the aggregated data;

transmit information identifying the data outlier and at least a portion of the aggregated data that includes the data outlier to a second communications device, the information instructing the second communications device to present the aggregated data portion to a user, and the data outlier being visually distinguishable within the presented aggregated data portion;

in response to the transmitted information, receive a request to modify the aggregated data from the second communications device; and

perform operations that modify at least a portion of the aggregated data in accordance with the received request.

2. The apparatus of claim 1, wherein the aggregated data comprises a first data portion and a second data portion, the first data portion including numerical values, and the second data portion including categorical data, and the categorical data including at least one non-numerical value.

3. The apparatus of claim 2, wherein at least one processor is further configured to:

apply at least one of a statistical or contextual techniques to a subset of the numerical data, the numerical data subset being characterized by a data parameter; and

identify the data outlier based on an outcome of the at least one statistical or contextual technique.

4. The apparatus of claim 3, wherein at least one processor is further configured to:

obtain information specifying at least one validation rule, the validation rule defining a valid relationship between elements of the aggregated data and the outcome of the at least one statistical or contextual technique;

determine that at least one element of the aggregated data fails to exhibit the valid relationship with the outcome of the at least one statistical or contextual technique; and

establish the at least one aggregated data element as the data outlier.

5. The apparatus of claim 2, wherein at least one processor is further configured to:

determine a probability of an occurrence of a first element of the categorical data in conjunction with at least one second element of the categorical data;

determine that the probability fails to exceed at threshold value; and

establish the first element of categorical data as the data outlier in response to the determination.

6. The apparatus of claim 2, wherein at least one processor is configured to perform operations that normalize at least a portion of the numerical data in accordance with one or more normalization criteria.

7. The apparatus of claim 1, wherein at least one processor is further configured to:

identify a first portion of the aggregated data collected by a corresponding one of the first communications devices;

determine a metric indicative of a degree of separation between the first portion of the aggregate data and other portions of aggregated data;

based on the determined metric, establish that at least one element of the first aggregated data portion represents potential falsified data.

8. The apparatus of claim 1, wherein at least one processor is further configured to:

detect a plurality of data outliers within the aggregated data; and

transmit information identifying the data outliers and at least a portion of the aggregated data that includes the data outliers torn the second communications device.

9. The apparatus of claim 1, wherein at least one processor is configured to re-format at least a portion of the aggregated data in accordance with a predetermined data format.

10. The apparatus of claim 1, wherein:

the first communications devices are associated with corresponding surveyors;

the second communications device is associated with a survey manager, the survey manager being a supervisor of a subset of the surveyors; and

at least a portion of the aggregated data comprises aggregated survey response data, the aggregated survey response data being collected by the first communications devices.

19. The method of claim 1, wherein:

the first communications devices comprise at least one mobile communications device

the first communications devices are associated with corresponding surveyors;

the second communications device is associated with a survey manager, the survey manager being a supervisor of a subset of the surveyors; and

at least a portion of the aggregated data comprises aggregated survey response data, the aggregated survey response data being collected by the first communications devices.

11. A computer-implemented method, comprising:

obtaining, by at least one processor, aggregated data collected by a plurality of first communications devices;

detecting, by at least one processor, at least one data outlier within the aggregated data, the data outlier corresponding to at least one element of the aggregated data;

generating, by at least one processor, an electronic command to transmit information identifying the data outlier and at least a portion of the aggregated data that includes the data outlier to a second communications device, the information instructing the second communications device to present the aggregated data portion to a user, and the data outlier being visually distinguishable within the presented aggregated data portion;

in response to the transmitted information, receiving, by at least one processor, a request to modify the aggregated data from the second communications device; and

performing, by at least one processor, operations that modify at least a portion of the aggregated data in accordance with the received request.

12. The method of claim 11, wherein the aggregated data comprises a first data portion and a second data portion, the first data portion including numerical values, and the second data portion including categorical data, and the categorical data including at least one non-numerical value.

13. The method of claim 12, further comprising;

applying at least one of a statistical or contextual techniques to a subset of the numerical data, the numerical data subset being characterized by a data parameter; and

identifying the data outlier based on an outcome of the at least one statistical or contextual technique.

14. The method of claim 13, further comprising:

obtaining information specifying at least one validation rule, the validation rule defining a valid relationship between elements of the aggregated data and the outcome of the at least one statistical or contextual technique;

determining that at least one element of the aggregated data fails to exhibit the valid relationship with the outcome of the at least one statistical or contextual technique; and

establishing the at least one aggregated data element as the data outlier.

15. The method of claim 12, further comprising:

determining a probability of an occurrence of a first element of the categorical data in conjunction with at least one second element of the categorical data;

determining that the probability fails to exceed at threshold value; and

establishing the first element of categorical data as the data outlier in response to the determination.

16. The method of claim 12, further comprising at least one of:

perform operations that re-format at least a portion of the aggregated data in accordance with a predetermined data format;

or

performing operations that normalize at least a portion of the numerical data in accordance with one or more normalization criteria.

17. The method of claim 11, further comprising:

identifying a first portion of the aggregated data collected by a corresponding one of the first communications devices;

determining a metric indicative of a degree of separation between the first portion of the aggregate data and other portions of aggregated data; and

based on the determined metric, establishing that at least one element of the first aggregated data portion represent potential falsified data.

18. The method of claim 11, further comprising:

detect a plurality of data outliers within the aggregated data; and

transmit information identifying the data outliers and at least a portion of the aggregated data that includes the data outliers to the second communications device.

20. A tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, perform a method comprising:

obtaining aggregated data collected by a plurality of first communications devices;

detecting at least one data outlier within the aggregated data, the data outlier corresponding to at least one element of the aggregated data;

generating an electronic command to transmit information identifying the data outlier and at least a portion of the aggregated data that includes the data outlier to a second communications device, the information instructing the second communications device to present the aggregated data portion to a user, and the data outlier being visually distinguishable within the presented aggregated data portion;

in response to the transmitted information, receiving a request to modify the aggregated data from the second communications device; and

performing operations that modify at least a portion of the aggregated data in accordance with the received request.