Method for statistical disclosure limitation

• US 20040049517A1
• Filed: 09/03/2002
• Published: 03/11/2004
• Est. Priority Date: 09/03/2002
• Status: Active Grant

• Current Assignee: Research Triangle Institute
• Original Assignee: Research Triangle Institute
• Inventors: Singh, Avinash C.
• Granted Patent: US 7,058,638 B2
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 21/6254 by anonymising data, e.g. decorrelating personal data from the owner's identification
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and system for ensuring statistical disclosure limitation (SDL) of categorical or continuous micro data, while maintaining the analytical quality of the micro data. The new SDL methodology exploits the analogy between (1) taking a sample (instead of a census,) along with some adjustments, including imputation, for missing information, and (2) releasing a subset, instead of the original data set, along with some adjustments for records still at disclosure risk. Survey sampling reduces monetary cost in comparison to a census, but entails some loss of information. Similarly, releasing a subset reduces disclosure cost in comparison to the full database, but entails some loss of information. Thus, optimal survey sampling methods can be used for statistical disclosure limitation. The method includes partitioning the database into risk strata, optimal probabilistic substitution, optimal probabilistic subsampling, and optimal sampling weight calibration.

Claims

1. A method of preserving confidentiality and analytical utility of an original database comprising a plurality of records, comprising:

partitioning the plurality of records into a plurality of risk strata based on a plurality of identifying variables, wherein each risk stratum includes at least one record; and

determining a respective rate of unique occurrence for each risk stratum in the plurality of risk strata.

2. The method of claim 1, wherein the partitioning step comprises:

determining a core risk stratum comprising those records in the plurality of records that have unique data values with respect to each identifying variable in a core subset of the plurality of identifying variables; and

determining a further risk stratum comprising those records in the plurality of records that have unique data values with respect to each identifying variable in a selected subset of the plurality of identifying variables, the selected subset including each identifying variable in the core subset.

3. The method of claim 2, further comprising:

repeating the step of determining a further risk stratum.

4. The method of claim 2, further comprising:

determining a residual risk stratum comprising those records in the plurality of records that do not have unique data values with respect to all of the plurality of identifying variables.

5. The method of claim 1, further comprising:

suppressing, in the plurality of records, data values associated with variables in the plurality of identifying variables that have low analytical utility and high identifying value.

6. The method of claim 1, further comprising:

performing a categorical transformation for at least one of the plurality of identifying variables to reduce a number of unique records in the plurality of records.

7. The method of claim 1, further comprising:

substituting at least one data value in at least one record in the plurality of records to create a substituted database.

8. The method of claim 7, further comprising:

selecting a subsample of records from the substituted database.

9. The method of claim 8, further comprising:

calibrating a set of sampling weights associated with the subsample of records so that, for a predetermined set of variables, data value totals in the subsample of records, weighted by the calibrated set of sampling weights, match data value totals in the original database.

10. The method of claim 9, wherein the calibrating step comprises:

minimizing a distance function subject to constraints on data value totals in the subsample of records for the predetermined set of variables.

11. The method of claim 8, wherein the step of selecting the subsample of records comprises:

partitioning each risk stratum in the plurality of risk strata into at least two substrata based on at least one outcome variable, thereby forming a plurality of substrata, each substratum comprising at least one record;

determining a respective subsampling probability for each substratum in the plurality of substrata; and

selecting, from the substituted database, the subsample of records based on the respective subsampling probabilities and the plurality of substrata.

12. The method of claim 11, wherein the step of determining the respective subsampling probabilities comprises:

minimizing a disclosure cost function subject to a set of variance constraints.

13. The method of claim 7, wherein the substituting step comprises:

selecting a partner record for each record in the plurality of records;

partitioning each risk stratum in the plurality of risk strata into at least two substrata based on at least one outcome variable, thereby forming a plurality of substrata, each substratum comprising at least one record;

determining a respective substitution probability for each substratum in the plurality of substrata; and

replacing data associated with at least one of the plurality of identifying variables in each record in a sample of records selected from the plurality of records, wherein (1) the sample of records is chosen based on the respective substitution probabilities, and (2) the replaced data is obtained from the corresponding partner record.

14. The method of claim 13, wherein the step of determining the respective substitution probabilities comprises:

minimizing a disclosure cost function subject to bias constraints.

15. The method of claim 13, wherein the step of selecting the partner record comprises:

selecting, for each record in the plurality of records, a partner record by minimizing, with respect to the plurality of identifying variables, a distance function between the record and a candidate partner record.

16. The method of claim 1, further comprising:

partitioning each risk stratum in the plurality of risk strata into at least two substitution substrata based on at least one outcome variable, thereby forming a plurality of substitution substrata, each substitution substratum comprising at least one record;

partitioning each risk stratum in the plurality of risk strata into at least two subsampling substrata based on at least one outcome variable, thereby forming a plurality of subsampling substrata, each subsampling substratum comprising at least one record;

determining a respective substitution probability for each substitution substratum in the plurality of substitution substrata;

determining a respective subsampling probability for each subsampling substratum in the plurality of subsampling substrata;

determining a respective misclassification probability for each record in the plurality of records, wherein each misclassification probability is the probability that the corresponding record is misclassified as a non-unique record in the subsample of records; and

calculating, for each record in the plurality of records, a measure of disclosure risk using the respective substitution probabilities, the respective subsampling probabilities, the respective misclassification probabilities, and the respective rates of unique occurrence.

17. The method of claim 16, further comprising:

determining, for each subsampling substratum in the plurality of subsampling substrata, a respective mean with respect to least one study variable;

determining, for each subsampling substratum in the plurality of subsampling substrata, a respective variance with respect to the least one study variable; and

calculating a measure of analytical utility for the at least one study variable using the respective substitution probabilities, the respective subsampling probabilities, the respective means, and the respective variances.

22. A system configured to preserve confidentiality and analytical utility of an original database comprised of a plurality of records by performing the steps recited in any one of claims 1-21.

23. A computer program product configured to store plural computer program instructions which, when executed by a computer, cause the computer perform the steps recited in any one of claims 1-21.

18. A method of substituting at least one data value in at least one record in a database comprising a plurality of records, comprising:

selecting a partner record for each record in the plurality of records; and

partitioning the plurality of records into a plurality of risk strata based on a plurality of identifying variables.

19. The method of 18, further comprising:

determining a respective substitution probability for each risk stratum in the plurality of risk strata by minimizing a disclosure loss function subject to a bias constraint; and

replacing data associated with at least one of the plurality of identifying variables in each record in a sample of records selected from the plurality of records, wherein (1) the sample of records is chosen based on the respective substitution probabilities, and (2) the replaced data is obtained from the corresponding partner record.

20. A method of selecting a subsample of records from a database comprising a plurality of records, comprising:

partitioning the plurality of records into a plurality of risk strata based on a plurality of identifying variables; and

determining a respective subsampling probability for each risk stratum in the plurality of risk strata by minimizing a disclosure loss function subject to a variance constraint.

21. The method of 20, further comprising:

selecting, from the plurality of records, the subsample of records based on the respective subsampling probabilities and the plurality of risk strata.