High-accuracy confidential data detection

• US 7,885,944 B1
• Filed: 03/28/2008
• Issued: 02/08/2011
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Bruening, Oskar
• Primary Examiner(s): Corrielus; Jean M
• Application Number:
  US12/079,647
• Time in Patent Office: 1,047 Days
• Field of Search: 707/694, 707/827, 726/26, 726/27
• US Class Current: 707/694
• CPC Class Codes: G06F 16/33 Querying

Abstract

A method and apparatus for providing accurate detection of confidential information is described. In one embodiment, the method includes searching a text document for multiple classified data patterns associated with confidential information that is represented as personal identifiers. The method further includes finding, in the text document, one or more personal identifier candidates matching any of the classified data patterns, and validating each of the personal identifier candidates using one or more personal identifier validators to provide accurate detection of the confidential information in the text document.

Claims

1. A computer-implemented method comprising:

storing, in memory, a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

identifying a text document to be searched;

searching the text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats;

finding, in the text document, one or more sets of data having the format that matches any of the plurality of classified data patterns, each found set of data representing a possible candidate of a personal identifier; and

validating each of the found sets of data from the text document using one or more personal identifier validators to determine whether the found set of data is a personal identifier or a false positive, wherein validating each of the found sets of data from the text document comprises;

eliminating false positives based on data immediately preceding or following each of the personal identifier candidates, wherein the data immediately preceding or following a personal identifier candidate indicates a false positive if the personal identifier candidate is immediately preceded, or immediately followed by, a number or a letter.

2. The method of claim 1 wherein the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number.

3. The method of claim 1 wherein a list of the plurality of classified data patterns is configurable based on user input.

4. The method of claim 1 wherein a list of the validators is configurable based on user input.

5. The method of claim 1 wherein searching the text document comprises:

performing a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks.

6. The method of claim 5 wherein the set of bitmasks comprises:

bitmasks associated with the plurality of classified data patterns;

bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link; and

bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

7. The method of claim 1 wherein finding, in the text document, one or more sets of data having the format that matches any of the plurality of classified data patterns comprises:

finding a plurality of matches; and

if any of the plurality of matches overlap, reconcile overlapped matches.

8. The method of claim 1 wherein validating each of the found sets of data from the text document comprises:

applying a list of validators to remaining candidates to eliminate additional false positives.

9. The method of claim 1 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.

10. A system comprising:

a memory to store a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

a processor, coupled to the memory;

a search engine, executed from the memory by the processor, to identify a text document to be searched, to search the text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats, and to find, in the text document, one or more sets of data having the format that matches any of the plurality of classified data patterns, each found set of data representing a possible candidate of a personal identifier; and

a validation engine, coupled to the search engine, to validate each of the found sets of data from the text document using one or more personal identifier validators to determine whether the found set of data is a personal identifier or a false positive, wherein the validation engine is to validate each of the found sets of data from the text document based on eliminating false positives based on data immediately preceding or following each of the personal identifier candidates, wherein the data immediately preceding or following a personal identifier candidate indicates a false positive if the personal identifier candidate is immediately preceded, or immediately followed by, a number or a letter.

11. The system of claim 10 wherein:

the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number;

a list of the plurality of classified data patterns is configurable based on user input; and

a list of the validators is configurable based on user input.

12. The system of claim 10 wherein the search engine is to perform a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks, the set of bitmasks comprising bitmasks associated with the plurality of classified data patterns, bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link, and bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

13. The system of claim 10 wherein the validation engine is to validate each of the found sets of data by

applying a list of validators to remaining candidates to eliminate additional false positives.

14. The system of claim 10 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.

15. A non-transitory computer readable storage medium that provides instructions, which when executed on a processing system cause the processing system to perform a method comprising:

storing, in memory, a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

identifying a text document to be searched;

searching the text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats;

finding, in the text document, one or more sets of data having the format that matches any of the plurality of classified data patterns, each found set of data representing a possible candidate of a personal identifier; and

validating each of the found sets of data from the text document using one or more personal identifier validators to determine whether the found set of data is a personal identifier or a false positive, wherein validating each of the found sets of data from the text document comprises;

eliminating false positives based on data immediately preceding or following each of the personal identifier candidates, wherein the data immediately preceding or following a personal identifier candidate indicates a false positive if the personal identifier candidate is immediately preceded, or immediately followed by, a number or a letter.

16. The non-transitory computer readable storage medium of claim 15 wherein:

the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number;

a list of the plurality of classified data patterns is configurable based on user input; and

a list of the validators is configurable based on user input.

17. The non-transitory computer readable storage medium of claim 15 wherein searching the text document comprises:

performing a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks, the set of bitmasks comprising bitmasks associated with the plurality of classified data patterns, bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link, and bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

18. The non-transitory computer readable storage medium of claim 15 wherein validating each of the found sets of data comprises:

applying a list of validators to remaining candidates to eliminate additional false positives.

19. The non-transitory computer readable storage medium of claim 15 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.