High-accuracy confidential data detection

• US 8,108,370 B1
• Filed: 12/30/2010
• Issued: 01/31/2012
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Bruening, Oskar
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US12/982,608
• Time in Patent Office: 397 Days
• Field of Search: 707/688, 707/689, 707/690, 707/692, 707/694, 707/827, 707/697, 707/698, 707/661, 726/26, 726/27
• US Class Current: 707/697
• CPC Class Codes: G06F 16/33 Querying

Abstract

A method and apparatus for providing accurate detection of confidential information is described. In one embodiment, the method includes searching a text document for multiple classified data patterns associated with confidential information that is represented as personal identifiers. The method further includes finding, in the text document, one or more personal identifier candidates matching any of the classified data patterns, and validating each of the personal identifier candidates using one or more personal identifier validators to provide accurate detection of the confidential information in the text document.

Claims

1. A computer-implemented method comprising:

storing, in memory, a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

identifying a text document to be searched;

searching the text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats;

finding, in the text document, one or more personal identifier candidates matching any of the plurality of classified data patterns; and

validating each of the personal identifier candidates using one or more personal identifier validators to provide accurate detection of the confidential information in the text document.

2. The method of claim 1 wherein the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number.

3. The method of claim 1 wherein a list of the plurality of classified data patterns is configurable based on user input.

4. The method of claim 1 wherein a list of the validators is configurable based on user input.

5. The method of claim 1 wherein searching the text document comprises:

performing a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks.

6. The method of claim 5 wherein the set of bitmasks comprises:

bitmasks associated with the plurality of classified data patterns;

bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link; and

bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

7. The method of claim 1 wherein finding, in the text document, one or more personal identifier candidates matching any of the plurality of classified data patterns comprises:

finding a plurality of matches; and

if any of the plurality of matches overlap, reconcile overlapped matches.

8. The method of claim 1 wherein validating each of the personal identifier candidates comprises:

eliminating preliminary false positives based on data immediately preceding or following each of the personal identifier candidates; and

applying a list of validators to remaining candidates to eliminate additional false positives.

10. The method of claim 1 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.

9. The method 8 wherein the data immediately preceding or following a personal identifier candidate indicates a false positive if the personal identifier candidate is immediately preceded, or immediately followed by, a number or a letter.

11. A system comprising:

a memory to store a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

a processor, coupled to the memory, to;

search a text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats,

find, in the text document, one or more personal identifier candidates matching any of the plurality of classified data patterns, and

validate each of the personal identifier candidates using one or more personal identifier validators to provide accurate detection of the confidential information in the text document.

12. The system of claim 11 wherein:

the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number;

a list of the plurality of classified data patterns is configurable based on user input; and

a list of the validators is configurable based on user input.

13. The system of claim 11 wherein the search engine is to perform a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks, the set of bitmasks comprising bitmasks associated with the plurality of classified data patterns, bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link, and bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

14. The system of claim 11 wherein the validation engine is to validate each of the personal identifier candidates by

eliminating preliminary false positives based on data immediately preceding or following each of the personal identifier candidates, and

applying a list of validators to remaining candidates to eliminate additional false positives.

15. The system of claim 11 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.

16. A non-transitory computer readable storage medium that provides instructions, which when executed on a processing system cause the processing system to perform a method comprising:

storing, in memory, a plurality of classified data patterns for personal identifiers, the plurality of classified data patterns corresponding to variations of personal identifier formats, the personal identifiers including confidential information of a plurality of entities;

identifying a text document to be searched;

searching the text document for data expressed in a format that matches any of the plurality of classified data patterns corresponding to the variations of personal identifier formats;

finding, in the text document, one or more personal identifier candidates matching any of the plurality of classified data patterns; and

validating each of the personal identifier candidates using one or more personal identifier validators to provide accurate detection of the confidential information in the text document.

17. The computer readable storage medium of claim 16 wherein:

the personal identifiers comprises one or more of a credit card number, a social security number, an account number, an employee number, a customer or patient number, an IP address, a driver license number, and a license plate number;

a list of the plurality of classified data patterns is configurable based on user input; and

a list of the validators is configurable based on user input.

18. The computer readable storage medium of claim 16 wherein searching the text document comprises:

performing a search for the plurality of classified data patterns in parallel using a single finite state machine (FSM) and a set of bitmasks, the set of bitmasks comprising bitmasks associated with the plurality of classified data patterns, bitmasks associated with a plurality of transition links of the FSM, the bitmasks identifying classified data patterns that share a relevant transition link, and bitmasks associated with a plurality of nodes of the FSM, each bitmask identifying data patterns exiting at a relevant node.

19. The computer readable storage medium of claim 16 wherein validating each of the personal identifier candidates comprises:

eliminating preliminary false positives based on data immediately preceding or following each of the personal identifier candidates; and

applying a list of validators to remaining candidates to eliminate additional false positives.

20. The computer readable storage medium of claim 16 wherein the validators comprise one or more of a checksum algorithm, a check against valid ranges of numbers within a personal identifier candidate, a check against invalid ranges of numbers within a personal identifier candidate, a check against invalid personal identifiers, a check for a valid prefix of a personal identifier candidate, a check for a valid suffix of a personal identifier candidate, a check for a valid format of a personal identifier candidate, a check for valid digits within a personal identifier candidate, a check for a valid number of digits or characters within a personal identifier candidate, or a search of the document for one or more keywords.