SYSTEM AND METHOD FOR PERFORMING CODE PROVENANCE REVIEW IN A SOFTWARE DUE DILIGENCE SYSTEM

• US 20100242028A1
• Filed: 03/18/2009
• Published: 09/23/2010
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Novell Incorporated (Open Text Corporation)
• Inventors: WEIGERT, JUERGEN
• Granted Patent: US 8,307,351 B2
• Time in Patent Office: Days
• US Class Current: 717/131
• CPC Class Codes: G06F 21/105 Arrangements for software license management or administration, e.g. for managing licenses at corporate level
• CPC Class Codes: G06F 21/16 Program or content traceability, e.g. by watermarking

Abstract

A system and method is provided for performing code provenance review in a software due diligence system. In particular, performing code provenance review may include sub-dividing source code under review and third-party source into logical fragments using a language-independent text fracturing algorithm. For example, the fracturing algorithm may include a set of heuristic rules that account for variations in coding style to create logical fragments that are as large as possible without being independently copyrightable. Unique fingerprints may then be generated for the logical fragments using a fingerprint algorithm that features arithmetic computation. As such, potentially related source code may be identified if sub-dividing the source code under review and the third-party source code produces one or more logical fragments that have identical fingerprints.

Claims

1. A method for performing code provenance review in a software due diligence system, comprising:

receiving source code subject to code provenance review;

retrieving third-party source code for comparison with the source code subject to code provenance review;

fracturing the source code subject to code provenance review into a first set of logical fragments, wherein a text fracturing algorithm is used to fracture the source code subject to code provenance review into the first set of logical fragments;

fracturing the third-party source code into a second set of logical fragments, wherein the text fracturing algorithm is used to fracture the third-party source code subject into the second set of logical fragments;

generating a first set of fingerprints corresponding to the first set of logical fragments, wherein a fingerprint algorithm is used to generate the first set of fingerprints;

generating a second set of fingerprints corresponding to the logical fragments in the second set of logical fragments, wherein the fingerprint algorithm is used to generate the second set of fingerprints; and

comparing the first set of fingerprints to the second set of fingerprints to determine whether the source code subject to code provenance review contains one or more potential code provenance issues.

2. The method of claim 1, wherein the source code subject to code provenance review contains one or more potential code provenance issues if the first set of fingerprints contains two or more fingerprints that match two or more respective fingerprints in the second set of fingerprints.

3. The method of claim 2, wherein the potential code provenance issues include one or more of undocumented open source usage, improperly documented open source usage, enumerated redundancies, false license declarations, alternate licensing terms and conditions, or outdated or divergent versions of identical source code.

4. The method of claim 1, wherein the text fracturing algorithm includes one or more parameters for the logical fragments, wherein the parameters include at least one of a minimum number of lines per logical fragment, a maximum number of lines per logical fragment, a maximum number of characters per line, or a maximum number of characters per logical fragment.

5. The method of claim 4, wherein the text fracturing algorithm includes one or more rules for determining borders for the logical fragments, wherein the rules are based on language-independent source code features.

6. The method of claim 5, wherein the one or more rules require a number of newline characters in one logical fragment to be equal to or less than the parameter for the maximum number of lines per logical fragment.

7. The method of claim 5, wherein the one or more rules treat a character at a position equaling the parameter for the maximum number of characters per line as a newline character if the character appears in a line that exceeds the parameter for the maximum number of characters per line.

8. The method of claim 5, wherein the one or more rules permit a line to end in a repeated sequence of characters without increasing a logical length of the line.

9. The method of claim 5, wherein the one or more rules require all closing braces or other brackets that precede a keyword to be included in one logical fragment.

10. The method of claim 5, wherein the one or more rules place an ending border of one logical fragment between two adjacent lines if a first one of the adjacent lines begins with whitespace and a second one of the adjacent lines does not begin with whitespace.

11. The method of claim 5, wherein the one or more rules place an ending border of one logical fragment between two adjacent lines if a first non-whitespace character in one of the adjacent lines is non-alphanumeric and a first non-whitespace character in the other of the adjacent lines is alphanumeric.

12. The method of claim 11, wherein non-alphanumeric characters that typically accompany keywords or variable names in source code are considered alphanumeric, and wherein non-alphanumeric characters that typically reflect comment sections or building block structures are considered non-alphanumeric.

13. The method of claim 1, wherein the fingerprint algorithm is configured to clear whitespace from the logical fragments, wherein clearing the whitespace from the logical fragments includes:

removing leading whitespace that begins a logical fragment;

removing trailing whitespace that ends a logical fragment;

replacing internal whitespace that does not begin or end a logical fragment with one space character if a word character delimits either side of the internal whitespace; and

removing the internal whitespace if a word character does not delimit either side of the internal whitespace.

14. The method of claim 13, further comprising determining borders for the logical fragments cleared of whitespace using one or more rules based on language-independent source code features.

15. The method of claim 14, wherein the one or more rules require each of the cleared logical fragments to have a number of newline characters that is equal to or less than a parameter for a maximum number of lines per logical fragment.

16. The method of claim 14, wherein if one or more of the cleared logical fragments include a line that exceeds a parameter for a maximum number of characters per line, the one or more rules treat a character at a position equaling the parameter for the maximum number of characters per line as a newline character.

17. The method of claim 14, wherein if one or more of the cleared logical fragments include a line that ends in a repeated sequence of characters, the one or more rules provide that the repeated sequence of characters does not increase a logical length of the line.

18. The method of claim 14, wherein if one or more of the cleared logical fragments include a keyword, the one or more rules require the one or more cleared logical fragments to include all closing braces or other brackets that precede the respective keyword.

19. The method of claim 14, wherein if one or more of the cleared logical fragments include adjacent lines that respectively begin with whitespace and non-whitespace, the one or more rules place a border between the adjacent lines.

20. The method of claim 14, wherein if one or more of the cleared logical fragments include adjacent lines that respectively have an alphanumeric character and a non-alphanumeric character for a first non-whitespace character, the one or more rules place a border between the adjacent lines.

21. The method of claim 20, wherein non-alphanumeric characters that typically accompany keywords or variable names in source code are considered alphanumeric, and wherein non-alphanumeric characters that typically reflect comment sections or building block structures are considered non-alphanumeric.

22. The method of claim 1, wherein the fingerprint algorithm is configured to clear non-printable characters from the logical fragments, wherein clearing the non-printable characters from the logical fragments includes:

replacing characters that exceed a predetermined ASCII character code with a predetermined character that does not exceed the predetermined ASCII character code; and

removing leading plus, minus, less than, and greater than characters that begin of a line of a logical fragment.

23. The method of claim 1, wherein the fingerprint for each respective logical fragment is derived from a length of the respective logical fragment and a hash value computed using a static set of random numbers taken from atmospheric noise.

24. A system for performing code provenance review, comprising:

a crawler configured to retrieve third-party source code from one or more third-party repositories; and

a code provenance engine configured to;

receive source code subject to code provenance review for comparison with the third-party source code;

fracture the source code subject to code provenance review into a first set of logical fragments, wherein the code provenance engine uses a text fracturing algorithm to fracture the source code subject to code provenance review into the first set of logical fragments;

fracture the third-party source code into a second set of logical fragments, wherein the code provenance engine uses the text fracturing algorithm to fracture the third-party source code subject into the second set of logical fragments;

generate a first set of fingerprints corresponding to the first set of logical fragments, wherein the code provenance engine uses a fingerprint algorithm to generate the first set of fingerprints;

generate a second set of fingerprints corresponding to the logical fragments in the second set of logical fragments, wherein the code provenance engine uses the fingerprint algorithm to generate the second set of fingerprints; and

compare the first set of fingerprints to the second set of fingerprints to determine whether the source code subject to code provenance review contains one or more potential code provenance issues.

25. The system of claim 24, wherein the code provenance engine is further configured to determine that the source code subject to code provenance review contains one or more potential code provenance issues if the first set of fingerprints contains two or more fingerprints that match two or more respective fingerprints in the second set of fingerprints.

26. The system of claim 24, wherein the text fracturing algorithm includes one or more rules for determining borders for the logical fragments, wherein the rules are based on language-independent source code features.

27. The system of claim 24, wherein the fingerprint algorithm is configured to clear whitespace and non-printable characters from the logical fragments, wherein clearing the whitespace and non-printable characters from the logical fragments includes:

removing leading whitespace that begins a logical fragment;

removing trailing whitespace that ends a logical fragment;

replacing internal whitespace that does not begin or end a logical fragment with one space character if a word character delimits either side of the internal whitespace;

removing the internal whitespace if a word character does not delimit either side of the internal whitespace;

replacing characters that exceed a predetermined ASCII character code with a predetermined character that does not exceed the predetermined ASCII character code; and

removing leading plus, minus, less than, and greater than characters that begin of a line of a logical fragment.

28. The system of claim 24, wherein the fingerprint for each respective logical fragment is derived from a length of the respective logical fragment and a hash value computed using a static set of random numbers taken from atmospheric noise.