Methods and systems for uniquely identifying digital content for eDiscovery

• US 9,880,983 B2
• Filed: 06/02/2014
• Issued: 01/30/2018
• Est. Priority Date: 06/04/2013
• Status: Active Grant

• Current Assignee: Pagefreezer Software Incorporated
• Original Assignee: X1 Discovery, Inc.
• Inventors: Botta, Brent, Torrecillas, Nicklaus, Sleigh, Jacob Edward
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): Smith, Benjamin J
• Application Number:
  US14/293,600
• Publication Number:
  US 20140359411A1
• Time in Patent Office: 1,338 Days
• CPC Class Codes: G06F 40/197 Version control for software G06F8/71

Abstract

Systems and methods provide for the collection of content, such as webpage content, and for detection of changes in content. Files composing a document at a different time periods may be accessed and sets of hash values corresponding to files composing the document at the different periods may be calculated. A determination is made as to whether a file in the identified files at the different time periods is an HTML file, and if so an additional hash value corresponding to the HTML file is calculated. Aggregated hash values may be calculated based on hash values in the sets of hash values. A report may be generated reporting hash values for the document as it exists at the different time periods, including the hash values for the files composing the document, the additional hash values for respective HTML files, and the aggregated hash values. Changes in hash values may be indicated.

Claims

1. A method of enabling changes in website content to be detected, the method comprising:

receiving an address;

accessing at a first time period, by a computerized system comprising at least one computing device, a web page corresponding to the address;

identifying, by the system, HTML web page text of the web page accessed at the first time period;

identifying, by the system, items of content linked to by the web page accessed at the first time period;

storing the identified HTML web page text accessed at the first time period;

accessing and storing the items of content linked to by the web page accessed at the first time period;

calculating, by the system, a first hash value corresponding to the identified HTML web page text accessed at the first time period, wherein calculating the first hash value does not include in the calculation of the first hash value the items of content linked to by the web page accessed at the first time period;

calculating a first set of hash values for respective items of content linked to by the web page accessed at the first time period;

calculating a first aggregated hash value based on the first hash value and the first set of hash values for respective items of content linked to by the web page accessed at the first time period;

storing the first hash value, the first set of hash values, and the first aggregated hash value in association with a date and time corresponding to the first time period and in association with a first identifier;

accessing, at a second time period, the web page corresponding to the address;

identifying, by the system, HTML web page text of the web page accessed at the second time period;

identifying, by the system, items of content linked to by the web page accessed at the second time period;

storing the identified HTML web page text accessed at the second time period;

accessing and storing the items of content linked to by the web page accessed at the second time period;

calculating, by the system, a second hash value corresponding to the identified HTML web page text accessed at the second time period;

calculating a second set of hash values for respective items of content linked to by the web page accessed at the second time period;

calculating a second aggregated hash value based on the second hash value and the second set of hash values for respective items of content linked to by the web page accessed at the second time period;

storing the second hash value, the second set of hash values, and the second aggregated hash value in association with a date and time corresponding to the second time period;

using the first hash value, corresponding to the identified HTML web page text accessed at the first time period, and the second hash value, corresponding to the identified HTML web page text accessed at the second time period, detecting whether the webpage HTML text has changed, and if the webpage HTML text has changed, and providing a first visual indication indicating changes in the webpage HTML text;

using the first set of hash values for respective items of content linked to by the web page accessed at the first time period, and the second set of hash values for respective items of content linked to by the web page accessed at the second time period, detecting if the respective items of content linked to by the web page have changed and providing a second visual indication indicating changes in the content linked to by the web page and indicating which content linked to by the web page has changed;

using the first aggregated hash value, calculated based on the first hash value and the first set of hash values for respective items of content linked to by the web page accessed at the first time period, the second aggregated hash value, calculated based on the second hash value and the second set of hash values for respective items of content linked to by the web page accessed at the second time period, to detect whether the webpage or respective items of content linked to by the web page have changed, and providing a corresponding third visual indication.

2. The method a defined in claim 1, wherein the items of content linked to by the web page at the first time period comprises first image content, and the items of content linked to by the web page at the second time period comprises second image content different than the first image content, and the link to the first image content is the same as the link to the second image content.

3. The method as defined in claim 1, the method further comprising:

providing a user interface configured to receive from a user;

a locator, and

a depth specification;

receiving a first locator provided by the user via the user interface;

receiving a first depth specification provided by the user via the user interface;

accessing, at a third time period, a first item of content corresponding to the first locator;

accessing, at the third time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the third time period;

accessing, at a fourth time period, the first item of content corresponding to the first locator;

accessing, at the fourth time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the fourth time period;

enabling the user to detect changes in layer content based at least in part on the calculated respective hash values for each layer of content accessed during the fourth time period.

4. The method as defined in claim 1, wherein calculating the first hash value further comprises calculating an MD5 hash value.

5. The method as defined in claim 1, the method further comprising:

providing a user interface configured to receive an indication as to whether;

video files embedded in or linked to by a given web page being collected are to be collected;

a visual image of the given web page is to be generated;

receiving at the system an indication from the user via the user interface indicating whether the video files embedded in or linked to a given web page being collected are to be collected and whether a visual image of the given web page is to be generated;

at least partly in response to the indication from the user, determining whether to collect video files embedded in or linked to the given web page being collected;

at least partly in response to the indication from the user, determining whether to generate a visual image of the given web page.

6. The method as defined in claim 1, wherein providing the first visual indication, the second visual indication and the third visual indication further comprises generating a user interface including:

the address;

the first hash value, the first set of hash values, and the first time period;

the second hash value, the second set of hash values, and the second time period.

7. The method as defined in claim 1, wherein the calculated first aggregated hash value includes a value calculated based upon text between a <

body> and

a <

/body>

tag.

8. The method as defined in claim 1, wherein the address is a URL, and the calculation of the first hash value does not include the URL in the calculation of the first hash value.

9. The method as defined in claim 1, the method further comprising enabling a user to specify for which file types a dedicated hash is to be calculated by the system.

10. A method comprising:

receiving an address for a document;

identifying, by a computer system comprising at least one computing device, files composing the document, including files linked to by the document, at a first time period;

calculating, by the computer system, a first set of hash values including respective hash values corresponding to the respective accessed files composing the document at the first time period, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period;

determining, by the computer system, if a file in the identified files composing the document at the first time period is an HTML file;

at least partly in response to determining that a file in the identified files composing the document at the first time period is an HTML file, calculating, by the computer system, a first additional hash value corresponding to the HTML file, wherein calculating the first additional hash value does not include in the calculation of the first additional hash value files linked to by the document accessed at the first time period;

calculating, by the computer system, a first aggregated hash value based on hash values in the first set of hash values and on the first additional hash value;

identifying, by the computer system, files composing the document, including files linked to by the document, at a second time period;

calculating, by the computer system, a second set of hash values including respective hash values corresponding to the respective accessed files composing the document at the second time period, wherein calculating the second additional hash value does not include in the calculation of the second additional hash value files linked to by the document accessed at the second time period;

determining, by the computer system, if a file in the identified files composing the document at the second time period is an HTML file;

at least partly in response to determining that a file in the identified files composing the document at the second time period is an HTML file, calculating, by the computer system, a second additional hash value corresponding to the HTML file, wherein the second additional hash value does not include hash values for respective files linked to by the document accessed at the second time period;

calculating, by the computer system, a second aggregated hash value based on hash values in the second set of hash values and on the second additional hash value;

using the first additional hash value, corresponding to the HTML file composing the document at the first time period, and the second additional hash value, corresponding to the HTML file composing the document at the second time period, detecting whether HTML file has changed, and if the HTML file has changed, providing a first visual indication indicating that the HTML file has changed;

using the first set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the first time period, and the second set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the second time period, detecting if the respective accessed files composing the document have changed, and if the respective accessed files composing the document have changed, providing a corresponding second visual indication indicating which accessed files composing the document have changed; and

using the first aggregated hash value, calculated based on hash values in the first set of hash values, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period, and on the first additional hash value, and the second aggregated hash value, calculated based on hash values in the second set of hash values, the second set of hash values including hash values for respective files linked to by the document accessed at the second time period, and on the second additional hash value, to detect whether the document or files composing the document have changed, and providing a corresponding third visual indication.

11. The method as defined in claim 10, wherein the files composing the document comprise linked to files.

12. The method as defined in claim 10, wherein the document comprises a web page.

13. The method as defined in claim 10, wherein the document comprises a microblog page.

14. The method as defined in claim 10, the method further comprising:

providing a user interface configured to receive from a user;

a locator, and

a depth specification;

receiving a first locator provided by the user via the user interface;

receiving a first depth specification provided by the user via the user interface;

accessing, at a third time period, a first item of content corresponding to the first locator;

accessing, at the third time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the third time period;

accessing, at a fourth time period, the first item of content corresponding to the first locator;

accessing, at the fourth time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the fourth time period;

enabling the user to detect changes in layer content based at least in part on the calculated respective hash values for each layer of content accessed during the fourth time period.

15. The method a defined in claim 10, wherein the document at the first time period includes a first link, wherein the first link links to a first image file, and the document at the second time period includes the first link, wherein the first link links to a second image file and not the first image file.

16. The method as defined in claim 10, the method further comprising:

providing a user interface configured to receive an indication as to whether;

video files embedded in or linked to a given document being collected are to be collected;

a visual image of the given document is to be generated;

receiving at the system an indication from the user via the user interface indicating whether the video files embedded in or linked to a given document being collected are to be collected and whether a visual image of the given document is to be generated;

at least partly in response to the indication, determining whether to collect video files embedded in or linked to a given document being collected;

at least partly in response to the indication, determining whether to generate a visual image of the given document.

17. A system, comprising:

a computing system comprising at least one computing device;

a non-transitory computer storage medium having stored thereon executable instructions that direct the computing system to perform operations comprising;

receiving an address for a document;

identifying, files composing the document, including files linked to by the document, at a first time period;

calculating, by the computer system, a first set of hash values including respective hash values corresponding to the respective accessed files composing the document at the first time period, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period;

determining, by the computer system, if a file in the identified files composing the document at the first time period is an HTML file;

at least partly in response to determining that a file in the identified files composing the document at the first time period is an HTML file, calculating, by the computer system, a first additional hash value corresponding to the HTML file, wherein calculating the first additional hash value does not include in the calculation of the first additional hash value files linked to by the document accessed at the first time period;

calculating, by the computer system, a first aggregated hash value based on hash values in the first set of hash values and on the first additional hash value;

identifying, by the computer system, files composing the document, including files linked to by the document, at a second time period;

calculating, by the computer system, a second set of hash values including respective hash values corresponding to the respective accessed files composing the document at the second time period, the second set of hash values including hash values for respective files linked to by the document accessed at the second time period;

determining, by the computer system, if a file in the identified files composing the document at the second time period is an HTML file;

at least partly in response to determining that a file in the identified files composing the document at the second time period is an HTML file, calculating, by the computer system, a second additional hash value corresponding to the HTML file, wherein calculating the second additional hash value does not include in the calculation of the second additional hash value files linked to by the document accessed at the second time period;

calculating, by the computer system, a second aggregated hash value based on hash values in the second set of hash values and on the second additional hash value;

using the first additional hash value, corresponding to the HTML file composing the document at the first time period, and the second additional hash value, corresponding to the HTML file composing the document at the second time period, detecting whether the HTML file has changed, and if the HTML file has changed, providing a first visual indication indicating that a change to the HTML file has occurred;

using the first set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the first time period, and the second set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the second time period, detecting if the respective accessed files composing the document have changed, and if the respective accessed files composing the document have changed, providing a corresponding second visual indication indicating which accessed files composing the document have changed; and

using the first aggregated hash value, calculated based on hash values in the first set of hash values, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period, and on the first additional hash value, and the second aggregated hash value, calculated based on hash values in the second set of hash values, the second set of hash values including hash values for respective files linked to by the document accessed at the second time period, and on the second additional hash value, to detect whether the document or files composing the document have changed, and providing a corresponding third visual indication.

18. The system as defined in claim 17, wherein the files composing the document comprise linked to files.

19. The system as defined in claim 17, wherein the document comprises a web page.

20. The system as defined in claim 17, wherein the document comprises a microblog page.

21. The system as defined in claim 17, the operations further comprising:

providing a user interface configured to receive from a user;

a locator, and

a depth specification;

receiving a first locator provided by the user via the user interface;

receiving a first depth specification provided by the user via the user interface;

accessing, at a third time period, a first item of content corresponding to the first locator;

accessing, at the third time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the third time period;

accessing, at a fourth time period, the first item of content corresponding to the first locator;

accessing, at the fourth time period, hyperlinked items of content from multiple layers of content in accordance with the first depth specification;

calculating respective hash values for each layer of content accessed during the fourth time period;

enabling the user to detect changes in layer content based at least in part on the calculated respective hash values for each layer of content accessed during the fourth time period.

22. The system as defined in claim 17, wherein the document at the first time period includes a first link, wherein the first link links to a first image file, and the document at the second time period includes the first link, wherein the first link links to a second image file and not the first image file.

23. The system as defined in claim 17, the operations further comprising:

providing a user interface configured to receive an indication as to whether;

video files embedded in or linked to a given document being collected are to be collected;

a visual image of the given document is to be generated;

receiving at the system an indication from the user via the user interface indicating whether the video files embedded in or linked to a given document being collected are to be collected and whether a visual image of the given document is to be generated;

at least partly in response to the indication, determining whether to collect video files embedded in or linked to a given document being collected;

at least partly in response to the indication, determining whether to generate a visual image of the given document.

24. A method comprising:

receiving an address for a document;

identifying, by a computer system comprising at least one computing device, files composing the document, including files linked to by the document, at a first time period;

calculating, by the computer system, a first hash value based at least on text present in the document at the first time period, wherein calculating the first hash value does not include in the calculation of the first hash value files linked to by the document accessed at the first time period;

calculating, by the computer system, a first set of hash values including respective hash values corresponding to the respective accessed files composing the document at the first time period, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period;

calculating, by the computer system, a first aggregated hash value based on hash values in the first set of hash values and the first hash value;

calculating, by the computer system, a second hash value based at least on text present in the document at a second time period, wherein calculating the second hash value does not include in the calculation of the second hash value files linked to by the document accessed at the second time period;

calculating, by the computer system, a second set of hash values including respective hash values corresponding to the respective accessed files composing the document at the second time period, the second set of hash values including hash values for respective files linked to by the document accessed at the second time period;

calculating, by the computer system, a second aggregated hash value based on hash values in the second set of hash values and the second hash value;

using the first hash value, based at least on text present in the document at the first time period, and the second hash value, based at least on text present in the document at a second time period, detecting whether document text has changed, and if the document text has changed, providing a first visual indication indicating that a change to the document text has occurred;

using the first set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the first time period, and the second set of hash values, including respective hash values corresponding to the respective accessed files composing the document, including files linked to by the document, at the second time period, detecting if the respective accessed files composing the document have changed, and if the respective accessed files composing the document have changed, providing a corresponding second visual indication indicating which accessed files composing the document have changed; and

using the first aggregated hash value, calculated based on hash values in the first set of hash values and the first hash value, the first set of hash values including hash values for respective files linked to by the document accessed at the first time period, and the second aggregated hash value, calculated second aggregated hash value based on hash values in the second set of hash values, the second set of hash values including hash values for respective files linked to by the document accessed at the second time period, and the second hash value, to detect whether the document or files composing the document have changed, and providing a corresponding third visual indication.

25. The method as defined in claim 24, wherein the first set of hash values comprises one or more content hashes, corresponding to one or more HTML portions of respective accessed files composing the document, and one or more acquisition hashes, corresponding to one or more non-HTML portions of the document at the first time period.