FILE IDENTIFICATION MANAGEMENT AND TRACKING

• US 20120254207A1
• Filed: 03/30/2011
• Published: 10/04/2012
• Est. Priority Date: 03/30/2011
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Blank, Mitchell Neuman Jr., Patel, Vishal, Sorkin, Stephen Phillip, Bath, Amritpal Singh
• Granted Patent: US 8,566,336 B2
• Time in Patent Office: Days
• US Class Current: 707/758
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data

Abstract

Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.

Claims

1. A network device, comprising:

a data store configured to include a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

a transceiver for receiving an item over a network; and

a processor for enabling actions, comprising;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein at least one of the generated two or more item identifiers is based on a predetermined amount of content at an end portion of the item;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers in the record; and

when it is determined that the item is a new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the item.

2. The network device of claim 1, wherein the processor further enables actions, comprising:

when it is determined that the item is an existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

3. The network device of claim 1, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating a first item identifier based on a name of the item;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item; and

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

4. The network device of claim 1, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating a first item identifier based on the decompressed content;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating a second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

5. The network device of claim 1, wherein generating the two or more item identifiers, further including:

generating a first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items having a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

6. The network device of claim 1, wherein generating the two or more item identifiers further includes employing a cyclic redundancy check on data associated with the item.

7. The network device of claim 1, wherein the processor further enables actions, comprising:

processing the item to a current end of the item;

obtaining a stored end of item pointer from the record about the existing item that matches the item;

determining whether the item is completely written to based on a comparison of the current end of the item and the stored end of item pointer; and

when it is determined the item is completely written to, returning a done command to indicate that the item is completely written to.

8. A method operating on one or more network devices, the method implemented as one or more computer executable instructions for performing actions, comprising:

storing a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

receiving an item;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein at least one of the generated two or more item identifiers is based on a predetermined amount of content at an end portion of the item;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers in the record; and

when it is determined that the item is a new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the item.

9. The method of claim 8, further comprising:

when it is determined that the item is an existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

10. The method of claim 8, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating a first item identifier based on a name of the item;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item; and

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

11. The method of claim 8, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating a first item identifier based on the decompressed content;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating a second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

12. The method of claim 8, wherein generating the two or more item identifiers, further including:

generating a first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

13. A system, comprising:

a storage device for storing a record about a existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item; and

one or more network devices that are configured to receive an item and to perform actions, including;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein at least one of the generated two or more item identifiers is based on a predetermined amount of content at an end portion of the item;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers in the record; and

when it is determined that the item is a new item based on the comparison, storing an other record about the new item in the storage device, wherein the other record includes at least the generated two or more item identifiers about the item.

14. The system of claim 13, wherein the one or more network devices perform actions, the actions further comprising:

when it is determined that the item is an existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

15. The system of claim 13, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating a first item identifier based on a name of the item;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item; and

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

16. The system of claim 13, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating a first item identifier based on the decompressed content;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating a second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

17. The system of claim 13, wherein generating the two or more item identifiers, further including:

generating a first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

18. A computer-readable storage device that includes data and instructions, that when installed within on a computer device causes the computer device to perform actions, comprising:

storing a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

receiving an item;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein at least one of the generated two or more item identifiers is based on a predetermined amount of content at an end portion of the item;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers in the record; and

when it is determined that the item is a new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the item.

19. The computer-readable storage device of claim 18, wherein the instructions perform actions, further comprising:

when it is determined that the item is an existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

20. The computer-readable storage device of claim 18, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating a first item identifier based on a name of the item;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item; and

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

21. The computer-readable storage device of claim 18, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating a first item identifier based on the decompressed content;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating a second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

22. The computer-readable storage device of claim 18, wherein generating the two or more item identifiers, further including:

generating a first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining a stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the item;

generating a second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.