File identification management and tracking

• US 8,566,336 B2
• Filed: 03/30/2011
• Issued: 10/22/2013
• Est. Priority Date: 03/30/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bath, Amritpal Singh, Blank, Mitchell Neuman Jr., Patel, Vishal, Sorkin, Stephen Phillip
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US13/076,296
• Publication Number:
  US 20120254207A1
• Time in Patent Office: 937 Days
• Field of Search: None
• US Class Current: 707/763
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data

Abstract

Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.

Claims

1. A network device, comprising:

a data store configured to include a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

a transceiver for receiving an item over a network; and

a processor for enabling actions, comprising;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein generating the two or more item identifiers includes;

obtaining a stored end of item pointer based on a first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the existing item; and

generating a second item identifier based on a predetermined amount of content at an end portion of the item, wherein the end portion is associated with the stored end of item pointer;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers about the existing item in the record, wherein the new item is separate from the existing item; and

when it is determined that the item is the new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the new item.

2. The network device of claim 1, wherein the processor further enables actions, comprising:

when it is determined that the item is the existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

3. The network device of claim 1, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating the first item identifier based on a name of the item;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item; and

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

4. The network device of claim 1, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating the first item identifier based on the decompressed content;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating the second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

5. The network device of claim 1, wherein generating the two or more item identifiers, further including:

generating the first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items having a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

6. The network device of claim 1, wherein generating the two or more item identifiers further includes employing a cyclic redundancy check on data associated with the item.

7. The network device of claim 1, wherein the processor further enables actions, comprising:

processing the item to a current end of the item;

obtaining the stored end of item pointer from the record about the existing item that matches the item;

determining whether the item is completely written to based on a comparison of the current end of the item and the stored end of item pointer; and

when it is determined the item is completely written to, returning a done command to indicate that the item is completely written to.

8. A method operating on one or more network devices, the method implemented as one or more computer executable instructions for performing actions, comprising:

storing a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

receiving an item;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein generating the two or more item identifiers includes;

obtaining a stored end of item pointer based on a first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the existing item; and

generating a second item identifier based on a predetermined amount of content at an end portion of the item, wherein the end portion is associated with the stored end of item pointer;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers about the existing item in the record, wherein the new item is separate from the existing item; and

when it is determined that the item is the new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the new item.

9. The method of claim 8, further comprising:

when it is determined that the item is the existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

10. The method of claim 8, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating the first item identifier based on a name of the item;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item; and

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

11. The method of claim 8, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating the first item identifier based on the decompressed content;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating the second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

12. The method of claim 8, wherein generating the two or more item identifiers, further including:

generating the first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

13. A system, comprising:

a storage device for storing a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item; and

one or more network devices that are configured to receive an item and to perform actions, including;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein generating the two or more item identifiers includes;

obtaining a stored end of item pointer based on a first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the existing item; and

generating a second item identifier based on a predetermined amount of content at an end portion of the item, wherein the end portion is associated with the stored end of item pointer;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers about the existing item in the record, wherein the new item is separate from the existing item; and

when it is determined that the item is the new item based on the comparison, storing an other record about the new item in the storage device, wherein the other record includes at least the generated two or more item identifiers about the new item.

14. The system of claim 13, wherein the one or more network devices perform actions, the actions further comprising:

when it is determined that the item is the existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

15. The system of claim 13, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating the first item identifier based on a name of the item;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item; and

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

16. The system of claim 13, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating the first item identifier based on the decompressed content;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating the second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

17. The system of claim 13, wherein generating the two or more item identifiers, further including:

generating the first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.

18. A computer-readable non-transitory storage device that includes data and instructions, that when installed within on a computer device causes the computer device to perform actions, comprising:

storing a record about an existing item that has been previously processed, wherein the record includes two or more item identifiers about the existing item;

receiving an item;

generating two or more item identifiers about the item, wherein each of the two or more item identifiers is generated from at least some different data associated with the item, and wherein generating the two or more item identifiers includes;

obtaining a stored end of item pointer based on a first item identifier from the record, wherein the stored end of item pointer is a reference to an end of the item from previously processing the existing item; and

generating a second item identifier is based on a predetermined amount of content at an end portion of the item, wherein the end portion is associated with the stored end of item pointer;

determining whether the item is a new item based on a comparison of the generated two or more item identifiers and the two or more item identifiers about the existing item in the record, and wherein the new item is separate from the existing item; and

when it is determined that the item is the new item based on the comparison, storing an other record about the new item in the data store, wherein the other record includes at least the generated two or more item identifiers about the new item.

19. The computer-readable non-transitory storage device of claim 18, wherein the instructions perform actions, further comprising:

when it is determined that the item is the existing item based on a match of the generated two or more item identifiers and the two or more item identifiers in the record, processing the item to determine if there is an actual change to the item based on a comparison of the item and the record; and

when it is determined that there is an actual change, storing the actual change in the record.

20. The computer-readable non-transitory storage device of claim 18, wherein generating the two or more item identifiers, further including:

based on a determination that an item size of the item is less than a defined amount of data for an item identifier, performing actions, including;

generating the first item identifier based on a name of the item;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item; and

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the predetermined amount of content is based on the stored end of item pointer, and wherein the end portion is associated with the stored end of item pointer.

21. The computer-readable non-transitory storage device of claim 18, wherein generating the two or more item identifiers, further including:

based on a determination that the item is compressed, performing actions, including;

decompressing a defined amount of content at a beginning of the item;

generating the first item identifier based on the decompressed content;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

decompressing additional content of the item up to the stored end of item pointer; and

generating the second item identifier from the additional decompressed content, such that the second item identifier is based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer.

22. The computer-readable non-transitory storage device of claim 18, wherein generating the two or more item identifiers, further including:

generating the first item identifier based on a defined amount of content at a beginning of the item;

determining whether the item has an identifier collision, at least in part based on two or more items have a same first item identifier; and

based on the determination that the item has an identifier collision, performing actions, including;

obtaining the stored end of item pointer based on the first item identifier from the record, wherein the stored end of item pointer is the reference to the end of the item from previously processing the item;

generating the second item identifier based on the predetermined amount of content at the end portion of the item, wherein the end portion is associated with the stored end of item pointer; and

generating a third item identifier based on a name of the item.