Extensible data path

• US 9,053,114 B1
• Filed: 08/07/2014
• Issued: 06/09/2015
• Est. Priority Date: 08/07/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Igneous Systems, Inc.
• Inventors: Lemar, Eric Michael, Rakitzis, Triantaphyllos Byron, Bhageshpur, Kiran V., Hughes, Jeffrey Douglas
• Primary Examiner(s): PEUGH, BRIAN R
• Application Number:
  US14/454,651
• Time in Patent Office: 306 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/0817 using directory methods
• CPC Class Codes: G06F 12/0831 using a bus scheme, e.g. with bus monitoring or watching means
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G11C 7/1039 using pipelining techniques, i.e. using latches between functional memory parts, e.g. row/column decoders, I/O buffers, sense amplifiers

Abstract

Embodiments are directed towards storing data in a storage system using an extensible data path. Data files may be provided to a caching tier in a storage system. If data files remain in the caching tier longer than a time limit, those data files may be removed from the caching tier and provided to a processing pipeline. The processing pipeline may be coupled to a capacity tier of the storage system. Filters to include in the processing pipeline may be determined based on the type of the data files. The data files may be updated based on applying each filter, such that each update corresponding to each filter may be cumulatively applied to each data file. Each updated data file may be stored in the capacity tier of the storage system after each filter in the processing pipeline has been applied.

Claims

1. A method for storing data in a storage system over a network using a network computer that executes instructions that perform actions, comprising:

providing at least one data file to a caching tier in a storage system; and

when a duration that the at least one data file remains in the caching tier exceeds a defined threshold, performing further actions, including;

providing the at least one data file from the caching tier to a processing pipeline coupled to a capacity tier of the storage system;

determining at least one filter to include in the processing pipeline based on at least a type of the at least one data file;

updating the at least one data file based on iteratively applying each determined filter to the at least one data file in the processing pipeline;

when a filter that is being cumulatively applied to the at least one data file exceeds a resource limit based on one or more defined threshold values that trigger resource violation during its application to the at least one data file,. canceling the application of the filter; and

storing each updated data file in the capacity tier of the storage system.

2. The method of claim 1, wherein providing the at least one data file to the caching tier further comprises, if the at least one data file is currently stored in the caching tier, resetting an age to new for the at least one data file currently stored in the caching tier.

3. The method of claim 1, wherein providing the at least one data file to the caching tier further comprises, communicating an acknowledgment to the network computer that provided the at least one data file, wherein the acknowledgment at least indicates that the at least one data file is currently stored in the caching tier.

4. The method of claim 1, further comprising, generating at least one index based on the applying of the at least one determined filter to the at least one data file.

5. The method of claim 1, further comprising, updating a filter trace that is associated with the at least one data file to include at least an identifier that corresponds to each filter that is applied to the at least one data file.

6. The method of claim 1, wherein updating the at least one data file based on applying each determined filter, further comprises, at least one of compressing the at least one data file or encrypting the at least one data file.

7. The method of claim 1, further comprising, responsive to a request to retrieve the at least one data file from the capacity tier of the storage system, applying each of the filters in a reverse order that they were previously applied to the at least one data file that was stored in the capacity tier of the storage system.

8. The method of claim 1, further comprising, initializing an execution environment for each determined filter that is at least one of a virtual machine or a process for applying the at least one data file.

9. A system for storing data in a storage system over a network, comprising:

a client computer, comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

providing at least one data file to a caching tier in a storage system; and

a network computer comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

when a duration that the at least one data file remains in the caching tier exceeds a defined threshold, performing further actions, including;

providing the at least one data file from the caching tier to a processing pipeline coupled to a capacity tier of the storage system;

determining at least one filter to include in the processing pipeline based on at least a type of the at least one data file;

updating the at least one data file based on iteratively applying each determined filter to the at least one data file in the processing pipeline,.

when a filter that is being cumulatively applied to the at least one data file exceeds a resource limit based on one or more defined threshold values that trigger resource violation during its application to the at least one data file, canceling the application of the filter; and

storing each updated data file in the capacity tier of the storage system.

10. The system of claim 9, wherein providing the at least one data file to the caching tier further comprises, if the at least one data file is currently stored in the caching tier, resetting an age to new for the at least one data file currently stored in the caching tier.

11. The system of claim 9, wherein providing the at least one data file to the caching tier further comprises, communicating an acknowledgment to the network computer that provided the at least one data file, wherein the acknowledgment at least indicates that the at least one data file is currently stored in the caching tier.

12. The system of claim 9, wherein the network computer processor device is operative to enable actions, further comprising, generating at least one index based on the applying of the at least one determined filter to the at least one data file.

13. The system of claim 9, wherein the network computer processor device is operative to enable actions, further comprising, updating a filter trace that is associated with the at least one data file to include at least an identifier that corresponds to each filter that is applied to the at least one data file.

14. The system of claim 9, wherein updating the at least one data file based on applying each determined filter, further comprises, at least one of compressing the at least one data file or encrypting the at least one data file.

15. The system of claim 9, wherein the network computer processor device is operative to enable actions, further comprising, responsive to a request to retrieve the at least one data file from the capacity tier of the storage system, applying each of the filters in a reverse order that they were previously applied to the at least one data file that was stored in the capacity tier of the storage system.

16. The system of claim 9, wherein the network computer processor device is operative to enable actions, further comprising, initializing an execution environment for each determined filter that is at least one of a virtual machine or a process for applying the at least one data file.

17. A processor readable non transitive non-transitory storage media that includes instructions for storing data in a storage system over a network, wherein execution of the instructions by a processor device enables actions, comprising:

providing at least one data file to a caching tier in a storage system; and

when a duration that the at least one data file remains in the caching tier exceeds a defined threshold, performing further actions, including;

providing the at least one data file from the caching tier to a processing pipeline coupled to a capacity tier of the storage system;

determining at least one filter to include in the processing pipeline based on at least a type of the at least one data file;

updating the at least one data file based on iteratively applying each determined filter to the at least one data file in the processing pipeline,.

when a filter that is being cumulatively applied to the at least one data file exceeds a resource limit based on one or more defined threshold values that trigger resource violation during its application to the at least one data file, canceling the application of the filter; and

storing each updated data file in the capacity tier of the storage system.

18. The media of claim 17, wherein providing the at least one data file to the caching tier further comprises, if the at least one data file is currently stored in the caching tier, resetting an age to new for the at least one data file currently stored in the caching tier.

19. The media of claim 17, wherein providing the at least one data file to the caching tier further comprises, communicating an acknowledgment to the network computer that provided the at least one data file, wherein the acknowledgment at least indicates that the at least one data file is currently stored in the caching tier.

20. The media of claim 17, further comprising, generating at least one index based on the applying of the at least one determined filter to the at least one data file.

21. The media of claim 17, further comprising, updating a filter trace that is associated with the at least one data file to include at least an identifier that corresponds to each filter that is applied to the at least one data file.

22. The media of claim 17, wherein updating the at least one data file based on applying each determined filter, further comprises, at least one of compressing the at least one data file or encrypting the at least one data file.

23. The media of claim 17, further comprising, responsive to a request to retrieve the at least one data file from the capacity tier of the storage system, applying each of the filters in a reverse order that they were previously applied to the at least one data file that was stored in the capacity tier of the storage system.

24. A network computer that is operative for storing data in a storage system over a network using a network, comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

providing at least one data file to a caching tier in a storage system; and

when a duration that the at least one data file remains in the caching tier exceeds a defined threshold, performing further actions, including;

providing the at least one data file from the caching tier to a processing pipeline coupled to a capacity tier of the storage system;

determining at least one filter to include in the processing pipeline based on at least a type of the at least one data file;

updating the at least one data file based on iteratively applying each determined filter to the at least one data file in the processing pipeline;

when a filter that is being cumulatively applied to the at least one data file exceeds a resource limit based on one or more defined threshold values that trigger resource violation during its application to the at least one data file,. canceling the application of the filter; and

storing each updated data file in the capacity tier of the storage system.

25. The network computer of claim 24, wherein providing the at least one data file to the caching tier further comprises, if the at least one data file is currently stored in the caching tier, resetting an age to new for the at least one data file currently stored in the caching tier.

26. The network computer of claim 24, wherein providing the at least one data file to the caching tier further comprises, communicating an acknowledgment to the network computer that provided the at least one data file, wherein the acknowledgment at least indicates that the at least one data file is currently stored in the caching tier.

27. The network computer of claim 24, wherein the network computer processor device is operative to enable actions, further comprising, generating at least one index based on the applying of the at least one determined filter to the at least one data file.

28. The network computer of claim 24, wherein the network computer processor device is operative to enable actions, further comprising, updating a filter trace that is associated with the at least one data file to include at least an identifier that corresponds to each filter that is applied to the at least one data file.

29. The network computer of claim 24, wherein updating the at least one data file based on applying each determined filter, further comprises, at least one of compressing the at least one data file or encrypting the at least one data file.

30. The network computer of claim 24, wherein the network computer processor device is operative to enable actions, further comprising, responsive to a request to retrieve the at least one data file from the capacity tier of the storage system, applying each of the filters in a reverse order that they were previously applied to the at least one data file that was stored in the capacity tier of the storage system.