SYSTEMS, METHODS AND DEVICES FOR INTEGRATING END-HOST AND NETWORK RESOURCES IN DISTRIBUTED MEMORY

• US 20160323237A1
• Filed: 07/08/2016
• Published: 11/03/2016
• Est. Priority Date: 07/17/2012
• Status: Active Grant

• Current Assignee: Open Invention Network LLC
• Original Assignee: Coho Data, Inc.
• Inventors: Warfield, Andrew, Wires, Jacob Taylor, Stodden, Daniel, Meyer, Dutch, Guyader, Jean Maurice Guy, Fraser, Keir, Deegan, Timothy John, Cully, Brendan Anthony, Clark, Christopher, Jamieson, Kevin, Lefebvre, Geoffrey
• Granted Patent: US 10,341,285 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 2101/686 using dual-stack hosts, e.g. in Internet protocol version 4 [IPv4]/Internet protocol version 6 [IPv6] networks
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 45/74 Address processing for routing
• CPC Class Codes: H04L 61/103 across network layers, e.g. resolution of network layer into physical layer addresses or address resolution protocol [ARP]
• CPC Class Codes: H04L 61/5007 Internet protocol [IP] addresses
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems, methods and devices for distributed memory management comprising a network component configured for network communication with one or more memory resources that store data and one or more consumer devices that use data, the network component comprising a switching device in operative communication with a mapping resource, wherein the mapping resource is configured to associate mappings between data addresses associated with memory requests from a consumer device relating to a data object and information relating to a storage location in the one or more memory resources associated with the data from the data object, wherein each data address has contained therein identification information for identifying the data from the data object associated with that data address; and the switching device is configured to route memory requests based on the mappings.

Claims

1. A network switching interface for a distributed digital data storage system, said network switching interface comprising:

a plurality of network interface ports for interfacing, over a digital network, a plurality of digital data storage devices for storing data and one or more client devices that send data requests, each of said data requests relating to a portion of said data and comprising a data address associated with the distributed digital data system, wherein said data address comprises therein embedded information that identifies one or more data characteristics of said portion;

a physical mapping database, comprising a forwarding table in accessible memory storage for associating each said data address with a given digital data storage device communicatively connected to the network interface ports; and

a switching device that routes a given data request to a given digital data storage device associated in the forwarding table with a given data address of said given data request;

wherein said data is remapped in said forwarding table such that at least one said data address is remapped from an association with a first one of said digital data storage devices to a second one of said data device storage devices based on said one or more data characteristics.

2. The network switching interface of claim 1, wherein the embedded information in the data address identifies at least one of said data characteristics of said portion based on an encoding convention.

3. The network switching interface of claim 2, wherein the embedded information is embedded in the data address in accordance with an embedding convention.

4. The network switching interface of claim 1, wherein the data address is an IP address.

5. The network switching interface of claim 4, wherein the IP address is selected from one of the following group:

an IPv4 address and an IPv6 address.

6. The network switching interface of claim 1, wherein the data address is an Ethernet address.

7. The network switching interface of claim 1, wherein the physical mapping database is further configured to dynamically generate associations between data addresses and information relating to storage locations in the plurality of digital data storage devices.

8. The network switching interface of claim 7, wherein the generation of mappings is in accordance with a routing methodology.

9. The network switching interface of claim 8, wherein the routing methodology is based one of the following:

a prefix of the data address, a portion of the data address, a static mapping, striping, hash function, round-robin assignment, assessment of operational characteristics, and a combination thereof.

10. The network switching interface of claim 1, wherein the data is remapped due to a redistribution of the data amongst the plurality of digital data storage devices.

11. The network switching interface of claim 1, wherein a given data remapping causes a redistribution of the data amongst the plurality of digital data storage devices based on the given data remapping.

12. The network switching interface of claim 1, wherein the data corresponding to the at least one remapped data address is stored on at least the second one of the digital data storage devices prior to having been remapped.

13. The network switching interface of claim 1, wherein the second one of the digital data storage devices has at least one first operational characteristic different from a corresponding operational characteristic of the first one of digital data storage devices.

14. The network switching interface of claim 13, wherein the at least one remapped data address is selectively remapped to the second one of the digital data storage devices based on at least one of the one or more data characteristics.

15. The network switching interface of claim 13, wherein the at least one remapped data address is selectively remapped to the second one of the digital data storage devices based on said difference.

16. The network switching interface of claim 1, wherein the one or more characteristics comprises at least one of:

a position of said portion in the data and an identifier of the data.

17. The network switching interface of claim 1, wherein the one or more characteristics comprises at least one of:

an operational characteristic of the data, a user of the data, a use of the data, and a type of the data.

18. The network switching interface of claim 1, wherein a data remapping causes a redistribution of data amongst the plurality of digital data storage devices based on said data remapping.

19. A distributed digital data system for storing and accessing data, said system comprising:

a network switching interface comprising a plurality of network interface ports for interfacing, via network communication, a plurality of digital data storage devices for storing said data and one or more client devices that send data requests relating to said data, the network switching interface further comprising a physical mapping database, wherein the physical mapping database comprises a forwarding table stored in accessible physical memory for associating data addresses with information relating to a storage location on the one or more digital data storage devices;

at least one client device comprising a client data processor and a stored set of instructions that, when executed;

associate a given data address from a range of data addresses associated with said distributed digital data storage system with a given portion of data, wherein said given data address comprises therein embedded information identifying one or more characteristics of said given portion of said data; and

send data requests relating to said given portion of the data with the given data address associated therewith; and

at least one digital data storage device for storing said portion of the data in storage locations in accordance with the corresponding associations in the forwarding table, wherein said data is remapped in said forwarding table amongst storage locations in at least one of said at least one digital data storage devices based on said one or more characteristics of said given portion.

20. The distributed digital data system of claim 19, wherein said data is remapped in response to a redistribution of data amongst the plurality of digital data storage devices.

21. The distributed digital data system of claim 19, wherein the one or more characteristics include at least one of:

a position of said portion in the data and an identifier of the data.

22. The distributed digital data system of claim 19, wherein the one or more characteristics include at least one of:

an operational characteristic of the data, a user of the data, a use of the data, and a type of the data.

23. A method for storing data across distributed digital data storage devices over a digital network, the method comprising:

generating for a portion of data a data address from a range of addresses associated with the distributed digital memory resources, the data address comprising therein embedded information that identifies one or more characteristics of said portion;

sending data requests relating to the portion of data over the digital network to a digital network switching interface using said data address, said digital network switching interface comprising a physical mapping database and plurality of network interface ports, said physical mapping database comprising access to a forwarding table stored on accessible physical memory;

receiving the data request at the digital network switching interface and checking to see if the data address has been associated in the forwarding table with information relating to a storage location in the distributed digital data storage devices communicatively coupled to the digital network switching interface;

forwarding the data request to the storage location associated with the data address in the data address index, if the data address is associated with information relating to a storage location in the forwarding table, else forwarding the data request to the distributed digital data storage devices in accordance with a data routing methodology; and

remapping the association between a given data address and the corresponding information relating to a given storage location in the forwarding table based on said one or more characteristics.

24. The method of claim 23, further comprising:

receiving operational characteristics of at least one of;

the digital data storage devices, the digital network, the data, the client devices, and users of the client devices; and

in response to the operational characteristics, remapping in the forwarding table at least some of the portions of the data to different storage locations in accordance with one or more pre-existing storage objectives based on the one or more characteristics of the one or more portions.

25. The method of claim 23, wherein the data routing methodology is based on one of:

a prefix of the data address, a portion of the data address, a static mapping, striping, a hashing function, round-robin assignment, assessment of operational characteristics, or a combination thereof.

26. The method of claim 23, wherein the remapping is caused by a redistribution of data amongst the plurality of digital data storage devices.

27. The method of claim 23, wherein the remapping causes a redistribution of data amongst the plurality of digital data storage devices based on said remapping.

28. The method of claim 23, wherein the one or more characteristics include at least one of the following:

a position of said portion in the data and an identifier of the data.

29. The method of claim 23, wherein the one or more characteristics include at least one of the following:

an operational characteristic of the data, a user of the data, a use of the data, and a type of the data.