Method and apparatus for data management

• US 20060190243A1
• Filed: 02/21/2006
• Published: 08/24/2006
• Est. Priority Date: 02/24/2005
• Status: Active Grant

• Current Assignee: Northend Networks Ltd.
• Original Assignee: XEROUND INC.
• Inventors: Klar, Nir, Levy, Roni, Broude, Zeev, Shomer, Ayelet, Kaminer, Iris, Romem, Yaniv, Vigder, Avi, Zlotkin, Gilad, Gilderman, Ilia, Barkai, Sharon
• Granted Patent: US 7,644,087 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast

Abstract

A data access system decouples the data processing from the data storage to provide improved accessibility, integrity, scalability and other features. The system comprises: database units arranged in virtual partitions each independently accessible, a plurality of data processing units, and a switching network for switching the data processing units between the virtual partitions, thereby to assign data processing capacity dynamically to respective virtual partitions.

Claims

1. A data access system comprising:

database units arranged in virtual partitions each independently accessible,

a plurality of data processing units, and

a switching network for switching said data processing units between said virtual partitions thereby to assign data processing capacity dynamically to respective virtual partitions.

2. The data access system of claim 1, wherein said switching network comprises an interconnection of switching units.

3. The data access system of claim 1, wherein each database unit is independently accessible as a respective network channel.

4. The data access system of claim 1, further comprising a hashing unit for carrying out a hashing process on data and wherein data is assigned to respective database units via a result of said hashing process.

5. The data access system of claim 4, wherein data is assigned in the form of records having a primary key and at least one secondary keys, and wherein the hashing process is carried out on said primary key.

6. The data access system of claim 5, configured with at least one router to tabulate a relationship between a secondary key and said hashed primary key, such that a search query based on a respective secondary key can be related via said router to said primary key.

7. The data access system of claim 5, configured with at least one additional automatically managed internal index table that maps the relationship between a secondary key and said hashed primary key, such that a search query based on a respective secondary key can be related via said internal index table to said primary key.

8. The data access system of claim 1, wherein data is replicated at least once over at least two data partitions.

9. The data access system of claim 1, comprising election functionality for dynamically assigning one of said data processing units as a coordinator to arbitrate between conflicting write operations.

10. The data access system of claim 9, wherein said coordinator is configured to signal regularly that it is continuing as coordinator, and wherein said election functionality is configured to repeat said dynamically assigning when said regular signal ceases.

11. The data access system of claim 10, wherein a write operation interrupted by said dynamic assigning is resumed from a most advanced recoverable position when said dynamic assigning is concluded.

12. The data access system of claim 9, wherein said coordinator is configured to assign a unique certificate to a record following a record-altering operation, thereby to render versions of said record comparable.

13. The data access system of claim 4, wherein data is assigned in the form of records having at least three keys and wherein each record is assigned a primary address based on said one of the keys and secondary addresses based on remaining ones of said keys.

14. The data access system of claim 13, comprising a resolution unit for resolving secondary addresses into corresponding primary addresses, therewith to find a record defined by a secondary key using a corresponding primary key.

15. The data access system of claim 14, wherein said resolution unit comprises at least one router.

16. The data access system of claim 15, wherein said resolution unit f urther comprises at least one backup router.

17. The data access system of claim 8, wherein said switching mechanism is configured to reassign data partition versions to remaining ones of said data processing units following a failure of one or more of said data processing units such that availability of all of said virtual partitions is maintained.

18. The data access system of claim 1, wherein each virtual partition is stored on a predetermined number of data processing units, such that after the failure of a given data processing unit, all data remains accessible.

19. The data access system of claim 18, wherein said predetermined number is at least three.

20. The data access system of claim 19, wherein said number being at least three is an odd number, thereby allowing majority voting between said copied virtual partitions to ensure integrity of said data.

21. The data access system of claim 20, wherein said odd number is at least five.

22. The data access system of claim 1, further comprising a use measurement function for measuring usage by individual customers of said data access system, and a billing functionality for billing said customers based on peak usage thereof.

23. A data access system comprising:

data processing units,

data storage units,

a switching system therebetween to dynamically switch between said data processing units and said data storage units, and

a use measurement function for measuring usage by individual customers of said data access system, and a billing functionality for billing said customers based on peak usage thereof.

24. A method of providing a high availability, high scalability data storage and query system comprising:

providing a data query arrangement, providing a data storage arrangement separate from said data query arrangement, and

providing a switching system to dynamically connect between said data storage arrangement and said data query arrangement under influence of a current query.

25. The method of claim 24, comprising providing said data storage arrangement as a plurality of channels.

26. The method of claim 25, comprising storing each data item as a record and providing copies of any given data record in a predetermined number of said channels.

27. The method of claim 26, wherein said predetermined number is an odd number.

28. The method of claim 27, further comprising using majority voting between said odd number of copies to ensure integrity of said data.

29. The method of claim 26, comprising setting a field of said data records as a primary key and hashing said primary key for addressing said channels.

30. The method of claim 29, comprising setting a field of said data records as a secondary key and providing at least one router for correlating between said secondary key and said primary key.

31. The method of claim 25, wherein said channels are publish subscribe channels.

32. The method of claim 24, wherein said data storage arrangement comprises a plurality of data storage units, the method comprising storing data in multiple copies at a plurality of data storage units, and upon detection of failure of any given data storage unit making additional copies from elsewhere of data units stored in said given data storage unit.

33. The method of claim 32, wherein said data is accessed via queries and wherein a response to a given queries depends on a current state in relation to said data.

34. The method of claim 33, further comprising retaining said current state in the event of a failure of at least one data storage unit by explicit state synchronization between respective data storage units.

35. The method of claim 34, wherein said explicit state synchronization is pull synchronization.

36. The method of claim 34, wherein said explicit state synchronization is push synchronization.

37. The method of claim 24, further comprising measuring usage by customers.

38. The method of claim 37, wherein said usage is measured as peak usage.

39. The method of claim 38 comprising billing said customers based on said peak usage.

40. A method of providing a data repository having a data storage resource and a data processing resource, comprising:

providing dynamic partitioning of said data storage resource;

providing dynamic allocation of said data processing resource; and

using dynamic switching between said data storage resource and said data processing resource such that said dynamic partitioning of said data storage resource is decoupled from said dynamic partitioning of said data processing resource.

41. The method of claim 40, comprising copying individual data items to at least two locations in said data storage resource and providing a group address said at least two locations.

42. The method of claim 40, comprising assigning to said data repository a plurality of states, such that upon detection of a fault said repository recovers to a state previous to said detection.

43. A shared nothing data repository comprising a plurality of data partitions and data items, each data item having a primary key, and one or more secondary keys, wherein said primary key defines said data partitions and each secondary key is implemented as an additional automatically managed internal index table that is partitioned by the secondary key, that maps the relationship between a secondary key and said partitioning primary key, such that a search query based on a respective secondary key can be related via said internal index table to said primary key.