Method and apparatus for data management

• US 7,644,087 B2
• Filed: 02/21/2006
• Issued: 01/05/2010
• Est. Priority Date: 02/24/2005
• Status: Expired due to Fees

• Current Assignee: Northend Networks Ltd.
• Original Assignee: XEROUND INC., XEROUND SYSTEMS LTD. (XEROUND INC.)
• Inventors: Barkai, Sharon, Zlotkin, Gilad, Vigder, Avi, Klar, Nir, Romem, Yaniv, Shomer, Ayelet, Kaminer, Iris, Levy, Roni, Broude, Zeev, Gilderman, Ilia
• Primary Examiner(s): Corrielus, Jean M
• Assistant Examiner(s): Nguyen, Loan T
• Application Number:
  US11/357,110
• Publication Number:
  US 20060190243A1
• Time in Patent Office: 1,414 Days
• Field of Search: None
• US Class Current: 707/770
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast

Abstract

A data access system decouples the data processing from the data storage to provide improved accessibility, integrity, scalability and other features. The system comprises: database units arranged in virtual partitions each independently accessible, a plurality of data processing units, and a switching network for switching the data processing units between the virtual partitions, thereby to assign data processing capacity dynamically to respective virtual partitions.

Claims

1. A data access system comprising a plurality of computers and a switching network, the plurality of computers respectively comprising multiple locations of physical data storage, the data access system comprising:

database units arranged to form virtual partitions, each virtual partition formed from a database unit being independently accessible, said virtual partitions being accessible as addressable channels in an addressing space independent of said physical data storage;

decoupling said physical data storage and said virtual partitions by using switched channel networking and creating an intermediate networking structure, wherein said data partitioning is statically mapped to the network channel, and wherein said decoupling data partitions permits reassignment of any given virtual partition between respectively independent locations of said physical data storage while maintaining said accessibility via respective ones of said addressable channels in said independent addressing space;

a plurality of data processing units, and

said switching network receiving data access requests to said database units as addressable channels of said independent addressing space, obtaining data from said physical data storage in accordance with a most recent reassignment of said database units and switching said data processing units between said virtual partitions to assign data processing capacity dynamically to respective virtual partitions.

2. The data access system of claim 1, wherein said switching network comprises an interconnection of switching units.

3. The data access system of claim 1, wherein each database unit is independently accessible as a respective network channel.

4. The data access system of claim 1, further comprising a hashing unit for carrying out a hashing process on data and wherein data is assigned to respective database units via a result of said hashing process.

5. The data access system of claim 4, wherein data is assigned in the form of records having a primary key and at least one secondary keys, and wherein the hashing process is carried out on said primary key.

6. The data access system of claim 5, configured with at least one router to tabulate a relationship between a secondary key and said hashed primary key, wherein said tabulated relationship, and a search query based on a respective secondary key are relatable via said router to said primary key.

7. The data access system of claim 5, configured with at least one additional automatically managed internal index table, said internal index table mapping the relationship between a secondary key and said hashed primary key, wherein said internal index table and a search query based on a respective secondary key are relatable to said primary key.

8. The data access system of claim 1, wherein data is replicated at least once over at least two data partitions.

9. The data access system of claim 1, comprising election functionality for dynamically assigning one of said data processing units as a coordinator to arbitrate between conflicting write operations.

10. The data access system of claim 9, wherein said coordinator is configured to signal regularly an indication that said coordinator is continuing as coordinator, and wherein said election functionality is configured to repeat said dynamically assigning when said regular signal ceases.

11. The data access system of claim 10, wherein a write operation interrupted by said dynamic assigning is resumed from a most advanced one of available recoverable positions when said dynamic assigning is concluded.

12. The data access system of claim 9, wherein said coordinator is configured to assign a unique certificate to a record following a record-altering operation, to render versions of said record comparable.

13. The data access system of claim 4, wherein data is assigned in the form of records having at least three keys and wherein each record is assigned a primary address based on said one of the keys and secondary addresses based on remaining ones of said keys.

14. The data access system of claim 13, comprising a resolution unit for resolving secondary addresses into corresponding primary addresses, and using said resolved addresses to find a record defined by a secondary key using a corresponding primary key.

15. The data access system of claim 14, wherein said resolution unit comprises at least one router.

16. The data access system of claim 15, wherein said resolution unit further comprises at least one backup router.

17. The data access system of claim 8, wherein said switching mechanism is configured to reassign data partition versions to remaining ones of said data processing units following a failure of one or more of said data processing units.

18. The data access system of claim 1, wherein each virtual partition is stored on a predetermined number of data processing units, said data processing units are in independent locations, after the failure of a given data processing unit, all data remains accessible.

19. The data access system of claim 18, wherein said predetermined number is at least three.

20. The data access system of claim 19, wherein said number being at least three is an odd number, the data access system polling to obtain a majority from said copied virtual partitions to ensure integrity of said data.

21. The data access system of claim 20, wherein said odd number is at least five.

22. The data access system of claim 1, further comprising a use measurement function for measuring usage by individual customers of said data access system, and a billing functionality for billing said customers based on peak usage of said data system.

23. The data access system of claim 1, further comprising:

a use measurement function for measuring usage by individual customers of said data access system, and a billing functionality for billing said customers based on peak usage of said data system.

43. The data access system of claim 1, wherein said database units comprise data items, each data hem having a primary key, and one or more secondary keys, wherein said primary key defines said virtual partitions and each secondary key is implemented as an additional automatically managed internal index table, said additional automatically managed internal index table being partitioned by the secondary key, said secondary key by said table mapping the relationship between a secondary key and said partitioning primary key.

24. A method of providing a high availability, high scalability data storage and query system for a database comprising database units, the method comprising:

providing a data query arrangement, providing multiple independent physical data storage locations;

providing a data storage arrangement separate from said data query arrangement, said data storage arrangement comprising virtual partitions formed from said database units, said virtual partitions being accessible as addressable channels in an addressing space independent of physical data storage;

decoupling said physical data storage and said virtual partitions by using switched channel networking and creating an intermediate networking structure, wherein said data partitioning is statically mapped to the network channel, and wherein said decoupling data partitions permits reassignment of any given virtual partition between respectively independent locations of said physical data storage while maintaining said accessibility via respective ones of said addressable channels in said independent addressing space; and

providing a switching system to dynamically connect between said data storage arrangement and said data query arrangement by using a query field of a current query and obtaining from said query field and said data query arrangement one of said addressable channels of said independent addressing space.

25. The method of claim 24, comprising providing said data storage arrangement as a plurality of channels.

26. The method of claim 25, comprising storing each data item as a record and providing copies of any given data record in a predetermined number of said channels.

27. The method of claim 26, wherein said predetermined number is an odd number.

28. The method of claim 27, further comprising polling to obtain a majority from between said odd number of copies to ensure integrity of said data.

29. The method of claim 26, comprising setting a field of said data records as a primary key and hashing said primary key for addressing said channels.

30. The method of claim 29, comprising setting a field of said data records as a secondary key and providing at least one router for correlating between said secondary key and said primary key.

31. The method of claim 25, wherein said channels are publish subscribe channels.

32. The method of claim 24, wherein said data storage arrangement comprises a plurality of data storage units;

storing data in multiple copies at a plurality of data storage units, and upon detection of failure of any given data storage unit making additional copies from at least one other of said plurality of data storage units of the data that was stored in said given data storage unit prior to failure.

33. The method of claim 32, wherein said data is accessed via queries and wherein a response to a given queries depends on a current state in relation to said data.

34. The method of claim 33, further comprising retaining said current state in the event of a failure of at least one data storage unit by explicit state synchronization between respective data storage units.

35. The method of claim 34, wherein said explicit state synchronization is pull synchronization.

36. The method of claim 34, wherein said explicit state synchronization is push synchronization.

37. The method of claim 24, further comprising measuring usage by customers.

38. The method of claim 37, wherein said usage is measured as peak usage.

39. The method of claim 38 comprising billing said customers based on said peak usage.

40. The method of claim 24, comprising:

providing dynamic allocation of said data processing resource to incoming queries.

41. The method of claim 40, comprising copying individual data hems to at least two locations in said data storage resource and providing a group address said at least two locations.

42. The method of claim 40, comprising assigning to said data repository a plurality of states, said states being usable to ensure that upon detection of a fault said repository recovers to a state previous to said detection.