Methods and Systems for Performing Transparent Object Migration Across Storage Tiers

• US 20110258225A1
• Filed: 04/19/2011
• Published: 10/20/2011
• Est. Priority Date: 04/19/2010
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Oliver, Kevin, Taylor, James, Eidson, Bill C.
• Granted Patent: US 9,824,108 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/2272 Management thereof

Abstract

In accordance with embodiments, there are provided mechanisms and methods for performing transparent object migration across storage tiers. In an embodiment and by way of example, a method for appending data to large data volumes is provided. The method embodiment includes a) setting a CustomEntityOption bit that determines (at object creation time) where the object is stored, either in the relational or the non-relational data store portion, b) loading the CustomEntityOption bit in a cached CustomEntityDefinition, c) showing the CustomEntityOption bit as EntityInfo, and d) allowing custom object definition and Metadata API functionality when the bit is shown.

Claims

1. In a combined data repository having a relational data store portion and a non-relational data store portion a method for controlling the operation of an API comprising:

a) setting a CustomEntityOption bit that determines (at object creation time) where the object is stored, either in the relational or the non-relational data store portion,

b) loading the CustomEntityOption bit in a cached CustomEntityDefinition;

c) showing the CustomEntityOption bit as EntityInfo, and

d) allowing custom object definition and Metadata API functionality when the bit is shown.

2. In a combined data repository having a relational data store portion and a non-relational data store portion a method for forming a search index row key comprising concatenating the byte value of indexed followed by the data row key byte.

3. The method of claim 2, wherein concatenating includes concatenating each indexed column.

4. The method of claim 2, wherein each column is terminated by a separator byte.

5. The method of claim 2, wherein single zero byte value will be used as a terminator for each column value in the key to allow a bytes comparator to be used to determine the sort order.

6. The method of claim 5, wherein a zero byte value is not used with a number that represents negative infinity with a single byte value of 0.

7. The method of claim 5, wherein a zero byte value is not used with a Boolean that represents False as a single byte value of 0.

8. The method of claim 2, wherein 1 is added to the first byte of an indexed column value when it has a length of 1.

9. In a combined data repository having a relational data store portion and a non-relational data store portion a method for updating search index metadata:

a) retrieving the replaced row from the non-relational data store,

b) updating the column values that are being changed, and

c) then doing an upsert by

i) sending an updated search index row with all of its row values to the non-relational data store, and

ii) replacing or shadowing a corresponding existing row.

10. The method of claim 9, wherein updating includes surfacing a SQL UPDATE statement that allows a subset of the columns to be updated.

11. The method of claim 9, further comprising:

a) defining the search index metadata by a cached Custom Index object,

b) sending the metadata through requests to the non-relational data store as part of a TableSpec object,

c) forcing a resend of the index metadata, by bumping the sequence number up for the corresponding table/org combination upon any index creation or deletion.

12. The method of claim 9, wherein the search index is maintained on the server side through the SyncTableCache object that is responsible for maintaining a snapshot of the unflushed rows for each indexed data table to ensure that all operations are working off of up-to-date data.

13. The method of claim 12, wherein the index table is updated at the same time that the data table is updated.

14. The method of claim 9, wherein deleted data rows cause the corresponding index row to be deleted, inserted data rows cause the insertion of the corresponding index row, updated data rows cause the deletion of the old index row and the insertion of the new index row, and index rows are flushed together with the data rows in the same flush operation.

15. The method of claim 9, further comprising:

a) calculating index statistics through normal queries of the non-relational data store,

b) storing the calculated index statistics in existing statistics tables, and

c) periodically updating the statistics using a scheduled batch process.

16. The method of claim 15, wherein calculating the index statistics is performed when an index has been completely built.

17. The method of claim 15, wherein calculating the index statistics is performed when an index is marked as active.

18. The method of claim 15, wherein the normal query is SELECT count(Id) totalCount,count_distinct(<

first indexed field>

) distinctCount FROM <

data table>

19. The method of claim 15, wherein the normal query is SELECT <

first indexed field>

value, COUNT(Id) totalCount FROM <

data table>

GROUP BY<

first indexed field>

HAVING COUNT(Id)>

100

20. The method of claim 9, wherein updating the column values further comprises:

a) running an optimize routine in a new index creation mode for all existing data segments,

b) writing out new index segments during index creation of the optimize routine,

c) creating index rows for any unflushed data rows,

d) enabling incremental index maintenance to cause the creation of index rows for any data that changes while index creation is in progress, and

e) marking created index rows as active when index creation is complete to enable queries to begin using it.

21. A custom index metadata structure for use in a combined data repository having a relational data store portion and a non-relational data store portion comprising:

a) IndexNum used as an index number,

b) FieldEnumOrId to define indexed column names,

c) Field2EnumOrId to define additional indexed column names as a composite with FieldEnumOrId,

d) IsActive to indicate when the index has been completely created and may start to be used by queries.

22. The structure of claim 21, further comprising more columns in core.custom_index

23. The structure of claim 21 further comprising a generalized bit to define more indexed column names.

24. The structure of claim 21, further comprising organization id.

25. The structure of claim 21, further comprising a field to store a subset of non indexed column names as covered columns.