×

Partition-based index management in hadoop-like data stores

  • US 9,460,147 B1
  • Filed: 01/12/2016
  • Issued: 10/04/2016
  • Est. Priority Date: 06/12/2015
  • Status: Expired due to Fees
First Claim
Patent Images

1. A method for maintaining an index into a dataset after a batch update of the dataset of a partitioned distributed storage system, the dataset stored in an HBase database having data stored in a base table and an index stored in an index table, the method comprising:

  • locking the base and index tables to prevent region split, merge and movement operations;

    receiving base and index table metadata from the partitioned distributed storage system, wherein the base and index table metadata includes respective table partition information;

    partitioning the dataset into a set of base-delta files according to the base table metadata and a first criteria;

    updating the partitioned distributed storage system a first time with the set of base-delta files;

    generating a set of index-delta files corresponding with the base-delta files by;

    determining a second criteria for generating keys for indexing the partitioned dataset,generating, based on the second criteria and partition information about the partitioned dataset, a set of index-delta files having keys for indexing the partitioned dataset;

    updating the partitioned distributed storage system a second time with the set of index-delta files,wherein a first update of the base table is synchronous with a second update of the index table; and

    unlocking, subsequent to the second update, the base and index tables;

    wherein, updating includes deleting the base-delta and index-delta files from a respective one or more computing systems having the base and index tables when the batch update includes a delete operation, and copying base-delta and index-delta files from the respective one or more computing systems having the base and index tables when the batch update includes a load operation.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×