Updating distributed shards without compromising on consistency
First Claim
1. A method comprising:
- receiving, at a queue manager executed on one or more hardware processors, from a first client application, a first data change to data stored in a distributed database;
receiving, at the queue manager, from a second client application different than the first client application, a second data change to the data stored in the distributed database;
batching, by the queue manager, the first data change and the second data change in a communication to an index manager;
reindexing, by the index manager, the distributed database based on the first data change and the second data change by creating a revised first shard index for a first shard in the distributed database and a revised second shard index for a second shard in the distributed database, wherein the revised first shard index is different than the revised second shard index, wherein the first shard is stored on a first device and the second shard is stored on a second device;
instructing, by the index manager, the first shard to update a first shard index based on the revised first shard index;
instructing, by the index manager, the second shard to update a second shard index based on the revised second shard index;
receiving, at the index manager, at a first time, confirmation, from the first device, that the first shard index has been updated;
receiving, at the index manager, at a second time, confirmation, from the second device, that the second shard index has been updated; and
in response to both the receiving the confirmation that the first shard index has been updated and the confirmation that the second shard index has been updated, instructing, by the index manager, the first shard to commit the revised first shard index and the second shard to commit the revised second shard index.
1 Assignment
0 Petitions
Accused Products
Abstract
In an example embodiment, a first data change and a second data change to data stored in a distributed database are received. The first data change and the second data change are batched in a communication to an index manager. The distributed database is reindexed based on the first data change and the second data change by creating a revised first shard index for a first shard in the distributed database and a revised second shard index for a second shard in the distributed database. The first shard and the second shard are instructed to update respective shard indexes. Confirmation that the first shard index has been updated is received. Then confirmation that the second shard index has been updated is received. In response to both of the confirmations, both the first shard and the second shard are instructed to commit their respective updates.
32 Citations
20 Claims
-
1. A method comprising:
-
receiving, at a queue manager executed on one or more hardware processors, from a first client application, a first data change to data stored in a distributed database; receiving, at the queue manager, from a second client application different than the first client application, a second data change to the data stored in the distributed database; batching, by the queue manager, the first data change and the second data change in a communication to an index manager; reindexing, by the index manager, the distributed database based on the first data change and the second data change by creating a revised first shard index for a first shard in the distributed database and a revised second shard index for a second shard in the distributed database, wherein the revised first shard index is different than the revised second shard index, wherein the first shard is stored on a first device and the second shard is stored on a second device; instructing, by the index manager, the first shard to update a first shard index based on the revised first shard index; instructing, by the index manager, the second shard to update a second shard index based on the revised second shard index; receiving, at the index manager, at a first time, confirmation, from the first device, that the first shard index has been updated; receiving, at the index manager, at a second time, confirmation, from the second device, that the second shard index has been updated; and in response to both the receiving the confirmation that the first shard index has been updated and the confirmation that the second shard index has been updated, instructing, by the index manager, the first shard to commit the revised first shard index and the second shard to commit the revised second shard index. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system comprising:
-
a distributed database; a search core; and an indexing core executable on one or more processors, the indexing core comprising; a plurality of different index adapters, each index adapter corresponding to a different document type of a different client application; an index builder; an index manager; a queue manager configured to; receive, from a first client application, a first data change to data stored in the distributed database; receive, from a second client application different than the first client application, a second data change to the data stored in the distributed database; and batch the first data change and the second data change in a communication to the index manager; the index manager configured to; cause the index builder to reindex the distributed database based on the first data change and the second data change by creating a revised first shard index for a first shard in the distributed database and a revised second shard index for a second shard in the distributed database, wherein the revised first shard index is different than the revised second shard index, wherein the first shard is stored on a first device and the second shard is stored on a second device; instruct the first shard to update a first shard index based on the revised first shard index; instruct the second shard to update a second shard index based on the revised second shard index; receive, at a first time, from the first device, confirmation that the first shard index has been updated; receive, at a second time, from the second device, confirmation that the second shard index has been updated; and in response to both the receiving the confirmation that the first shard index has been updated and the confirmation that the second shard index has been updated, instruct the first shard to commit the revised first shard index and the second shard to commit the revised second shard index. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:
-
receiving, at a queue manager executed on one or more hardware processors, from a first client application, a first data change to data stored in a distributed database; receiving, at the queue manager, from a second client application different than the first client application, a second data change to the data stored in the distributed database; batching, by the queue manager, the first data change and the second data change in a communication to an index manager; reindexing, by the index manager, the distributed database based on the first data change and the second data change by creating a revised first shard index for a first shard in the distributed database and a revised second shard index for a second shard in the distributed database, wherein the revised first shard index is different than the revised second shard index, wherein the first shard is stored on a first device and the second shard is stored on a second device; instructing, by the index manager, the first shard to update a first shard index based on the revised first shard index; instructing, by the index manager, the second shard to update a second shard index based on the revised second shard index; receiving, at the index manager, at a first time, confirmation, from the first device, that the first shard index has been updated; receiving, at the index manager, at a second time, confirmation, from the second device, that the second shard index has been updated; and in response to both the receiving the confirmation that the first shard index has been updated and the confirmation that the second shard index has been updated, instructing, by the index manager, the first shard to commit the revised first shard index and the second shard to commit the revised second shard index. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification