Method and system for creating indices and loading key-value pairs for NoSQL databases

• US 9,378,263 B2
• Filed: 04/10/2013
• Issued: 06/28/2016
• Est. Priority Date: 06/19/2012
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Khimich, Oleksandr, Kudriavtsev, Dmytro, Nachnani, Pawan
• Primary Examiner(s): Leroux, Etienne
• Assistant Examiner(s): Nguyen, Cindy
• Application Number:
  US13/860,220
• Publication Number:
  US 20130339366A1
• Time in Patent Office: 1,175 Days
• Field of Search: 707/741
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/28 Databases characterised by their database models, e.g. relational or object models

Abstract

Systems and methods are provided for creating indices and loading key-value pairs for NoSQL databases. Attributes are created that correspond to records in a NoSQL database based on corresponding record fields. An index is created based on the attributes. A memory is loaded with attributes that correspond to a subset of the index as keys in a key-value pair and identifiers that correspond to records that correspond to the attributes as values in the key-value pair. The attributes that correspond to the subset of the index are sorted in the memory. Any duplicate attributes are identified from the sorted attributes in the memory. Any identifiers that correspond to any duplicate attributes also identify records in the NoSQL database to be evaluated as potential duplicate records.

Claims

1. An apparatus for creating indices and loading key-value pairs for NoSQL databases, the apparatus comprising:

a processor; and

one or more stored sequences of instructions which, when executed by the processor,

cause the processor to carry out the steps of;

creating a plurality of attributes that correspond to a plurality of records in a NoSQL database, wherein each attribute of the plurality of attributes comprises data from a corresponding plurality of record fields;

creating an index based on the plurality of attributes;

loading, in a memory, a plurality of attributes that correspond to a subset of the index as keys in a key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as values in the key-value pair;

sorting, in the memory, the plurality of attributes that correspond to the subset of the index; and

identifying, in the memory, any duplicate attributes from the sorted plurality of attributes, wherein any identifiers that correspond to the any duplicate attributes also identify records in the NoSQL database to be evaluated as to whether the identified records are duplicates.

2. The apparatus of claim 1, wherein the plurality of attributes that correspond to the plurality of records in a NoSQL database is based on an alphanumeric combination of the corresponding plurality of record fields and the index is based on an alphanumeric subset of the alphanumeric combination.

3. The apparatus of claim 1, wherein the steps further comprise:

determining whether to delete a record that is associated with a duplicate attribute; and

deleting the record from the memory in response to a determination to delete the record associated with the duplicate attribute.

4. The apparatus of claim 1, wherein the steps further comprise:

determining whether to merge a plurality of records that are associated with a plurality of duplicate attributes; and

merging the plurality of records in the memory in response to a determination to merge the plurality of records associated with the plurality of duplicate attributes.

5. The apparatus of claim 1, wherein the steps further comprise:

loading, in the memory, a plurality of attributes that correspond to the index as the keys in the key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as the values in the key-value pair;

receiving a request for at least one record that corresponds to an attribute of the plurality of attributes; and

loading, in the memory, the at least one record based on an identifier that corresponds to the attribute of the plurality of attributes.

6. A non-transitory machine-readable medium carrying one or more sequences of instructions for creating indices and loading key-value pairs for NoSQL databases, which instructions, when executed by one or more processors, cause the one or more processors to carry out the steps of:

creating a plurality of attributes that correspond to a plurality of records in a NoSQL database, wherein each attribute of the plurality of attributes comprises data from a corresponding plurality of record fields;

creating an index based on the plurality of attributes;

loading, in a memory, a plurality of attributes that correspond to a subset of the index as keys in a key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as values in the key-value pair;

sorting, in the memory, the plurality of attributes that correspond to the subset of the index; and

identifying, in the memory, any duplicate attributes from the sorted plurality of attributes, wherein any identifiers that correspond to the any duplicate attributes also identify records in the NoSQL database to be evaluated as to whether the identified records are duplicates.

7. The non-transitory machine-readable medium of claim 6, wherein the plurality of attributes that correspond to the plurality of records in a NoSQL database is based on an alphanumeric combination of the corresponding plurality of record fields and the index is based on an alphanumeric subset of the alphanumeric combination.

8. The non-transitory machine-readable medium of claim 6, wherein the steps further comprise:

determining whether to delete a record that is associated with a duplicate attribute; and

deleting the record from the memory in response to a determination to delete the record associated with the duplicate attribute.

9. The non-transitory machine-readable medium of claim 6, wherein the steps further comprise:

determining whether to merge a plurality of records that are associated with a plurality of duplicate attributes; and

merging the plurality of records in the memory in response to a determination to merge the plurality of records associated with the plurality of duplicate attributes.

10. The non-transitory machine-readable medium of claim 6, wherein the steps further comprise:

loading, in the memory, a plurality of attributes that correspond to the index as the keys in the key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as the values in the key-value pair;

receiving a request for at least one record that corresponds to an attribute of the plurality of attributes; and

loading, in the memory, the at least one record based on an identifier that corresponds to the attribute of the plurality of attributes.

11. A method for creating indices and loading key-value pairs for NoSQL databases, the method comprising:

creating a plurality of attributes that correspond to a plurality of records in a NoSQL database, wherein each attribute of the plurality of attributes comprises data from a corresponding plurality of record fields;

creating an index based on the plurality of attributes;

loading, in a memory, a plurality of attributes that correspond to a subset of the index as keys in a key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as values in the key-value pair;

sorting, in the memory, the plurality of attributes that correspond to the subset of the index; and

identifying, in the memory, any duplicate attributes from the sorted plurality of attributes, wherein any identifiers that correspond to the any duplicate attributes also identify records in the NoSQL database to be evaluated as to whether the identified records are duplicates.

12. The method of claim 11, wherein the plurality of attributes that correspond to the plurality of records in a NoSQL database is based on an alphanumeric combination of the corresponding plurality of record fields and the index is based on an alphanumeric subset of the alphanumeric combination.

13. The method of claim 11, wherein the method further comprises:

determining whether to delete a record that is associated with a duplicate attribute; and

deleting the record from the memory in response to a determination to delete the record associated with the duplicate attribute.

14. The method of claim 11, wherein the method further comprises:

determining whether to merge a plurality of records that are associated with a plurality of duplicate attributes; and

merging the plurality of records in the memory in response to a determination to merge the plurality of records associated with the plurality of duplicate attributes.

15. The method of claim 11, wherein the method further comprises:

loading, in the memory, a plurality of attributes that correspond to the index as the keys in the key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as the values in the key-value pair;

receiving a request for at least one record that corresponds to an attribute of the plurality of attributes; and

loading, in the memory, the at least one record based on an identifier that corresponds to the attribute of the plurality of attributes.

16. A method for transmitting code for creating indices and loading key-value pairs for NoSQL databases on a transmission medium, the method comprising:

transmitting code to create a plurality of attributes that correspond to a plurality of records in a NoSQL database, wherein each attribute of the plurality of attributes comprises data from a corresponding plurality of record fields;

transmitting code to create an index based on the plurality of attributes;

transmitting code to load, in a memory, a plurality of attributes that correspond to a subset of the index as keys in a key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as values in the key-value pair;

transmitting code to sort, in the memory, the plurality of attributes that correspond to the subset of the index; and

transmitting code to identify, in the memory, any duplicate attributes from the sorted plurality of attributes, wherein any identifiers that correspond to the any duplicate attributes also identify records in the NoSQL database to be evaluated as to whether the identified records are duplicates.

17. The method for transmitting code of claim 16, wherein the plurality of attributes that correspond to the plurality of records in a NoSQL database is based on an alphanumeric combination of the corresponding plurality of record fields and the index is based on an alphanumeric subset of the alphanumeric combination.

18. The method for transmitting code of claim 16, wherein the method further comprises:

determining whether to delete a record that is associated with a duplicate attribute; and

deleting the record from the memory in response to a determination to delete the record associated with the duplicate attribute.

19. The method for transmitting code of claim 16, wherein the method further comprises:

determining whether to merge a plurality of records that are associated with a plurality of duplicate attributes; and

merging the plurality of records in the memory in response to a determination to merge the plurality of records associated with the plurality of duplicate attributes.

20. The method for transmitting code of claim 16, wherein the method further comprises:

loading, in the memory, a plurality of attributes that correspond to the index as the keys in the key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as the values in the key-value pair;

receiving a request for at least one record that corresponds to an attribute of the plurality of attributes; and

loading, in the memory, the at least one record based on an identifier that corresponds to the attribute of the plurality of attributes.