System and method utilizing a shared cache to provide zero copy memory mapped database

• US 9,652,291 B2
• Filed: 06/03/2016
• Issued: 05/16/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Biswal, Punya, Liu, Beyang, Marinelli, Eugene, Ghamsari, Nima
• Primary Examiner(s): Rigol, Yaima
• Assistant Examiner(s): Gossage, Glenn
• Application Number:
  US15/173,526
• Publication Number:
  US 20160283277A1
• Time in Patent Office: 347 Days
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/60 Details of cache memory
• CPC Class Codes: G06F 2212/65 Details of virtual memory and virtual address translation
• CPC Class Codes: G06F 9/5016 the resource being the memory

Abstract

Methods and systems for providing a plurality of applications with concurrent access to data are disclosed. One such method includes identifying attributes of an expected data set to be accessed concurrently by the applications, initializing a shared cache with a column data store configured to store the expected data set in columns, and creating a memory map for accessing a physical memory location in the shared cache. The method may also include mapping the applications'"'"' data access requests to the shared cache with the memory map. Only one instance of the expected data set is stored in memory, so each application is not required to create additional instances of the expected data set in the application'"'"'s memory address space. Therefore, larger expected data sets may be entirely stored in memory without limiting the number of applications running concurrently.

Claims

1. A method comprising:

receiving a list of models to include in an expected data set for a plurality of applications to analyze concurrently, the models including a combination of data attributes from a database repository, wherein the expected data set includes different types of models and different types of models include different data attributes;

identifying in the database repository, a portion of the database repository comprising a plurality of data attributes of the expected data set to be accessed concurrently by the plurality of applications;

allocating a memory space for a shared cache, wherein the shared cache comprises a column data store configured to store data for each of the plurality of attributes of the expected data set in columns;

retrieving the expected data set from a database;

populating the shared cache with the expected data set;

storing memory address locations corresponding to the columns of the column data store of the shared cache; and

providing, to the plurality of applications, one or more of the stored memory address locations corresponding to the columns of the column data store of the shared cache.

2. The method of claim 1, wherein each of the plurality of applications accesses the data set by:

requesting data from one or more memory locations in a virtual address space allocated to the application;

mapping memory locations in the virtual address space to corresponding memory address locations of the shared cache, via a memory map; and

accessing the requested data from the mapped memory locations in the shared cache.

3. The method of claim 1, wherein storing the data for each of the plurality of attributes of the expected data set in columns, comprises:

dividing one or more data records retrieved from the database into a plurality of attribute values;

identifying a contiguous memory location in the shared cache for each attribute value; and

storing each attribute value in one of the identified contiguous memory locations.

4. The method of claim 1, wherein a plurality of object-oriented representations of the expected data set are provided to the plurality of applications by:

initializing a plurality of objects with data access methods;

storing the memory address locations corresponding to the columns of the column data store of the shared cache in the objects; and

providing the plurality of objects for access by the plurality of applications.

5. The method of claim 4, wherein at least one of the plurality of applications access the data set by:

calling a data access method of one of the objects; and

receiving requested data from the data access method of the object.

6. The method of claim 1, wherein the shared cache is configured to provide read only access to the plurality of applications.

7. The method of claim 1, wherein the shared cache is updated by:

identifying a plurality of attributes of an updated expected data set;

re-allocating the memory space for the shared cache;

retrieving an updated expected data set from the database;

populating the shared cache with the updated expected data set; and

storing updated memory address locations corresponding to the columns of the column data store in the shared cache.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform an operation for providing a plurality of applications with concurrent access to an expected data set from a database repository, the operation comprising:

receiving a list of models to include in the expected data set for a plurality of applications to analyze concurrently, the models including a combination of data attributes from the database repository, wherein the expected data set includes different types of models and different types of models include different data attributes;

identifying in the database repository, a portion of the database repository comprising a plurality of data attributes of the expected data set to be accessed concurrently by the plurality of applications;

allocating a memory space for a shared cache, wherein the shared cache comprises a column data store configured to store data for each of the plurality of attributes of the expected data set in columns;

retrieving the expected data set from a database;

populating the shared cache with the expected data set;

storing memory address locations corresponding to the columns of the column data store of the shared cache; and

providing, to the plurality of applications, one or more of the stored memory address locations corresponding to the columns of the column data store of the shared cache.

9. The non-transitory computer-readable storage medium of claim 8, wherein each of the plurality of applications accesses the data set by:

requesting data from one or more memory locations in a virtual address space allocated to the application;

mapping memory locations in the virtual address space to corresponding memory address locations of the shared cache, via a memory map; and

accessing the requested data from the mapped memory locations in the shared cache.

10. The non-transitory computer-readable storage medium of claim 8, wherein storing the data for each of the plurality of attributes of the expected data set in columns, comprises:

dividing one or more data records retrieved from the database into a plurality of attribute values;

identifying a contiguous memory location in the shared cache for each attribute value; and

storing each attribute value in one of the identified contiguous memory locations.

11. The non-transitory computer-readable storage medium of claim 8, wherein a plurality of object-oriented representations of the expected data set are provided to the plurality of applications by:

initializing a plurality of objects with data access methods;

storing the memory address locations corresponding to the columns of the column data store of the shared cache in the objects; and

providing the plurality of objects for access by the plurality of applications.

12. The non-transitory computer-readable storage medium of claim 11, wherein at least one of the plurality of applications access the data set by:

calling a data access method of one of the objects; and

receiving requested data from the data access method of the object.

13. The non-transitory computer-readable storage medium of claim 8, wherein the shared cache is configured to provide read only access to the plurality of applications.

14. The non-transitory computer-readable storage medium of claim 8, wherein the shared cache is updated by:

identifying a plurality of attributes of an updated expected data set;

re-allocating the memory space for the shared cache;

retrieving an updated expected data set from the database;

populating the shared cache with the updated expected data set; and

storing updated memory address locations corresponding to the columns of the column data store in the shared cache.

15. A computer system, comprising:

a memory storing one or more programs; and

a processor configured to execute the one or more programs to perform an operation for providing a plurality of applications with concurrent access to data, the operation comprising;

receiving a list of models to include in an expected data set for multiple applications to analyze concurrently, the models including a combination of data attributes from the database repository, wherein the expected data set includes different types of models and different types of models include different data attributes;

identifying in the database repository, a portion of the database repository comprising a plurality of data attributes of the expected data set to be accessed concurrently by the plurality of applications;

allocating a memory space for a shared cache, wherein the shared cache comprises a column data store configured to store data for each of the plurality of attributes of the expected data set in columns;

retrieving the expected data set from a database;

populating the shared cache with the expected data set;

storing memory address locations corresponding to the columns of the column data store of the shared cache; and

providing, to the plurality of applications, one or more of the stored memory address locations corresponding to the columns of the column data store of the shared cache.

16. The system of claim 15, wherein each of the plurality of applications accesses the data set by:

requesting data from one or more of the memory locations in a virtual address space allocated to the application;

mapping memory locations in the virtual address space to corresponding memory address locations of the shared cache, via a memory map; and

accessing the requested data from the mapped memory locations in the shared cache.

17. The system of claim 15, wherein storing the data for each of the plurality of attributes of the expected data set in columns, comprises:

dividing one or more data records retrieved from the database into a plurality of attribute values;

identifying a contiguous memory location in the shared cache for each attribute value; and

storing each attribute value in one of the identified contiguous memory locations.

18. The system of claim 15, wherein a plurality of object-oriented representations of the expected data set are provided to the plurality of applications by:

initializing a plurality of objects with data access methods;

storing the memory address locations corresponding to the columns of the column data store of the shared cache in the objects; and

providing the plurality of objects for access by the plurality of applications.

19. The system of claim 18, wherein at least one of the plurality of applications access the data set by:

calling a data access method of one of the objects; and

receiving requested data from the data access method of the object.

20. The system of claim 15, wherein the shared cache is configured to provide read only access to the plurality of applications.

21. The system of claim 15, wherein the shared cache is updated by:

identifying a plurality of attributes of an updated expected data set;

re-allocating the memory space for the shared cache;

retrieving an updated expected data set from the database;

populating the shared cache with the updated expected data set; and

storing updated memory address locations corresponding to the columns of the column data store in the shared cache.