HIGH-PERFORMANCE GRIDDED DATA STORAGE, ARRANGEMENT AND EXTRACTION

• US 20160019214A1
• Filed: 07/16/2014
• Published: 01/21/2016
• Est. Priority Date: 07/16/2013
• Status: Active Grant

• Current Assignee: Iteris Incorporated, DTN, LLC (TBG AG)
• Original Assignee: Iteris Incorporated
• Inventors: MEWES, JOHN J., PEDERSEN, LEIF, ZARNS, KRISTOPHER A., RAND, DOUGLAS K., SALENTINY, DUSTIN
• Granted Patent: US 9,922,062 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/29 Geographical information databases

Abstract

A high-performance gridded database protocol for storing, arranging, and extracting gridded data includes associating values for a single grid cell and storing them together to extract as many useful values as possible from a single read operation. Gridded data is stored in a geographically-indexed cylindrical grid that permits efficient data extraction for a particular location while maximizing efficiency of read operations. Cylinders of values are built by grouping grids that are related to each other so that when data for a location is to be extracted, a minimal number of read operations is needed to retrieve an entire stack of data relevant to the location.

Claims

1. A method of storing and arranging gridded data for high-performance extraction of information from a database, comprising:

ingesting meteorological data from one or more sources of weather information, and identifying grids relative to a specific geographical location to populate individual grid cells with the meteorological data in a database; and

indexing the populated grid cells so that a plurality of values are grouped together in stacks relative to the specific geographic location, in which a payload of the plurality of values are stored in cylinders includes a height representing a number of different values in a stack of values, and a spread covering several grid cells represented by a footprint having a width and a length covering more than a single grid cell, to permit a read of the plurality of values to extract meteorological data for a relevant geographical location,

wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.

2. The method of claim 1, further comprising initially selecting one or more grids to intelligently build the stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

3. The method of claim 1, further comprising arranging grids into tiles comprised of a single value for a grid, and storing cylinders of tiles.

4. The method of claim 1, further comprising dynamically expiring grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database.

5. The method of claim 1, further comprising enforcing constraints to maintain a timeliness of values stored in the database.

6. The method of claim 5, wherein the constraints include at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time.

7. The method of claim 1, further comprising prioritizing a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

8. A method comprising:

populating individual grid cells within a database by collecting and writing meteorological data from one or more sources of weather information in a manner that groups values for a specific geographic location together in related grid cells within a stack;

arranging a payload of the values into cylinders that have both height representing a number of different values in a stack of values, and a spread covering several grid cells represented by a footprint having a width and a length covering more than a single grid cell; and

extracting gridded data by retrieving a whole stack of grid cells so that only a single read operation is needed to obtain values for a relevant geographic location.

9. The method of claim 8, further comprising initially selecting one or more grids to intelligently build stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

10. The method of claim 8, further comprising arranging grids into tiles comprised of a single value for a grid, and storing cylinders of tiles.

11. The method of claim 8, further comprising dynamically expiring grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database.

12. The method of claim 8, further comprising enforcing constraints to maintain a timeliness of values stored in the database.

13. The method of claim 12, wherein the constraints include at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time.

14. The method of claim 8, further comprising prioritizing a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

15. The method of claim 8, wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.

16. A high-performance gridded database protocol, comprising:

in a computing environment that includes a plurality of software and hardware components coupled to at least one processor, the at least one processor configured to carry out one or more program instructions to execute a database protocol having a plurality operations;

a write operation in the plurality of operations, the write operation configured to individual grid cells within a database by collecting and writing meteorological data from one or more sources of weather information in a manner that groups values for a specific geographic location together in related grid cells within a stack;

a store operation in the plurality of operations, the store operation configured to arrange a payload of the values into cylinders that have both height representing a number of different values in a stack of values, and a spread covering several grid cells represented by a footprint having a width and a length covering more than a single grid cell; and

a read operation in the plurality of operations, the read operation configured to extract gridded data by retrieving a whole stack of grid cells so that only a single read operation is needed to obtain values for a relevant geographic location.

17. The protocol of claim 16, wherein the store operation initially selects one or more grids to intelligently build stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

18. The protocol of claim 16, wherein the store operation arranges grids into tiles comprised of a single value for a grid, and stores cylinders of tiles.

19. The protocol of claim 16, wherein the store operation dynamically expires grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database.

20. The protocol of claim 16, wherein the store operation enforces constraints to maintain a timeliness of values stored in the database.

21. The protocol of claim 20, wherein the constraints include at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time.

22. The protocol of claim 17, wherein the read operation prioritizes a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

23. The protocol of claim 16, wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.