High-performance gridded data storage, arrangement and extraction

• US 9,922,062 B2
• Filed: 07/16/2014
• Issued: 03/20/2018
• Est. Priority Date: 07/16/2013
• Status: Active Grant

• Current Assignee: Iteris Incorporated, DTN, LLC (TBG AG)
• Original Assignee: ClearAg, Inc. (TBG AG)
• Inventors: Rand, Douglas K., Mewes, John J., Pedersen, Leif, Zarns, Kristopher A., Salentiny, Dustin
• Primary Examiner(s): Fan, Shiow-Jy
• Application Number:
  US14/332,768
• Publication Number:
  US 20160019214A1
• Time in Patent Office: 1,343 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/29 Geographical information databases

Abstract

A high-performance gridded database protocol for storing, arranging, and extracting gridded data includes associating values for a single grid cell and storing them together to extract as many useful values as possible from a single read operation. Gridded data is stored in a geographically-indexed cylindrical grid that permits efficient data extraction for a particular location while maximizing efficiency of read operations. Cylinders of values are built by grouping grids that are related to each other so that when data for a location is to be extracted, a minimal number of read operations is needed to retrieve an entire stack of data relevant to the location.

Claims

1. A method of storing and arranging gridded data for high-performance extraction of information from a database, comprising:

ingesting meteorological data from one or more sources of weather information, and identifying grids relative to a specific geographical location to populate individual grid cells with the meteorological data in a database;

separating the grids into stacks, each stack comprising grid cells corresponding to the specific geographic location and having a coverage area of a length X and a width Y, and each grid cell comprised of a cylinder of a height Z, so that each stack includes X times Y times Z values;

indexing the populated grid cells so that a plurality of values are grouped together in stacks relative to the specific geographic location, in which a payload of the plurality of values that are stored in the cylinders includes the height representing a number of different values in multiple variables, and a spread covering several of the grid cells represented by a footprint having a width and a length covering more than a single grid cell, to permit a read of the plurality of values to extract meteorological data for the specific geographical location;

storing one stack in a database cell with the cylinder height and coverage area dynamically adjusted to optimize storage, so that fetching a single database cell returns a stack of meteorological data that is spatially and temporally related and represents meteorological characteristics stored on a common grid projection for the specific geographical location;

dynamically expiring grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database; and

enforcing constraints to maintain a timeliness of values stored in the database, wherein the constraints include at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time, and

wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.

2. The method of claim 1, further comprising initially selecting one or more grids to intelligently build the stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

3. The method of claim 1, further comprising arranging grids into tiles comprised of a single value for a grid, and storing cylinders of tiles.

4. The method of claim 1, further comprising prioritizing a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

5. A method comprising:

populating individual grid cells within a database by collecting and writing meteorological data from one or more sources of weather information in a manner that groups values for a specific geographic location together in related grid cells within a stack by separating the grids into stacks, each stack comprising grid cells corresponding to the specific geographic location and having a coverage area of a length X and a width Y, and each grid cell comprised of a cylinder of a height Z, so that each stack includes X times Y times Z values;

arranging a payload of the values into cylinders so that the cylinders have both the height representing a number of different values in multiple variables, and a spread covering several of the grid cells represented by a footprint having a width and a length covering more than a single grid cell, by storing one stack in a database cell with the cylinder height and coverage area dynamically adjusted to optimize storage, so that fetching a single database cell returns a stack of meteorological data that is spatially and temporally related and represents meteorological characteristics stored on a common grid projection for the specific geographical location;

extracting gridded data by retrieving a whole stack of grid cells so that only a single read operation is needed to obtain values for the specific geographic location; and

dynamically expiring grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database; and

enforcing constraints to maintain a timeliness of values stored in the database, wherein the constraints include at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time, and

wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.

6. The method of claim 5, further comprising initially selecting one or more grids to intelligently build stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

7. The method of claim 5, further comprising arranging grids into tiles comprised of a single value for a grid, and storing cylinders of tiles.

8. The method of claim 5, further comprising prioritizing a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

9. A high-performance gridded database protocol, comprising:

in a computing environment that includes a plurality of software and hardware components coupled to at least one processor, the at least one processor configured to carry out one or more program instructions to execute a database protocol having a plurality of operations;

a write operation in the plurality of operations, the write operation configured to populate individual grid cells within a database by collecting and writing meteorological data from one or more sources of weather information in a manner that groups values for a specific geographic location together in related grid cells within a stack by separating the grids into stacks, each stack comprising grid cells corresponding to the specific geographic location and having a coverage area of a length X and a width Y, and each grid cell comprised of a cylinder of a height Z, so that each stack includes X times Y times Z values;

a store operation in the plurality of operations, the store operation configured to arrange a payload of the values into cylinders so that the cylinders have both the height representing a number of different values multiple variables, and a spread covering several of the grid cells represented by a footprint having a width and a length covering more than a single grid cell, by storing one stack in a database cell with the cylinder height and coverage area dynamically adjusted to optimize storage, so that fetching a single database cell returns a stack of meteorological data that is spatially and temporally related and meteorological characteristics stored on a common grid projection for the specific geographical location,

wherein the store operation dynamically expires grids by setting a timer within the database for a stack of grid cells, wherein the timer is set when the values in the stack of grid cells are ingested into the database, and wherein the store operation enforces constraints to maintain a timeliness of values stored in the database, the constraints including at least one of a latest limit constraint to impose a limit on a time in between receiving grids, a mb-limit constraint to impose a limit on a stack size, a global-mb-limit constraint to impose a limit on total memory available across all stacks processed, a maximum number of grids in a stack, and a maximum number of grids in all stacks processed at any time; and

a read operation in the plurality of operations, the read operation configured to extract gridded data by retrieving a whole stack of grid cells so that only a single read operation is needed to obtain values for the specific geographic location, and

wherein a selection of the height and the spread includes a consideration of physical memory, a consideration of cache advantages of having adjacent grid cells stored in the same database object, and a size of the database object being stored, relative to an extraction of the payload of stored data in the specific geographical location.

10. The protocol of claim 9, wherein the store operation initially selects one or more grids to intelligently build stacks comprising the values so that grid cells that are related by geographical location are grouped into the same cylinder.

11. The protocol of claim 10, wherein the read operation prioritizes a stack so that meteorological data may be drawn from an ordered preference of resources to fulfill specific requests.

12. The protocol of claim 9, wherein the store operation arranges grids into tiles comprised of a single value for a grid, and stores cylinders of tiles.