Vehicle domain multi-level parallel buffering and context-based streaming data pre-processing system

• US 9,537,914 B1
• Filed: 12/01/2015
• Issued: 01/03/2017
• Est. Priority Date: 12/01/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dong, Wei Shan, Duan, Ning, Gao, Peng, Li, Chang Sheng, Ma, Chun Yang, Zhang, Xin, Zhu, Jun
• Primary Examiner(s): Pollack, Melvin H
• Application Number:
  US14/955,218
• Time in Patent Office: 399 Days
• Field of Search: 709/219
• US Class Current: 1/1
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A vehicle domain multi-level parallel buffering and context-based streaming data pre-processing system includes a first data processing level and a second data processing level. The first data processing level includes a first-level buffer configured to buffer data provided from a plurality of raw data streams output from a plurality of vehicles. The second data processing level includes an electronic task-queue-dictionary (TQD) module and a plurality of second-level data processing buffers. The TQD module is configured to create a plurality of tasks in response to receiving a serial data stream output from the first-level buffer. The TQD module is further configured to assign each task to a corresponding second-level buffer, and separate the serial data stream into individual data values that are delivered to a specific second-level buffer based on the task so as to generate a multi-level parallel context-based buffering operation.

Claims

1. A vehicle domain multi-level parallel buffering and context-based streaming data pre-processing system comprising:

a first data processing level including a first-level buffer configured to serially buffer data provided from a plurality of raw data streams output from a plurality of vehicles; and

a second data processing level including an electronic task-queue-dictionary (TQD) module and a plurality of second-level data processing buffers, the TQD module configured to create a plurality of tasks in response to receiving a serial data stream output from the first-level buffer, the tasks corresponding to available processing operations performed by a local vehicle operating the vehicle domain multi-level parallel buffering and context-based streaming and data pre-processing system, each task corresponding to a request from a respective vehicle among the plurality of vehicles;

wherein the TQD module compares the requested tasks to the available processing operations and assigns each task to a corresponding second-level buffer, and after serially buffering the data in the first-level buffer, separates the serial data stream into individual data values that are delivered in parallel to a specific second-level buffer based on the task.

2. The system of claim 1, wherein the data values include vehicle identification information, and the TQD module assigns a buffer identification value to each second-level buffer such that the data values are loaded into a corresponding second-level buffer having a buffer identification value that matches the vehicle identification information.

3. The system of claim 1, wherein the TQD module limits each vehicle to a single working task created in the second data processing level.

4. The system of claim 1, wherein the TQD module loads each second-level buffer based on a timestamp indicated by a respective data value.

5. A method of processing data streams in a vehicle domain, the method comprising:

buffering data provided from a plurality of raw data streams output from a plurality of vehicles using a first-level buffer installed at a first data processing level included in a multi-level parallel buffering and context-based streaming data pre-processing system;

delivering a serial data stream output from the first-level buffer to an electronic task-queue-dictionary (TQD) module installed in a second data processing level included in the multi-level parallel buffering and context-based streaming data pre-processing system;

creating a plurality of tasks via the TQD module in a in response to receiving the serial data stream, the tasks corresponding to available processing operations performed by a local vehicle operating the vehicle domain multi-level parallel buffering and context-based streaming and data pre-processing system, each task corresponding to a request from a respective vehicle among the plurality of vehicles,

wherein, after serially buffering the data in the first-level buffer, the TQD module compares the requested tasks to the available processing operations and assigns each task to a corresponding second-level buffer so as to separate the serial data stream into individual data values that are delivered in parallel to a specific second-level buffer based on the task.

6. The method of claim 5, further comprising embedding vehicle identification information in the data values, wherein the TQD module assigns a buffer identification value to each second-level buffer such that the data values are loaded into a corresponding second-level buffer having a buffer identification value that matches the vehicle identification information.

7. The method of claim 5, wherein the TQD module limits each vehicle to a single working task created in the second data processing level.

8. The method of claim 5, wherein the TQD module loads each second-level buffer based on a timestamp indicated by a respective data value.

9. A vehicle domain multi-level parallel buffering and context-based streaming data pre-processing system comprising:

a first-level buffer installed in a first data processing level, the first-level buffer configured to concurrently queue data values provided from a plurality of data streams output from a plurality of vehicles;

an electronic task-queue-dictionary (TQD) module installed in a second data processing level different from the first data processing level, the TQD module configured to receive a serial data stream output from the first-level buffer and separate the data values from the serial data stream based on a context of the data values; and

a plurality of second-level buffers installed in the second data processing level, the second-level buffers having context buffer identification values assigned thereto and configured to queue data values having a context that matches the context buffer identification value,

wherein, after serially buffering the data in the first-level buffer, the TQD module loads data values into respective second-level buffers according to a parallel data delivery scheme to generate respective separated data streams.

10. The system of claim 9, further comprising at least one electronic data porter module configured to determine a requested task by a respective vehicle based on the context of the data values, and commands the TQD module to perform at least one of creating a new task in the second data processing level and updating an existing task in response to determining the requested task, wherein the tasks correspond to available processing operations performed by a local vehicle operating the vehicle domain multi-level parallel buffering and context-based streaming and data pre-processing system, each task corresponding to a request from a respective vehicle among the plurality of vehicles.

11. The system of claim 10, further comprising:

at least one electronic message handler configured to execute at least one data process necessary to complete a requested task; and

an electronic working task dictionary (WTD) module configured to receive a separated data stream, determine at least one process necessary to complete the requested task corresponding to the separated data stream, and deliver the separated data stream to the at least one message handler based on the determined at least one process.

12. The system of claim 11, wherein the TQD module assigns a single second-level buffer to a single working task per each vehicle, and updates the second-level buffer to include a new task for a respective vehicle after the working task is completed.

13. A method of performing parallel buffering and context-based streaming data pre-processing in a vehicle domain, the method comprising:

concurrently queuing in a first data processing level data values provided from a plurality of data streams output from a plurality of vehicles;

receiving from a first-level buffer a serial data stream in a second data processing level and separating the data values from the serial data stream based on a context of the data values;

assigning a context buffer identification value to each of a second-level buffers; and

queuing data into respective second-level data buffers based on a comparison between the context of the data values and the context buffer identification values,

wherein queuing the data includes loading data values into respective second-level buffers according to a parallel data delivery scheme to generate respective separated data streams.

14. The method of claim 13, further comprising determining a requested task from a respective vehicle based on the context of the data values, and commanding an electronic task-queue-dictionary (TQD) module to perform at least one of creating a new task in the second data processing level and updating an existing task in response to determining the requested task, the tasks corresponding to available processing operations performed by a local vehicle operating the vehicle domain multi-level parallel buffering and context-based streaming and data pre-processing system, each task corresponding to a request from a respective vehicle among the plurality of vehicles.

15. The method of claim 14, further comprising:

determining at least one process necessary to complete the requested task corresponding to the separated data stream; and

delivering the separated data stream to at least one message handler module based on the determined at least one process so as to execute at least one data process necessary to complete a requested task.

16. The method of claim 15, further comprising:

assigning a single second-level buffer to a single working task per each vehicle; and

updating the second-level buffer to include a new task for a respective vehicle after the working task is completed.

17. A computer program product to control a vehicle domain multi-level parallel buffering and context-based streaming data pre-processing system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic computer processor such that the system performs operations comprising:

buffering data provided from a plurality of raw data streams output from a plurality of vehicles using a first-level buffer installed at a first data processing level included in a multi-level parallel buffering and context-based streaming data pre-processing system;

delivering a serial data stream output from the first-level buffer to an electronic task-queue-dictionary (TQD) module installed in a second data processing level included in the multi-level parallel buffering and context-based streaming data pre-processing system;

creating a plurality of tasks via the TQD module in a in response to receiving the serial data stream; and

commanding the TQD module to assign each task to a corresponding second-level buffer so as to separate the serial data stream into individual data values that are delivered to a specific second-level buffer based on the task; and

delivering the data values in parallel to the corresponding second-level buffer.

18. The computer program product of claim 17, wherein the operations further comprise embedding vehicle identification information in the data values, and assigning a buffer identification value to each second-level buffer such that the data values are loaded into a corresponding second-level buffer having a buffer identification value that matches the vehicle identification information.

19. The computer program product of claim 17, wherein the operations further comprise limiting each vehicle to a single working task created in the second data processing level.

20. The computer program product of claim 17, wherein the operations further comprise loading each second-level buffer based on a timestamp indicated by a respective data value.