Method and system of managing data files

• US 9,947,072 B2
• Filed: 10/19/2016
• Issued: 04/17/2018
• Est. Priority Date: 09/16/2010
• Status: Active Grant

• Current Assignee: Cubic Digital Intelligence, Inc. (Cubic Corporation)
• Original Assignee: Pixia Corp. (Cubic Corporation)
• Inventors: Thakkar, Rahul C., Pakula, Scott L., Ernst, Rudolf O.
• Primary Examiner(s): Do, Anh H
• Application Number:
  US15/297,987
• Publication Number:
  US 20170039675A1
• Time in Patent Office: 545 Days
• Field of Search: 382305, 382205, 382173, 382283, 382224, 382282, 382175, 382298, 1 1, 360 60, 360137, 369 5345, 370452, 370455, 705 11, 705 3, 705 712, 705310, 707737
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2343 Locking methods, e.g. distributed locking or locking implementation details
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82

Abstract

Methods and systems of managing data files are described herein. One method includes updating a container file of data files having a container file metadata section and one or more empty records, each record of the one or more empty records having: (i) a data file section reserved for storing a data file, (ii) a file metadata section reserved for storing metadata about the data file, and (iii) a record metadata section including information about the record and having at least a record status mark indicating that the record is empty. The method includes setting the record status mark of the one or more empty records to being updated; writing one or more records containing the data file into the one or more empty records within the container file. After successfully writing the one or more records resetting the record status mark to valid.

Claims

1. A method of updating a container file of data files, the container file having a container file metadata section and one or more empty records, each record of the one or more empty records comprising:

(i) a data file section reserved for storing a data file, (ii) a file metadata section reserved for storing metadata about the data file, and (iii) a record metadata section including information about the record and having at least a record status mark indicating that the record is empty, the method being implemented by a computer system that includes one or more processors configured to execute processing operations, the method comprising;

setting, by the one or more processors, the record status mark of the one or more empty records to being updated;

writing, by the one or more processors, one or more records containing the data file into the one or more empty records within the container file;

if writing the one or more records is successful, resetting, by the one or more processors, the record status mark to valid after writing the one or more records; and

updating, by the one or more processors, the container file metadata to reflect a number of records in the container file.

2. The method according to claim 1, further comprising:

setting, by the one or more processors, a container status mark in the container file metadata section to unavailable or being updated while updating or writting the one or more records in the container file, and

resetting, by the one or more processors, the container status mark to available, after updating the container file metadata.

3. The method according to claim 1, wherein writing the one or more records comprises writing the one or more records sequentially or in parallel into the one or more empty records within the container file.

4. The method according to claim 1, wherein the data file comprises an image file.

5. A computer system of managing a container file of data files, the container file having a container file metadata section and one or more empty records, each record of the one or more empty records comprising:

(i) a data file section reserved for storing a data file, (ii) a file metadata section reserved for storing metadata about the data file, and (iii) a record metadata section including information about the record and having at least a record status mark indicating that the record is empty, the computer system comprising one or more processors configured to;

set the record status mark of the one or more empty records to being updated;

write one or more records containing the data file into the one or more empty records within the container file;

if writing the one or more records is successful, reset the record status mark to valid after writing the one or more records; and

update the container file metadata to reflect a number of records in the container file.

6. The computer system according to claim 5, wherein the one or more processors are further configured to:

set a container status mark in the container file metadata section to unavailable or being updated while updating or writing the one or more records in the container file; and

reset the container status mark to available, after the container file metadata is updated.

7. The computer system according to claim 5, wherein the one or more processors are further configured to write the one or more records sequentially or in parallel into the one or more empty records within the container file.

8. The computer system according to claim 5, wherein the data file comprises an image file.

9. A method of monitoring and controlling insertion of data files into a container file on a storage device, the method being implemented by a computer system that includes one or more processors configured to execute processing operations comprising:

inserting, by a plurality of insert services executed by the one or more processors, a plurality of data files into the plurality of container files, each of the plurality of data files being inserted by a corresponding one insert service in the plurality of insert services, each one insert service being executed by the one or more processors being configured to;

open a container file in a plurality of container files on the storage device, the container file comprising one or more records, each record including;

(i) a data file section reserved for storing a data file,

(ii) a metadata section reserved for storing data about the data file, and

(iii) a record metadata section having a status flag or status mark indicating whether the data file section is empty;

write the data file into a corresponding record in the one or more records and write the record into a corresponding container file and mark the status mark of the record metadata section of the record as being updated while writing the record into the container file; and

after writing the record into the corresponding container file, marking the status flag or status mark in the record metadata section of the record as being valid, if the writing is successful, or marking the status flag or status mark in the record metadata section of the record as being invalid, if the writing is not successful.

10. The method according to claim 9, further comprising prior to inserting, retrieving, by a plurality of job managers executed by the one or more processors, the plurality of insert services of inserting the plurality of data files into the plurality of container files from a queue manager, each job manager being assigned to a single container file in the plurality of container files, and the queue manager overseeing a submission, by the plurality of job managers of the plurality of data files to the plurality of container files stored on the storage device.

11. The method according to claim 10, wherein the retrieving by the plurality of job managers of the plurality of jobs of inserting the plurality of data files into the plurality of containers is performed in parallel.

12. The method according to claim 9, wherein the inserting by the plurality of insert services comprises running a plurality of insert services in parallel, each insert service using a hypertext transfer protocol (HTTP) request.

13. The method according to claim 9, wherein the data file comprises an image file.

14. A computer system for monitoring and controlling insertion of data files into a container file on a storage device, the computer system comprising one or more processors configured to:

insert, by a plurality of insert services executed by the one or more processors, a plurality of data files into the plurality of container files, each of the plurality of data fides being inserted by a corresponding one insert service in the plurality of insert services, each one insert service being executed by the one or more processors being configured to;

open a container file in a plurality of container files on the storage device, the container file comprising one or more records, each record including;

(i) a data file section reserved for storing a data file,

(ii) a metadata section reserved for storing data about the data file, and

(iii) a record metadata section having a status flag or status mark indicating whether the data file section is empty,

write the data file into a corresponding record in the one or more records and write the record into a corresponding container file and mark the status mark of the record metadata section of the record as being updated while writing the record into the container file; and

after writing the record into the corresponding container file, mark the status flag or status mark in the record metadata section of the record as being valid, if the writing is successful, or marking the status flag or status mark in the record metadata section of the record as being invalid, if the writing is not successful.

15. The computer system according to claim 14, wherein the one or more processors are further configured to retrieve, prior to inserting, using a plurality of job managers executed by the one or more processors, the plurality of insert services of inserting the plurality of data files into the plurality of container files from a queue manager, each job manager being assigned to a single container file in the plurality of container files, and the queue manager overseeing a submission, by the plurality of job managers of the plurality of data files to the plurality of container files stored on the storage device.

16. The computer system according to claim 14, wherein the one or more processors are further configured to insert the plurality of data files into the plurality of containers in parallel.

17. The computer system according to claim 14, wherein the one or more processors are further configured to insert the plurality of data files using the plurality of insert services running in parallel, each insert service using a hypertext transfer protocol (HTTP) request.

18. The computer system according to claim 14, wherein the data file comprises an image file.

19. A method for providing a video codestream from a plurality of containers, the method being implemented by a computer system that includes one or more processors configured to execute computer program modules, the method comprising:

receiving, by the one or more processors, a request comprising a plurality of request parts, each request part being a request for generating a partial video codestream of a plurality of partial video codestreams from a corresponding specified container in the plurality of containers, each partial video codestream containing a sequence of viewports, each viewport in the sequence of viewports being contained within one image in a plurality of temporally sequential images located within a plurality of records stored in the corresponding specified container;

retrieving in parallel, by the one or more processors, the sequence of viewports for each partial video codestream in the plurality of partial video codestreams from the corresponding specified container;

encoding in parallel, by the one or more processors, the sequence of viewports for each of the plurality of partial video codestreams to generate the plurality of partial video codestreams; and

delivering, by the one or more processors, the plurality of partial video codestreams to a user computer in response to the request comprising the plurality of request parts.

20. The method according to claim 19, further comprising combining, by the one or more processors, the plurality of partial video codestreams into a single video codestream and delivering the single video codestream to the user computer.

21. The method according to claim 19, further comprising interleaving data between adjacent viewports in the sequence of viewports.

22. The method according to claim 21, wherein interleaving the data includes interleaving audio information, close captioned data, or Key Length Value (KLV) metadata, or any combination thereof.

23. A computer system for providing a video codestream from a plurality of containers, the computer system comprising one or more processors configured to:

receive a request comprising a plurality of request parts, each request part being a request to generate a partial video codestream of a plurality of partial video codestreams from a corresponding specified container in a plurality of containers, each partial video codestream containing a sequence of viewports, each viewport in the sequence of viewports being contained within one image in a plurality of temporally sequential images located within a plurality of records stored in the corresponding specified container;

retrieve in parallel the sequence of viewports for each partial video codestream in the plurality of partial video codestreams from the corresponding specified container;

encode in parallel the sequence of viewports for each of the plurality of partial video codestreams to generate the plurality of partial video codestreams; and

deliver the plurality of partial video codestreams to a user computer in response to the request comprising the plurality of request parts.

24. The computer system according to claim 23, wherein the one or more processors are further configured to combine the plurality of partial video codestreams into a single video codestream and to deliver the single video codestream to the user computer.

25. The computer system according to claim 23, wherein the one or more processors are further configured to in data between adjacent viewports in the sequence of viewports.

26. The computer system according to claim 25, wherein the data includes audio information, close captioned data, or Key Length Value (KLV) metadata, or any combination thereof.