Apparatus and methods for embedding metadata into video stream

• US 10,431,258 B2
• Filed: 02/07/2018
• Issued: 10/01/2019
• Est. Priority Date: 10/22/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US15/891,136
• Publication Number:
  US 20180166102A1
• Time in Patent Office: 601 Days
• Field of Search: 386210, 386223-229
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/19 by using information detectable on the record carrier
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Apparatus and methods for combining metadata with video into a video stream using a 32-bit aligned payload, that is computer storage efficient and human discernable. The metadata is stored in a track in a self-describing structure. Metadata track may be decoded using an identifier reference table that is substantially smaller than typical fourCC identifier tables. The combined metadata/video stream is compatible with a standard video stream convention and may be played using conventional media player applications that reads media files compliant with MP4/MOV container format. The proposed format may enable decoding of metadata during streaming, partitioning of combined video stream without loss of metadata. The proposed format and/or metadata protocol provides for temporal synchronization of metadata with video frames.

Claims

1. A system for generating a multimedia streaming file, the system comprising:

information storage; and

one or more processors configured by computer instructions to;

obtain a series of images captured by an imaging sensor;

obtain information captured by one or more sensors other than the imaging sensor, the obtained information being relevant to one or more images within the series of images, the one or more sensors other than the imaging sensor including a first sensor;

generate an encoded video track that includes images from the series of images;

generate a sensor track that includes a first sensor record based on the obtained information;

generate a combined multimedia stream comprised of the encoded video track and the sensor track; and

store the combined multimedia stream in the information storage;

wherein;

the first sensor record comprises;

a header portion comprising a tag field comprising a sensor tag selected from potential sensor tags, the sensor tag identifying type of the obtained information;

a type size field comprising at least one of a value type field identifying a value type of a given value of the obtained information that is within the first sensor record;

an item size field indicating size of the given value of the obtained information that is within the first sensor record; and

/or a repeat field indicating a number of values of the obtained information that is within the first sensor record; and

a data portion comprising the values of the obtained information; and

wherein individual ones of the values of the obtained information correspond temporally to specific ones of the one or more images within the series of images.

2. The system of claim 1, wherein the obtained information comprises a parameter characterizing the imaging sensor during capture of the series of images.

3. The system of claim 2, wherein the parameter is selected from a group comprising white balance, imaging sensor temperature, shutter speed, contrast parameter, imaging sensor gain, and/or saturation.

4. The system of claim 1, wherein:

individual values of the number of values of the obtained information within the first sensor record correspond temporally to one or more images in a first portion of the series of images, the first portion corresponding to a first time window;

the sensor track comprises a second sensor record comprising a number of values of the obtained information, individual ones of the number of values within the second sensor record correspond temporally to one or more images in a second portion of the series of images, the second portion corresponding to a second time window; and

the first time window is configured subsequent to and non-overlapping/spaced from the second time window.

5. The system of claim 4, further comprising:

a communications interface configured to enable communication of at least a first portion of the combined multimedia stream, the first portion of the combined multimedia stream comprising the first sensor record and the first portion of the series of images;

wherein;

the first sensor record comprises first sensor information portion characterizing at least one aspect of the imaging sensor within the first time window, the second sensor record comprises second sensor information portion characterizing the at least one aspect of the imaging sensor within the second time window; and

communication of the first portion of the combined multimedia stream in absence of the communication of a second portion of the combined multimedia stream is configured to provide information related to the at least one aspect of the imaging sensor during the first time window.

6. The system of claim 5, wherein the aspect comprises one or more of the imaging sensor motion, the imaging sensor orientation, and the imaging sensor position.

7. The system of claim 5, further comprising:

a sensor coupled to the sensor interface, the sensor configured to provide the obtained information comprising one or more of the imaging sensor motion, the imaging sensor orientation, and the imaging sensor position; and

a housing configured to enclose the processor, the information storage, and the sensor.

8. The system of claim 5, further comprising:

a housing configured to enclose the one or more processors and the information storage;

wherein the sensor interface comprises a wireless communications interface configured to receive data communication from a sensor disposed external to the housing, the data communication configured to convey the obtained information.

9. The system of claim 7, wherein:

the sensor comprises a gyroscope; and

and the header portion includes a four character code “

GYRO”

.

10. The system of claim 7, wherein:

the sensor comprises an accelerometer; and

the header portion includes a four character code “

ACCL”

.

11. The system of claim 7, wherein:

the sensor comprises a 6-axis inertial measurement unit; and

the header portion includes a four character code “

IMU6”

.

12. The system of claim 7, wherein:

the sensor comprises an 9-axis inertial measurement unit; and

the header portion includes a four character code “

IMU9”

.

13. The system of claim 7, wherein:

the sensor comprises a magnetic field sensor; and

the header portion includes a four character code “

MAGN”

.

14. The system of claim 1, wherein generation of the combined multimedia stream is configured in accordance with a multimedia container format comprised of a video track and a text track;

the encoded video track being stored in the video track; and

the sensor track being stored in the text track.

15. The system of claim 1, wherein the type size field comprises the value type field and the item size field.

16. The system of claim 1, wherein the type size field comprises the value type field and the repeat field.

17. The system of claim 1, wherein the type size field comprises the item size field and the repeat field.

18. The system of claim 1, wherein the type size field is 32 bit.

19. The system of claim 1, wherein the type size field is 64 bit.

20. A non-transitory computer readable medium comprising a plurality of computer instructions configured to, when executed by a processor, decode sensor information from a multimedia stream by at least:

accessing one or more image frames from a video track of the multimedia stream, the one or more frames corresponding to a time interval;

accessing a text track of the multimedia stream corresponding to the time interval, the accessing the text track comprises steps of;

reading from the text track a sensor tag field value;

identifying one or more of type, origin, and/or meaning of the sensor information based on the sensor tag field value;

reading from the text track a type size field comprising at least one of a type portion configured to identify type of a given value of the sensor information within a sensor record;

an item size field indicating size of the given value of the sensor information; and

/or a repeat field indicating a number of values of the sensor information within the sensor record; and

reading from a data portion comprising the number of values of the sensor information;

wherein;

individual values of the number of values of the sensor information correspond temporally to the one or more images; and

the sensor tag field, the type size field and the data portion are configured to form the sensor record, the sensor record being stored in the text track.