Annotating media content for automatic content understanding

• US 10,553,252 B2
• Filed: 06/28/2019
• Issued: 02/04/2020
• Est. Priority Date: 04/24/2012
• Status: Active Grant

• Current Assignee: LiveClips, LLC
• Original Assignee: LiveClips, LLC
• Inventors: Petajan, Eric David, Weite, David Eugene, Vunic, Douglas W.
• Primary Examiner(s): Hunter, Mishawn N.
• Application Number:
  US16/457,113
• Publication Number:
  US 20190318765A1
• Time in Patent Office: 221 Days
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/19 by using information detectable on the record carrier
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/854 Content authoring

Abstract

A system for annotating frames in a media stream 114 includes a pattern recognition system (PRS) 108 to generate PRS output metadata for a frame; an archive 106 for storing ground truth metadata (GTM); a device to merge the GTM and PRS output metadata and thereby generate proposed annotation data (PAD) 110; and a user interface 109 for use by the human annotator HA 118. The user interface 104 includes an editor 111 and an input device 107 used by the HA 118 to approve GTM for the frame. An optimization system 105 receives the approved GTM and metadata output by the PRS 108, and adjusts input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTM and PRS output metadata.

Claims

1. A non-transitory, machine-readable storage medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, comprising:

performing pattern recognition on video frames of a media stream to generate pattern recognition metadata associated with the video frames of the media stream;

comparing the pattern recognition metadata associated with the video frames of the media stream and ground-truth metadata associated with the video frames of the media stream to generate a single distance metric, wherein the ground-truth metadata and the pattern recognition metadata comprise a first type of metadata; and

adjusting a set of input parameters associated with the pattern recognition according to the single distance metric, wherein the single distance metric is weighted according to the first type of metadata.

2. The non-transitory, machine-readable storage medium of claim 1, wherein the operations further comprise generating proposed annotation data associated with the video frames of the media stream.

3. The non-transitory, machine-readable storage medium of claim 2, wherein the generating the proposed annotations data comprises merging the pattern recognition metadata associated with the video frames of the media stream with the ground-truth metadata associated with the video frames of the media stream.

4. The non-transitory, machine-readable storage medium of claim 2, wherein the operations further comprise receiving approval of the proposed annotation data via a user interface to generate the ground-truth metadata associated with the video frames of the media stream.

5. The non-transitory, machine-readable storage medium of claim 2, wherein the operations further comprise presenting the proposed annotation data for the video frames via a user interface, wherein the user interface includes an editor and an input device.

6. The non-transitory, machine-readable storage medium of claim 5, wherein the presenting of the proposed annotation data comprises overlaying the proposed annotation data onto the media stream via the user interface.

7. The non-transitory, machine-readable storage medium of claim 1, wherein the adjusting the set of input parameters associated with the pattern recognition minimizes the single distance metric.

8. The non-transitory, machine-readable storage medium of claim 1, wherein the first type of metadata is associated with spatial position.

9. The non-transitory, machine-readable storage medium of claim 1, wherein the operations further comprise storing the ground-truth metadata associated with the video frames of the media stream in an archive.

10. The non-transitory, machine-readable storage medium of claim 1, wherein the operations further comprise correcting a time delay between third party metadata and the media stream via an alignment process.

11. A method, comprising:

comparing, by a processing system including a processor, pattern recognition metadata associated with video frames of a media stream and ground-truth metadata associated with the video frames of the media stream to generate a single distance metric;

adjusting, by the processing system, a set of input parameters associated with pattern recognition according to the single distance metric, wherein the single distance metric is weighted according to a first type of metadata; and

merging, by the processing system, the pattern recognition metadata associated with the video frames of the media stream with the ground-truth metadata associated with the video frames of the media stream to generate proposed annotation data associated with the video frames of the media stream.

12. The method of claim 11, further comprising performing, by the processing system, pattern recognition on the video frames of the media stream to generate the pattern recognition metadata associated with the video frames of the media stream.

13. The method of claim 11, wherein the adjusting the set of input parameters associated with the pattern recognition minimizes the single distance metric.

14. The method of claim 11, further comprises storing, by the processing system, the ground-truth metadata associated with the video frames of the media stream in an archive, wherein the ground-truth metadata and the pattern recognition metadata comprise the first type of metadata.

15. The method of claim 11, further comprising comprise presenting, by the processing system, the proposed annotation data for the video frames via a user interface, wherein the user interface includes an editor and an input device, wherein the presenting of the proposed annotation data comprises overlaying the proposed annotation data onto the media stream via the user interface.

16. The method of claim 11, further comprising receiving, by the processing system, approval of the proposed annotation data via a user interface to generate the ground-truth metadata associated with the video frames of the media stream.

17. The method of claim 11, further comprising correcting, by the processing system, a time delay between third party metadata and the media stream via an alignment process.

18. A system, comprising:

a pattern recognition system, including a processing system comprising a processor, that facilitates generating pattern recognition metadata associated with video frames of a media stream according to a set of input parameters associate with the pattern recognition system; and

an optimization system, including the processing system, that facilitates comparing the pattern recognition metadata associated with the video frames of the media stream and ground-truth metadata associated with the video frames of the media stream to generate a single distance metric, and wherein the set of input parameters associated with the pattern recognition system are adjusted according to the single distance metric.

19. The system of claim 18, further comprising an encoder system, including the processing system, that facilitates generating proposed annotation data associated with the video frames of the media stream by merging the pattern recognition metadata with the ground-truth metadata associated with the video frames of the media stream.

20. The system of claim 19, further comprising receiving approval of the proposed annotation data via a user interface to generate the ground-truth metadata associated with the video frames of the media stream.