AUGMENTING VIDEO DATA TO PRESENT REAL-TIME METRICS

• US 20180084310A1
• Filed: 09/19/2017
• Published: 03/22/2018
• Est. Priority Date: 09/21/2016
• Status: Active Grant

• Current Assignee: Gumgum Sports Incorporated
• Original Assignee: GumGum, Inc.
• Inventors: Katz, Jeffrey Benjamin, Carter, Cambron Neil, Kim, Brian Jongmin
• Granted Patent: US 10,255,505 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0275 Auctions
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30224 Ball; Puck
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 2201/09 : Recognition of logos
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 30/19173 : Classification techniques
• CPC Class Codes: G06V 30/1983 : Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: H04H 60/47 : for recognising genres
• CPC Class Codes: H04H 60/48 : for recognising items expressed in broadcast information
• CPC Class Codes: H04H 60/59 : of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/63 : for services of sales data processing systems or methods specially adapted for electronic commerce G06Q30/00
• CPC Class Codes: H04H 60/66 : for using the result on distributors' side
• CPC Class Codes: H04N 21/2187 : Live feed
• CPC Class Codes: H04N 21/232 : Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/23418 : involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2387 : Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/2407 : Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/2547 : Third Party Billing, e.g. billing of advertiser
• CPC Class Codes: H04N 21/26241 : involving the time of distribution, e.g. the best time of the day for inserting an advertisement or airing a children program
• CPC Class Codes: H04N 21/2665 : Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/44008 : involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44204 : Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/6581 : Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8133 : specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

Systems and methods are described for augmenting video data based on automated identification of one or more objects depicted in the video data. One or more classification models may identify an object of interest in video data. An aggregated duration count may be maintained that reflects a length of time that the object of interest has been depicted in the video data. This duration or additional metric data derived in part from the duration may be displayed in association with display of the video data and continuously updated during playback of the video data.

Claims

1. A computing system comprising:

a memory; and

a processor in communication with the memory and configured with processor-executable instructions to perform operations comprising;

receiving video data depicting a sporting event;

providing at least a first frame of the video data to each of a plurality of classification models that are each trained to identify depiction of one or more sponsor logos in image or video data;

identifying a sponsor logo within at least the first frame based at least in part on output of one or more of the plurality of classification models;

tracking an in-frame location of the sponsor logo across a plurality of subsequent frames in which the sponsor logo is depicted;

augmenting image data in at least the plurality of subsequent frames to visually mark the sponsor logo;

determining an aggregated duration of time that the sponsor logo has been depicted in the video data; and

during presentation of the video data, causing display of at least one of (a) the aggregated duration of time that the sponsor logo has appeared in the video data, or (b) an aggregated media value attributable to appearance of the sponsor logo in the video data.

2. The computing system of claim 1, wherein the video data is received by the system from a broadcast source substantially in real time as the sporting event occurs.

3. The computing system of claim 1, wherein the aggregated duration and the aggregated media value are each displayed during presentation of the video data.

4. The computing system of claim 3, wherein the operations further comprise updating display of the aggregated duration and the aggregated media value substantially in real time during presentation of additional frames of the video data to reflect that the computing system detects that the sponsor logo is depicted in the additional frames.

5. The computing system of claim 1, wherein the operations further comprise:

identifying a second sponsor logo depicted in the first frame of the video data; and

tracking an in-frame location of the second sponsor logo across at least a subset of the plurality of subsequent frames.

6. The computing system of claim 1, wherein augmenting the image data to visually mark the sponsor logo comprises at least one of:

displaying a visual bounding shape around the sponsor logo, or changing a brightness or contrast of the sponsor logo.

7. The computing system of claim 1, wherein the operations further comprise:

identifying at least one frame in which the sponsor logo is no longer depicted; and

updating display of the at least one of the aggregated duration of time or the aggregated media value to visually indicate that the sponsor logo is not depicted in a current portion of the video data.

8. The computing system of claim 7, wherein updating the display of the at least one of the aggregated duration of time or the aggregated media value comprises at least one of applying a shading effect or increasing a transparency level of the at least one of the aggregated duration of time or the aggregated media value as displayed.

9. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific executable instructions,

receiving a stream of video data depicting a sporting event;

substantially in real time as the stream of video data is received, identifying a sponsor logo depicted in individual frames of the video data, wherein the sponsor logo is identified based at least in part on output of a classification model that is trained to identify depiction of the sponsor logo in image or video data;

continuously updating an aggregated duration of time that the sponsor logo has been depicted in the video data as the stream of video data is received;

continuously updating an aggregated media value attributable to appearance of the sponsor logo in the video data based at least in part on the aggregated duration of time that the sponsor logo has been depicted; and

as the stream of video data is presented for playback, causing display of a current aggregated media value attributable to appearance of the sponsor logo in the video data, wherein the current aggregated media value displayed changes to reflect the continuous updating of the aggregated media value during playback of the stream of video data.

10. The computer-implemented method of claim 9, further comprising augmenting image data in each of a plurality of frames of the video data to visually mark the sponsor logo.

11. The computer-implemented method of claim 9, wherein the video data is broadcast by a television network, and wherein the aggregated media value attributable to appearance of the sponsor logo is based at least in part on a cost to purchase an advertisement from the television network.

12. The computer-implemented method of claim 11, wherein the aggregated media value is based at least in further part on real time ratings or viewership data associated with the broadcast of the video data by the television network.

13. The computer-implemented method of claim 9, further comprising identifying that a portion of the video data has been shared on a social media network, and wherein the aggregated media value attributable to appearance of the sponsor logo is based at least in part on an estimated viewership of the portion of the video data on the social media network.

14. The computer-implemented method of claim 9, wherein the video data is received via the Internet as a live video stream while the sporting event occurs.

15. The computer-implemented method of claim 9, wherein the aggregated media value attributable to appearance of the sponsor logo is based at least in part on at least one of size, clarity, duration or position of the sponsor logo in one or more frames of the video data.

16. The computer-implemented method of claim 9, wherein causing display of a current aggregated media value comprises augmenting the video data to display the current aggregated media value over content of the video data.

17. The computer-implemented method of claim 9, wherein causing display of a current aggregated media value comprises generating a user interface that is presented in association with playback of the video data.

18. The computer-implemented method of claim 9, further comprising:

tracking an in-frame location of the sponsor logo across a plurality of frames of the video.

19. The computing system of claim 18, further comprising augmenting the video data to visually mark the in-frame location of the sponsor logo.

20. The computing system of claim 19, wherein augmenting the video data to visually mark the in-frame location of the sponsor logo comprises at least one of:

displaying a visual bounding shape around the sponsor logo, or changing a brightness or contrast of the sponsor logo.