Video object tag creation and processing

• US 9,596,516 B1
• Filed: 05/15/2014
• Issued: 03/14/2017
• Est. Priority Date: 11/30/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fink, Michael, Tavory, Ran, Kolodny, Tamar Rachel, Marmasse, Natalia
• Primary Examiner(s): Amelunxen, Barbara Joan
• Application Number:
  US14/278,473
• Time in Patent Office: 1,034 Days
• Field of Search: 348/E7.071, 375/E7.024, 715/716, 725/32
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06Q 20/123 Shopping for digital content
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0254 based on statistics
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/6175 involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Methods, and systems, including computer programs encoded on computer-readable storage mediums, including a method for presenting a video content item in a first display area; concurrently presenting, with the video content item in the first display area, objects that are displayed during the presentation of the video content item in a second display area, wherein the objects persist in the second display area after the object is no longer displayed during the presentation of the video content item in the first display area; receiving an indication identifying one of the objects presented in the first display area or the second display area; and processing a tag associated with the object, the tag comprising a reference to a media item, wherein the processing comprises: accessing the media item referenced by the tag; and presenting the media item at least partially in the first display area or the second display area.

Claims

1. A method comprising:

receiving, by a video preprocessor executed by a video content item provider, an image depicting a first object;

pre-processing a video content item separate from the received image, by the video preprocessor, to identify one or more occurrences of the first object being depicted in one or more frames of the video content item;

causing presentation of the video content item at a user device, by a video server executed by the video content item provider, including the one or more frames of the video content item that depicts the first object;

receiving, by the video server from the user device, a selection of the first object depicted in a first frame, the first frame being associated with a video content item;

receiving, by the video server from the user device, a first reference to additional content for the first object; and

generating a first tag for the first object, by a tag attacher executed by the video content item provider, the first tag including the first reference, and the first tag including instructions that cause presentation of the additional content referenced by the first reference concomitant with playback of the video content item.

2. The method of claim 1, wherein pre-processing the video content item comprises:

performing, by a video analyzer executed by the video content item provider, an object recognition technique for each of the plurality of frames included in the video content item; and

for each occurrence of the first object in a frame of the plurality of frames, storing, for the video content item, an association between the first object and the frame in a memory device of the video content item provider.

3. The method of claim 1, further comprising causing presentation of the first frame of the video content item including

causing a portion of the first frame that includes the first object to be highlighted during presentation of the first frame.

4. The method of claim 1, wherein receiving, from a user device, a selection of the first object depicted in the first frame comprises:

receiving user input indicating a selection of a portion of the first frame.

5. The method of claim 1, further comprising causing presentation of the first frame of the video content item including:

causing a portion of the first frame that includes the first object to be displayed in a second display area that is separate from a first display area in which the first frame is presented.

6. The method of claim 5, wherein receiving, from the user device, the selection of the first object depicted in the first frame comprises:

receiving, by the video server from the user device, user input indicating a selection, from the second display area, of the portion of the first frame that includes the first object.

7. The method of claim 1, further comprising:

in response to receiving the selection of the first object, causing presentation of a prompt for user input, the prompt including a portion for receiving input specifying the first reference to additional content for the first object.

8. The method of claim 7, wherein the prompt includes user-selectable options for controlling presentation of the additional content during playback of the video content item, the user-selectable options including one or more of:

an option to cause presentation of the additional content to interrupt playback of the video content item;

an option to cause presentation of the additional content in a new application window;

an option to cause the first object to be highlighted and user-selectable during playback of the video content item;

an option to cause presentation of the additional content in a content display area that is separate from a display area used for playback of the video content item;

or

an option to cause presentation of the additional content when the video content item has finished playback.

9. The method of claim 1, further comprising:

identifying one or more additional occurrences of the first object in one or more other frames of the plurality of frames included in the video content item; and

tagging at least one of the additional occurrences of the first object with the first tag.

10. The method of claim 1, wherein the additional content for the first object comprises media content that is different from content included in the video content item.

11. A system comprising:

one or more data processors executing a video preprocessor, a video server, and a tag attacher; and

instructions stored on a computer readable storage medium that when executed by the one or more data processors cause the one or more data processors to perform operations comprising;

receiving, by the video preprocessor, an image depicting a first object;

pre-processing a video content item separate from the received image, by the video preprocessor, to identify one or more occurrences of the first object being depicted in one or more frames of the video content item;

causing presentation of the video content item at a user device, by the video server, including the one or more frames of the video content item that depicts the first object;

receiving, by the video server from the user device, a selection of the first object depicted in a first frame, the first frame being associated with a video content item;

receiving, by the video server from the user device, a first reference to additional content for the first object; and

generating a first tag for the first object, by the tag attacher, the first tag including the first reference, and the first tag including instructions that cause presentation of the additional content referenced by the first reference concomitant with playback of the video content item.

12. The system of claim 11, wherein the one or more data processors execute a video analyzer, and wherein pre-processing the video content item further comprises:

performing, by the video analyzer, an object recognition technique for each of the plurality of frames included in the video content item; and

for each occurrence of the first object in a frame of the plurality of frames, storing, for the video content item, an association between the first object and the frame in the computer readable storage medium.

13. The system of claim 11, wherein execution of the instructions by the one or more data processors further cause the one or more data processors to cause a portion of the first frame that includes the first object to be highlighted during presentation of the first frame.

14. The system of claim 11, wherein receiving, from a user device, a selection of the first object depicted in the first frame comprises:

receiving user input indicating a selection of a portion of the first frame.

15. The system of claim 11, wherein execution of the instructions by the one or more data processors further cause the one or more data processors to cause a portion of the first frame that includes the first object to be displayed in a second display area that is separate from a first display area in which the first frame is presented.

16. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more data processors, causes the one or more data processors to perform operations comprising:

receiving, by a video preprocessor executed by a video content item provider, an image depicting a first object;

pre-processing a video content item separate from the received image, by the video preprocessor, to identify one or more occurrences of the first object being depicted in one or more frames of the video content item;

causing presentation of the video content item at a user device, by a video server executed by the video content item provider, including the one or more frames of the video content item that depicts the first object;

receiving, by the video server from the user device, a selection of the first object depicted in a first frame, the first frame being associated with a video content item;

receiving, by the video server from the user device, a first reference to additional content for the first object; and

generating a first tag for the first object, by a tag attacher executed by the video content item provider, the first tag including the first reference, and the first tag including instructions that cause presentation of the additional content referenced by the first reference concomitant with playback of the video content item.