System and method for obtaining information relating to video images

• US 9,906,840 B2
• Filed: 10/03/2016
• Issued: 02/27/2018
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Clayton, Nik
• Primary Examiner(s): HANCE, ROBERT J
• Application Number:
  US15/284,145
• Publication Number:
  US 20170026708A1
• Time in Patent Office: 512 Days
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/43 Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware real-time communication protocols in data switching networks H04L65/00
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

A system and method is provided where, in one aspect, a user signals that the user is interested in a particular image being displayed as part of a video, wherein objects in the image are detected and recognized, and wherein an action is taken based on information associated with the recognized objects. By way of example only, the actions may include recording a show (with or without a user'"'"'s prior confirmation) or displaying information about a product in the image.

Claims

1. A method for presenting supplemental content, the method comprising:

receiving, using a hardware processor, a request regarding an item of interest that was presented in a media content item;

obtaining, using the hardware processor, a plurality of image data including a video frame of the media content item corresponding to a time of the received request and at least one of a preceding video frame presented before the video frame and a subsequent frame presented after the video frame;

comparing, using the hardware processor, the plurality of image data to identify the item of interest;

identifying, using the hardware processor, supplemental content that is associated with the item of interest by performing a search with a search query based on the item of interest and receiving supplemental content as search results responsive to the search query;

transmitting, using the hardware processor, a response to the request that includes the supplemental content corresponding to the item of interest identified in the media content item; and

causing, based on the transmitted response to the request, a computing device to present the supplemental content corresponding to the item of interest identified in the media content item and perform an action on the computing device in connection with the supplemental content.

2. The method of claim 1, wherein the supplemental content includes a second media content item that is different than the media content item and wherein the method further comprises:

identifying first data relating to the item of interest based on visual characteristics of the determined item of interest, wherein the first data comprises information relating to the second media content item; and

determining a message to be displayed based on the first data and an action to be performed on the second media content item based on the first data.

3. The method of claim 2, wherein transmitting the response further comprises transmitting structured data that comprises a first node that includes information relating to displaying the message and a second node that includes instructions for performing the action by the computing device, wherein the transmitted structured data causes the computing device to display the message relating to the second media content item and perform the action on the second media content item.

4. The method of claim 3, wherein the action comprises displaying a webpage related to the item of interest and wherein the structured data includes a URL of the webpage.

5. The method of claim 2, wherein the visual characteristics of the item of interest in the first media content item are similar to the visual characteristics of an object appearing in the second media content item.

6. The method of claim 2, wherein the visual characteristics comprise text captured in the video frame of the media content item, wherein identifying the first data comprises using character recognition to identify the characters of the text, and wherein the first data is retrieved based on a query including the characters.

7. The method of claim 2, wherein identifying the first data comprises identifying other images that are visually similar to the plurality of image data, wherein the other images are associated with metadata, and wherein the first data is retrieved based on the metadata.

8. The method of claim 2, wherein the first data is used to form the search query that is performed on the item of interest to identify the supplemental content and wherein the action is determined based on a type of search result responsive to the search query.

9. A system for presenting supplemental content, the system comprising:

a hardware processor that;

receives a request regarding an item of interest that was presented in a media content item;

obtains a plurality of image data including a video frame of the media content item corresponding to a time of the received request and at least one of a preceding video frame presented before the video frame and a subsequent frame presented after the video frame;

compares the plurality of image data to identify the item of interest;

identifies supplemental content that is associated with the item of interest by performing a search with a search query based on the item of interest and receiving supplemental content as search results responsive to the search query;

transmits a response to the request that includes the supplemental content corresponding to the item of interest identified in the media content item; and

causes, based on the transmitted response to the request, a computing device to present the supplemental content corresponding to the item of interest identified in the media content item and perform an action on the computing device in connection with the supplemental content.

10. The system of claim 9, wherein the supplemental content includes a second media content item that is different than the media content item and wherein the hardware processor is further configured to:

identify first data relating to the item of interest based on visual characteristics of the determined item of interest, wherein the first data comprises information relating to the second media content item; and

determine a message to be displayed based on the first data and an action to be performed on the second media content item based on the first data.

11. The system of claim 10, wherein the hardware processor is further configured to transmit the response further comprises transmitting structured data that comprises a first node that includes information relating to displaying the message and a second node that includes instructions for performing the action by the computing device, wherein the transmitted structured data causes the computing device to display the message relating to the second media content item and perform the action on the second media content item.

12. The system of claim 11, wherein the action comprises displaying a webpage related to the item of interest and wherein the structured data includes a URL of the webpage.

13. The system of claim 10, wherein the visual characteristics of the item of interest in the first media content item are similar to the visual characteristics of an object appearing in the second media content item.

14. The system of claim 10, wherein the visual characteristics comprise text captured in the video frame of the media content item, wherein identifying the first data comprises using character recognition to identify the characters of the text, and wherein the first data is retrieved based on a query including the characters.

15. The system of claim 10, wherein identifying the first data comprises identifying other images that are visually similar to the plurality of image data, wherein the other images are associated with metadata, and wherein the first data is retrieved based on the metadata.

16. The system of claim 10, wherein the first data is used to form the search query that is performed on the item of interest to identify the supplemental content and wherein the action is determined based on a type of search result responsive to the search query.

17. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a hardware processor, cause the hardware processor to perform a method for presenting supplemental content, the method comprising:

receiving a request regarding an item of interest that was presented in a media content item;

obtaining a plurality of image data including a video frame of the media content item corresponding to a time of the received request and at least one of a preceding video frame presented before the video frame and a subsequent frame presented after the video frame;

comparing the plurality of image data to identify the item of interest;

identifying supplemental content that is associated with the item of interest by performing a search with a search query based on the item of interest and receiving supplemental content as search results responsive to the search query;

transmitting a response to the request that includes the supplemental content corresponding to the item of interest identified in the media content item; and

causing, based on the transmitted response to the request, a computing device to present the supplemental content corresponding to the item of interest identified in the media content item and perform an action on the computing device in connection with the supplemental content.

18. The non-transitory computer-readable medium of claim 17, wherein the supplemental content includes a second media content item that is different than the media content item and wherein the method further comprises:

identifying first data relating to the item of interest based on visual characteristics of the determined item of interest, wherein the first data comprises information relating to the second media content item; and

determining a message to be displayed based on the first data and an action to be performed on the second media content item based on the first data.

19. The non-transitory computer-readable medium of claim 18, wherein transmitting the response further comprises transmitting structured data that comprises a first node that includes information relating to displaying the message and a second node that includes instructions for performing the action by the computing device, wherein the transmitted structured data causes the computing device to display the message relating to the second media content item and perform the action on the second media content item.

20. The non-transitory computer-readable medium of claim 19, wherein the action comprises displaying a webpage related to the item of interest and wherein the structured data includes a URL of the webpage.

21. The non-transitory computer-readable medium of claim 18, wherein the visual characteristics of the item of interest in the first media content item are similar to the visual characteristics of an object appearing in the second media content item.

22. The non-transitory computer-readable medium of claim 18, wherein the visual characteristics comprise text captured in the video frame of the media content item, wherein identifying the first data comprises using character recognition to identify the characters of the text, and wherein the first data is retrieved based on a query including the characters.

23. The non-transitory computer-readable medium of claim 18, wherein identifying the first data comprises identifying other images that are visually similar to the plurality of image data, wherein the other images are associated with metadata, and wherein the first data is retrieved based on the metadata.

24. The non-transitory computer-readable medium of claim 18, wherein the first data is used to form the search query that is performed on the item of interest to identify the supplemental content and wherein the action is determined based on a type of search result responsive to the search query.