System and method for obtaining information relating to video images

• US 9,258,597 B1
• Filed: 03/13/2013
• Issued: 02/09/2016
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Clayton, Nik
• Primary Examiner(s): HANCE, ROBERT J
• Application Number:
  US13/801,843
• Time in Patent Office: 1,063 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/43 Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware real-time communication protocols in data switching networks H04L65/00
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

A system and method is provided where, in one aspect, a user signals that the user is interested in a particular image being displayed as part of a video, wherein objects in the image are detected and recognized, and wherein an action is taken based on information associated with the recognized objects. By way of example only, the actions may include recording a show (with or without a user'"'"'s prior confirmation) or displaying information about a product in the image.

Claims

1. A method comprising:

receiving, from a remote device, a request to receive supplemental information relating to an item of interest from a video frame in a first video;

obtaining, from a memory connected to the remote device, a plurality of image data including the video frame corresponding to a time of the request, at least one preceding video frame presented before the video frame, and at least one subsequent frame presented after the video frame;

comparing the plurality of image data from the video frame, the at least one preceding video frame, and the at least one subsequent frame to determine the item of interest in the received request;

performing, using a hardware processor, a search query based on the determined item of interest and device information associated with the remote device to identify a second video that is related to the first video;

identifying, using the hardware processor, first data relating to the determined item of interest based on the visual characteristics of the determined item of interest, wherein the first data comprises information relating to the second video;

determining, using the hardware processor, a message to be displayed by the remote device based on the first data and an action to be performed on the second video by the remote device based on the first data, wherein the action comprises recording the second video; and

transmitting, to the remote device, structured data that comprises a first node that includes information relating to displaying the message and a second node that includes instructions for performing the action by the remote device, wherein the transmitted structured data causes the remote device to display the message relating to the second video and perform the action on the second video.

2. The method of claim 1, wherein the remote device is a set top box.

3. The method of claim 1, wherein the first data comprises an identification of the second video, wherein visual characteristics of the object are visually similar to the visual characteristics of an object associated with the second video, and wherein the action comprises receiving the second video at the remote device.

4. The method of claim 3, wherein the second video is a television show and the action comprises recording the show at the remote device.

5. The method of claim 3, wherein the visual characteristics comprise the visual characteristics of a person appearing in both videos.

6. The method of claim 1, wherein the visual characteristics comprise text captured in the image, and wherein identifying the first data comprises using character recognition to identify the characters of the text, and wherein the first data is retrieved based on a query including the characters.

7. The method of claim 1, wherein identifying the first data comprises identifying other images that are visually similar to the image received from the remote device, wherein the other images are associated with metadata, and wherein the first data is retrieved based on the metadata.

8. The method of claim 7, wherein the other images comprise reference images of objects and the metadata comprises the name of the object.

9. The method of claim 1, wherein the action comprises displaying a webpage related to the object.

10. The method of claim 9, wherein the action comprises transmitting the URL of a webpage to be retrieved by the remote device.

11. The method of claim 1, wherein the first data is used to form a query, and wherein the action is determined based on the type of a search result returned in response to the query.

12. The method of claim 11, wherein the action comprises displaying a web page when the type of search result identifies a webpage, and wherein the action comprises receiving a video when the type of search result identifies a video that is or will be accessible to the remote device.

13. A device comprising:

a processor,

a video output structured to output video to a display;

a memory containing instructions executable by the processor; and

a user interface;

wherein the instructions comprise;

providing video via the video output;

receiving a request from the user interface indicating that a user is interested in receiving supplemental information relating to an item of interest appearing in a video frame of the video;

obtaining, from the memory, a plurality of image data including the video frame corresponding to a time of the request, at least one preceding video frame presented before the video frame, and at least one subsequent frame presented after the video frame;

comparing the plurality of image data from the video frame, the at least one preceding video frame, and the at least one subsequent frame to determine the item of interest in the received request;

transmitting the determined item of interest to perform a search query based on the determined item of interest and device information associated with the device;

receiving structured data that comprises a first node that includes information relating to displaying a message and a second node that includes instructions for performing an action to be taken by the device in response to the request, wherein the action is determined based on the visual characteristics of the image displayed to the user, and wherein the action comprises recording a second video; and

causing the message relating to the second video from the first node of the structured data to be displayed and the action from the second node of the structured data to be performed on the second video.

14. The device of claim 13, wherein the action is determined by a processor contained in the device.

15. The device of claim 13, wherein the action is determined by transmitting the image to a computer over a network and receiving the action in response.

16. The device of claim 15, further comprising receiving a data file identifying information for display to the user and the action.

17. The device of claim 13, further comprising a memory for storing video data, wherein receiving an action comprises receiving a plurality of different actions, and wherein at least one action comprises recording the second video at a future date and time.

18. The device of claim 13, wherein the device is a set top box and the user interface is a remote control operated by the user.

19. The device of claim 13, wherein the action is determined by comparing the visual characteristics of the image with the visual characteristics of other images, identifying text associated with another image if the other image is visually similar to the image displayed to the user, and wherein the action is determined based on the identified text.

20. A method of recording a video, the method comprising:

displaying, using a hardware processor, a first video to a user,

receiving a signal from a user interface that the user is interested in receiving supplemental information relating to an item of interest appearing in a video frame of the first video;

obtaining, from a memory connected to the hardware processor, a plurality of image data including the video frame corresponding to a time of the signal, at least one preceding video frame presented before the video frame, and at least one subsequent frame presented after the video frame;

comparing the plurality of image data from the video frame, the at least one preceding video frame, and the at least one subsequent video frame to identify the item of interest contained in the first video based on the signal from the user interface;

transmitting an identification of the item of interest to a device over a network;

receiving an identification of a second video based on the identification of the image and structured data that comprises a first node that includes information relating to displaying a message related to the second video and a second node that includes instructions for performing an action in connection with the second video; and

displaying, using the hardware processor, the second video using the received structured data while causing the message relating to the second video from the first node of the structured data to be displayed and the action from the second node of the structured data to be performed on the second video.

21. The method of claim 20, wherein transmitting an identification of the image comprises transmitting pixel data representing the image.

22. The method of claim 20, wherein the second video is identified based on the image'"'"'s visual similarity with other images, wherein non-image information that is associated with visually similar images is retrieved, and wherein the second video is identified based on the non-image information.