Identification and presentation of internet-accessible content associated with currently playing television programs

• US 10,237,612 B2
• Filed: 10/06/2017
• Issued: 03/19/2019
• Est. Priority Date: 09/19/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Verrilli, Michael, Sandland, Nathan Leslie
• Primary Examiner(s): Flynn, Nathan J
• Assistant Examiner(s): Trinh, Tung T
• Application Number:
  US15/727,394
• Publication Number:
  US 20180035162A1
• Time in Patent Office: 529 Days
• Field of Search: 725 14
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image

Abstract

Features of various implementations are used to enable identifying and presenting, in real time, Internet-accessible and/or Internet-based functionality, content and/or services associated with a particular television program concurrently playing on a television or display, as a part of an interactive television suite of technologies. In some implementations, the process of identifying Internet-accessible and/or Internet-based functionality, content and/or services associated with a particular television program is enabled to reduce communication between a primary set-top box and a supplemental set-top box and/or television integrated device, thereby reducing the need for one or more provider specific APIs. For example, one aspect of the disclosure is a computer-implemented method of identifying a media program by capturing display data from the media program, and extracting information from the display data in response to determining that the display data includes a program information overlay associated with the media program.

Claims

1. A computer-implemented method, comprising:

at an electronic device that is communicatively coupled to a display device, a second screen client device, and an Internet search engine, and while a media program is being provided to the display device for display thereon;

receiving from the second screen client device a content search request associated with the media program displayed on the display device;

in response to the content search request;

capturing screen display data associated with the media program displayed on the display device;

determining whether a program information overlay is being displayed on the display device, including analyzing the captured screen display data to determine whether the screen display data includes one or more predefined indicators of the program information overlay;

in accordance with a determination that the program information overlay is being displayed on the display device, extracting information displayed on the program information overlay;

generating search terms from the extracted information;

performing an Internet search based on a subset of the generated search terms via the Internet search engine to identify content associated with the media program; and

while the media program is being displayed on the display device, transmitting search results of the Internet search to at least one of the display device and the second screen client device for concurrent and synchronous display thereon.

2. The method of claim 1, wherein the Internet search is selected from a group consisting of a general Internet search, a targeted search for associated news items, a targeted search for associated images, a targeted search for associated Internet-accessible media content, and a targeted search for associated social media content.

3. The method of claim 1, wherein the one or more predefined indicators include at least one of:

a color of the program information overlay, a percentage of display area the program information overlay typically covers when visible, an arrangement of text with the program information overlay, contrast between a portion of a display likely to be displaying the media program and another portion of the display likely to be displaying the program information overlay.

4. The method of claim 1, wherein capturing the screen display data includes performing a screen capture of the display device, and extracting the information displayed on the program information overlay comprises applying an optical character recognition process to the captured screen display data.

5. The method of claim 1, wherein a set-top box is coupled to the display device, further comprising:

obtaining screen display data of the program information overlay from one of a display card of the set-top box, an operating system of the set-top box, and a television application running on the set-top box.

6. The method of claim 1, further comprising:

obtaining screen display data of the program information overlay by decoding the media program streamed by the electronic device to the display device.

7. The method of claim 1, further comprising:

comparing the extracted information to electronic program guide data to confirm identification of the media program that is played on the display device.

8. The method of claim 1, further comprising:

transmitting a code that causes the display device to receive the program information overlay in the screen display data, and

capturing the screen display data associated with the media program in response to transmitting the code that causes the display device to receive the program information overlay in the screen display data.

9. An electronic device, wherein the electronic device is communicatively coupled to a display device, a second screen client device, and an Internet search engine, comprising:

one or more processors; and

memory storing one or more programs, the one or more programs including instructions that, when executed by the one or more processors, cause the device to implement operations for, while a media program is being provided to the display device for display thereon;

receiving from the second screen client device a content search request associated with the media program displayed on the display device;

in response to the content search request;

capturing screen display data associated with the media program displayed on the display device;

determining whether a program information overlay is being displayed on the display device, including analyzing the captured screen display data to determine whether the screen display data includes one or more predefined indicators of the program information overlay;

in accordance with a determination that the program information overlay is being displayed on the display device, extracting information displayed on the program information overlay;

generating search terms from the extracted information;

performing an Internet search based on a subset of the generated search terms via the Internet search engine to identify content associated with the media program; and

while the media program is being displayed on the display device, transmitting search results of the Internet search to at least one of the display device and the second screen client device for concurrent and synchronous display thereon.

10. The electronic device of claim 9, wherein generating search terms comprises:

identifying noun phrases in the extracted information; and

selecting a threshold number of the noun phrases to be search terms.

11. The electronic device of claim 10, wherein identifying noun phrases comprises:

removing program showing data from the extracted information to produce one or more descriptive phrases; and

removing at least one verb from the one or more descriptive phrases to produce one or more of the noun phrases.

12. The electronic device of claim 11, wherein identifying noun phrases comprises comparing segments of the one or more descriptive phrases to known names of television shows, movies, sporting events and known proper names.

13. The electronic device of claim 10, the one or more programs further comprising instructions for refining each of the selected threshold number of search terms by at least one of:

checking and correcting for typographical errors;

cross-referencing against known names of television shows, movies, sporting events and known proper names; and

cross-referencing against prior search data.

14. The electronic device of claim 10, wherein performing the Internet search comprises performing a respective Internet search for each of the threshold number of noun phrases, and wherein each respective Internet search produces corresponding search results.

15. The electronic device of claim 9, the one or more programs further

comprising instructions for:

scoring the search results; and

enabling display of at least some of the search results based on the scoring.

16. The electronic device of claim 15, wherein the scoring comprises:

categorizing the search results;

assigning a query score based on at least one of location data, search history and user preferences;

assigning a proximity score to each of the threshold number of the noun phrases selected as search terms based on the order in which the noun phrases appear in the extracted information; and

multiplying each query score by a corresponding proximity score to produce a respective relevance score, wherein the results of the search are displayed based at least on the respective relevance scores.

17. A non-transitory computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an electronic device cause the device to implement operations for:

at the electronic device, wherein the electronic device is communicatively coupled to a display device, a second screen client device, and an Internet search engine, while a media program is being provided to the display device for display thereon;

receiving from the second screen client device a content search request associated with the media program displayed on the display device;

in response to the content search request;

capturing screen display data associated with the media program displayed on the display device;

determining whether a program information overlay is being displayed on the display device, including analyzing the captured screen display data to determine whether the screen display data includes one or more predefined indicators of the program information overlay;

in accordance with a determination that the program information overlay is being displayed on the display device, extracting information displayed on the program information overlay;

generating search terms from the extracted information;

performing an Internet search based on a subset of the generated search terms via the Internet search engine to identify content associated with the media program; and

while the media program is being displayed on the display device, transmitting search results of the Internet search to at least one of the display device and the second screen client device for concurrent and synchronous display thereon.

18. The non-transitory computer readable storage medium of claim 17, wherein the Internet search is performed when the electronic device submits a search query based on the at least some of the generated search terms to the Internet search engine that is configured to provide information stored on one or more third party content servers according to heuristic data of prior searches of a plurality of users and content providers.

19. The non-transitory computer readable storage medium of claim 17, wherein the Internet search is selected from a group consisting of a general Internet search, a targeted search for associated news items, a targeted search for associated images, a targeted search for associated Internet-accessible media content, and a targeted search for associated social media content.

20. The non-transitory computer readable storage medium of claim 17, the one or more programs further comprising instructions for:

initiating the display of the program information overlay; and

ending the display of the program information overlay in response to capturing the screen display data associated with the media program displayed on the display device.