Verbal queries relative to video content

• US 10,271,109 B1
• Filed: 09/16/2015
• Issued: 04/23/2019
• Est. Priority Date: 09/16/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Davoust, Peter Martin
• Primary Examiner(s): Kumar, Pankaj
• Assistant Examiner(s): Tilahun, Alazar
• Application Number:
  US14/855,995
• Time in Patent Office: 1,315 Days
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/7343 Query language or query format
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/47815 Electronic shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

Disclosed are various embodiments for processing verbal queries relative to video content. A verbal query that is associated with a portion of video content is received. The verbal query specifies a relative frame location. An item depicted in the portion of the video content at the relative frame location is identified. Information about the item is determined as an answer to the verbal query. Information about the item is then presented to a user.

Claims

1. A non-transitory computer-readable medium embodying a program executable in at least one computing device, wherein when executed the program causes the at least one computing device to at least:

receive a verbal query via a microphone associated with a user account;

perform natural language processing on the verbal query to determine a region of a video frame expressed in relative terms in the verbal query;

identify a portion of video content that is currently being presented via a display associated with the user account;

identify an item depicted in the portion of the video content at the region;

determine information about the item as an answer to the verbal query; and

cause the information about the item to be presented via an audio device associated with the user account using a speech synthesizer.

2. The non-transitory computer-readable medium of claim 1, wherein when executed the program further causes the at least one computing device to at least:

identify an ambiguity in the relative terms of the verbal query;

cause a grid to be rendered as an overlay on top of the video content via the display, the grid containing a plurality of cells; and

receive a verbal confirmation via the microphone, the verbal confirmation identifying one of the plurality of cells.

3. The non-transitory computer-readable medium of claim 1, wherein the item corresponds to a cast member in the video content, and the information comprises at least one of a cast member name of the cast member or a character name of the cast member.

4. A system, comprising:

at least one computing device; and

at least one application executed in the at least one computing device, wherein when executed the at least one application causes the at least one computing device to at least;

receive a verbal query associated with a user account, the verbal query specifying a region of a video frame expressed in relative terms;

identify a portion of video content that is currently being presented via the user account;

identify an item depicted in the portion of the video content at the region;

determine information about the item as an answer to the verbal query; and

cause the information about the item to be presented.

5. The system of claim 4, wherein when executed the at least one application further causes the at least one computing device to at least:

determine that an ambiguity exists in the verbal query;

cause a follow-up question to be presented;

receive a verbal response to the follow-up question; and

wherein the item is identified based at least in part on the verbal response.

6. The system of claim 4, wherein when executed the at least one application further causes the at least one computing device to at least:

receive a verbal order command after the information about the item is presented; and

initiate an order of the item via the user account in response to the verbal order command.

7. The system of claim 4, wherein when executed the at least one application further causes the at least one computing device to at least perform natural language processing on the verbal query to determine the relative terms specified in the verbal query.

8. The system of claim 4, wherein the information comprises an action performed by the item in the portion of the video content.

9. The system of claim 4, wherein when executed the at least one application further causes the at least one computing device to at least:

identify a plurality of items that appear during the portion of the video content;

cause a plurality of indicia corresponding to individual ones of the plurality of items to be rendered as an overlay on top of the video content via a display; and

wherein the region specified by the verbal query is an identification of a particular indicium of the plurality of indicia.

10. The system of claim 4, wherein when executed the at least one application further causes the at least one computing device to at least:

cause a grid to be rendered as an overlay on top of the video content via a display, the grid containing a plurality of cells; and

wherein the region specified by the verbal query is an identification of a particular cell of the plurality of cells.

11. The system of claim 4, wherein causing the information about the item to be presented further comprises causing a speech synthesizer to announce the information via an audio device.

12. The system of claim 4, wherein causing the information about the item to be presented further comprises causing a graphical overlay to be rendered on top of the video content via a display.

13. A method, comprising:

receiving, via at least one computing device, a verbal query associated with a portion of video content, the verbal query specifying a region in a video frame expressed in relative terms;

identifying, via the at least one computing device, an item depicted in the portion of the video content at the region;

determining, via the at least one computing device, information about the item as an answer to the verbal query; and

causing, via the at least one computing device, the information about the item to be presented to a user.

14. The method of claim 13, further comprising:

identifying, via the at least one computing device, a plurality of items that appear during the portion of the video content;

causing, via the at least one computing device, a plurality of indicia corresponding to individual ones of the plurality of items to be rendered as an overlay on top of the video content via a display; and

wherein the region specified by the verbal query is an identification of a particular indicium of the plurality of indicia.

15. The method of claim 13, further comprising:

causing, via the at least one computing device, a grid to be rendered as an overlay on top of the video content via a display, the grid containing a plurality of cells; and

wherein the region specified by the verbal query is an identification of a particular cell of the plurality of cells.

16. The method of claim 13, further comprising:

determining, via the at least one computing device, a user account corresponding to the verbal query;

identifying, via the at least one computing device, a current time in a playback of the video content associated with the user account; and

determining, via the at least one computing device, the portion of the video content based at least in part on the current time.

17. The method of claim 13, wherein causing the information about the item to be presented to the user further comprises causing, via the at least one computing device, a speech synthesizer to announce the information to the user via an audio device.

18. The method of claim 13, wherein causing the information about the item to be presented to the user further comprises causing, via the at least one computing device, a graphical overlay to be rendered on top of the video content via a display.

19. The method of claim 13, wherein the information comprises an action performed by the item in the portion of the video content.

20. The method of claim 13, wherein the item corresponds to a cast member in the video content, and the information comprises at least one of a cast member name of the cast member or a character name of the cast member.