Scene-by-scene plot context for cognitively impaired

• US 10,264,330 B1
• Filed: 01/03/2018
• Issued: 04/16/2019
• Est. Priority Date: 01/03/2018
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Candelore, Brant, Nejat, Mahyar, Shintani, Peter, Blanchard, Robert Noel
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US15/861,445
• Time in Patent Office: 468 Days
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 21/26603 for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/488 Data services, e.g. news ticker systems specially adapted for using meteorological information in broadcast systems H04H60/71
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

To improve cognitive comprehension of plots in videos and thus assist not only people with cognitive issues but also inattentive people that have trouble following along with the plot, information is sent about each scene in a video. Image recognition or speech recognition may be implemented on a video to recognize where in the video the viewer currently is at, and ancillary information pertaining to that spot in the video presented immediately or at a convenient scene break.

Claims

1. A computer program product comprising:

at least one computer readable storage medium that is not a transitory signal and that comprises instructions executable by a processor to configure the processor for;

for a first scene in plural scenes of a piece of content, capturing at least a portion of at leas one frame in the content;

providing at least a portion of the at least one frame to an image recognition device that recognizes video elements including non-textual elements in the frame;

receiving from the image recognition device first plot information relating to the first scene, the first plot information not relating to scenes occurring in the content after the first scene occurs; and

presenting the first plot information, wherein the first plot information is obtained at least in part by;

receiving a first command from a remote control (RC);

responsive to the first command, presenting at least one code on a display;

capturing the code;

providing the code to the image recognition device that recognizes video elements including non-textual elements in the frame; and

receiving from the image recognition device the first plot information.

2. The computer program product of claim 1, comprising the processor.

3. The computer program product of claim 1, comprising at least one display controllable by the processor to present the content.

4. The computer program product of claim 1, wherein the instructions are executable for:

capturing the at least one frame in response to one or more commands received form a remote control (RC).

5. The computer program product of claim 1, wherein the instructions are executable for:

playing the content using a first computer application; and

capturing the at least one frame using a second computer application different than the first computer application.

6. The computer program product of claim 1, wherein the instructions are executable for:

playing the content using a first computer application; and

capturing the at least one frame using the first computer application.

7. The computer program product of claim 1, wherein the first plot information comprises at least one non-textual symbol.

8. The computer program product of claim 1, wherein the instructions are executable for:

presenting the first plot information at a predetermined location in the content occurring after the at least one frame is captured, the content being paused at the predetermined location, the first plot information that is presented while the content is paused at the predetermined location comprising at least one video clip that is not part of the content.

9. The computer program product of claim 1, wherein the instructions are executable for:

presenting on a display at least one user interface (UI), the UI comprising at least;

a first selector selectable to cause presentation of plot information relating to the content from the start of the content to a current location of the content, but not beyond the current location; and

a second selector selectable to cause presentation of plot information relating to the content from the start of the first scene, but not before, to a current location of the content, but not beyond the current location.

10. The computer program product of claim 9, wherein the UI comprises:

a third selector selectable to cause presentation of plot information relating to the content in its entirety from beginning to end of the content.

11. An assembly comprising:

at least one non-transitory computer readable storage medium with instructions executable by a processor to configure the processor for;

displaying at least one image from content played on at least one audio video device (AVD);

the content comprising a start and a first scene;

presenting on a display at least one user interface (UI), the UI comprising at least;

a first selector selectable to cause presentation of plot information relating to the content from the start of the content to a current location of the content, but not beyond the current location; and

a second selector selectable to cause presentation of plot information relating to the content from the start of the first scene, but not before, to a current location of the content, but not beyond the current location;

receiving at least one selection of at least one of the selectors; and

responsive to the selection, presenting the plot information.

12. The assembly of claim 11, comprising the processor.

13. The assembly of claim 11, wherein non-textual portion used to identify the plot information comprises an image of a video object.

14. The assembly of claim 11, wherein non-textual portion used to identify the plot information comprises a computer code.

15. The assembly of claim 11, wherein the assembly comprises the AVD.

16. The assembly of claim 11, wherein the assembly comprises a mobile communication device (MCD) that is not the AVD.

17. The assembly of claim 11, wherein the plot information comprises a video clip.

18. A method, comprising:

at an audio video device (AVD), receiving a command to present plot information to be presented at a predetermined location in video being presented on the AVD;

responsive to the command, continuing to play the video until the predetermined location is reached;

responsive to reaching the predetermined location, automatically pausing the video; and

with the video paused, presenting the plot information on the AVD.

19. The method of claim 18, wherein the plot information comprises a video clip different from the content.