SYSTEMS AND METHODS FOR CONTENT ANALYSIS TO SUPPORT NAVIGATION AND ANNOTATION IN EXPOSITORY VIDEOS

• US 20140245152A1
• Filed: 07/10/2013
• Published: 08/28/2014
• Est. Priority Date: 02/22/2013
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Carter, Scott, Cooper, Matthew L., Denoue, Laurent
• Granted Patent: US 10,482,777 B2
• Time in Patent Office: Days
• US Class Current: 715/720
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312

Abstract

Online educational videos are often difficult to navigate. Furthermore, most video interfaces do not lend themselves to note-taking. Described system detects and reuses boundaries that tend to occur in these types of videos. In particular, many educational videos are organized around distinct breaks that correspond to slide changes, scroll events, or a combination of both. Described algorithms can detect these structural changes in the video content. From these events the system can generate navigable overviews to help users searching for specific content. Furthermore, these boundary events can help the system automatically associate rich media annotations to manually-defined bookmarks. Finally, when manual or automatically recovered spoken transcripts are available, the spoken text can be combined with the temporal segmentation implied by detected events for video indexing and retrieval. This text can also be used to seed a set of text annotations for user selection or be combined with user text input.

Claims

1. A computer-implemented method performed in a computerized system comprising a central processing unit, a display device and a memory, the computer-implemented method performed in connection with a video of a content being added to a medium, the computer-implemented method comprising:

a. using the central processing unit to detect at least one boundary event in the video;

b. using the central processing unit to segment the video into at least one video segment based on the detected at least one boundary event;

c. using the central processing unit to generate an overview image using at least a portion of the video;

d. using the central processing unit to map a portion of the overview image to the at least one video segment;

e. generating a graphical user interface on the display device, the graphical user interface displaying at least the mapped portion of an overview image; and

f. in response to detection of a selection event of the mapped portion of the overview image by a user, playing the video segment associated with the mapped portion of the overview image.

2. The computer-implemented method of claim 1, wherein the at least one detected boundary event is a medium scroll event.

3. The computer-implemented method of claim 2, wherein the overview image is stitched from a plurality of frames of the video based on a scroll amount associated with the detected medium scroll event.

4. The computer-implemented method of claim 1, wherein the boundary event is a slide change event.

5. The computer-implemented method of claim 1, wherein the mapped portion of the overview image is a content cluster obtained by clustering of the content and wherein the content clustering is performed based on locations of bounding boxes enclosing spatial regions on the medium containing portions of the added content.

6. The computer-implemented method of claim 1, further comprising obtaining a textual transcript of a narration associated with the video and segmenting the textual transcript based on the at least one detected boundary event.

7. The computer-implemented method of claim 1, wherein in c. a plurality of overview images are generated and wherein the graphical user interface comprises an overview image selection portion for selecting the overview image incorporating the displayed mapped portion from the generated plurality of overview images.

8. The computer-implemented method of claim 1, wherein the at least one boundary event is a plurality of boundary events and wherein the graphical user interface comprises a boundary event selection portion for selecting boundary events in the plurality of boundary events.

9. The computer-implemented method of claim 1, wherein the graphical user interface comprises a timeline portion, the timeline portion comprises at least one boundary event marker indicating a timing of the at least one boundary event in the video.

10. The computer-implemented method of claim 9, further comprising, in response to detection of a second selection event of the at least one boundary event marker by the user, playing the video segment corresponding to the boundary event.

11. The computer-implemented method of claim 1, wherein the graphical user interface comprises a timeline portion, the computer-implemented method further comprising, in response to detection of a user interaction event with the mapped portion of the overview image, displaying a marker on the timeline portion, the marker indicating a timing of the video segment associated with the mapped portion of the overview image involved in the user interaction event.

12. The computer-implemented method of claim 1, wherein the graphical user interface comprises a timeline portion, the timeline portion comprises at least one boundary event indicator indicating a timing of the at least one boundary event in the video.

13. The computer-implemented method of claim 1, further comprising, in response to detection of a user bookmark event, using the at least one detected boundary event to automatically select an information to be associated with a user bookmark corresponding to the user bookmark event.

14. The computer-implemented method of claim 13, wherein the information automatically selected to be associated with the user bookmark comprises a second overview image corresponding to a second detected boundary event closest in time to the user bookmark event.

15. The computer-implemented method of claim 13, wherein the information automatically selected to be associated with the user bookmark comprises a segment of a textual transcript of a narration associated with the video, the segment being closest in time to the user bookmark event.

16. A non-transitory computer-readable medium embodying a set of computer-executable instructions, which, when executed in a computerized system comprising a central processing unit, a display device and a memory in connection with a video of a content being added to a medium, cause the computerized system to perform a method comprising:

a. using the central processing unit to detect at least one boundary event in the video;

b. using the central processing unit to segment the video into at least one video segment based on the detected at least one boundary event;

c. using the central processing unit to generate an overview image using at least a portion of the video;

d. using the central processing unit to map a portion of the overview image to the at least one video segment;

e. generating a graphical user interface on the display device, the graphical user interface displaying at least the mapped portion of an overview image; and

f. in response to detection of a selection event of the mapped portion of the overview image by a user, playing the video segment associated with the mapped portion of the overview image.

17. A computerized system comprising a central processing unit, a display device and a memory storing a set of instructions executable in connection with a video of a content being added to a medium, the set of instructions comprising instructions for:

a. detecting at least one boundary event in the video;

b. segmenting the video into at least one video segment based on the detected at least one boundary event;

c. generating an overview image using at least a portion of the video;

d. mapping a portion of the overview image to the at least one video segment;

e. generating a graphical user interface on the display device, the graphical user interface displaying at least the mapped portion of an overview image; and

f. in response to detection of a selection event of the mapped portion of the overview image by a user, playing the video segment associated with the mapped portion of the overview image.

18. The computerized system of claim 17, wherein the at least one detected boundary event is a medium scroll event.

19. The computerized system of claim 18, wherein the overview image is stitched from a plurality of frames of the video based on a scroll amount associated with the detected medium scroll event.

20. The computerized system of claim 17, wherein the mapped portion of the overview image is a content cluster obtained by clustering of the content and wherein the content clustering is performed based on locations of bounding boxes enclosing spatial regions on the medium containing portions of the added content.