Producing video bits for space time video summary

• US 20150003805A1
• Filed: 09/15/2014
• Published: 01/01/2015
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: Vertex Capital LLC
• Original Assignee: ViLynx, Inc.
• Inventors: Insua, Juan Carlos Riveiro, Fouren, Nils Hakan, Abril, Francisco Jose Blasco, Gome, Nicolas Francisco Amezquita
• Granted Patent: US 10,410,679 B2
• Time in Patent Office: Days
• US Class Current: 386/230
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/30 on the same track as the main recording
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/45455 applied to a region of the image
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/4756 for rating content, e.g. scoring a recommended movie
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A computer-assisted method for producing a space time summary for one or more original videos includes automatically recognizing a key element in an original video, extracting pixels related to the key element from a series of video frames of the original video, producing a video bit comprising a series of video frames comprising the pixels and audio information extracted from the original video, wherein at least one video frame of the video bit is formed by a subset of pixels of the corresponding video frame in the original video, automatically displaying a plurality of video bits in a user interface, wherein the plurality of video bits are extracted from one or more original videos, and provide a space time summary for the one or more original videos, and allowing two of the plurality of video bits to be played simultaneously with audio and motion in the user interface.

Claims

1. A computer-assisted method for producing a space time summary for one or more original videos, comprising:

automatically recognizing a key element in one or more original videos by a computer processor;

extracting pixels related to the key element from a series of video frames of the one or more original videos;

producing a plurality of video bits each comprising a series of video frames comprising the pixels and audio information extracted from the one or more original videos, wherein the video frames of the video bit follow the key element and dynamically change positions in the video frames of the one or more original videos, wherein at least one video frame of the video bit is formed by a subset of pixels of the corresponding video frame in the one or more original videos;

assigning weights to the plurality of video bits;

selecting one or more video bits from the plurality of video bits based on the weights to obtain a selected video bit to represent the one or more original videos; and

allowing the one or more selected video bits to be displayed and played at a user interface to provide a space time summary for the one or more original videos.

2. The computer-assisted method of claim 1, wherein the weights associated with the plurality of video bits are determined at least in part by content in the one or more original videos.

3. The computer-assisted method of claim 2, wherein the weights associated with the plurality of video bits are determined at least in part by objects, people, events or actions, types of key elements, or frequencies of appearances of the key element in the one or more original videos.

4. The computer-assisted method of claim 1, wherein the weights associated with the plurality of video bits are determined at least in part by meta data associated with the one or more original videos.

5. The computer-assisted method of claim 1, wherein the weights associated with the plurality of video bits are determined at least in part by number of times that the respective video bits have been played, remarks made by viewers of the video bits, people'"'"'s relationship in the key elements in the plurality of video.

6. The computer-assisted method of claim 1, further comprising:

dynamically changing the one or more video bits displayed at a user interface based on the weights associated with the plurality of video bits.

7. The computer-assisted method of claim 6, further comprising:

allowing the one or more video bits to be zoomed in at a user interface based on the weights associated with the plurality of video bits.

8. The computer-assisted method of claim 1, further comprising:

dynamically changing the one or more video bits displayed at a user interface based on the number of times that user(s) has played the respective video bits.

9. The computer-assisted method of claim 1, further comprising:

selecting two or more of the plurality of video bits based on the weights to obtain selected video bits to represent the one or more original videos;

allowing the selected video bits to be displayed at a user interface to provide a space time summary for the one or more original videos; and

allowing one of the selected video bits to be zoomed in the user interface based on the relative weights of the selected video bits.

10. The computer-assisted method of claim 9, wherein the two or more video bits are played simultaneously in a same window in the user interface.

11. The computer-assisted method of claim 1, wherein the weights associated with the plurality of video bits are automatically determined based on the types of key elements in the one or more original videos.

12. The computer-assisted method of claim 1, wherein the weights associated with the plurality of video bits are automatically determined based on the frequency of appearance of the key elements in the one or more original videos.

13. The computer-assisted method of claim 1, wherein the weights for the plurality of video bits are determined by relationship of one or more persons recognized in the key element with an owner of the one or more original videos.

14. The computer-assisted method of claim 1, wherein the weights for the plurality of video bits are determined at least in part by input from a user.

15. The computer-assisted method of claim 14, further comprising:

receiving a remark from a user about at least one of the plurality of video bits in the user interface; and

storing the remark in association with the one of the video bits, wherein the weights for the plurality of video bits are determined at least in part by the remark.

16. The computer-assisted method of claim 1, wherein at least two video frames in the one of the plurality of video bits have different shapes or different numbers of pixels.

17. The computer-assisted method of claim 1, wherein at least one video frame in the one of the plurality of video bits has an irregular frame shape that is not a rectangle.

18. A computer-assisted method for producing a space time summary for one or more original videos, comprising:

providing stored feature descriptors associated with object features;

providing stored feature vectors generated by the feature descriptors in the database;

selecting a first element and a second element from an original video;

extracting feature descriptors from the first element and the second element in the original video by a computer processor to produce extracted feature descriptors;

matching the extracted feature descriptors to the stored feature descriptors, wherein the stored feature descriptors are associated with a feature vector; and

if a positive matching is obtained based on predetermined criteria, automatically recognizing a key element in the first element and/or the second element based on the stored feature descriptors in the database.

19. The computer-assisted method of claim 18, wherein the object features comprise a person, an object, an occasion, an event, an action, or a location.

20. The computer-assisted method of claim 18, wherein the step of matching the extracted feature descriptors to the stored feature descriptors is based on a plurality of matching thresholds each associated with a different one of the stored feature descriptors in the database.

21. The computer-assisted method of claim 18, wherein the predetermined criteria are determined by a combination of degrees of matching between different pairs of extracted feature descriptor and stored feature descriptor.

22. The computer-assisted method of claim 18, wherein the key element is automatically recognized in part based on user input, owner of the original video, social relationships of the owner, and recording location and recording time of the original video.

23. The computer-assisted method of claim 18, wherein the first element relates to an object, wherein the second element relates to a background, wherein the step of automatically recognizing a key element comprises:

extracting the background in the original video; and

extracting the first element from the background in the original video by motion segmentation.

24. The computer-assisted method of claim 23, wherein the first element is extracted from the background by motion segmentation.

25. The computer-assisted method of claim 23, wherein the first element is extracted from the background by color segmentation and/or optical flow.

26. The computer-assisted method of claim 18, further comprising:

forming a complex element by the first element and the second element;

segmenting the complex element from the original video by the computer processor; and

extracting the complex element from the original video to form part of the video bit.

27. The computer-assisted method of claim 26, further comprising:

merging multiple video bits respectively comprising the first element and the second element into a complex video bit comprising the key element.