PRODUCING VIDEO BITS FOR SPACE TIME VIDEO SUMMARY

• US 20130081082A1
• Filed: 09/28/2011
• Published: 03/28/2013
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: Vertex Capital LLC
• Original Assignee: ViLynx, Inc.
• Inventors: Riveiro Insua, Juan Carlos, Fouren, Nils Hakan, Amezquita Gomez, Nicolas Francisco, Blasco Abril, Francisco Jose
• Granted Patent: US 8,869,198 B2
• Time in Patent Office: Days
• US Class Current: 725/38
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/30 on the same track as the main recording
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/45455 applied to a region of the image
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/4756 for rating content, e.g. scoring a recommended movie
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A computer-assisted method for producing a space time summary for one or more original videos includes automatically recognizing a key element in an original video, extracting pixels related to the key element from a series of video frames of the original video, producing a video bit comprising a series of video frames comprising the pixels and audio information extracted from the original video, wherein at least one video frame of the video bit is formed by a subset of pixels of the corresponding video frame in the original video, automatically displaying a plurality of video bits in a user interface, wherein the plurality of video bits are extracted from one or more original videos, and provide a space time summary for the one or more original videos, and allowing two of the plurality of video bits to be played simultaneously with audio and motion in the user interface.

Claims

1. A computer-assisted method for producing a space time summary for one or more original videos, comprising:

automatically recognizing a key element in an original video by a computer processor;

extracting pixels related to the key element from a series of video frames of the original video;

producing a video bit comprising a series of video frames comprising the pixels and audio information extracted from the original video, wherein at least one video frame of the video bit is formed by a subset of pixels of the corresponding video frame in the original video;

automatically displaying a plurality of video bits in a user interface, wherein the plurality of video bits are extracted from one or more original videos, and provide a space time summary for the one or more original videos; and

allowing at least two of the plurality of video bits to be played simultaneously with audio and in motion in the user interface.

2. The computer-assisted method of claim 1, wherein the video frames of the video bit follow the key element and dynamically change positions in the video frames of the original video.

3. The computer-assisted method of claim 1, wherein at least two video frames in the video bit have different shapes.

4. The computer-assisted method of claim 1, wherein at least two video frames in the video bit have different numbers of pixels.

5. The computer-assisted method of claim 1, wherein at least one video frame in the video bit has an irregular frame shape other than a rectangle.

6. The computer-assisted method of claim 1, wherein two video bits in the user interface overlap in time or overlap in locations in the video frames of the original video.

7. The computer-assisted method of claim 6, wherein the two video bits played simultaneously in the user interface have an overlapping location, wherein the pixels of the two video bits in the overlapping location are transparent to each other.

8. The computer-assisted method of claim 6, wherein the two video bits played simultaneously in the user interface have an overlapping location, wherein at least some of the pixels of the two video bits in the overlapping location are semi-transparent or opaque to each other.

9. The computer-assisted method of claim 1, wherein the key element comprises a person, an occasion, an event, an action, or a location.

10. The computer-assisted method of claim 1, wherein the step of automatically recognizing a key element in an original video comprises:

receiving a search; and

automatically detecting the key element in one or more original videos in response to the search.

11. The computer-assisted method of claim 1, further comprising:

receiving a search; and

automatically detecting a video bit in response to the search.

12. The computer-assisted method of claim 1, wherein the one or more original videos summarized by the plurality of video bits in the user interface were recorded by two or more users.

13. The computer-assisted method of claim 1, further comprising:

storing weights in association with the plurality of video bits; and

allowing one of the video bits to be zoomed in the user interface according to the weight of the one of the video bits.

14. The computer-assisted method of claim 13, wherein the weights for the plurality of video bits are determined by a user.

15. The computer-assisted method of claim 13, wherein the weights for the plurality of video bits are determined automatically based on the types of key elements or the frequency of appearance of the key elements in the original video.

16. The computer-assisted method of claim 13, wherein the weights for the plurality of video bits are determined by people'"'"'s relationship in the key elements in the plurality of video bits.

17. The computer-assisted method of claim 1, further comprising:

receiving a remark from a user about at least one of the video bits in the user interface; and

storing the remark in association with the one of the video bits.

18. The computer-assisted method of claim 17, wherein the remark determines the importance and the weight of the one of the video bits.

19. The computer-assisted method of claim 1, further comprising:

allowing one of the video bits to be played by itself in the user interface.

20. The computer-assisted method of claim 1, wherein the two video bits are played simultaneously in a same window in the user interface.

21. A computer-assisted method for producing a space time summary for an original videos, comprising:

automatically recognizing a key element in an original video by a computer processor;

extracting pixels related to the key element from a series of video frames of the original video;

producing a video bit comprising a series of video frames comprising the pixels and audio information extracted from the original video, wherein the video frames of the video bit follow the key element and dynamically change positions in the video frames of the original video, wherein at least one video frame of the video bit is formed by a subset of pixels of the corresponding video frame in the original video; and

allowing the video bit to be played at a user interface with audio and motion to provide a space time summary for the original video.

22. The computer-assisted method of claim 21, wherein at least two video frames in the video bit have different shapes or different numbers of pixels.

23. The computer-assisted method of claim 21, wherein at least one video frame in the video bit has an irregular frame shape other than a rectangle.

24. The computer-assisted method of claim 21, wherein the key element comprises a person, an occasion, an event, an action, or a location.

25. The computer-assisted method of claim 21, wherein the step of automatically recognizing a key element in an original video comprises:

receiving a search; and

automatically detecting the key element in one or more original videos in response to the search.

26. The computer-assisted method of claim 21, further comprising:

receiving a search; and

automatically detecting a video bit in response to the search.

27. The computer-assisted method of claim 21, wherein the one or more original videos summarized by the plurality of video bits in the user interface were recorded by two or more users.

28. The computer-assisted method of claim 21, further comprising:

receiving a remark from a user about at least one of the video bits in the user interface; and

storing the remark in association with the one of the video bits.

29. The computer-assisted method of claim 28, wherein the remark determines the importance and the weight of the one of the video bits.

30. The computer-assisted method of claim 21, wherein the two video bits are played simultaneously in a same window in the user interface.

31. A computer-assisted method for producing a space time summary for one or more original videos, comprising:

storing feature vectors each associated with a person, an object, an occasion, an event, an action, or a location in a database;

selecting the at least two elements from an original video.

extracting feature vectors of the at least two elements from the original video by a computer processor;

matching feature vectors extracted from the original video to feature vectors stored in the database, wherein the feature vectors in the database are associated with a same feature descriptor;

if a positive matching is obtained, automatically recognizing a key element based the feature descriptor associated with the feature vectors stored in the database;

32. The computer-assisted method of claim 31, wherein the key element is automatically recognized in the original video further based on user input, the owner of the video, social relationships of the owner, and location and time when the original video is recorded.

33. The computer-assisted method of claim 31, wherein the first element relates to an object, wherein the second element relates to a background, wherein the step of automatically recognizing a key element comprises:

extracting the background in the original video; and

extracting the first element from the background in the original video by motion segmentation.

34. The computer-assisted method of claim 33, wherein the motion segmentation is based on color segmentation and optical flow.

35. The computer-assisted method of claim 31, wherein the first element and the second element both relate to objects.

36. The computer-assisted method of claim 34, further comprising:

forming a complex element by the first element and the second element;

segmenting the complex element from the original video by the computer processor;

extracting the complex element from the original video to form part of the video bit.

37. The computer-assisted method of claim 37, further comprising:

merging multiple video bits respectively comprising the first element and the second element into a complex video bit comprising the complex element.