Method and system for video indexing and video synopsis

• US 8,311,277 B2
• Filed: 12/09/2007
• Issued: 11/13/2012
• Est. Priority Date: 02/01/2007
• Status: Active Grant

• Current Assignee: BriefCam, Ltd. (Canon Inc.)
• Original Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Inventors: Peleg, Shmuel, Pritch, Yael, Rav-Acha, Alexander, Gutman, Avital
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Ansari, Tahmina
• Application Number:
  US12/522,843
• Publication Number:
  US 20100092037A1
• Time in Patent Office: 1,801 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

In a system and method for generating a synopsis video from a source video, at least three different source objects are selected according to one or more defined constraints, each source object being a connected subset of image points from at least three different frames of the source video. One or more synopsis objects are sampled from each selected source object by temporal sampling using image points derived from specified time periods. For each synopsis object a respective time for starting its display in the synopsis video is determined, and for each synopsis object and each frame a respective color transformation for displaying the synopsis object may be determined. The synopsis video is displayed by displaying selected synopsis objects at their respective time and color transformation, such that in the synopsis video at least three points that each derive from different respective times in the source video are displayed simultaneously.

Claims

1. A method for generating a synopsis video from a substantially endless source video stream as generated by a video surveillance camera, the method comprising:

receiving in real time object-based descriptions of at least three different source objects in said source video stream, each source object being a connected subset of image points from at least three different frames of the source video stream;

continuously maintaining a queue of said received object-based descriptions including for each respective source object its duration and location;

selecting a subset of at least three source objects from said queue based on given criteria, and sampling from each selected source object one or more synopsis objects by temporal sampling;

determining for each synopsis object a respective display time for starting its display in the synopsis video; and

generating the synopsis video by displaying selected synopsis objects or objects derived therefrom each at its respective predetermined display time;

such that at least three points, each derived from different respective times in the source video stream, are displayed simultaneously in the synopsis video and at least two points, both derived from the same time, are displayed at different times in the synopsis video.

2. The method according to claim 1, further including:

determining for each synopsis object and each frame in the synopsis video a respective color transformation for displaying the synopsis object; and

displaying said selected synopsis objects or the objects derived therefrom at their respective color transformation.

3. The method according to claim 1, wherein one of the objects is a background object.

4. The method according to claim 3, including stitching the objects and the background into a seamless video.

5. The method according to claim 1, wherein source objects are selected from the queue and a respective time for starting the display of each synopsis object is determined so as to optimize a cost function.

6. The method according to claim 1, wherein the background object is generated synthetically.

7. The method according to claim 1, wherein each object in the synopsis video points to a time segment in the source video stream where the respective object is visible.

8. The method according to claim 7, wherein selecting an object causes the time segment in the source video stream pointed to by the selected object to be played.

9. The method according to claim 1, wherein at least one object in the synopsis video is formed by replacing a corresponding object in the source video stream by a predetermined symbol.

10. The method according to claim 1, wherein objects are first clustered into similar classes, and the synopsis video includes objects from at least a pre-determined number of classes.

11. The method according to claim 1, wherein objects are first clustered into similar classes, and objects from at least one selected class are not displayed.

12. The method according to claim 7, wherein objects are first clustered into similar classes and selecting an object points to a video synopsis including objects only from the same class as the selected object.

13. The method according to claim 1, wherein selecting one or more source objects includes:

computing a cost function for stitching the synopsis objects onto the synopsis video; and

selecting synopsis objects for which the cost function is considered as close to optimal as can be achieved.

14. The method according to claim 1, wherein selecting at least three non-overlapping source objects from the queue includes filtering the source objects based on user-defined constraints and limiting filtered source objects to source objects that appear within a specified time window.

15. The method according to claim 1, wherein selecting at least three non-overlapping source objects includes determining an interest score.

16. The method according to claim 15, wherein the interest score is a measure of activity.

17. The method according to claim 1, wherein the synopsis video contains all objects of interest in the source video stream.

18. The method according to claim 1, wherein a number of objects of interest in the source video stream that appear also in the synopsis video is a tradeoff between maximizing said number while maintaining visual appeal of the synopsis video.

19. The method according to claim 1, wherein the source video stream is captured by a single camera.

20. The method according to claim 19, including maintaining said single camera at a fixed location.

21. The method according to claim 20, wherein the camera is rotated relative to an axis at said fixed location.

22. The method according to claim 1, including spatially warping at least one of said synopsis objects prior to display thereof.

23. The method according to claim 1, including pre-aligning the source video stream so as to produce a stabilized source video stream by:

(a) computing image motion parameters between frames in the source video stream;

(b) warping the video frames in the source video stream so that stationary objects will appear stationary in the stabilized source video stream.

24. The method according to claim 1, being used for video surveillance.

25. The method according to claim 1, being used for at least one in the group of:

video indexing, video browsing and video retrieval.

26. The method according to claim 25, including maintaining for pixels in the synopsis video a pointer to corresponding pixels in the source video stream.

27. A system for generating a synopsis video from a substantially endless a source video stream as generated by a surveillance camera, the system comprising:

a source object selector adapted to be coupled to an object memory that stores a continuously maintained queue of object-based descriptions of at least three different source objects in said source video stream, said object-based descriptions including for each respective source object its duration and location, the source object selector being adapted to select at least three different source objects according to one or more defined constraints, each source object being a connected subset of image points from at least three different frames of the source video stream;

a synopsis object sampler coupled to the source object selector for sampling from each selected source object one or more synopsis objects by temporal sampling using image points derived from specified time periods;

a time selection unit coupled to the synopsis object sampler for determining for each synopsis object a respective display time for starting its display in the synopsis video;

a stitching unit coupled to the time selection unit for stitching each of the selected synopsis objects or objects derived therefrom at a respective display time so as to generate successive synopsis video frames, such that in the synopsis video frames at least three points that each derive from different respective times in the source video stream are displayed simultaneously; and

a synopsis frame memory coupled to the stitching unit for storing said synopsis video frames.

28. The system according to claim 27, further including a display unit coupled to the stitching unit for displaying the synopsis video.

29. The system according to claim 27, further including a color transformation unit coupled to the time selection unit for determining for each synopsis object and each frame a respective color transformation for displaying the synopsis object;

the stitching unit being coupled to the color transformation unit for stitching the selected synopsis objects or objects derived therefrom at their respective color transformation.

30. The system according to claim 27, further including a user interface coupled to the object memory for allowing user-defined constraints to be defined.

31. The system according to claim 27, further including a clustering unit for clustering objects according to defined criteria.

32. The system according to claim 27, further including a pre-processor for processing captured video on line to detect said objects in the source video stream, said pre-processor being adapted for coupling to the object memory for storing said objects therein.

33. The system according to claim 32, wherein the pre-processor includes an alignment unit for pre-aligning video frames in the source video stream.

34. The system according to claim 33, wherein the alignment unit is adapted to:

compute image motion parameters between frames in the first sequence; and

warp the video frames in the first sequence so that stationary objects in the first dynamic scene will be stationary in the video.

35. The system according to claim 27, wherein the frame generator includes a warping unit for spatially warping at least one object prior to stitching to the synopsis video.

36. The system according to claim 27, being adapted for at least one in the group of:

video indexing, video browsing and video retrieval.

37. A computer program product for generating a synopsis video from a substantially endless a source video stream as generated by a surveillance camera, the computer program product comprising:

a non-transitory readable computer medium having computer readable program embodied therewith, the computer readable program comprising;

computer readable program configured to store continuously maintained queue of object-based descriptions of at least three different source objects in said source video stream, said object-based descriptions including for each respective source object its duration and location, the source object selector being adapted to select at least three different source objects according to one or more defined constraints, each source object being a connected subset of image points from at least three different frames of the source video stream;

computer readable program configured to sample from each selected source object one or more synopsis objects by temporal sampling using image points derived from specified time periods;

computer readable program configured to determine for each synopsis object a respective display time for starting its display in the synopsis video;

computer readable program configured to stitch each of the selected synopsis objects or objects derived therefrom at a respective display time so as to generate successive synopsis video frames, such that in the synopsis video frames at least three points that each derive from different respective times in the source video stream are displayed simultaneously; and

computer readable program configured to store said synopsis video frames.