Methods and interfaces for event timeline and logs of video streams

• US 20060288288A1
• Filed: 01/03/2006
• Published: 12/21/2006
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Girgensohn, Andreas, Wilcox, Lynn D., Shipman, Frank M. III
• Granted Patent: US 7,996,771 B2
• Time in Patent Office: Days
• US Class Current: 715/716
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system

Abstract

Techniques for generating timelines and event logs from one or more fixed-position cameras based on the identification of activity in the video are presented. Various embodiments of the invention include an assessment of the importance of the activity, the creation of a timeline identifying events of interest, and interaction techniques for seeing more details of an event or alternate views of the video. In one embodiment, motion detection is used to determine activity in one or more synchronized video streams. In another embodiment, events are determined based on periods of activity and assigned importance assessments based on the activity, important locations in the video streams, and events from other sensors. In different embodiments, the interface consists of a timeline, event log, and map.

Claims

1. A method of identifying and visualizing events in one or more video frames comprising the steps of:

(a) determining a measure of interest;

(b) generating an importance score for each video frame based on the measure of interest;

(c) computing one or more threshold values based on the importance scores; and

(d) selecting video frames identifying events based on the threshold values.

2. The method of claim 1, where in step (a) the measure of interest is based on criteria selected from the group consisting of the amount of activity in the video frames, points of interest in the video frames, distance to points of interest from the camera, detected features in the video frames, facial features in the video frames, if the activity is of interest, if the feature is of interest, activities detected by other sensors and events detected by other sensors.

3. The method of claim 1, where in step (a) determining the measure of interest further comprises the steps of:

(e) determining one or more points of interest in the video frames;

(f) determining one or more distances from one or more video camera positions to the one or more points of interest in the video frames; and

(g) determining the measure of interest based on the distances from the camera positions to the points of interest.

4. The method of claim 1, where in step (b) generating the importance score further comprises the step of smoothing.

5. The method of claim 1, where in step (c) one or more threshold values are computed for measures selected from the group consisting of the minimum measure of interest, the minimum fraction of changed pixels to be considered activity, the minimum pause in activity to start a new video segment and the minimum length of the activity segment to ignore noise.

6. The method of claim 5, where in step (d) selecting video frames further comprises the steps of:

(h) including video frames in the event if the smoothed importance score is above a minimum measure of interest threshold; and

(i) merging selected consecutive video frames into a single event if the gap between the selected consecutive video frames is below the minimum pause in activity to start a new video segment threshold value.

7. The method of claim 1, wherein the video frames are part of a video stream and the method further comprises generating a timeline of at least one of the events in the video stream.

8. The method of claim 7, wherein the duration of events in the timeline are identified using a horizontal line, wherein the line begins at the time the event was first present in the video stream and ends at the time the event was last present in the video stream, wherein a keyframe is used to visualize the content of each event, wherein the keyframe is associated with the duration of events.

9. The method of claim 8, wherein two or more video streams simultaneously recorded with two or more cameras are represented on a single timeline, wherein the duration of the event present in each video stream is represented with a horizontal line, wherein a code is used to indicate the camera used to record the stream, wherein the same code is used for different events present in the video from the same camera, wherein the same code is used to frame the keyframe for each event from the same camera.

10. The method of claim 9, wherein the code uses different colors to indicate an event shot with a different camera, wherein the same color is used for different events present in the video from the same camera, wherein the same color is used to frame the keyframe for each event from the same camera.

11. The method of claim 9, wherein a map is used to show the geographic position of two or more cameras used to film the two or more video streams, wherein a code is used to indicate a camera, where a keyframe is used to show the video stream observed from the camera, wherein the keyframe frame is indicated using that code, wherein a different code is used to indicate a different camera, wherein different keyframes used to show the video stream observed from the different cameras are framed with the different codes associated with the different cameras, wherein the keyframes vary as a cursor moves along the timeline.

12. The method of claim 11, wherein the code uses different colors to show the geographic position of two or more cameras used to film the two or more video streams, wherein a color is used to indicate a camera, wherein the keyframe is framed in that color, wherein a different color is used to indicate a different camera, wherein the different keyframes are framed with the different colors associated with the different cameras, wherein the keyframes vary as a cursor moves along the timeline.

13. The method of claim 8, wherein keyframes of the identified event are presented, wherein the keyframes are numbered according to the timeline.

14. The method of claim 13, wherein the keyframes are selected from the group consisting of single action keyframes representative of the period of activity and time-lapse visualization of the period of activity.

15. The method of claim 8, wherein keyframes are used to visualize a composition in a video stream, wherein the keyframes are numbered according to the single timeline.

16. The method of claim 11, wherein an event-log is used with keyframes on a map to visualize the event.

17. The method of claim 1, wherein the event is represented using a medium selected from the group consisting of a map, an event-log and a timeline.

18. The program of instructions executable by a computer to generate a timeline of events displaying events of interest in a video stream as per claim 7, comprising the steps of:

determining a measure of interest;

generating an importance score for one or more video frames in the video stream based on the measure of interest;

computing one or more threshold values;

selecting video frames identifying events based on the threshold values; and

generating a timeline displaying keyframes of the one or more selected events in the video stream.

19. A system or apparatus for generating a timeline of events in a video stream, wherein generating a timeline comprises:

a) one or more processors capable of specifying one or more sets of parameters;

capable of transferring the one or more sets of parameters to a source code;

capable of compiling the source code into a series of tasks for visualizing an event in a video stream; and

b) a machine readable medium including operations stored thereon that when processed by one or more processors cause a system to perform the steps of specifying one or more sets of parameters;

transferring one or more sets of parameters to a source code;

compiling the source code into a series of tasks for generating a timeline of events in a video stream.

20. A machine-readable medium having instructions stored thereon to cause a system to:

determine a measure of interest;

generate an importance score for each video frame based on the measure of interest;

compute one or more threshold values;

select video frames identifying events based on the threshold values; and

generate a timeline displaying keyframes of the one or more selected events in the video stream.