Motion based event detection system and method

• US 5,969,755 A
• Filed: 02/05/1997
• Issued: 10/19/1999
• Est. Priority Date: 02/05/1996
• Status: Expired due to Term

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Courtney, Jonathan D.
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): VOISINET, JOHN A
• Application Number:
  US08/795,432
• Time in Patent Office: 986 Days
• Field of Search: 348/135, 348/142, 348/152, 348/154, 348/155, 348/143, 348/171, 348/172, 382/103, 382/107, 382/236
• US Class Current: 348/143
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A method to provide automatic content-based video indexing from object motion is described. Moving objects in video from a surveillance camera 11 detected in the video sequence using motion segmentation methods by motion segmentor 21. Objects are tracked through segmented data in an object tracker 22. A symbolic representation of the video is generated in the form of an annotated graphics describing the objects and their movement. A motion analyzer 23 analyzes results of object tracking and annotates the graph motion with indices describing several events. The graph is then indexed using a rule based classification scheme to identify events of interest such as appearance/disappearance, deposit/removal, entrance/exit, and motion/rest of objects. Clips of the video identified by spatio-temporal, event, and object-based queries are recalled to view the desired video.

Claims

1. A method of providing video indexing comprising the steps of:

(a) detecting objects in video to provide detected objects comprising the step of performing motion segmentation including image differencing, thresholding, morphology and connected component analysis;

(b) analyzing motion of said detected objects comprising the step of tracking said detected objects using linear prediction of object position;

(c) generating a symbolic motion description of object motion; and

(d) placing index marks in said symbolic motion description to identify occurrence of events in the video.

2. The method of claim 1 wherein said step of detecting objects includes performing motion segmentation comprising the steps of image differencing, thresholding, connected component analysis, image smoothing and morphology.

3. The method of claim 1 wherein said analyzing step includes associating objects in successive frames of said video using a mutual-nearest-neighbor criterion.

4. The method of claim 1 wherein said analyzing step includes determining paths of said objects and intersection with paths of other objects.

5. The method of claim 1 wherein said generating step includes generating a directed graph reflective of the paths and path intersections of said objects.

6. The method of claim 1 wherein said generating step includes generating a record of image statistics for objects in every video frame.

7. The method of claim 6 wherein said generating a record of image statistics step includes generating size, shape, position, and time-stamp of objects in every video frame.

8. The method of claim 1 wherein said generating step includes generating hierarchical graph node groupings reflective of paths and intersections of said objects.

9. The method of claim 8 wherein said placing index marks step includes placement of index marks corresponding to motion events in accordance with said hierarchical graph node groupings.

10. The method of claim 9 wherein said placing index marks step includes use of a rule-based classifier.

11. The method of claim 9 wherein said placing index marks step corresponding to motion events includes placement of one or more marks corresponding to appearance, disappearance, deposit, removal, entrance, exit, motion, or rest of objects.

12. A method of providing video indexing comprising the steps of:

(a) detecting objects in a video to provide detected objects;

(b) analyzing motion of said detected objects;

(c) generating a symbolic motion description of object motion; and

(d) placing index marks in said symbolic motion description to identify occurrence of events in the video, said generating step includes generating primary and secondary graph links reflective of the likelihood of accurate motion analysis.

13. A method of providing video indexing comprising the steps of:

(a) detecting objects in a video to provide detected objects;

(b) analyzing motion of said detected objects;

(c) generating a symbolic motion description of object motion; and

(d) placing index marks in said symbolic motion description to identify occurrence of events in the video, said generating step includes generating hierarchial graph node groupings reflective of paths and intersections of said objects, said hierarchial graph node groupings reflect the likelihood of accurate motion analysis.

14. A method for real-time detection of video events comprising the steps of:

(a) detecting object in real-time video to provide detected objects;

said step of detecting including performing motion segmentation comprising the steps of image differencing, thresholding, connected components analysis and morphology;

(b) analyzing motion of said detected objects including the step of tracking said detected objects using linear prediction of object positions;

(c) generating a symbolic motion description of object motion;

(d) placing index marks in said symbolic motion description to identify occurrence of events in video; and

(e) providing a signal in response to the occurrence of said video events.

15. The method of claim 14 wherein said step of providing a signal for the purpose of sounding an alarm.

16. The method of claim 14 wherein said step of providing a signal to make a record.

17. The method of claim 15 wherein said step of providing a signal comprises initiation of an automated action by signaling one of a computer program or electronic device.

18. A method to assist human analysis of video data comprising the steps of:

(a) detecting objects in a video to provide detected objects;

said detecting step including performing motion segmentation comprising the steps of image differencing, thresholding, connected component analysis and morphology;

(b) analyzing motion of said objects;

said analyzing step including the step of tracking said detected objects using linear prediction of said object positions;

(c) generating a symbolic motion description of object motion;

(d) placing index marks in said symbolic motion description to identify occurrence of events in video;

(e) receiving content-base queries;

(f) matching queries with symbolic video information and said index marks; and

(g) providing video sequences corresponding to the query.

19. The method of claim 18 wherein said step of receiving content-base queries includes receiving queries with constraints involving one or more of a video clip, time interval, object, spatial region, or motion event.

20. The method of claim 19 wherein constraints for said queries may be specified by manipulation of video clips.

21. The method of claim 19 wherein said step of matching queries with symbolic video information includes filtering symbolic information to meet the query constraints of one or more video clip, time interval, object, spatial region, or motion event.

22. A method to assist human analysis of video data comprising the steps of:

(a) detecting objects in a video to provide detected objects;

(b) analyzing motion of said objects;

(c) generating a symbolic motion description of object motion;

(d) placing index marks in said symbolic motion description to identify occurrence of events in video;

(e) receiving content-based queries;

(f) matching queries with symbolic video information and said index marks; and

(g) providing video sequences corresponding to the query, said step of providing video sequences includes a system clipboard with sets of video clips for progressive refinement of content based queries and query results.