System and method for object identification and behavior characterization using video analysis

• US 6,678,413 B1
• Filed: 11/24/2000
• Issued: 01/13/2004
• Est. Priority Date: 11/24/2000
• Status: Expired due to Term

• Current Assignee: Clever Sys Incorporated
• Original Assignee: Clever Sys Incorporated
• Inventors: Wolf, Wayne, Liang, Yiqing, Kobla, Vikrant, Crnic, Linda
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/718,374
• Time in Patent Office: 1,145 Days
• Field of Search: 382/167, 382/110, 382/162, 382/163, 382/160, 382/106, 382/190, 382/193, 382/215, 382/107, 382/294, 382/182, 382/284, 382/181, 348/169, 324/671, 324/663, 324/662, 324/687, 324/716, 384/143, 384/152, 384/154, 384/155
• US Class Current: 382/181
• CPC Class Codes: A01K 1/031 Cages for laboratory animals; Cages for measuring metabolism of animals
• CPC Class Codes: A01K 29/005 Monitoring or measuring activity, e.g. detecting heat or mating
• CPC Class Codes: A61B 2503/40 Animals
• CPC Class Codes: A61B 2503/42 for laboratory research
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

In general, the present invention is directed to systems and methods for finding the position and shape of an object using video. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide object segmentation and identification, object motion tracking (for moving objects), object position classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for object identification and tracking, probabilistic approach with expectation-maximization for tracking the motion detection and object classification, and decision tree classification for behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various objects and the object'"'"'s movements within the image. The image may be provided in real time or from storage. The invention is particularly useful for monitoring and classifying animal behavior for testing drugs and genetic mutations, but may be used in any of a number of other surveillance applications.

Claims

1. A system comprising:

a computer configured to determine a position and shape of an object of interest from video images and to characterize activity of said object of interest based on analysis of changes in said position and said shape over time;

wherein said computer includes an object identification and segregation module receiving said video images; and

wherein said object identification and segregation module operates using a background subtraction algorithm in which a plurality of said video images are grouped into a set, a standard deviation map of the set of video images is created, a bounding box where a variation is greater than a predetermined threshold is remove from said set of video images, and the set of images less said bounding boxes is averaged to produce a back ground image.

2. The system of claim 1, further comprising:

a video camera coupled to said computer for providing said video images.

3. The system of claim 2, further comprising:

a video digitization unit couple to said video camera and said computer for converting said video images provided by said video camera from analog to digital format.

4. The system of claim 3, further comprising:

a storage/retrieval unit coupled to said video digitization unit, said video camera, and said computer, for storing video images and standard object video images.

5. The system of claim 1, wherein said computer further includes a behavior identification module for characterizing activity of said object, said behavior identification module being coupled to said object identification and segregation module.

6. The system of claim 5, wherein said computer further includes an object tracking module for tracking said object from one frame of said video images to another frame, and an object shape and location change classifier for classifying the activity of said object, coupled to each other, said object identification and segregation module, and said behavior identification module.

7. The system of claim 6, wherein said computer further includes a standard object behavior storage module that stores information about known behavior of a predetermined standard object for comparing the activity of said object, said standard object behavior storage module being coupled to said behavior identification module, and a standard object classifier module coupled to said standard object behavior module.

8. The system of claim 1, wherein said computer further includes a standard object behavior storage module that stores information about known behavior of a predetermined standard object for comparing the activity of said object, said standard object behavior storage module being coupled to said behavior identification module.

9. The system of claim 1, wherein said object is a living object.

10. The system of claim 1, wherein said object is an animal.

11. The system of claim 1, wherein said object is a mouse.

12. The system of claim 1, wherein said object is a human.

13. The system of claim 1, wherein said object is a man made machine.

14. A method of characterizing activity of an object using a computer comprising:

detecting a foreground object of interest in video images;

tracking said foreground object over a plurality of said video images;

classifying said foreground object in said plurality of video images; and

characterizing said activity of said foreground object based on comparison of said classifications to activity of a standard object;

wherein said characterizing said activity includes;

describing a sequence of postures as behavior primitives; and

aggregating behavior primitives into actual behavior over a range of images;

wherein said describing said behavior primitives further includes;

identifying patterns of postures over a sequence of images; and

analyzing temporal information selected from the group consisting of direction and magnitude of movement of the centroid, increase and decrease of the eccentricity, increase and decrease of the area, increase and decrease of the aspect ratio of the bounding box, change in the b-spline representation points, change in the convex hull points, and direction and magnitude of corner points.

15. The method of claim 14, wherein said foreground object detection includes the step of generating a background image from an average of a set of individual frames of said video images.

16. The method of claim 15, wherein said step of generating a background image includes the step of determining variation in intensity of pixels within said individual frames to identify a region where said foreground object is located.

17. The method of claim 16, wherein said step of generating a background image further includes the step of using non-variant pixels of the video images to generate said background image.

18. The method of claim 17, wherein said step of generating a background image is performed periodically to correct for changes in background objects and small movements of a camera capturing said video images.

19. The method of claim 14, wherein said detecting a foreground object includes using a background subtraction method comprising the steps of:

multiply frames in a neighborhood of current image;

apply a lenient threshold on a difference between a current image and a background so as to determine a broad region of interest;

classify by intensity various pixels in said region of interest to obtain said foreground object; and

apply edge information to refine contours of said foreground object image.

20. The method of claim 14, wherein said step of detecting said foreground includes the step of manual identification of foreground objects to be tracked and characterized.

21. The method of claim 14, wherein said posture determination and description includes using statistical and contour-based shape information.

22. The method of claim 21, wherein said step of identifying and classifying changes to said foreground object includes using statistical shape information selected from the group consisting of:

area of the foreground object;

centroid of the foreground object;

bounding box and its aspect ratio of the foreground object;

eccentricity of the foreground object; and

a directional orientation of the foreground object relative to an axis as generated with a Principal Component Analysis.

23. The method of claim 21, wherein said step of identifying and classifying changes to said foreground object uses contour-based shape information selected from the group consisting of b-spline representation, convex hull representation, and comer points.

24. The method of claim 21, wherein said step of identifying and classifying changes to said foreground object includes identifying a set of model postures and their description information, said set of model postures including horizontal posture, vertical posture, eating posture, or sleeping posture.

25. The method of claim 24, wherein said step of identifying and classifying changes to said foreground object includes classifying changes to said foreground object includes classifying the statistical and contour-based shape information from a current image to assign a best-matched posture.

26. The method of claim 25, wherein the said step of determining actual behavior by aggregating behavior primitives includes the step of analyzing temporal ordering of the primitives, such as using information about a transition from a previous behavior primitive to a next behavior primitive.

27. The method of claim 26, wherein said temporal analysis is a time-series analysis such as Hidden Markov Model (HMMs).

28. The method of claim 26, wherein the said step of determining actual behavior includes identifying actual behavior selected from a group consisting of sleeping, eating, roaming around, grooming, and climbing.

29. A method for background subtraction of a video image, comprising the steps of:

grouping a number of images into a set of video images;

creating a standard deviation map of the grouped images;

removing a bounding box area of said image where variation is above a predetermined threshold to create a/partial image; and

combining said partial image with,an existing set of partial images by averaging the set of images to generate a complete background image deplete of a desired foreground object.

30. The method of claim 29, further comprising the step of subtracting said complete background image from a current image so a to obtain said desired foreground object.

31. The method of claim 29, wherein said steps are repeated periodically to update said complete background package.