System and method for object identification and behavior characterization using video analysis

• US 7,068,842 B2
• Filed: 09/19/2003
• Issued: 06/27/2006
• Est. Priority Date: 11/24/2000
• Status: Expired due to Term

• Current Assignee: Clever Sys Incorporated
• Original Assignee: Clever Sys Incorporated
• Inventors: Crnic, Linda, Wolf, Wayne, Liang, Yiqing, Kobla, Vikrant
• Primary Examiner(s): Patel, Kanjibhai
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US10/666,741
• Publication Number:
  US 20040131254A1
• Time in Patent Office: 1,012 Days
• Field of Search: 382/103, 382/106, 382/107, 382/108, 382/110, 382/115, 382/154, 382/181, 382/194, 382/199, 382/203, 382/216, 382/224, 382/287, 382/294, 382/305, 382/318, 382/190, 382/282, 382/285, 119/421, 128/897, 348/143, 340/573.1, 702/139
• US Class Current: 382/181
• CPC Class Codes: A01K 1/031 Cages for laboratory animals; Cages for measuring metabolism of animals
• CPC Class Codes: A01K 29/005 Monitoring or measuring activity, e.g. detecting heat or mating
• CPC Class Codes: A61B 2503/40 Animals
• CPC Class Codes: A61B 2503/42 for laboratory research
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

In general, the present invention is directed to systems and methods for finding the position and shape of an object using video. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide object segmentation and identification, object motion tracking (for moving objects), object position classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for object identification and tracking, probabilistic approach with expectation-maximization for tracking the motion detection and object classification, and decision tree classification for behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various objects and the object'"'"'s movements within the image. The image may be provided in real time or from storage. The invention is particularly useful for monitoring and classifying animal behavior for testing drugs and genetic mutations, but may be used in any of a number of other surveillance applications.

Claims

1. A device comprising:

a first module configured to receive video images and to identify objects of interest in response to the video images;

a second module coupled to the first module and configured to classify a plurality of shape and posture categories based on a plurality of observed states of the objects of interest; and

a third module coupled to the second module, wherein the third module is configured to identify behaviors of the objects of interest using the plurality of shape and posture categories and standard object behaviors, and is further configured to analyze temporal ordering of behavior primitives time-series analysis to indentify a transition from a previous behavior primitive to a next behavior primitive.

2. The device of claim 1, wherein the second module further obtains feature information by tracking the objects of interest over a plurality of video images.

3. The device of claim 2, wherein the third module is further configured to identify behaviors of the objects of interest by comparing feature information of the objects of interest with predefined categories exhibited by the standard object behaviors.

4. The device of claim 2, wherein the third module is further configured to identify at least one of eating, rearing, jumping, drinking, running, walking, chewing, grooming, hanging, cuddling, landing vertically, repetitively jumping, circling, sleeping, twitching, awakening, digging, foraging, sniffing, pausing, urinating, stretching, licking, scratching, spinning, sitting, standing, and lying.

5. The device of claim 1, wherein the first module is an object identification and segregation module that identifies and segregates predetermined types of objects from the video images.

6. The device of claim 5, wherein the second module further includes:

an object tracking module configured to track the objects of interest in a series of video image frames; and

an object shape and posture classifier coupled to the object tracking module and configured to provide the plurality of shape and posture categories.

7. The device of claim 6, wherein the third module includes a behavior identification module, which is coupled to a standard object behavior storage module.

8. A system for characterizing activity of objects comprising:

a video camera;

a video digitization unit; and

a digital processing unit that includes;

a device of claim 1; and

a memory device.

9. A method of characterizing an activity comprising:

obtaining a stream of video image frames from a monitoring device;

identifying objects of interest from the stream of video image frames;

providing feature information of the objects of interest in response to the stream of video image frames;

classifying postures of the objects of interest in response to the feature information;

obtaining standard object behaviors from a storage memory device; and

characterizing behaviors of the objects of interest in response to a comparison between the postures of the objects of interest and the standard object behaviors, wherein the characterizing the behaviors of the objects of interest further includes analyzing temporal ordering of behavior primitive using time-series analysis to identify a transition from a previous behavior primitive to a next behavior primitive.

10. The method of claim 9, wherein the time-series analysis further includes employing Hidden Markov Models (HMMs).

11. The method of claim 9, wherein the providing feature information further includes tracking physical changes of the objects of interest over multiple video image frames.

12. The method of claim 9, wherein the identifying objects of interest from the stream of video image frames further includes detecting foreground objects of interest from the video image frames.

13. The method of claim 12, wherein the detecting foreground objects of interest further includes subtracting a background utilizing a background subtraction algorithm.

14. The method of claim 9, wherein the identifying objects of interest further includes identifying humans, vehicles, and other moving and non-moving objects.

15. The method of claim 9, wherein the identifying objects of interest further includes identifying an animal.

16. The method of claim 15, wherein the identifying an animal further includes identifying a biological mouse or rat.

17. The method of claim 9, wherein the classifying postures of the objects of interest further includes:

describing a sequence of postures as behavior primitives; and

aggregating behavior primitives to identify behaviors of the objects of interest over a plurality of video image frames.

18. The method of claim 17, wherein the classifying of postures of the objects of interest includes detecting the shapes of the objects of interest.

19. The method of claim 9, wherein the characterizing the behaviors of the objects of interest further includes identifying one of eating, rearing, jumping, drinking, running, walking, chewing, grooming, hanging, cuddling, landing vertically, repetitively jumping, circling, sleeping, twitching, awakening, digging, foraging, sniffing, pausing, urinating, stretching, licking, scratching, spinning, sitting, standing, and lying.

20. The method of claim 9, wherein the identifying an objects of interest further includes detecting one of vertical position side view, horizontal position side view, vertical position front view, horizontal position front view, and moving of the objects of interest.

21. An apparatus for characterizing an activity comprising:

means for obtaining a stream of video image frames from a monitoring device;

means for identifying objects of interest from the stream of video image frames;

means for providing feature information of the objects of interest in response to the stream of video image frames;

means for classifying postures of the objects of interest in response to the feature information;

means for obtaining standard object behaviors from a storage memory device; and

means for characterizing behaviors of the objects of interest in response to comparison between the postures of the objects of interest and the standard object behaviors;

wherein the means for characterizing further includes means for analyzing temporal ordering of behavior primitives using transitional information from a previous behavior primitive to a next behavior primitive;

wherein the means for analyzing temporal ordering of behavior primitives further includes means for utilizing Hidden Markov Model time-series analysis.

22. The apparatus of claim 21, wherein the means for providing feature information further includes means for tracking physical changes of the objects of interest over multiple video image frames.

23. The apparatus of claim 21, wherein the means for identifying objects of interest from the stream of video image frames further includes means for detecting foreground objects of interest from the video image frames.

24. The apparatus of claim 23, wherein the means for detecting foreground objects of interest further includes means for subtracting a background utilizing a background subtraction algorithm.

25. The apparatus of claim 21, wherein the means for identifying objects of interest further includes means for identifying humans, vehicles, and other moving and non-moving objects.

26. The apparatus of claim 21, wherein the means for identifying objects of interest further includes means for identifying an animal.

27. The apparatus of claim 26, wherein the means for identifying an animal further includes means for identifying a biological mouse or rat.

28. The apparatus of claim 21, wherein the means for classifying postures of the objects of interest further includes:

means for describing a sequence of postures as behavior primitives; and

means for aggregating behavior primitives to identify behaviors of the objects of interest over a plurality of video image frames.

29. The apparatus of claim 28, wherein the means for classifying of postures of the objects of interest includes means for detecting shapes of the objects of interest.

30. The apparatus of claim 21, wherein the means for characterizing the behaviors of the objects of interest further includes means for identifying one of eating, rearing, jumping, drinking, running, walking, chewing, grooming, hanging, cuddling, landing vertically, repetitively jumping, circling, sleeping, twitching, awakening, digging, foraging, sniffing, pausing, urinating, stretching, licking, scratching, spinning, sitting, standing, and lying.

31. The apparatus of claim 21, wherein the means for identifying objects of interest further includes means for detecting one of vertical position side view, horizontal position side view, vertical position front view, horizontal position front view, and moving of the objects of interest.

32. A machine-readable medium having stored thereon computer executable instructions, which cause a digital processing system to perform a function, the function comprising:

obtaining a stream of video image frames from a monitoring device;

identifying objects of interest from the stream of video image frames;

providing feature information of the objects of interest in response to the stream of video image frames;

classifying postures of the objects of interest in response to the feature information;

obtaining standard object behaviors from a storage memory device; and

characterizing behaviors of the objects of interest in response to comparison between the postures of the objects of interest and the standard object behaviors, wherein the characterizing the behaviors of the objects of interest further includes analyzing temporal ordering of behavior primitives.

33. The machine-readable medium of claim 32, wherein the analyzing temporal ordering of the behavior primitives further includes utilizing time-series analysis to identify a transition from a previous behavior primitive to a next behavior primitive.

34. The machine-readable medium of claim 33, wherein the time-series analysis further includes employing Hidden Markov Models (HMMs).

35. The machine-readable medium of claim 32, wherein the providing feature information further includes tracking physical changes of the objects of interest over multiple video image frames.

36. The machine-readable medium of claim 32, wherein the identifying objects of interest from the stream of video image frames further includes detecting foreground objects of interest from the video image frames.

37. The machine-readable medium of claim 36, wherein the detecting foreground objects of interest further includes subtracting a background utilizing a background subtraction algorithm.