Unified system and method for animal behavior characterization in home cages using video analysis

• US 20040141636A1
• Filed: 10/30/2003
• Published: 07/22/2004
• Est. Priority Date: 11/24/2000
• Status: Active Grant

• Current Assignee: Clever Sys Incorporated
• Original Assignee: Clever Sys Incorporated
• Inventors: Zhang, Yi, Liang, Yiqing, Kobla, Vikrant, Bai, Xuesheng
• Granted Patent: US 7,209,588 B2
• Time in Patent Office: Days
• US Class Current: 382/110
• CPC Class Codes: A01K 1/031 Cages for laboratory animals; Cages for measuring metabolism of animals
• CPC Class Codes: A01K 29/005 Monitoring or measuring activity, e.g. detecting heat or mating
• CPC Class Codes: A61B 2503/40 Animals
• CPC Class Codes: A61B 2503/42 for laboratory research
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

In general, the present invention is directed to systems and methods for finding the position and shape of an animal using video. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide animal segmentation and identification, animal motion tracking (for moving animals), animal-posture classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for animal identification and tracking, and a combination of decision tree classification and rule-based classification for posture and behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various animals and the animal'"'"'s movements within the image. The image may be provided in real time or from storage. The invention is particularly useful for monitoring and classifying animal behavior for testing drugs and genetic mutations, but may be used in any of a number of other surveillance applications.

Claims

1. A video-based animal behavior analysis system, comprising:

a computer configured to determine a position and shape of an animal from video images and characterize activity of said animal based on analysis of changes in said position and said shape over time.

2. The system of claim 1, further comprising:

a video camera and a video digitization unit coupled to said computer for capturing said video images and converting said video images from analog to digital format.

3. The system of claim 2, further comprising:

an animal identification, segregation, and tracking module receiving said video images.

4. The system of claim 3, wherein said computer further includes a behavior identification module for characterizing activity of said animal, said behavior identification module being coupled to said animal identification, segregation, and tracking module.

5. The system of claim 4, wherein said computer further includes a standard animal behavior storage module that stores information about known behavior of a predetermined standard animal for comparing the activity of said animal, said standard animal behavior storage module being coupled to said behavior identification module.

6. The system of claim 1, wherein said animal is a mouse.

7. The system of claim 1, wherein said animal is a rat.

8. A method of determining and characterizing activity of an animal using computer processing of video images, comprising the steps of:

detecting an animal in said video images;

tracking changes to said animal over a plurality of said video images;

identifying and classifying said changes to said animal; and

characterizing said activity of said animal based on comparison to pre-trained models or rules of such activity.

9. The method of claim 8, wherein said step of characterizing said activity includes the steps of:

describing a sequence of postures as behavior primitives; and

aggregating behavior primitives into actual behavior over a range of images;

10. The method of claim 9, wherein said step of characterizing said activity by describing and aggregating behavior primitives further includes the steps of:

describing a set of conditions and rules required for characterizing said activities; and

matching and testing generated features to see if said conditions and rules are satisfied;

11. The method of claim 8, wherein said detecting an animal includes using a background subtraction method comprising the steps of:

apply a lenient threshold on a difference between a current image and a background so as to determine a broad region of interest;

classify by intensity values various pixels in said region of interest to obtain said animal, by selecting only those intensity values that belong to the set of model intensity values of the animal; and

refine contours of said animal image by smoothing.

12. The method of claim 10, wherein said posture determination and description includes using statistical and contour-based shape information.

13. The method of claim 12, wherein said step of identifying and classifying changes to said animal includes using statistical shape information selected from the group consisting of:

area of the animal;

centroid position of the animal;

bounding box and its aspect ratio of the animal;

eccentricity of the animal; and

a directional orientation of the animal relative to an axis as generated with a Principal Component Analysis.

14. The method of claim 12, wherein said step of identifying and classifying changes to said animal uses contour-based shape information selected from the group consisting of curvature measures, thickness measures, relative orientation measures, length measures, and corner points.

15. The method of claim 12, wherein said step of identifying and classifying changes to said animal includes identifying a set of model postures and their description information, said set of model postures including horizontal side view posture, vertical posture, cuddled posture, horizontal front/back view posture, partially reared posture, stretched posture, hang vertical posture, hang cuddled posture, eating posture, or drinking posture.

16. The method of claim 15, wherein said step of identifying and classifying changes to said animal includes classifying the statistical and contour-based shape information from a current image to assign a best-matched posture.

17. The method of claim 10, wherein the said step of describing said behavior primitives includes the step of identifying patterns of postures over a sequence of images.

18. The method of claim 17, wherein said step of describing said behavior primitives step further includes the step of analyzing temporal information selected from the group consisting of direction and magnitude of movement of the centroid, increase and decrease of the eccentricity, increase and decrease of the area, increase and decrease of the aspect ratio of the bounding box, and change in contour information.

19. The method of claim 10, wherein the said step of determining actual behavior by aggregating behavior primitives includes the step of analyzing temporal ordering of the primitives, such as using information about a transition from a previous behavior primitive to a next behavior primitive, and applying all applicable conditions and rules.

20. The method of claim 19, wherein said temporal analysis is a time-series analysis such as Hidden Markov Model (HMMs).

21. The method of claim 19, wherein the said step of determining actual behavior includes identifying actual behavior selected from a group of pre-trained behavior models.

22. The method of claim 21, wherein said group of behavior models includes the behavior of rearing up to a fully reared up or partially reared up position, and said rearing behavior is determined by the sequence of posture starting from cuddled, horizontal side-view, or horizontal front/back view postures to ending in a vertical or partially reared posture;

23. The method of claim 21, wherein said group of behavior models includes the behavior of coming down from a reared up or partially reared up position, and said come down behavior is determined by the sequence of postures starting from vertical or partially reared postures to ending in a cuddled, horizontal side view or horizontal front/back view posture;

24. The method of claim 21, wherein said group of behavior models includes the behavior of eating, and said eating behavior is determined by a sequence of eating postures where the mouth of the animal is in touch with a food container;

25. The method of claim 21, wherein said group of behavior models includes the behavior of drinking, and said drinking behavior is determined by a sequence of drinking postures where the mouth of the animal is in touch with a water spout;

26. The method of claim 21, wherein said group of behavior models includes the behavior of digging, and said digging behavior is determined by the aft movement of bedding by the animal with its fore and hind limbs;

27. The method of claim 21, wherein said group of behavior models includes the behavior of foraging, and said foraging behavior is determined by the movement of bedding using the mouth and forelimbs;

28. The method of claim 21, wherein said group of behavior models includes the behavior of jumping, and said jumping behavior is determined by a single up and down movement of the animal;

29. The method of claim 21, wherein said group of behavior models includes the behavior of jumping repetitively and said repetitive jumping behavior is determined by several continuous up and down movement of the animal;

30. The method of claim 21, wherein said group of behavior models includes the behavior of sniffing, and said sniffing behavior is determined by random brisk movement of the head while the rest of the body remains stationary;

31. The method of claim 21, wherein said group of behavior models includes the behavior of hanging from the top of the cage, and said hanging behavior is determined by a sequence of postures starting from vertical posture to ending in a hang vertical or hang cuddled posture;

32. The method of claim 21, wherein said group of behavior models includes the behavior of landing after hanging, and said landing behavior is determined by a sequence of postures starting from a hang vertical or hang cuddled posture to ending in a vertical posture;

33. The method of claim 21, wherein said group of behavior models includes the behavior of sleeping, and said sleeping behavior is determined by the absence of major movements of the contour of the animal for a prolonged period of time;

34. The method of claim 21, wherein said group of behavior models includes the behavior of twitching during sleep, and said twitch behavior is determined by the detection of a brief period of substantial movement and the resumption of sleep activity;

35. The method of claim 21, wherein said group of behavior models includes the behavior of awakening from sleep, and said awaken behavior is determined by a prolonged movement of the animal after sleep has set in;

36. The method of claim 21, wherein said group of behavior models includes the behavior of grooming, and said grooming behavior determined by brisk movement of limbs and mouth in a cyclical and periodic pattern;

37. The method of claim 21, wherein said group of behavior models includes the behavior of pausing briefly, and said pause behavior is determined by brief absence of movement of the animal;

38. The method of claim 21, wherein said group of behavior models includes the behavior of urinating, and said urinate behavior is determined by the detection of the tail being raised up and the animal remaining stationary briefly;

39. The method of claim 21, wherein said group of behavior models includes the behavior of turning, and said turn behavior is determined by a sequence of postures starting from horizontal side view or cuddled posture to ending in a horizontal front/back view posture, and vice versa;

40. The method of claim 21, wherein said group of behavior models includes the behavior of circling, and said circling behavior is determined by a 3 or more successive turns;

41. The method of claim 21, wherein said group of behavior models includes the behavior of walking or running, and said walking or running behavior is determined by the continuous sideways movement of the centroid of the animal;

42. The method of claim 21, wherein said group of behavior models includes the behavior of stretching its body vertically or horizontally, and said stretch behavior is determined by a concave shape of the animal'"'"'s back;

43. The method of claim 21, wherein said group of behavior models includes the behavior of chewing, and said chewing behavior is determined by the movement of the mouth while the mouth is not in touch with a food container;

44. The method of claim 21, wherein said group of behavior models includes the behavior of remaining stationary, and said stationary behavior is determined by the animal remaining in the same place and not performing any of the other behaviors;

45. The method of claim 21, wherein if the activity cannot be characterized by any of the behavior models, the behavior is deemed to be unknown;

46. The method of claim 8, wherein said steps are also performed in night conditions by using red light to simulate such night conditions, or by using infra-red cameras to capture the images with no light;

47. The method of claim 8, wherein said steps are also performed with a plurality of cages or arenas, each of which contains a single animal;

48. The method of claim 8, wherein said step of detecting animal includes the step of detecting body parts of the animal;

49. The method of claim 48, wherein said body parts include the head;

50. The method of claim 48, wherein said body parts include the tail;

51. The method of claim 48, wherein said body parts include the ear;

52. The method of claim 48, wherein said body parts include the upper and lower back;

53. The method of claim 48, wherein said body parts include the abdomen;

54. The method of claim 48, wherein said body parts include the hind-limbs;

55. The method of claim 48, wherein said body parts include the forelimbs;