System and method for stereo-view multiple animal behavior characterization

• US 8,634,635 B2
• Filed: 10/30/2008
• Issued: 01/21/2014
• Est. Priority Date: 10/30/2008
• Status: Expired due to Fees

• Current Assignee: Clever Sys Incorporated
• Original Assignee: Clever Sys Incorporated
• Inventors: Bai, Xuesheng, Liang, Yiqing, Kobla, Vikrant N., Li, FaYin, Liu, Dongliang
• Primary Examiner(s): Fitzpatrick, Atiba O
• Assistant Examiner(s): JAMES, THOMAS AIDEN
• Application Number:
  US12/261,691
• Publication Number:
  US 20100111359A1
• Time in Patent Office: 1,909 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: A01K 1/031 Cages for laboratory animals; Cages for measuring metabolism of animals
• CPC Class Codes: A01K 29/005 Monitoring or measuring activity, e.g. detecting heat or mating
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

In general, the present invention is directed to systems and methods for identifying behaviors of two or more animal using video. The invention includes a system with at least two video cameras coupled to a computer where the computer is configured to automatically perform animal segmentation and identification, animal body parts identification, and behavior identification.

Claims

1. A behavior analysis system, comprising:

an arena;

at least a first camera and a second camera, said first camera positioned to capture images of the arena from a first view, and said second camera positioned to capture images of the arena from a second view, the first view being different from the second view;

a database containing definitions of behaviors of living organisms;

a computer coupled to the first camera, the second camera and the database;

said computer comprising memory storing a computer program;

said computer program configures the computer to receive as an input at least the first view images and the second view images;

said computer program including a behavior analysis module;

said behavior analysis module configured to;

identify at least first view objects from the first view images and second view objects from the second view images;

determine at least a first living organism and a second living organism using the first view objects and the second view objects;

determine at least one spatial relationship over time between the first living organism and the second living organism based on stereo-view analysis of spatial relationships between the first living organism and the second living organism from the first view images and the second view images; and

identify which of the behaviors stored in the database correspond to the at least one spatial relationship over time between the first living organism and the second living organism.

2. The system of claim 1 wherein the first view is from the top of the arena and the second view is from the side of the arena.

3. The system of claim 1 wherein the first view is from the bottom of the arena and the second view is from the side of the arena.

4. The system of claim 1 wherein the computer program further comprises:

a design module to facilitate design of experiment;

an execute module to execute the experiment; and

a review and process module to facilitate review of the output of the behavior analysis module; and

wherein the first view images and the second view images are captured during the experiment.

5. The system of claim 1 wherein said behavior analysis module is further configured to determine the first view living organisms using the first view objects and determine the second view living organisms using the first view living organisms and the second view objects.

6. The system of claim 1 wherein said behavior analysis module is further configured to determine the spatial relationships over time between at least the first living organism and the other parts of the arena.

7. The system of claim 1 wherein the behavior analysis module is further configured to use an inter-view matching procedure to determine at least the first living organism and the second living organism from the first view objects and the second view objects.

8. The system of claim 1 wherein the behavior analysis module is further configured to determine connected second view objects and perform connected object segmentation of the connected second view objects using the first view living organisms.

9. The system of claim 1 wherein the behavior analysis module is further configured to perform animal size normalization on the second view living organisms using the first view living organisms.

10. The system of claim 1 wherein the identification of the behaviors is performed using probabilistic approach.

11. The system of claim 1 wherein the identification of the behaviors is performed using deterministic approach.

12. The system of claim 1 wherein the identification of the behaviors is performed using at least one of the methods selected from the group consisting of rule-based analysis, token parsing procedure, sequence analysis, state change analysis and Hidden Markov Modeling.

13. The system of claim 1, wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of active social contact behavior, passive social contact behavior, sniff at another animal and its specific body parts behavior, approach behavior, leave behavior, follow behavior, and stare at another animal behavior.

14. The system of claim 1, wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of threat behavior, thrust behavior, chase behavior, bite behavior, fully aggressive behavior, offensive upright behavior, offensive sideways behavior, aggressive grooming behavior, tail rattle behavior, fight behavior, flag behavior, evade behavior, retreat behavior, flee behavior, freeze behavior, crouch behavior, elevated crouch behavior, full submissive behavior, defensive upright behavior, and defensive sideways behavior.

15. The system of claim 1, wherein the behavior analysis module is configured to identify dominant/submissive behaviors using the living organisms'"'"' interactions with a reward.

16. The system of claim 1, wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of retrieve offspring behavior, lick and groom offspring behavior, nurse offspring behavior, on nest behavior, off nest behavior, scattered offspring behavior, transport offspring behavior, arch-back nurse offspring behavior, prone nurse offspring behavior, and build nest behavior.

17. The system of claim 1, wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of hop behavior, dart behavior, ear wiggle behavior, lordosis behavior, chase behavior, grooming behavior, sniffing genital area behavior, mount behavior, single trust behavior, intromission behavior, and ejaculate behavior.

18. The system of claim 1 wherein the computer program configures the computer to divide the arena into virtual zones.

19. The system of claim 1, wherein the living organisms are animals.

20. The system of claim 19, wherein the animals are mice.

21. The system of claim 19, wherein the animals are rats.

22. The system of claim 19, wherein the animals are a combination of mice and rats.

23. The system of claim 1, wherein the first view objects and the second view objects include body parts.

24. The system of claim 23, wherein the body parts include at least one of the body parts selected from the group consisting of a head, a tail, a waist, a fore body, a hind body, hind limbs, forelimbs, upper back, lower back, abdomen, feet, mouth, and nose.

25. The system of claim 1, wherein the images are captured under various types of light conditions including visible, colored, sodium and infra-red light.

26. The system of claim 1 further comprising:

a second arena; and

a third camera and a fourth camera, said third camera positioned to capture images of the second arena from a third view, and said fourth camera positioned to captured images of the second arena from a fourth view; and

wherein;

the computer is further coupled to the third camera and the fourth camera;

the computer program configures the computer to receive as an input at least the third view images and the fourth view images;

said behavior analysis module further configured to;

identify at least third view objects from the third view images and fourth view objects from the fourth view images;

determine second arena living organisms from the third view objects and the fourth view objects;

determine at least one spatial relationship over time for at least one of the second arena living organisms; and

identify which of the behaviors stored in the database correspond to the spatial relationships over time of at least one of the second arena living organisms.

27. The system of claim 1 further comprising a multiplexer wherein at least the first camera and the second camera are coupled to the computer by the multiplexer.

28. The system of claim 1 wherein the behavior analysis module is further configured to determine at least the first living organism and the second living organism using information about the living organisms.

29. The system of claim 1 wherein the behavior analysis module is further configured to convert the images from analog to digital format if the images are captured in analog format.

30. The system of claim 1 wherein the spatial relationships comprise spatial relationships between the first living organism and the second living organism.

31. The system of claim 1 wherein the spatial relationships comprise spatial relationships between body parts of the first living organism and the second living organism.

32. The system of claim 1 wherein the spatial relationships comprise spatial relationships between body parts of the first living organism and body parts of the second living organism.

33. The system of claim 1 wherein the spatial relationships comprise spatial relationships between the first living organism and inanimate objects.

34. The system of claim 1 wherein the spatial relationships comprise spatial relationships between body parts of the first living organism and inanimate objects.

60. The system of claim 1 wherein the first view and the second view are perpendicular to each other.

35. A behavior analysis system, comprising:

a database containing definitions of behaviors of living organisms; and

a computer coupled to the database said computer comprising a memory storing a computer program that configures the computer to receive as an input at least a first view image of an arena from a first view and a second view image of the arena from a second view, where the first view is different from the second view;

said computer program comprising;

a behavior analysis module configured to;

identify at least first view objects from the first view image and second view objects from the second view image;

determine at least a first living organism and a second object from the first view objects and the second view objects;

perform stereo-view analysis to determine;

a first spatial relationship between the first living organism and the second object from the first view image of the arena from the first view; and

a second spatial relationship between the first living organism and the second object from the second view image of the arena from the second view;

determine a spatial relationship over time between the first living organism and the second object based at least in part on the stereo-view analysis; and

identify which of the behaviors stored in the database correspond to the spatial relationship over time between the first living organism and the second object.

36. The system of claim 35 wherein the computer program further comprises:

a design module to facilitate design of experiment;

an execute module to execute the experiment; and

a review and process module to facilitate review of the output of the behavior analysis module; and

wherein the first view image and the second view image are captured during the experiment.

37. The system of claim 35 wherein the second object is a second living organism.

38. The system of claim 37 wherein the behavior analysis module is further configured to use an inter-view matching procedure to determine the first living organism and the second living organism from each of the first view objects and the second view objects.

39. The system of claim 37 wherein the behavior analysis module is further configured to determine connected second view objects and perform connected object segmentation of the connected second view objects using the first view objects.

40. The system of claim 37 wherein the behavior analysis module is further configured to perform animal size normalization on the second view objects using the first view objects.

41. The system of claim 37 wherein the identification of the behaviors is performed using probabilistic approach.

42. The system of claim 37 wherein the identification of the behaviors is performed using deterministic approach.

43. The system of claim 37 wherein the identification of the behaviors is performed using at least one of the methods selected from the group consisting of rule-based analysis, token parsing procedure, sequence analysis, state change analysis and Hidden Markov Modeling.

44. The system of claim 37 wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of active social contact behavior, passive social contact behavior, sniff at another animal and its specific body parts behavior, approach behavior, leave behavior, follow behavior, and stare at another animal behavior.

45. The system of claim 37 wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of threat behavior, thrust behavior, chase behavior, bite behavior, fully aggressive behavior, offensive upright behavior, offensive sideways behavior, aggressive grooming behavior, tail rattle behavior, fight behavior, flag behavior, evade behavior, retreat behavior, flee behavior, freeze behavior, crouch behavior, elevated crouch behavior, full submissive behavior, defensive upright behavior, and defensive sideways behavior.

46. The system of claim 37 wherein the behavior analysis module is configured to identify dominant/submissive behaviors using the living organisms'"'"' interactions with a reward.

47. The system of claim 37 wherein the behavior analysis module is at least one of the behaviors selected from the group consisting of retrieve offspring behavior, lick and groom offspring behavior, nurse offspring behavior, on nest behavior, off nest behavior, scattered offspring behavior, transport offspring behavior, arch-back nurse offspring behavior, prone nurse offspring behavior, and build nest behavior.

48. The system of claim 37 wherein the behavior analysis module is configured to identify at least one of the behaviors selected from the group consisting of hop behavior, dart behavior, ear wiggle behavior, lordosis behavior, chase behavior, grooming behavior, sniffing genital area behavior, mount behavior, single trust behavior, intromission behavior, and ejaculate behavior.

49. The system of claim 35 wherein the computer program is configured to divide the arena into virtual zones.

50. The system of claim 35 wherein the first view objects and the second view objects include body parts.

51. The system of claim 35 wherein the behavior analysis module is further configured to determine the first living organism using information about the first living organism.

52. The system of claim 35 wherein the spatial relationship over time comprises spatial relationships between the first living organism and an inanimate object located within the arena.

53. The system of claim 37 wherein the spatial relationship over time comprises spatial relationships between body parts of the first living organism and the second living organism.

54. The system of claim 37 wherein the spatial relationship over time comprises spatial relationships between body parts of the first living organism and body parts of the second living organism.

55. The system of claim 37 wherein the spatial relationship over time comprises spatial relationships between the first living organism and an inanimate object determined from the first view objects and the second view objects.

56. The system of claim 35 wherein the spatial relationship over time comprises comprise spatial relationships between body parts of the first living organism and the second object, the second object being an inanimate object.

59. The system of claim 35 wherein the first view and the second view are perpendicular to each other.

57. A computer-implemented method of identifying behaviors of living organisms comprising:

acquiring at least a first view image of an arena from a first view over time and a second view image of the arena from a second view over time, the first view being different from the second view;

identifying first view objects in the first view images;

identifying second view objects in the second view images;

determining a living organism and a second object from the first view objects and second view objects;

performing stereo-view analysis to determine;

a first spatial relationship between the living organism and the second object from the first view image of the arena from the first view; and

a second spatial relationship for the first living organism and the second object from the second view image of the arena from the second view;

determining spatial relationships over time between the first living organism and the second object based at least in part on the stereo-view analysis; and

identifying behaviors that correspond to the spatial relationships over time of the between the first living organism and the second object.

58. The computer-implemented method of claim 57 wherein the first view and the second view are perpendicular to each other.