System and method for automatically discovering, characterizing, classifying and semi-automatically labeling animal behavior and quantitative phenotyping of behaviors in animals

• US 9,317,743 B2
• Filed: 11/10/2014
• Issued: 04/19/2016
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: President and Fellows of Harvard College (Harvard Corporation)
• Original Assignee: President and Fellows of Harvard College (Harvard Corporation)
• Inventors: Datta, Sandeep Robert, Wiltschko, Alexander B.
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): SCHWARTZ, RAPHAEL M
• Application Number:
  US14/537,246
• Publication Number:
  US 20150146939A1
• Time in Patent Office: 526 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A01K 2227/105 Murine
• CPC Class Codes: A01K 2267/0356 Animal model for processes and diseases of the central nervous system, e.g. stress, learning, schizophrenia, pain, epilepsy
• CPC Class Codes: A01K 29/005 Monitoring or measuring activity, e.g. detecting heat or mating
• CPC Class Codes: A01K 67/00 Rearing or breeding animals, not otherwise provided for; New or modified breeds of animals
• CPC Class Codes: A61B 2503/40 Animals
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/112 Gait analysis
• CPC Class Codes: A61B 5/7203 for noise prevention, reduction or removal
• CPC Class Codes: A61B 5/7225 Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation A61B5/30, A61B5/316 take precedence; transformers for specific applications H01F38/00; amplifiers H03F
• CPC Class Codes: F04C 2270/041 Controlled or regulated
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G16B 20/00 ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method for studying the behavior of an animal in an experimental area including stimulating the animal using a stimulus device; collecting data from the animal using a data collection device; analyzing the collected data; and developing a quantitative behavioral primitive from the analyzed data. A system for studying the behavior of an animal in an experimental area including a stimulus device for stimulating the animal; a data collection device for collecting data from the animal; a device for analyzing the collected data; and a device for developing a quantitative behavioral primitive from the analyzed data. A computer implemented method, a computer system and a nontransitory computer readable storage medium related to the same. Also, a method and apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area. Further, use of a depth camera and/or a touch sensitive device related to the same.

Claims

1. A method for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:

(a) using a 3D depth camera to obtain a video stream having a plurality of images of the experimental area with the animal in the experimental area, the images having both area and depth information;

(b) removing background noise from each of the plurality of images to generate processed images having light and dark areas;

(c) determining contours of the light areas in the plurality of processed images;

(d) extracting parameters from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time and identifying the animal'"'"'s head and tail based on a derivative of the contour curvature;

(e) clustering the data points at each specific time to output a set of clusters that are segmented form each other so that each cluster represents an animal behavior;

(f) assigning each cluster a label that represents an animal behavior; and

(g) outputting a visual representation of the set of clusters and corresponding labels.

2. The method of claim 1 wherein step (b) comprises:

(b1) using the 3D depth camera to obtain a video stream having a plurality of baseline images of the experimental area without an animal present;

(b2) generating a median of the baseline images to form a baseline depth image;

(b3) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;

(b4) performing a median filtering operation on each difference image to generate a filtered difference image; and

(b5) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

3. The method of claim 1, wherein step (c) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

4. The method of claim 1, wherein the label for each cluster is requested by a user interface after displaying video data representing each cluster.

5. The method of claim 1, wherein the behavior comprises a quantitative behavior primitive.

6. An apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:

a 3D depth camera that generates a video stream having a plurality of images of the experimental area with the animal in the experimental area, the images having both area and depth information;

a data processing system having a processor and a memory containing program code which, when executed;

(a) removes background noise from each of the plurality of images to generate processed images having light and dark areas;

(b) determines contours of the light areas in the plurality of processed images;

(c) extracts parameters from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time and identifies the animal'"'"'s head and tail based on a derivative of the contour curvature; and

(d) clusters the data points to output a set of clusters that each represent an animal behavior;

(e) assigns each cluster a label that represents an animal behavior; and

(f) outputs a visual representation of the set of clusters and corresponding labels.

7. The apparatus of claim 6, wherein the 3D depth camera obtains a video stream having a plurality of baseline images of the experimental area without an animal present and wherein step (a) comprises:

(a1) generating a median of the baseline images to form a baseline depth image;

(a2) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;

(a3) performing a median filtering operation on each difference image to generate a filtered difference image; and

(a4) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

8. The apparatus of claim 6, wherein the label for each cluster is requested after displaying video data representing each cluster.

9. The apparatus of claim 6, wherein the behavior comprises a quantitative behavior primitive.

10. The apparatus of claim 6, wherein step (b) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

11. The apparatus of claim 6 wherein in step (c) parameters extracted from area information with each contour include at least one of perimeter, surface area rotation angle and length.

12. The apparatus of claim 6, wherein in step (c) parameters extracted from depth information with each contour include at least one of height, width, depth, velocity spine curvature and limb position.

13. The apparatus of claim 6, wherein step (d) comprises reducing covariance between data points and clustering the reduced covariance data points with a clustering method.

14. The apparatus of claim 6, wherein covariance between data points is reduced by reducing dimensionality of each data point.

15. The apparatus of claim 14, wherein the dimensionality of each data point is reduced by applying at least one of principal components analysis, singular value decomposition, independent components analysis and locally linear embedding to the points.