Multi-view object detection using appearance model transfer from similar scenes

• US 9,224,046 B2
• Filed: 01/19/2015
• Issued: 12/29/2015
• Est. Priority Date: 07/15/2011
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Feris, Rogerio S., Pankanti, Sharathchandra U., Siddiquie, Behjat
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US14/599,616
• Publication Number:
  US 20150131861A1
• Time in Patent Office: 344 Days
• Field of Search: 382/103, 382/107, 382/236, 348/143, 348/152, 348/154, 348/155, 348/169, 348/170, 348/171, 348/172, 348/208.1, 348/352, 701/117, 701/118, 701/120, 701/414, 701/417, 701/423, 340/907, 340/917, 340/919, 340/931
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

View-specific object detectors are learned as a function of scene geometry and object motion patterns. Motion directions are determined for object images extracted from a training dataset and collected from different camera scene viewpoints. The object images are categorized into clusters as a function of similarities of their determined motion directions, the object images in each cluster are acquired from the same camera scene viewpoint. Zenith angles are estimated for object image poses in the clusters relative to a position of a horizon in the cluster camera scene viewpoint, and azimuth angles of the poses as a function of a relation of the determined motion directions of the clustered images to the cluster camera scene viewpoint. Detectors are thus built for recognizing objects in input video, one for each of the clusters, and associated with the estimated zenith angles and azimuth angles of the poses of the respective clusters.

Claims

1. A computer-implemented method for learning a view-specific object detector, the method comprising executing on a central processing unit the steps of:

determining a position of a horizon in a target camera viewpoint scene;

determining a motion direction for an object within an image of the target camera viewpoint scene;

determining a zenith angle for a pose of the object within the target scene relative to the estimated target camera viewpoint scene horizon;

determining an azimuth angle for a pose of the object within the target scene relative to the determined motion direction for the object within the image of the target camera viewpoint scene; and

selecting one of a plurality of built detectors for recognizing objects in video data, as a function of the selected built detector having an associated cluster zenith angle that best matches the zenith angle determined for the pose of the object relative to cluster zenith angles that are associated with each of others of the plurality of built detectors, and an associated cluster azimuth angle that best matches the azimuth angle determined for the pose of the object relative to azimuth angles associated with each of the others of the plurality of built detectors.

2. The method of claim 1, further comprising:

determining motion directions for each of a plurality of object images that are extracted from a source training video dataset input and that each have size and motion dimension values that meet an expected criterion of an object of interest, wherein the object images are collected from each of a plurality of different camera scene viewpoints;

categorizing the plurality of object images into each of the plurality of clusters as a function of similarities of their determined motion directions, wherein the object images in each of the clusters are also acquired from one of the different camera scene viewpoints;

estimating zenith angles for poses of objects within the object images in each of the clusters relative to a position of a horizon in the camera scene viewpoint from which the clustered object images are acquired;

approximating azimuth angles for the poses of the object images in each of the clusters relative to the determined motion directions for the objects within the object images in each of the clusters;

building the plurality of built detectors for recognizing objects within input video, one for each of the clusters of the object images; and

associating each of the built detectors with the estimated zenith angles and approximated azimuth angles of the poses of the clusters for which the detectors are built.

3. The method of claim 2, further comprising:

representing variations of the poses of the objects in each of the clusters with respect to a camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

representing variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

4. The method of claim 3, wherein the clusters are optical flow map clusters, and

wherein the method further comprises:

estimating a direction of motion of objects appearing in each scene for each respective camera viewpoint through an optical flow process;

representing each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorizing the plurality of object images into the plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by using a self-tuning variant of spectral clustering that automatically selects a scale of analysis and the total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by the dominant direction of motion of the points within each optical flow map cluster and by the location of the cluster in the image plane.

5. The method of claim 4, further comprising:

estimating the position of the horizon in each of the clustered training camera views and the target camera view by utilizing structures in images of the camera scene viewpoint that have an inherent geometric relationship to an image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images.

6. The method of claim 5, wherein the step of estimating the position of the horizon in at least one of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images comprises:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

7. The method of claim 6, wherein the step of building the plurality of detectors for recognizing objects in input video comprises:

building Deformable Parts Model (DPM)-based object detectors that treat positions of parts of the objects as latent variables; and

employing a latent Support Vector Machine (SVM) to infer the positions of the parts of the objects from image data in the input video.

8. A system comprising:

a central processing unit;

a computer readable memory in circuit communication with the central processing unit; and

a computer-readable storage medium in circuit communication with the central processing unit; and

wherein the central processing unit executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

determines a position of a horizon in a target camera viewpoint scene;

determines a motion direction for an object within an image of the target camera viewpoint scene;

determines a zenith angle for a pose of the object within the target scene relative to the estimated target camera viewpoint scene horizon;

determines an azimuth angle for a pose of the object within the target scene relative to the determined motion direction for the object within the image of the target camera viewpoint scene; and

selects one of a plurality of built detectors for recognizing objects in video data, as a function of the selected built detector having an associated cluster zenith angle that best matches the zenith angle determined for the pose of the object relative to cluster zenith angles that are associated with each of others of the plurality of built detectors, and an associated cluster azimuth angle that best matches the azimuth angle determined for the pose of the object relative to azimuth angles associated with each of the others of the plurality of built detectors.

9. The system of claim 8, wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

determines motion directions for each of a plurality of object images that are extracted from a source training video dataset input and that each have size and motion dimension values that meet an expected criterion of an object of interest, wherein the object images are collected from each of a plurality of different camera scene viewpoints;

categorizes the plurality of object images into each of the plurality of clusters as a function of similarities of their determined motion directions, wherein the object images in each of the clusters are also acquired from one of the different camera scene viewpoints;

estimates zenith angles for poses of objects within the object images in each of the clusters relative to a position of a horizon in the camera scene viewpoint from which the clustered object images are acquired;

approximates azimuth angles for the poses of the object images in each of the clusters relative to the determined motion directions for the objects within the object images in each of the clusters;

builds the plurality of built detectors for recognizing objects within input video, one for each of the clusters of the object images; and

associates each of the built detectors with the estimated zenith angles and approximated azimuth angles of the poses of the clusters for which the detectors are built.

10. The system of claim 9, wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

represents variations of the poses of the objects in each of the clusters with respect to a camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

represents variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

11. The system of claim 10, wherein the clusters are optical flow map clusters, and wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

estimates a direction of motion of objects appearing in each scene for each respective camera viewpoint through an optical flow process;

represents each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorizes the plurality of object images into the plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by using a self-tuning variant of spectral clustering that automatically selects a scale of analysis and the total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by the dominant direction of motion of the points within each optical flow map cluster and by the location of the cluster in the image plane.

12. The system of claim 11, wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

estimates the position of the horizon in each of the clustered training camera views and the target camera view by utilizing structures in images of the camera scene viewpoint that have an inherent geometric relationship to an image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images.

13. The system of claim 12, wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further estimates the position of the horizon in at least one of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images by:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing via the processing unit that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

14. The system of claim 13, wherein the central processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further builds the plurality of detectors for recognizing objects in input video by:

building Deformable Parts Model (DPM)-based object detectors that treat positions of parts of the objects as latent variables; and

employing a latent Support Vector Machine (SVM) to infer the positions of the parts of the objects from image data in the input video.

15. An article of manufacture, comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a processor that cause the processor to;

determine a position of a horizon in a target camera viewpoint scene;

determine a motion direction for an object within an image of the target camera viewpoint scene;

determine a zenith angle for a pose of the object within the target scene relative to the estimated target camera viewpoint scene horizon;

determine an azimuth angle for a pose of the object within the target scene relative to the determined motion direction for the object within the image of the target camera viewpoint scene; and

select one of a plurality of built detectors for recognizing objects in video data, as a function of the selected built detector having an associated cluster zenith angle that best matches the zenith angle determined for the pose of the object relative to cluster zenith angles that are associated with each of others of the plurality of built detectors, and an associated cluster azimuth angle that best matches the azimuth angle determined for the pose of the object relative to azimuth angles associated with each of the others of the plurality of built detectors.

16. The article of manufacture of claim 15, wherein the computer readable program code instructions for execution by the processor further cause the processor to:

determine motion directions for each of a plurality of object images that are extracted from a source training video dataset input and that each have size and motion dimension values that meet an expected criterion of an object of interest, wherein the object images are collected from each of a plurality of different camera scene viewpoints;

categorize the plurality of object images into each of the plurality of clusters as a function of similarities of their determined motion directions, wherein the object images in each of the clusters are also acquired from one of the different camera scene viewpoints;

estimate zenith angles for poses of objects within the object images in each of the clusters relative to a position of a horizon in the camera scene viewpoint from which the clustered object images are acquired;

approximate azimuth angles for the poses of the object images in each of the clusters relative to the determined motion directions for the objects within the object images in each of the clusters;

build the plurality of built detectors for recognizing objects within input video, one for each of the clusters of the object images; and

associate each of the built detectors with the estimated zenith angles and approximated azimuth angles of the poses of the clusters for which the detectors are built.

17. The article of manufacture of claim 16, wherein the computer readable program code instructions for execution by the processor further cause the processor to:

represent variations of the poses of the objects in each of the clusters with respect to a camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

represent variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

18. The article of manufacture of claim 17, wherein the clusters are optical flow map clusters, and wherein the computer readable program code instructions for execution by the processor further cause the processor to:

estimate a direction of motion of objects appearing in each scene for each respective camera viewpoint through an optical flow process;

represent each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorize the plurality of object images into the plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by using a self-tuning variant of spectral clustering that automatically selects a scale of analysis and the total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by the dominant direction of motion of the points within each optical flow map cluster and by the location of the cluster in the image plane.

19. The article of manufacture of claim 18, wherein the computer readable program code instructions for execution by the processor further cause the processor to:

estimate the position of the horizon in each of the clustered training camera views and the target camera view by utilizing structures in images of the camera scene viewpoint that have an inherent geometric relationship to an image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images.

20. The article of manufacture of claim 19, wherein the computer readable program code instructions for execution by the processor further cause the processor to estimate the position of the horizon in at least one of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images by:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing via the processing unit that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.