Multi-view object detection using appearance model transfer from similar scenes

• US 8,983,133 B2
• Filed: 06/07/2013
• Issued: 03/17/2015
• Est. Priority Date: 07/15/2011
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Feris, Rogerio S., Pankanti, Sharathchandra U., Siddiquie, Behjat
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US13/912,391
• Publication Number:
  US 20130272573A1
• Time in Patent Office: 648 Days
• Field of Search: 382/103, 382/107, 382/236, 348/143, 348/152, 348/154, 348/155, 348/169, 348/170, 348/171, 348/172, 348/208.1, 348/352, 340/907, 340/917, 340/919, 340/931, 701/117, 701/118, 701/120, 701/414, 701/417, 701/423
• US Class Current: 382/103
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

View-specific object detectors are learned as a function of scene geometry and object motion patterns. Motion directions are determined for object images extracted from a training dataset and collected from different camera scene viewpoints. The object images are categorized into clusters as a function of similarities of their determined motion directions, the object images in each cluster are acquired from the same camera scene viewpoint. Zenith angles are estimated for object image poses in the clusters relative to a position of a horizon in the cluster camera scene viewpoint, and azimuth angles of the poses as a function of a relation of the determined motion directions of the clustered images to the cluster camera scene viewpoint. Detectors are thus built for recognizing objects in input video, one for each of the clusters, and associated with the estimated zenith angles and azimuth angles of the poses of the respective clusters.

Claims

1. A method for determining motion directions for video dataset object images, the method comprising:

estimating directions of motion, through an optical flow process executing on a processor, for each of a plurality of object images that are extracted from a source training video dataset input, for each of a plurality of different camera scene viewpoints, wherein the object images are collected from each of the different camera scene viewpoints;

representing each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorizing the plurality of object images into a plurality of optical flow map clusters as a function of similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by clustering that automatically selects a scale of analysis and a total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by a dominant direction of motion of the points within each optical flow map cluster and by a location of the cluster in the image plane.

2. The method of claim 1, further comprising:

estimating zenith angles for poses of the object images in each of the clusters relative to a position of a horizon in the camera scene viewpoint from which the clustered object images are acquired, and azimuth angles of the poses as a function of a relation of the determined motion directions of the clustered object images to the camera scene viewpoint from which the clustered object images are acquired; and

building a plurality of detectors for recognizing objects input video, one for each of the clusters of the object images, and associating each of the built detectors with the estimated zenith angles and azimuth angles of the poses of the cluster for which the detectors are built.

3. The method of claim 2, further comprising:

estimating the position of the horizon in a target camera viewpoint scene, wherein the target camera viewpoint scene is different from and not included in the source domain camera scene viewpoints;

determining a motion direction for the target scene object image;

estimating a zenith angle for a pose of the target scene objects relative to the estimated target camera viewpoint scene horizon and an azimuth angle of the target scene object pose as a function of the determined target scene object image motion direction to the target camera scene viewpoint; and

selecting one or more of the built detectors that have an associated cluster zenith angle and an associated cluster azimuth angle that best match the target scene object image pose zenith angle and target scene object image pose azimuth angle, wherein the built detector is selected for recognizing objects in video data of the target domain acquired from the target camera viewpoint that have the size and motion dimension values that meet the expected criterion of the object of interest.

4. The method of claim 3, further comprising:

representing variations of the poses of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

representing variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

5. The method of claim 4, further comprising:

estimating the position of the horizon in each of the clustered training camera views and the target camera view by utilizing structures in images of the camera scene viewpoint that have an inherent geometric relationship to an image horizon inferred from the real-world, three- dimensional geometry of the structures in the camera scene viewpoint images.

6. The method of claim 5, wherein the step of estimating the position of the horizon in at least one of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images comprises:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

7. The method of claim 6, wherein the step of building the plurality of detectors for recognizing objects in input video comprises:

building Deformable Parts Model (DPM)-based object detectors that treat positions of parts of the objects as latent variables; and

employing a latent Support Vector Machine (SVM) to infer the positions of the parts of the objects from image data in the input video.

8. A method of providing a service for determining motion directions for video dataset object images, the method comprising:

providing a motion direction determiner that estimates directions of motion, through an optical flow process, for each of a plurality of object images that are extracted from a source training video dataset input, for each of a plurality of different camera scene viewpoints, wherein the object images are collected from each of the different camera scene viewpoints, and represents each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of each space-time point in an image plane, a magnitude and a direction of its optical flow; and

providing an object classifier that categorizes the plurality of object images into a plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by clustering that automatically selects a scale of analysis and a total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by a dominant direction of motion of the points within the each optical flow map cluster and by a location in the image plane.

9. The method of claim 8, further comprising:

providing a pose parameterizer that estimates zenith angles for poses of the object images in each of the clusters relative to a position of a horizon in the camera scene viewpoint from which the clustered object images are acquired, and azimuth angles of the poses as a function of a relation of the determined motion directions of the clustered object images to the camera scene viewpoint from which the clustered object images are acquired; and

providing an object detector modeler that builds a plurality of detectors for recognizing objects, one for each of the clusters of the object images, and associates each of the built detectors with the estimated zenith angles and azimuth angles of the poses of the cluster for which the detectors are built.

10. The method of claim 9, wherein the motion direction determiner further determines a motion direction for the target scene object image, wherein the target camera viewpoint scene is different from and not included in the source domain camera scene viewpoints;

wherein the pose parameterizer further estimates a position of a horizon in a target camera viewpoint scene of an object image acquired from a target camera, a zenith angle for the pose of the target scene object relative to the estimated target camera viewpoint scene horizon and an azimuth angle of the target scene object pose as a function of a relation of the determined target scene object image motion direction to the target camera scene viewpoint; and

wherein the method further comprises;

providing a detector selector that selects one of the built detectors that has an associated cluster zenith angle and an associated cluster azimuth angle that best matches the target scene object image pose zenith angle and target scene object image pose azimuth angle; and

a detector applicator that applies the selected previously learned detector to video data of the target domain acquired from the target camera viewpoint to recognize objects in the target domain video data.

11. The method of claim 10, wherein the object classifier further:

represents variations of the poses of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

represents variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

12. The method of claim 11, wherein the pose parameterizer further:

estimates the position of the horizon in each of the clustered training camera views and the target camera view by utilizing structures in images of the camera scene viewpoint that have an inherent geometric relationship to an image horizon inferred from the real-world, three- dimensional geometry of the structures in the camera scene viewpoint images.

13. The method of claim 12, wherein the pose parameterizer estimates the position of the horizon in at least one of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images by:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

14. The method of claim 13, wherein the object detector modeler builds the plurality of detectors for recognizing objects in input video comprises:

building Deformable Parts Model (DPM)-based object detectors that treat positions of parts of the objects as latent variables; and

employing a latent Support Vector Machine (SVM) to infer the positions of the parts of the objects from image data in the input video.

15. A system comprising:

a processing unit;

a computer readable memory in circuit communication with the processing unit; and

a computer-readable storage medium in circuit communication with the processing unit; and

wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory;

estimates directions of motion, through an optical flow process, for each of a plurality of object images that are extracted from a source training video dataset input, for each of a plurality of different camera scene viewpoints, wherein the object images are collected from each of the different camera scene viewpoints;

represents each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorizes the plurality of object images into a plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by clustering that automatically selects a scale of analysis and a total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by a dominant direction of motion of the points within the each optical flow map cluster and by a location in the image plane.

16. The system of claim 15, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further:

estimates zenith angles for poses of the object images in each of the clusters relative to the position of the horizon in the camera scene viewpoint from which the clustered object images are acquired, and azimuth angles of the poses as a function of the determined motion directions of the clustered object images; and

builds a plurality of detectors for recognizing objects input video, one for each of the clusters of the object images, and associates each of the built detectors with the estimated zenith angles and azimuth angles of the poses of the cluster for which the detectors are built.

17. The system of claim 16, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further:

determines a motion direction for the target scene object image, wherein the target camera viewpoint scene is different from and not included in the source domain camera scene viewpoints;

estimates a position of the horizon in a target camera viewpoint scene of an object image acquired from a target camera, a zenith angle for the pose of the target scene object relative to the estimated target camera viewpoint scene horizon and an azimuth angle of the target scene object pose as a function of the determined target scene object image motion direction;

selects one of the built detectors that has an associated cluster zenith angle and an associated cluster azimuth angle that best matches the target scene object image pose zenith angle and target scene object image pose azimuth angle; and

applies the selected previously learned detector to video data of the target domain acquired from the target camera viewpoint to recognize objects in the target domain video data.

18. The system of claim 17, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further:

represents variations of the poses of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

represents variations of the determined directions of motion of each of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

19. The system of claim 18, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further estimates the position of the horizon in each of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three- dimensional geometry of the structures in the camera scene viewpoint images by:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

20. An article of manufacture, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising instructions that, when executed by the computer processor, cause the computer processor to;

estimate directions of motion, through an optical flow process, for each of a plurality of object images that are extracted from a source training video dataset input, for each of a plurality of different camera scene viewpoints, wherein the object images are collected from each of the different camera scene viewpoints;

represent each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and

categorize the plurality of object images into a plurality of optical flow map clusters as a function of the similarities of their determined motion directions by;

discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise;

after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by clustering that automatically selects a scale of analysis and a total number of the clusters; and

representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by a dominant direction of motion of the points within the each optical flow map cluster and by a location of the cluster in the image plane.

21. The article of manufacture of claim 20, wherein the computer readable program code instructions, when executed by the computer processor, further cause the computer processor to:

estimate zenith angles for poses of the object images in each of the clusters relative to the position of a horizon in the camera scene viewpoint from which the clustered object images are acquired, and azimuth angles of the poses as a function of the determined motion directions of the clustered object images to the camera scene viewpoint from which the clustered object images are acquired; and

build a plurality of detectors for recognizing objects input video, one for each of the clusters of the object images, and associates each of the built detectors with the estimated zenith angles and azimuth angles of the poses of the cluster for which the detectors are built.

22. The article of manufacture of claim 21, wherein the computer readable program code instructions, when executed by the computer processor, further cause the computer processor to:

determine a motion direction for the target scene object image, wherein the target camera viewpoint scene is different from and not included in the source domain camera scene viewpoints;

estimate a position of a horizon in a target camera viewpoint scene of an object image acquired from a target camera, a zenith angle for the pose of the target scene object relative to the estimated target camera viewpoint scene horizon and an azimuth angle of the target scene object pose as a function of the determined target scene object image motion directions;

select one of the built detectors that has an associated cluster zenith angle and an associated cluster azimuth angle that best matches the target scene object image pose zenith angle and target scene object image pose azimuth angle; and

apply the selected previously learned detector to video data of the target domain acquired from the target camera viewpoint to recognize objects in the target domain video data.

23. The article of manufacture of claim 22, wherein the computer readable program code instructions, when executed by the computer processor, further cause the computer processor to:

represent variations of the poses of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, by a range of the zenith angles determined for the cluster objects from the minimum determined zenith angle to the maximum determined zenith angle; and

represent variations of the determined directions of motion of the objects in each of the clusters with respect to the camera viewpoint from which the clustered objects are acquired, as a range of the azimuth angles determined for the cluster objects from the minimum determined azimuth angle to the maximum determined azimuth angle.

24. The article of manufacture of claim 23, wherein the computer readable program code instructions, when executed by the computer processor, further cause the computer processor to estimate the position of the horizon in each of the clustered camera views and the target camera view by utilizing the structures in images of the camera scene viewpoint that have an inherent geometric relationship to the image horizon inferred from the real-world, three- dimensional geometry of the structures in the camera scene viewpoint images by:

identifying a plurality of structures in the images of the camera scene viewpoint through geometric parsing that are generally parallel to each other in the real-world, three-dimensional geometry of the structures in the camera scene viewpoint images;

using the plurality of structures to define multiple sets of parallel lines in the camera scene viewpoint images that are each aligned with structures such as buildings, roads etc. wherein each set of parallel lines intersects at a vanishing point; and

estimating the horizon line as a line passing through the vanishing points of at least two sets of parallel lines.

25. The article of manufacture of claim 24, wherein the computer readable program code instructions, when executed by the computer processor, further cause the computer processor to build the plurality of detectors for recognizing objects in input video by:

building Deformable Parts Model (DPM)-based object detectors that treat positions of parts of the objects as latent variables; and

employing a latent Support Vector Machine (SVM) to infer the positions of the parts of the objects from image data in the input video.