DEVICE FOR DETERMINING DISAPPEARING DIRECTION AND METHOD THEREOF, APPARATUS FOR VIDEO CAMERA CALIBRATION AND METHOD THEREOF

• US 20120293663A1
• Filed: 05/02/2012
• Published: 11/22/2012
• Est. Priority Date: 05/16/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: LIU, Yuyu, NIE, Zhi
• Granted Patent: US 9,202,277 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 3/06 Topological mapping of higher dimensional structures onto lower dimensional surfaces
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A disappearing direction determination device and method, a video camera calibration apparatus and method, a video camera and a computer program product are provided. The device comprises: a moving target detecting unit for detecting in the video image a moving target area where a moving object locates; a feature point extracting unit for extracting at least one feature point on the moving object in the detected moving target area; a moving trajectory obtaining unit for tracking a movement of the feature point in a predetermined number of video image frames to obtain a movement trajectory of the feature point; and a disappearing direction determining unit for determining, according to the movement trajectories of one or more moving objects in the video image, a disappearing direction pointed by a major moving direction of the moving objects. Thus, a disappearing direction and video camera gesture parameters can be determined accurately.

Claims

1. A device for determining a disappearing direction, the device comprising:

a moving target detecting unit configured to detect in a plurality of video images a moving target area where a moving object locates;

a feature point extracting unit configured to extract at least one feature point on the moving object in the detected moving target area;

a moving trajectory obtaining unit configured to track a movement of the feature point in a predetermined number of video image frames to obtain a movement trajectory of the feature point; and

a disappearing direction determining unit configured to determine, according to the movement trajectories of one or more moving objects in the video images, a disappearing direction pointed by a major moving direction of the moving objects.

2. The device for determining a disappearing direction according to claim 1, wherein, the feature point is a corner of the moving object, or a corner of texture of the moving object.

3. A video camera, comprising:

an apparatus for video camera calibration configured to calibrate the video camera based on a plurality of video images shot by the video camera, the apparatus comprising a device for determining a disappearing direction and a video camera calibration device,

wherein the device for determining a disappearing direction comprises;

a moving target detecting unit configured to detect in the video images a moving target area where a moving object locates,

a feature point extracting unit configured to extract at least one feature point on the moving object in the detected moving target area,

a moving trajectory obtaining unit configured to track a movement of the feature point in a predetermined number of video image frames to obtain a movement trajectory of the feature point, and

a disappearing direction determining unit configured to determine, according to the movement trajectories of one or more moving objects in the video images, a disappearing direction pointed by a major moving direction of the moving objects; and

the video camera calibration device is configured to determine a gesture of the video camera relative to the ground by using the disappearing direction calculated by the device for determining a disappearing direction.

4. A method of determining a disappearing direction, comprising:

detecting in a plurality of video images a moving target area where a moving object locates;

extracting at least one feature point on the moving object in the detected moving target area;

tracking a movement of the feature point in a predetermined number of video image frames to obtain a movement trajectory of the feature point; and

according to the movement trajectories of one or more moving objects in the video images, determining a disappearing direction pointed by a major moving direction of the moving objects.

5. A device for determining a disappearing direction, comprising:

an edge-detection unit configured to detect a plurality of edges of one or more objects in a video image to obtain corresponding line segments on the objects;

an initial estimating unit configured to group the line segments into a plurality of line segment groups by a predetermined rule, to initially estimate the disappearing direction according to the plurality of line segment groups and obtain an initial estimation of the disappearing direction;

a closest line segment selecting unit configured to select at least one line segment which is closest to the initial estimation from a predetermined number of line segment groups among all the line segment groups; and

a disappearing direction obtaining unit configured to determine the disappearing direction using the selected line segment.

6. The device for determining a disappearing direction according to claim 5, wherein, the predetermined rule is:

the line segments of the same object in the video image are grouped into same line segment group.

7. The device for determining a disappearing direction according to claim 5, wherein the initial estimation unit configured to initially estimate the disappearing direction by using a second disappearing direction perpendicular to the disappearing direction to be estimated.

8. The device for determining a disappearing direction according to claim 7, wherein the initial estimation unit comprises:

a group combining unit configured to combine the groups into one or more combinations, each combination including at least two groups.

9. The device for determining a disappearing direction according to claim 8, wherein, each combination is consisted of a first group and a second group, the initial estimation unit further comprising:

a vertical restraint calculating unit configured to, for each combination in a predetermined number of combinations in all combinations, calculate an outer product of edge plane normal vectors of every two line segments respectively of the first group and the second group, and seek an inner product of the calculated outer product and the second disappearing direction;

an initial value determining unit configured to compare all inner products calculated by the vertical restraint calculating unit for the predetermined number of combinations with a first threshold, and determine a direction of an edge plane normal vector corresponding to any inner product smaller than the first threshold as an initial value of the initial estimation of the disappearing direction.

10. The device for determining a disappearing direction according to claim 9, wherein, the initial estimation unit further comprises:

an initial estimation determining unit configured to select, from the initial values determined by the initial value determining unit, an initial values with a maximum number of support groups as a final value of the initial estimation of the disappearing direction, wherein, for each support group, a minimum inner product in all the inner products calculated with the initial value and edge plane normal vectors of all line segments in the support group, is smaller than a second threshold.

11. The device for determining a disappearing direction according to claim 10, wherein, when there are more than one initial values with the same maximum number of support groups, the initial estimation determining unit selects the initial value which makes a target function smallest as the final value of the initial estimation, wherein, the target function includes a measurement of a perpendicular degree between the initial value and the second disappearing direction, and a measurement of a parallel degree between the initial value and line segments most parallel to the initial value in other support groups.

12. The device for determining a disappearing direction according to claim 11, wherein, the target function is:

13. A video camera, comprising:

an apparatus for video camera calibration configured to calibrate the video camera based on a video image shot by the video camera, the apparatus comprising a device for determining a disappearing direction and a video camera calibration device,

wherein the device for determining a disappearing direction comprises;

an edge-detection unit configured to detect a plurality of edges of one or more objects in the video image shot by the video camera to obtain corresponding line segments on the object,

an initial estimating unit configured to group the line segments into a plurality of line segment groups by a predetermined rule, to initially estimate a disappearing direction according to the grouped line segments and obtain an initial estimation of the disappearing direction,

a closest line segment selecting unit configured to select at least one line segment which is closest to the initial estimation from a predetermined number of line segment groups among all the line segment groups, and

a disappearing direction obtaining unit configured to determine the disappearing direction using the selected line segment; and

the video camera calibration unit is configured to determine a gesture of the video camera relative to the ground by using the disappearing direction calculated by the device for determining a disappearing direction.

14. A method of determining a disappearing direction, comprising:

detecting a plurality of edges of one or more objects in a video image to obtain corresponding line segments on the objects;

grouping the line segments into a plurality of line segment groups by a predetermined rule, and initially estimating the disappearing direction according to the plurality of line segment groups to obtain an initial estimation of the disappearing direction;

selecting at least one line segment which is closest to the initial estimation from a predetermined number of line segment groups among all the line segment groups; and

determining the disappearing direction by using the selected line segments.

15. A computer program product, comprising program codes which, when executed by a processor of a computer, cause the processor to perform the following steps of:

detecting in a plurality of input video images a moving target area where a moving object locates;

extracting at least one feature point on the moving object in the detected moving target area;

tracking a movement of the feature point in a predetermined number of video image frames to obtain a movement trajectory of the feature point; and

according to the movement trajectories of one or more moving objects in the video images, determining a disappearing direction pointed by a major moving direction of the moving objects.

16. A computer program product, comprising program codes which, when executed by a processor of a computer, cause the processor to perform the following steps of:

detecting a plurality of edges of one or more objects in an input video image to obtain corresponding line segments on the objects;

grouping the line segments into a plurality of line segment groups by a predetermined rule, and initially estimating the disappearing direction according to the plurality of line segment groups to obtain an initial estimation of the disappearing direction;

selecting at least one line segment which is closest to the initial estimation from a predetermined number of line segment groups among all the line segment groups; and

determining a disappearing direction by using the selected line segments.