Operation Support System, Vehicle, And Method For Estimating Three-Dimensional Object Area

• US 20100246901A1
• Filed: 09/24/2008
• Published: 09/30/2010
• Est. Priority Date: 11/20/2007
• Status: Abandoned Application

• Current Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Yang, Changhui
• Application Number:
  US12/743,383
• Publication Number:
  US 20100246901A1
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

Camera images at times (t1 and t2) are acquired from a camera installed on a vehicle, and the camera images are converted to bird'"'"'s-eye view images at the times (t1 and t2). A plurality of feature points are extracted from the camera image at the time (t1) and the movement vectors of the respective feature points between first and second camera images are detected. Then, the respective feature points and the respective movement vectors are mapped onto a bird'"'"'s-eye image coordinate plane. Two or more feature points on the bird'"'"'s-eye view images are permitted to be targets, the positions and the movement vectors of the target feature points are applied to an equation of constraint which ground surface feature points have to satisfy, thereby discriminating whether the target feature points are the ground surface feature points. Then, from position information and movement vector information on the two or more feature points discriminated to be the ground surface feature points, movement information on the vehicle is obtained and used for taking the difference between the bird'"'"'s-eye images at the times (t1 and t2) to estimate a three-dimensional object area.

Claims

1. An operation support system which is provided with a camera fitted to a mobile object to shoot surroundings of the mobile object, and which estimates, based on camera images on a camera coordinate plane obtained from the camera, a three-dimensional object area in an image based on the camera image, the operation support system comprising:

an image acquisition portion which acquires first and second camera images shot by the camera at first and second time points, respectively, while the mobile object is moving, the first and second time points being different from each other;

a movement vector detection portion which extracts n feature points (where n is an integer of 2 or more) from the first camera image, and which also detects movement vectors, on the camera coordinate plane, of the feature points between the first and second camera images;

a bird'"'"'s eye conversion portion which projects the camera images, and the feature points and the movement vectors on the camera coordinate plane onto a bird'"'"'s eye view coordinate plane which is parallel to ground to thereby convert the first and second camera images into first and second bird'"'"'s eye view images, respectively, and detect positions of the feature points on the first bird'"'"'s eye view image and movement vectors of the feature points on the bird'"'"'s eye view coordinate plane between the first and second bird'"'"'s eye view images;

a determination portion which determines, by use of a restraint condition for a ground feature point located on the ground to satisfy, whether or not a target feature point on the first bird'"'"'s eye view image is the ground feature point;

a movement information estimation portion which estimates movement information of the mobile object between the first and second time points based on positions on the first bird'"'"'s eye view image, and movement vectors on the bird'"'"'s eye view coordinate plane, of two or more feature points which are each judged as the ground feature point; and

a three-dimensional object area estimation portion which estimates the three-dimensional object area based on the first and second bird'"'"'s eye view images and the movement information.

2. The operation support system of claim 1, wherein the restraint condition defines a relationship which should be satisfied by a rotation angle and a parallel movement amount of the mobile object between the first and second time points and a position on the first bird'"'"'s eye view image, and a movement vector on the bird'"'"'s eye view coordinate plane, of the ground feature point.

3. The operation support system of claim 2, wherein the determination portion extracts, as target feature points, two or more feature points from among the n feature points on the first bird'"'"'s eye view image, and determines whether or not the target feature points are each the ground feature point by determining whether or not the target feature points satisfy the restraint condition.

4. The operation support system of claim 2, wherein the determination portion

extracts, as target feature points, two or more feature points from among the n feature points on the first bird'"'"'s eye view image,

obtains two or more estimation values of the rotation angle and two or more estimation values of the parallel movement amount by applying the two or more target feature points to the relationship, on an assumption that the two or more target feature points are each the ground feature point, and

determines whether or not the target feature points are each the ground feature point, based on a variation among the estimation values of the rotation angle and a variation among the estimation values of the parallel movement amount.

5. The operation support system of claim 2, wherein the movement information includes information which indicates the rotation angle and the parallel movement amount of the mobile object.

6. The operation support system of claim 1, wherein

the three-dimensional object area estimation portion

corrects, based on the movement information, displacement between the first and second bird'"'"'s eye view images attributable to the mobile object moving between the first and second time points, and

estimates the three-dimensional object area based on a comparison result between the first and second bird'"'"'s eye view images after the displacement is corrected.

7. The operation support system of claim 1, wherein the three-dimensional object area which is estimated corresponds to an area where a three-dimensional object appears in the first camera image, in the second camera image, in the first bird'"'"'s eye view image, or in the second bird'"'"'s eye view image.

8. A vehicle as a mobile object, comprising the operation support system of claim 1.

10. The operation support system of claim 3, wherein the movement information includes information which indicates the rotation angle and the parallel movement amount of the mobile object.

11. The operation support system of claim 4, wherein the movement information includes information which indicates the rotation angle and the parallel movement amount of the mobile object.

12. The operation support system of claim 2, wherein

the three-dimensional object area estimation portion

corrects, based on the movement information, displacement between the first and second bird'"'"'s eye view images attributable to the mobile object moving between the first and second time points, and

estimates the three-dimensional object area based on a comparison result between the first and second bird'"'"'s eye view images after the displacement is corrected.

13. The operation support system of claim 3, wherein

the three-dimensional object area estimation portion

corrects, based on the movement information, displacement between the first and second bird'"'"'s eye view images attributable to the mobile object moving between the first and second time points, and

estimates the three-dimensional object area based on a comparison result between the first and second bird'"'"'s eye view images after the displacement is corrected.

14. The operation support system of claim 4, wherein

the three-dimensional object area estimation portion

corrects, based on the movement information, displacement between the first and second bird'"'"'s eye view images attributable to the mobile object moving between the first and second time points, and

estimates the three-dimensional object area based on a comparison result between the first and second bird'"'"'s eye view images after the displacement is corrected.

15. The operation support system of claim 2, wherein the three-dimensional object area which is estimated corresponds to an area where a three-dimensional object appears in the first camera image, in the second camera image, in the first bird'"'"'s eye view image, or in the second bird'"'"'s eye view image.

16. The operation support system of claim 3, wherein the three-dimensional object area which is estimated corresponds to an area where a three-dimensional object appears in the first camera image, in the second camera image, in the first bird'"'"'s eye view image, or in the second bird'"'"'s eye view image.

17. The operation support system of claim 4, wherein the three-dimensional object area which is estimated corresponds to an area where a three-dimensional object appears in the first camera image, in the second camera image, in the first bird'"'"'s eye view image, or in the second bird'"'"'s eye view image.

18. A vehicle as a mobile object, comprising the operation support system of claim 2.

19. A vehicle as a mobile object, comprising the operation support system of claim 3.

20. A vehicle as a mobile object, comprising the operation support system of claim 4.

9. A three-dimensional object area estimation method for estimating, based on a camera image on a camera coordinate plane obtained from a camera fitted to a mobile object to shoot surroundings of the mobile object, a three-dimensional object area in an image based on the camera image, the three-dimensional object area estimation method comprising:

an image acquisition step for acquiring first and second camera images shot by the camera at first and second time points, respectively, while the mobile object is moving, the first and second time points being different from each other;

a movement vector detection step for extracting n feature points (where n is an integer of 2 or more) from the first camera image, and detecting movement vectors of the feature points on the camera coordinate plane between the first and second camera images;

a bird'"'"'s eye conversion step for projecting the camera images, and the feature points and the movement vectors on the camera coordinate plane onto a bird'"'"'s eye view coordinate plane which is parallel to ground, to thereby convert the first and second camera images into first and second bird'"'"'s eye view images, respectively, and detect positions of the feature points on the first bird'"'"'s eye view image and movement vectors of the feature points on the bird'"'"'s eye view coordinate plane between the first and second bird'"'"'s eye view images;

a determination step for determining, by use of a restraint condition for a ground feature point located on the ground to satisfy, whether or not a target feature point on the first bird'"'"'s eye view image is the ground feature point;

a movement information estimation step for estimating movement information of the mobile object between the first and second time points based on positions on the first bird'"'"'s eye view image, and movement vectors on the bird'"'"'s eye view coordinate plane, of two or more feature points which are each judged as the ground feature point; and

a three-dimensional object area estimation step for estimating the three-dimensional object area based on the first and second bird'"'"'s eye view images and the movement information.