Method and apparatus for environment recognition

• US 6,535,114 B1
• Filed: 03/22/2000
• Issued: 03/18/2003
• Est. Priority Date: 03/22/2000
• Status: Expired due to Term

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Suzuki, Toshihiko, Kanade, Takeo
• Primary Examiner(s): Hofsass, Jeffrey
• Assistant Examiner(s): Goins, Davetta W.
• Application Number:
  US09/533,167
• Time in Patent Office: 1,091 Days
• Field of Search: 340/435, 340/937, 340/903, 340/901, 340/436, 318/480, 348/148, 348/162, 348/164, 348/149, 348/150, 701/25, 701/300, 701/301
• US Class Current: 340/435
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Images are captured using a single camera in motion. A recognition process section detects a possible object in a photographed image, tracks the possible object within the moving image, and generates object shape information from the trajectory information. A motion and pose determination section determines camera motion and pose using the photographed images for recognition processing. The determined data are used for object recognition along with the tracking information. The motion and pose determination section converts the optical flow parameters obtained from the photographed images into physical parameters in three-dimensional space. An Extended Kalman Filter is used for both the determination of motion and pose and for object recognition.

Claims

1. An environment recognition method for recognizing, through image processing, an object captured in images, comprising:

a step of capturing a sequence of images using a single moving camera; and

a step of recognition processing including identifying a possible object captured in an image, tracking the identified possible object within the image sequence, and generating three-dimensional information on the object based on a magnitude of a positional change within an image frame of the possible object, wherein, in the recognition processing step, flow parameters of the sequence of images in image coordinates are converted into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.

2. The environment recognition method defined in claim 1 wherein:

the camera is mounted on a vehicle;

the possible object is a possible obstacle on the road; and

the three-dimensional information on the object includes the height of the possible object.

3. The environment recognition method defined in claim 1 further comprising:

a step of determining camera motion during the tracking;

wherein

in the recognition processing step, the recognition processing is performed by relating the camera motion to the tracking information.

4. The environment recognition method defined in claim 3 wherein:

the motion determining step further includes determining camera pose during the tracking; and

in the recognition processing step, the recognition processing is performed by relating the camera motion and the camera pose to the tracking information.

5. The environment recognition method defined in claim 3 wherein

in the motion determining step, the instantaneous camera motion at a point along the camera'"'"'s movement is determined from the image sequence.

6. The environment recognition method defined in claim 5 wherein

in the motion determining step, the camera motion is determined using a detection signal from a motion sensor in addition to the image sequence.

7. The environment recognition method defined in claim 5 wherein

in the motion determining step, the camera motion is determined by converting flow parameters of the image sequence in image coordinates into physical parameters in three-dimensional coordinates.

8. The environment recognition method defined in claim 7 wherein

a Kalman Filter capable of non-linear processing is used in the motion determining step.

9. The environment recognition method defined in claim 3 further comprising:

a step of determining the gradient of a surface on which the camera moves; and

in the recognition processing step, the recognition processing is performed by relating the camera motion and the gradient to the tracking information.

10. The environment recognition method defined in claim 9 wherein

in the gradient determining step, the gradient information is generated based on a difference between an estimated pitch angle estimated from the image sequence as the camera motion and a detected pitch angle detected using a sensor.

11. The environment recognition method defined in claim 9 wherein

a Kalman Filter capable of non-linear processing is used in the gradient determining step.

12. The environment recognition method defined in claim 1 wherein

a Kalman Filter capable of non-linear processing is used in the recognition processing step.

13. The environment recognition method defined in claim 12 wherein

a plurality of Kalman Filters are used to simultaneously perform the recognition processing for a plurality of possible objects by activating at least one Kalman Filter for a new possible object when such a new possible object is detected.

14. The environment recognition method defined in claim 1 further comprising

a step of dividing each of the images into a plurality of sections and generating information concerning irregularities in the region based on results of the recognition processing for the respective divided sections.

15. An environment recognition apparatus for recognizing, through image processing, an object captured in camera images photographing a surrounding region, comprising:

a camera mounted on a moving structure for obtaining images of at least a portion of a surrounding region; and

a recognition process unit for generating three-dimensional information concerning an object captured in a sequence of images photographed by the camera;

wherein

the recognition process unit identifies a possible object captured in an image, tracks the identified possible object within the image sequence, and generates the three-dimensional information on the object based on a magnitude of a positional change within an image frame of the possible object, wherein, in the recognition processing unit, flow parameters of the sequence of images in image coordinates are converted into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.

16. The environment recognition apparatus defined in claim 15 wherein

the recognition process unit further includes means for determining the instantaneous motion of the camera based on the photographed image sequence, and performs recognition-processing by relating the camera motion to the tracking information.

17. The environment recognition apparatus defined in claim 16 wherein

the recognition process unit further includes means for determining camera pose along with the camera motion, and performs the recognition processing by relating the camera motion and the camera pose to the tracking information.

18. The environment recognition apparatus defined in claim 17 wherein:

the recognition process unit includes a Kalman Filter capable of non-linear processing; and

the Kalman Filter calculates the camera motion by converting flow parameters of the image sequence in image coordinates into physical parameters in three-dimensional coordinates.

19. The environment recognition apparatus defined in claim 16 wherein

the recognition process unit further includes means for determining gradient of a surface on which the camera is moving, and performs recognition processing by relating the camera motion and the gradient to the tracking information.

20. The environment recognition apparatus defined in claim 19 comprising:

a pitch sensor for detecting a pitch angle of the camera;

wherein

the gradient determining means include a Kalman Filter capable of non-linear processing, and the Kalman Filter generates gradient information based on a difference between an estimated pitch angle estimated from the image sequence and a detected pitch angle detected using the sensor.

21. The environment recognition apparatus defined in claim 16 wherein

the recognition process unit includes a Kalman Filter capable of non-linear processing used for the recognition processing, and performs accumulation of the camera motion and object recognition based on the accumulated information using the Kalman Filter for the recognition processing.

22. The environment recognition apparatus defined in claim 21 wherein

the recognition process unit includes a plurality of Kalman Filter processors for the recognition processing, and assigns at least one Kalman Filter to each of successively identified new possible objects.

23. The environment recognition apparatus defined in claim 15 wherein

the recognition process unit divides each of the images into a plurality of sections, performs the recognition processing for the respective divided sections, and generates information concerning unevenness between the sections.

24. An environment recognition method for recognizing, through image processing, a photographed object captured in camera images, comprising:

a step of obtaining a sequence of images using a single moving camera; and

a step of recognition processing including tracking the photographed object within the image sequence, and generating three-dimensional information on the photographed object based on a magnitude of a positional change within an image frame of the photographed object, wherein, in the recognition processing step, flow parameters of the sequence of images in image coordinates are converted into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.

25. The environment recognition method defined in claim 24, further comprising:

a step of determining the motion of photographing side based on the image sequence;

wherein

in the recognition processing step, the recognition processing is performed by relating the motion to the tracking information.

26. The environment recognition method defined in claim 24, further comprising:

a step of determining motion and pose of the photographing side based on the image sequence, wherein

in the recognition processing step, the recognition processing is performed by relating the motion and the pose to the tracking information.

27. A method for determining motion of a camera or a moving structure on which the camera is mounted, comprising the steps of:

capturing a plurality of images using a single moving camera; and

determining the motion using the plurality of images;

wherein

in the motion determining step, flow parameters of the plurality of images in image coordinates are converted into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.

28. An apparatus for determining motion of a camera or a moving structure on which the camera is mounted, comprising:

a camera mounted on a moving structure for obtaining images of at least a portion of a surrounding region of the moving structure while the moving structure moves; and

means for determining the motion using the plurality of images;

wherein

the motion determining means convert flow parameters of the plurality of images in image coordinates into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.

29. The motion determination apparatus defined in claim 28 wherein

a detection signal from a motion sensor is used in addition to the plurality of images to determine the motion.

30. The motion determination apparatus defined in claim 28 wherein

the motion determining means include a Kalman Filter capable of non-linear processing.

31. A method for determining gradient of a surface on which a camera moves using images photographed by the camera, comprising the steps of:

capturing a plurality of images using a single moving camera;

estimating a pitch angle of the camera or of a moving structure on which the camera is mounted by converting flow parameters obtained from the plurality of images into physical parameters in three-dimensional coordinates;

detecting pitch angle information of the camera or the moving structure on which the camera is mounted using a pitch sensor; and

determining the gradient of a surface on which the camera is moving based on a difference between the estimated pitch angle and the detected pitch angle.

32. A map creation method for creating a three-dimensional map comprising a step of capturing a sequence of images using a single moving camera;

and a step of recognition processing which includes identifying a possible object captured in an image, tracking the identified possible object within the image sequence, and generating three-dimensional information on the object based on information obtained by the tracking concerning changes in images of the possible object, wherein in the recognition processing step flow parameters of the sequence of images in image coordinates are converted into physical parameters in three-dimensional coordinates, the physical parameters indicating motion information of the camera side.