Image processing apparatus, image processing method, and recording medium

• US 20100134325A1
• Filed: 09/09/2009
• Published: 06/03/2010
• Est. Priority Date: 11/28/2008
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Yamada, Hiroshi, Shimizu, Seiya, Gomi, Toshiaki
• Granted Patent: US 8,319,618 B2
• Time in Patent Office: Days
• US Class Current: 340/995.140
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

An image processing apparatus, includes: a state information obtaining unit obtaining state information that indicates a state of driving operation of the vehicle; an image storage unit storing a plurality of two-dimensional image data pieces obtained from the plurality of imaging devices whose viewing fields partly overlap with each other; a specifying unit compositing a plurality of two-dimensional image data pieces into a shape of a three-dimensional projection plane, and then specifying, in correspondence to the state information, a compositing method for overlapping portions of viewing fields to be used for transformation from a three-dimensional image into projection image data viewed from a position of a given viewpoint along a given line-of-sight direction; and an image transformation unit transforming the plurality of two-dimensional image data pieces into projection image data viewed from the position of the given viewpoint along the given line-of-sight direction, by using the compositing method.

Claims

1. An image processing apparatus processing image data obtained by surroundings image of a vehicle through a plurality of imaging devices mounted on the vehicle, comprising:

a state information obtaining unit obtaining state information that indicates a state of driving operation of the vehicle;

an image storage unit storing a plurality of two-dimensional image data pieces obtained from the plurality of imaging devices whose viewing fields partly overlap with each other;

a specifying unit compositing a plurality of two-dimensional image data pieces into a shape of a three-dimensional projection plane when the state information obtaining unit has obtained given state information, and then specifying, in correspondence to the state information, a compositing method for overlapping portions of viewing fields or transformation information to be used for transformation from a three-dimensional image based on the obtained three-dimensional image data into projection image data viewed from a position of a given viewpoint along a given line-of-sight direction; and

an image transformation unit transforming the plurality of two-dimensional image data pieces stored in the image storage unit into projection image data viewed from the position of the given viewpoint along the given line-of-sight direction, by using the compositing method or the transformation information specified by the specifying unit.

3. The image processing apparatus according to claim 1, wherein

the specifying unit includes information concerning a compositing position that indicates a separation which relates to two-dimensional image data adopted in the overlap range of the plurality of two-dimensional image data pieces in the three-dimensional image data to be used when the plurality of two-dimensional image data pieces are composited into a shape of a three-dimensional projection plane or information that indicates a position of the viewpoint and a line-of-sight direction.

5. The image processing apparatus according to claim 1, further comprising:

a location obtaining unit obtaining location information that indicates a location; and

a map storage unit storing map data and characteristics information that indicates characteristics of each place in the map data, wherein

the state information obtaining unit obtaining from the map as the state information storage unit the characteristics information of the location indicated by the location information obtained by the location obtaining unit, and

the specifying unit specifies compositing method or transformation information in accordance with the characteristics information obtained by the state information obtaining unit.

7. The image processing apparatus according to claim 3, further comprising:

a location obtaining unit obtaining location information that indicates a location; and

a map storage unit storing map data and characteristics information that indicates characteristics of each place in the map data, wherein

the state information obtaining unit obtains from the map storage unit as the state information the characteristics information of the location indicated by the location information obtained by the location obtaining unit, and

the specifying unit specifies compositing method or transformation information in accordance with the characteristics information obtained by the state information obtaining unit.

9. The image processing apparatus according to claim 5, wherein

the characteristics information contains information that indicates which place is a highway or information that indicates which place is an urban area.

10. The image processing apparatus according to claim 1, wherein

the state information obtaining unit obtains, as the state information, running state information that indicates a running state of the vehicle, and

the specifying unit specifies compositing method or transformation information in accordance with the running state information obtained by the state information obtaining unit.

12. The image processing apparatus according to claim 1, wherein

the state information obtaining unit obtains, as the state information, surroundings state information that indicates a surroundings state of the vehicle, and

the specifying unit specifies compositing method or transformation information in accordance with the surroundings state information obtained by the state information obtaining unit.

14. The image processing apparatus according to claim 12, wherein

the surroundings state information contains information that indicates an obstacle or a moving object located in surroundings of the vehicle or information that indicates a shape of a road where the vehicle runs.

15. The image processing apparatus according to claim 1, wherein

the three-dimensional image data contains vehicle image data that indicates the vehicle, plane image data that indicates a plane extending from a rectangular arrangement plane in which the vehicle is arranged, and curved surface image data that indicates a curved surface extending from the plane,

the plane image data contains:

rectangular extended surface data that indicates an extended surface of approximately rectangular shape which is extended from each side of the arrangement plane and in which a plurality of polygons are arranged along the extension direction; and

sector extended surface data that indicates an extended surface of approximately sector shape which is extended from each vertex of the arrangement plane and in which a plurality of polygons are arranged radially, and

the specifying unit specifies information concerning a straight line having an end at each vertex of the arrangement plane in the extended surface of approximately sector shape, as information concerning a compositing position that indicates a separation which relates two-dimensional image data adopted in the overlap range of the plurality of two-dimensional image data pieces in the three-dimensional image data to be used when the plurality of two-dimensional image data pieces are composited into a shape of a three-dimensional projection plane.

17. The image processing apparatus according to claim 1, further comprising:

an image output unit outputting to a display device the projection image data obtained by the transformation in the image transformation unit.

2. An image processing apparatus processing image data obtained by surroundings image of a vehicle through a plurality of imaging devices mounted on the vehicle, comprising:

a state information obtaining unit obtaining state information that indicates a state of driving operation of the vehicle, from the vehicle at regular intervals or alternatively at occasions of a change;

an image storage unit storing a plurality of two-dimensional image data pieces obtained from the plurality of imaging devices whose viewing fields partly overlap with each other;

a specifying unit compositing a plurality of two-dimensional image data pieces stored in the image storage unit into a shape of a three-dimensional projection plane, and then specifying, in correspondence to the state information obtained by the state information obtaining unit, a compositing method for overlapping portions of viewing fields or the transformation information to be used for transformation from a three-dimensional image based on the obtained three-dimensional image data into projection image data viewed from a position of a given viewpoint along a given line-of-sight direction; and

an image transformation unit transforming the plurality of two-dimensional image data pieces stored in the image storage unit into projection image data viewed from the position of the given viewpoint along the given line-of-sight direction, by using the compositing method or the transformation information specified by the specifying unit.

4. The image processing apparatus according to claim 2, wherein

the specifying unit includes information concerning a compositing position that indicates a separation which relates to two-dimensional image data adopted in the overlap range of the plurality of two-dimensional image data pieces in the three-dimensional image data to be used when the plurality of two-dimensional image data pieces are composited into a shape of a three-dimensional projection plane or information that indicates a position of the viewpoint and a line-of-sight direction.

6. The image processing apparatus according to claim 2, further comprising:

a location obtaining unit obtaining location information that indicates a location; and

a map storage unit storing map data and characteristics information that indicates characteristics of each place in the map data, wherein

the state information obtaining unit obtains from the map storage unit as the state information the characteristics information of the location indicated by the location information obtained by the location obtaining unit, and

the specifying unit specifies compositing method or transformation information in accordance with the characteristics information obtained by the state information obtaining unit.

8. The image processing apparatus according to claim 4, further comprising:

a location obtaining unit obtaining location information that indicates a location; and

a map storage unit storing map data and characteristics information that indicates characteristics of each place in the map data, wherein

the state information obtaining unit obtains from the map storage unit as the state information the characteristics information of the location indicated by the location information obtained by the location obtaining unit, and

the specifying unit specifies compositing method or transformation information in accordance with the characteristics information obtained by the state information obtaining unit.

11. The image processing apparatus according to claim 2, wherein

the state information obtaining unit obtains, as the state information, running state information that indicates a running state of the vehicle, and

the specifying unit specifies compositing method or transformation information in accordance with the running state information obtained by the state information obtaining unit.

13. The image processing apparatus according to claim 2, wherein

the state information obtaining unit obtains, as the state information, surroundings state information that indicates a surroundings state of the vehicle, and

the specifying unit specifies compositing method or transformation information in accordance with the surroundings state information obtained by the state information obtaining unit.

16. The image processing apparatus according to claim 2, wherein

the three-dimensional image data contains vehicle image data that indicates the vehicle, plane image data that indicates a plane extending from a rectangular arrangement plane in which the vehicle is arranged, and curved surface image data that indicates a curved surface extending from the plane,

the plane image data contains:

rectangular extended surface data that indicates an extended surface of approximately rectangular shape which is extended from each side of the arrangement plane and in which a plurality of polygons are arranged along the extension direction; and

sector extended surface data that indicates an extended surface of approximately sector shape which is extended from each vertex of the arrangement plane and in which a plurality of polygons are arranged radially, and

the specifying unit specifies information concerning a straight line having an end at each vertex of the arrangement plane in the extended surface of approximately sector shape, as information concerning a compositing position that indicates a separation which relates to two-dimensional image data adopted in the overlap range of the plurality of two-dimensional image data pieces in the three-dimensional image data to be used when the plurality of two-dimensional image data pieces are composited into a shape of a three-dimensional projection plane.

18. The image processing apparatus according to claim 2, further comprising:

an image output unit outputting to a display device the projection image data obtained by the transformation in the image transformation unit.

19. An image processing method performed with surroundings image of a vehicle through a plurality of imaging devices mounted on the vehicle, comprising:

obtaining state information that indicates a state of driving operation of the vehicle;

compositing a plurality of two-dimensional image data pieces obtained from the plurality of imaging devices whose viewing fields partly overlap with each other into a shape of a three-dimensional projection plane when given state information is obtained;

specifying, in correspondence to the state information, a compositing method for overlapping portions of viewing field or transformation information to be used for transformation from three-dimensional image based on the obtained three-dimensional image data into projection image data viewed from a position of a given viewpoint along a given line-of-sight direction; and

transforming, by using the transformation information or the compositing method having been specified, the plurality of two-dimensional image data pieces into projection image data viewed from the position of the given viewpoint along the given line-of-sight direction.

20. A recording medium capable of being read by a computer in which program for making the computer execute processing image data obtained by surroundings image of a vehicle through a plurality of imaging devices mounted on the vehicle, comprising:

obtaining state information that indicates a state of driving operation of the vehicle;

compositing a plurality of two-dimensional image data pieces obtained from the plurality of imaging devices whose viewing fields partly overlap with each other into a shape of a three-dimensional projection plane when given state information is obtained;

specifying, in correspondence to the state information, a compositing method for overlapping portions of viewing fields or transformation information to be used for transformation from three-dimensional image based on the obtained three-dimensional image data into projection image data viewed from a position of a given viewpoint along a given line-of-sight direction; and

transforming, by using the compositing method or the transformation information having been specified, the plurality of two-dimensional image data pieces into projection image data viewed from the position of the given viewpoint along the given line-of-sight direction.