ELECTRONIC CONTROL SYSTEM, ELECTRONIC CONTROL UNIT AND ASSOCIATED METHODOLOGY OF ADAPTING 3D PANORAMIC VIEWS OF VEHICLE SURROUNDINGS BY PREDICTING DRIVER INTENT

• US 20110074916A1
• Filed: 09/29/2009
• Published: 03/31/2011
• Est. Priority Date: 09/29/2009
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: DEMIRDJIAN, David
• Granted Patent: US 8,502,860 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/103 using camera systems provided with artificial illumination device, e.g. IR light source
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An electronic control system, electronic control unit and an associated methodology for adapting three dimensional panoramic views of vehicle surroundings by predicting driver intent are provided. A plurality of cameras mounted on a vehicle generate images of a surrounding area of the vehicle. A visual sensor detects a three dimensional profile of an occupant of the vehicle. An electronic control unit generates a three dimensional panoramic view based on the images generated by the plurality of cameras, determines three dimensional locations and orientations of a plurality of body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor, and adapts the three dimensional panoramic view based on the determined three dimensional locations and orientations. A display unit displays the adapted three dimensional panoramic view.

Claims

1. An electronic control system for adapting three dimensional panoramic views of vehicle surroundings by predicting driver intent, comprising:

a plurality of cameras mounted on a vehicle that generate images of a surrounding area of the vehicle;

a visual sensor that detects a three dimensional profile of an occupant of the vehicle;

an electronic control unit that is configured to generate a three dimensional panoramic view based on the images generated by the plurality of cameras, that is configured to determine three dimensional locations and orientations of a plurality of body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor, and that is configured to adapt the three dimensional panoramic view based on the determined three dimensional locations and orientations; and

a display unit that displays the adapted three dimensional panoramic view.

2. The electronic control system according to claim 1,

wherein said electronic control unit is configured to determine a velocity or acceleration of at least one of the plurality of body parts of the occupant of the vehicle based on the three dimensional profile of the occupant of the vehicle detected by the visual sensor, and

wherein said electronic control unit is configured to adapt the three dimensional panoramic view based on the determined velocity or acceleration of the at least one of the plurality of body parts of the occupant of the vehicle.

3. The electronic control system according to claim 1, wherein the electronic control unit is configured to predict the vehicle occupant'"'"'s intentions relating to operation of the vehicle based on the determined three dimensional locations and orientations, and the electronic control unit is configured to adapt the three dimensional panoramic view based on the predicted vehicle occupant'"'"'s intentions.

4. The electronic control system according to claim 3, wherein the electronic control unit is configured to assess potential risks that relate to the vehicle'"'"'s proximity to other objects, and the electronic control unit is configured to adapt the three dimensional panoramic view based on the assessed potential risks.

5. The electronic control system according to claim 4, wherein the electronic control unit is configured to determine a potential risk exists by performing object recognition on the images generated by the plurality of camera.

6. The electronic control system according to claim 1, wherein the electronic control unit is configured to generate the three dimensional panoramic view by

creating a two dimensional panoramic reconstruction by stitching together views from each of the plurality of cameras,

generating a three dimensional panoramic image by projecting the two dimensional panoramic reconstruction on a three dimensional shape, and

generating the three dimensional panoramic view based on a virtual perspective viewpoint of the three dimensional panoramic image.

7. The electronic control system according to claim 6, wherein the electronic control unit is configured to adapt the three dimensional panoramic view by changing the virtual perspective viewpoint.

8. The electronic control system according to claim 6, wherein the electronic control unit is configured to adapt the three dimensional panoramic view by changing the three dimensional shape on which the two dimensional panoramic reconstruction is projected.

9. The electronic control system according to claim 1, further comprising a plurality of markers mounted on the vehicle body,

wherein the electronic control unit is configured to calibrate the plurality of cameras by detecting the markers in the images generated by the plurality of cameras.

10. The electronic control system according to claim 1, wherein the visual sensor is a three dimensional time of flight camera.

11. A method for adapting three dimensional panoramic views of vehicle surroundings by predicting driver intent, comprising:

generating images of a surrounding area of a vehicle with a plurality of cameras mounted on the vehicle;

detecting a three dimensional profile of an occupant of the vehicle with a visual sensor mounted on the vehicle;

generating a three dimensional panoramic view based on the images generated by the plurality of cameras;

determining three dimensional locations and orientations of a plurality of body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor;

adapting the three dimensional panoramic view based on the determined three dimensional locations and orientations; and

displaying the adapted three dimensional panoramic view.

12. The method according to claim 11, further comprising predicting the vehicle occupant'"'"'s intentions relating to operation of the vehicle based on the determined three dimensional locations and orientations,

wherein the adapting the three dimensional panoramic view includes adapting the three dimensional panoramic view based on the predicted vehicle occupant'"'"'s intentions.

13. The electronic control system according to claim 11, wherein the generating the three dimensional panoramic view includes

creating a two dimensional panoramic reconstruction by stitching together views from each of the plurality of cameras,

generating a three dimensional panoramic image by projecting the two dimensional panoramic reconstruction on a three dimensional shape, and

generating the three dimensional panoramic view based on a virtual perspective viewpoint of the three dimensional panoramic image.

14. The electronic control system according to claim 11, further comprising calibrating the plurality of cameras by detecting a plurality of markers in the images generated by the plurality of cameras.

15. The electronic control system according to claim 12, wherein calibrating the cameras includes detecting one of the plurality of markers in two different images that are respectively generated by two different cameras of the plurality of cameras.

16. An electronic control unit for adapting three dimensional panoramic views of vehicle surroundings by predicting driver intent, comprising:

a data acquisition interface configured to receive a plurality of images of a surrounding area of the vehicle and a three dimensional profile of an occupant of a vehicle;

a data processor configured to generate a three dimensional panoramic view based on the plurality of images, that is configured to determine three dimensional locations and orientations of a plurality of body parts of an occupant of a vehicle based on the three dimensional profile, and that is configured to adapt the three dimensional panoramic view based on the determined three dimensional locations and orientations; and

a display interface configured to send display information corresponding to the adapted three dimensional panoramic view.

17. An electronic control system for calibrating cameras used to generate three dimensional panoramic views of vehicle surroundings, comprising:

a plurality of cameras mounted on a vehicle that generate images of a surrounding area of the vehicle;

a plurality of calibration markers mounted on a body of the vehicle;

an electronic control unit that is configured to generate a three dimensional panoramic view based on the images generated by the plurality of cameras, and that is configured to calibrate the plurality of cameras by detecting the calibration markers in the images generated by the plurality of cameras.

18. The electronic control system according to claim 17, wherein the electronic control unit continuously calibrates the cameras each time the plurality of cameras generate images.

19. The electronic control system according to claim 17, wherein the plurality of calibration markers and the plurality of cameras are positioned such that adjacent cameras have overlapping fields of view that share at least one calibration marker of the plurality of calibration markers in common.

20. The electronic control system according to claim 17, wherein each calibration marker includes a geometry that includes more than a single corner and a color that provides contrast with a background on which the calibration marker is mounted.