Electronic control system, electronic control unit and associated methodology of adapting 3D panoramic views of vehicle surroundings by predicting driver intent

• US 8,502,860 B2
• Filed: 09/29/2009
• Issued: 08/06/2013
• Est. Priority Date: 09/29/2009
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Demirdjian, David
• Primary Examiner(s): BOUTAH, ALINA A
• Application Number:
  US12/569,182
• Publication Number:
  US 20110074916A1
• Time in Patent Office: 1,407 Days
• Field of Search: 348/36, 348/148, 348/E13.001, 348/42, 701 36- 38, 701/45
• US Class Current: 348/36
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/103 using camera systems provided with artificial illumination device, e.g. IR light source
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An electronic control system, electronic control unit and an associated methodology for adapting three dimensional panoramic views of vehicle surroundings by predicting driver intent are provided. A plurality of cameras mounted on a vehicle generate images of a surrounding area of the vehicle. A visual sensor detects a three dimensional profile of an occupant of the vehicle. An electronic control unit generates a three dimensional panoramic view based on the images generated by the plurality of cameras, determines three dimensional locations and orientations of a plurality of body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor, and adapts the three dimensional panoramic view based on the determined three dimensional locations and orientations. A display unit displays the adapted three dimensional panoramic view.

Claims

1. An electronic control system for adapting three dimensional panoramic views of vehicle surroundings, comprising:

a plurality of cameras mounted on a vehicle that generate images of a surrounding area of the vehicle, the surrounding area being exterior to the vehicle;

a visual sensor that detects a three dimensional profile of at least a portion of an occupant of the vehicle;

an electronic control unit that is configured to;

generate a three dimensional panoramic view of the surrounding area of the vehicle based on the images generated by the plurality of cameras,

determine three dimensional locations and orientations of one or more body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor,

determine movement of the one or more body parts by determining a velocity or an acceleration of the one or more body parts based on the three dimensional locations and orientations, and

adapt the three dimensional panoramic view of the surrounding area of the vehicle based on the movement of the one or more body parts of the occupant; and

a display unit to output a display of the adapted three dimensional panoramic view.

2. The electronic control system according to claim 1, wherein the electronic control unit is configured to predict the vehicle occupant'"'"'s intentions relating to operation of the vehicle based on the determined three dimensional locations and orientations, and the electronic control unit is configured to adapt the three dimensional panoramic view based on the predicted vehicle occupant'"'"'s intentions.

3. The electronic control system according to claim 2, wherein the electronic control unit is configured to assess potential risks that relate to the vehicle'"'"'s proximity to other objects, and the electronic control unit is configured to adapt the three dimensional panoramic view based on the assessed potential risks.

4. The electronic control system according to claim 3, wherein the electronic control unit is configured to determine a potential risk exists by performing object recognition on the images generated by the plurality of cameras.

5. The electronic control system according to claim 1, wherein the electronic control unit is configured to generate the three dimensional panoramic view by:

creating a two dimensional panoramic reconstruction by stitching together views from each of the plurality of cameras,

generating a three dimensional panoramic image by projecting the two dimensional panoramic reconstruction on a three dimensional shape, and

generating the three dimensional panoramic view based on a virtual perspective viewpoint of the three dimensional panoramic image.

6. The electronic control system according to claim 5, wherein the electronic control unit is configured to adapt the three dimensional panoramic view by changing the virtual perspective viewpoint.

7. The electronic control system according to claim 5, wherein the electronic control unit is configured to adapt the three dimensional panoramic view by changing the three dimensional shape on which the two dimensional panoramic reconstruction is projected.

8. The electronic control system according to claim 1, wherein the electronic control unit is configured to calibrate the plurality of cameras by detecting markers mounted on an exterior of the vehicle body, which are detected in the images generated by the plurality of cameras.

9. The electronic control system according to claim 1, wherein the visual sensor is a three dimensional time of flight camera.

10. A method for adapting three dimensional panoramic views of vehicle surroundings, comprising:

generating images of a surrounding area of a vehicle with a plurality of cameras mounted on the vehicle, the surrounding area being exterior to the vehicle;

detecting a three dimensional profile of an occupant of the vehicle with a visual sensor mounted on the vehicle;

generating a three dimensional panoramic view of the surrounding area of the vehicle based on the images generated by the plurality of cameras;

determining three dimensional locations and orientations of one or more body parts of the occupant of the vehicle based on the three dimensional profile detected by the visual sensor;

determining movement of the one or more body parts by determining a velocity or an acceleration of the one or more body parts based on the three dimensional locations and orientations;

adapting the three dimensional panoramic view of the surrounding area of the vehicle based on the movement of the one or more body parts of the occupant; and

outputting a display of the adapted three dimensional panoramic view.

11. The method according to claim 10, further comprising predicting the vehicle occupant'"'"'s intentions relating to operation of the vehicle based on the determined three dimensional locations and orientations,

wherein the adapting the three dimensional panoramic view includes adapting the three dimensional panoramic view based on the predicted vehicle occupant'"'"'s intentions.

12. The electronic control system according to claim 10, wherein the generating the three dimensional panoramic view includes:

creating a two dimensional panoramic reconstruction by stitching together views from each of the plurality of cameras,

generating a three dimensional panoramic image by projecting the two dimensional panoramic reconstruction on a three dimensional shape, and

generating the three dimensional panoramic view based on a virtual perspective viewpoint of the three dimensional panoramic image.

13. The electronic control system according to claim 10, further comprising calibrating the plurality of cameras by detecting a plurality of markers on an exterior of a body of the vehicle in the images generated by the plurality of cameras.

14. The electronic control system according to claim 13, wherein calibrating the cameras includes detecting one of the plurality of markers in two different images that are respectively generated by two different cameras of the plurality of cameras.

15. An electronic control unit for adapting three dimensional panoramic views of vehicle surroundings, comprising:

a data acquisition circuit configured to receive a plurality of images of a surrounding area of the vehicle and a three dimensional profile of an occupant of a vehicle, the surrounding area being exterior to the vehicle;

a data processor configured to;

generate a three dimensional panoramic view based on the plurality of images,

determine three dimensional locations and orientations of a plurality of body parts of an occupant of a vehicle based on the three dimensional profile,

determine movement of the body parts by determining a velocity or an acceleration of the body parts based on the three dimensional locations and orientations, and

adapt the three dimensional panoramic view based on the movement of the body parts; and

a display circuit configured to output display information corresponding to the adapted three dimensional panoramic view.

16. An electronic control system for adapting the three dimensional panoramic views of vehicle surroundings, comprising:

a plurality of cameras mounted on a vehicle that generate images of a surrounding area of the vehicle, the surrounding area being exterior to the vehicle;

a plurality of calibration markers mounted on an exterior of a body of the vehicle;

an electronic control unit that is configured to;

generate a three dimensional panoramic view of the surrounding area of the vehicle based on the images generated by the plurality of cameras,

calibrate the plurality of cameras by detecting the calibration markers in the images of the surrounding area of the vehicle generated by the plurality of cameras,

determine movement of one or more body parts of an occupant of the vehicle by determining a velocity or an acceleration of the one or more body parts based on three dimensional locations and orientations of the one or more body parts, and

adapt the three dimensional panoramic view of the surrounding area of the vehicle based on the movement of the one or more body parts of the occupant of the vehicle.

17. The electronic control system according to claim 16, wherein the electronic control unit continuously calibrates the cameras each time the plurality of cameras generate images.

18. The electronic control system according to claim 16, wherein the plurality of calibration markers and the plurality of cameras are positioned such that adjacent cameras have overlapping fields of view that share at least one calibration marker of the plurality of calibration markers in common.

19. The electronic control system according to claim 16, wherein each calibration marker includes a geometry that includes more than a single corner and a color that provides contrast with a background on which the calibration marker is mounted.