Filling in surround view areas blocked by mirrors or other vehicle parts

• US 9,902,322 B2
• Filed: 10/30/2015
• Issued: 02/27/2018
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Bendix Commercial Vehicle Systems LLC (Estate of Heinz Hermann Thiele)
• Original Assignee: Bendix Commercial Vehicle Systems LLC (Estate of Heinz Hermann Thiele)
• Inventors: Kuehnle, Andreas U., Boon, Cathy L., Li, Zheng, Molin, Hans M.
• Primary Examiner(s): Beck, Leron
• Application Number:
  US14/927,983
• Publication Number:
  US 20170120817A1
• Time in Patent Office: 851 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/202 displaying a blind spot scene on the vehicle part responsible for the blind spot
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An imaging system, method, and computer readable medium filling in blind spot regions in images of peripheral areas of a vehicle. Intrinsic or extrinsic blind spot data is used together with vehicle movement data including vehicle speed and steering angle information to determine one or more portions of a series of images of the peripheral areas that include or will include one or more blind spot obstructions in the images. Portions of the images predicted to be obstructed at a future time, portions of overlapping images obtained concurrently from plural sources, or both, are obtained and used as an image patch. A blind spot region restoration unit operates to stitch together a restored image without the blind spot obstruction by merging one or more image patches into portions of the images that include the one or more blind spot obstructions.

Claims

1. An imaging system filling in blind spot regions in peripheral areas of an associated vehicle, the system comprising:

a processor;

a non-transient memory operatively coupled with the processor, the non-transient memory storing intrinsic blind spot data representative of a blind spot region of a peripheral area of the associated vehicle, and movement data representative of a speed of movement of the associated vehicle;

an image obtaining unit operatively coupled with the processor, the image obtaining unit being configured to receive first image data representative of a first image of the peripheral area of the associated vehicle captured at a first time, and to receive second image data representative of a second image of the peripheral area of the associated vehicle captured at a second time after the first time;

a predicted blind spot region determining unit operatively coupled with the processor, the predicted blind spot region determining unit being configured to determine, at the first time, a portion of the first image predicted to be in the blind spot region at the second time in accordance with the intrinsic blind spot data and the movement data; and

a current blind spot region restoration unit operatively coupled with the processor, the current blind spot region restoration unit being configured to generate, at the second time, restored second image data by merging;

i) a selected portion of the first image data corresponding to the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.

2. The imaging system according to claim 1, further comprising:

an image output unit operatively coupled with the processor and configured to generate a visual representation of the restored second image on an associated human readable display device in accordance with the generated restored second image data.

3. The imaging system according to claim 1, further comprising:

a vehicle movement determining unit operatively coupled with the processor and comprising;

a vehicle speed determining unit receiving a vehicle speed signal representative of a speed of the associated vehicle; and

a vehicle steering determining unit receiving a vehicle steering angle signal representative of a steering angle of the associated vehicle,

wherein the vehicle movement determining unit is configured to generate the movement data in accordance with the vehicle speed and vehicle steering angle signals.

4. The imaging system according to claim 1, further comprising:

a planarity determination unit operatively coupled with the processor and configured to determine, in accordance with a planarity determination model;

a first planarity measure of the first portion of the first image predicted to be in the blind spot region at the second time; and

a second planarity measure of the second image data; and

a planarity assessment unit operatively coupled with the processor and configured to;

determine a planarity conformance level of the first and second planarity measures relative to a predetermined planarity metric; and

selectively generate a planarity conformance signal in accordance with the determined planarity conformance level.

5. The imaging system according to claim 4, wherein:

the planarity determination unit is configured to determine the first and second planarity measures in accordance with a planarity assessment algorithm; and

the current blind spot region restoration unit is configured to selectively merge default blank image data representative of a predetermined neutral image with the second image data in accordance with a logical level of the planarity conformance signal.

6. The imaging system according to claim 1, wherein:

the image obtaining unit comprises a camera configured to capture image data as a sequence of image data frame sets, each image data frame set of the frame sets being acquired in succession following a predetermined time interval; and

a time between the first time and the second time is an integer multiple of the predetermined time interval.

7. The imaging system according to claim 1, wherein the image obtaining unit comprises:

a first camera configured to;

capture, at the first time, from a first perspective relative to the associated vehicle, the first image data representative of the first image of the peripheral area of the associated vehicle; and

capture, at the second time, from the first perspective, the second image data representative of the second image of the peripheral area of the associated vehicle at the second time; and

a second camera configured to capture, at the first time, from a second perspective relative to the associated vehicle different than the first perspective, auxiliary image data representative of an auxiliary image of the peripheral area of the associated vehicle.

8. The imaging system according to claim 7, further comprising:

a blind spot overlap region determining unit operatively coupled with the processor and configured to determine, at the first time in accordance with the intrinsic blind spot data and the movement data;

a first portion of the first image in the blind spot region at the first time; and

a first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time,

wherein the current blind spot region restoration unit is configured to generate restored first image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, with ii) the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time, wherein the generated restored first image data is representative of a restored first image of the peripheral area of the associated vehicle at the first time unobstructed by the blind spot region.

9. The imaging system according to claim 7, further comprising:

a blind spot overlap region determining unit operatively coupled with the processor and configured to determine, in accordance with the intrinsic blind spot data and the movement data, a first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time,

wherein the current blind spot region restoration unit is configured to generate the restored second image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data at a first portion of the second image data corresponding to the first portion of the second image in the blind spot in accordance with the intrinsic blind spot data and the movement data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.

10. An imaging system filling in blind spot regions in peripheral areas of an associated vehicle, the system comprising:

a processor;

a non-transient memory operatively coupled with the processor, the non-transient memory storing intrinsic blind spot data representative of a blind spot region of a peripheral area of the associated vehicle;

an image obtaining unit operatively coupled with the processor and configured to receive first image data representative of a first image of the peripheral area of the associated vehicle captured at a first time, and to receive auxiliary image data representative of an auxiliary image of the peripheral area of the associated vehicle captured at the first time;

a blind spot overlap region determining unit operatively coupled with the processor and configured to determine, at the first time in accordance with the intrinsic blind spot data;

a first portion of the first image in the blind spot region at the first time; and

a first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time; and

a current blind spot region restoration unit operatively coupled with the processor and configured to generate restored first image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, with ii) the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time, wherein the generated restored first image data is representative of a restored first image of the peripheral area of the associated vehicle at the first time unobstructed by the blind spot region.

11. The imaging system according to claim 10, further comprising:

an image output unit operatively coupled with the processor and configured to generate a visual representation of the restored first image on an associated human readable display device in accordance with the generated restored first image data.

12. The imaging system according to claim 10, wherein the image obtaining unit comprises:

a first camera configured to capture, at the first time, from a first perspective relative to the associated vehicle, the first image data representative of the first image of the peripheral area of the associated vehicle; and

a second camera configured to capture, at the first time, from a second perspective relative to the associated vehicle different than the first perspective, the auxiliary image data representative of the auxiliary image of the peripheral area of the associated vehicle.

13. The imaging system according to claim 10, further comprising:

a planarity determination unit operatively coupled with the processor and configured to determine, in accordance with a planarity determination model, a planarity measure of the first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time; and

a planarity assessment unit operatively coupled with the processor and configured to;

determine a planarity conformance level of the planarity measure relative to a predetermined planarity metric; and

selectively generate a planarity conformance signal in accordance with the determined planarity conformance level.

14. The imaging system according to claim 13, wherein:

the planarity determination unit is configured to determine the planarity measure in accordance with a planarity assessment algorithm; and

the current blind spot region restoration unit is configured to selectively merge default blank image data representative of a predetermined neutral image with the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time in accordance with a logical level of the planarity conformance signal.

15. A method in an associated imaging system for filling in a blind spot region in a peripheral area of an associated vehicle, the method comprising:

storing intrinsic blind spot data representative of a blind spot region of a peripheral area of the associated vehicle in a non-transient memory operatively coupled with a processor of the associated imaging system;

storing movement data representative of a speed of movement of the associated vehicle in the non-transient memory;

obtaining, at a first time, first image data using an image obtaining unit operatively coupled with the processor of the associated imaging system, the first image data being representative of a first image of the peripheral area of the associated vehicle at the first time;

obtaining, at a second time after the first time, second image data using the image obtaining unit, the second image data being representative of a second image of the peripheral area of the associated vehicle at a second time after the first time;

determining, at the first time by a predicted blind spot region determining unit operatively coupled with the processor of the associated imaging system, a portion of the first image predicted to be in the blind spot region at the second time in accordance with the intrinsic blind spot data and the movement data; and

generating, at the second time by a current blind spot region restoration unit operatively coupled with the processor of the associated imaging system, restored second image data by merging;

i) a selected portion of the first image data corresponding to the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.

16. The method according to claim 15, further comprising:

generating, by an image output unit operatively coupled with the processor of the associated imaging system, a visual representation of the restored second image on an associated human readable display device in accordance with the generated restored second image data.

17. The method according to claim 15, further comprising:

receiving, by a vehicle speed determining unit of a vehicle movement determining unit operatively coupled with the processor of the associated imaging system, a vehicle speed signal representative of a speed of the associated vehicle;

receiving, by a vehicle steering determining unit of the vehicle movement determining unit, a vehicle steering angle signal representative of a steering angle of the associated vehicle; and

generating, by the vehicle movement determining unit, the movement data in accordance with the vehicle speed and vehicle steering angle signals.

18. The method according to claim 15, further comprising:

determining, by a planarity determination unit operatively coupled with the processor, in accordance with a planarity determination model;

a first planarity measure of the first portion of the first image predicted to be in the blind spot region at the second time; and

a second planarity measure of the second image data;

determining, by a planarity assessment unit operatively coupled with the processor, a planarity conformance level of the first and second planarity measures relative to a predetermined planarity metric; and

selectively generating, by the planarity assessment unit, a planarity conformance signal in accordance with the determined planarity conformance level.

19. The method according to claim 18, wherein:

the determining the first and second planarity measures comprises determining, by the planarity determination unit, the first and second planarity measures in accordance with a planarity assessment algorithm; and

the merging of the selected portion of the first image with the second image data by the current blind spot region restoration unit comprises selectively merging default blank image data representative of a predetermined neutral image with the second image data in accordance with a logical level of the planarity conformance signal.

20. The method according to claim 15, wherein:

the receiving the first and second image data comprises capturing the image data as a sequence of image data frame sets using a camera of the imaging unit configured to capture image data, each image data frame set of the frame sets being acquired by the camera in succession following a predetermined time interval; and

a time between the first time and the second time is an integer multiple of the predetermined time interval.

21. The method according to claim 15, further comprising:

capturing, at the first time by a first camera from a first perspective relative to the associated vehicle, the first image data representative of the first image of the peripheral area of the associated vehicle;

capturing, at the second time by the first camera from the first perspective, the second image data representative of the second image of the peripheral area of the associated vehicle at the second time; and

capturing, at the first time by a second camera from a second perspective relative to the associated vehicle different than the first perspective, auxiliary image data representative of an auxiliary image of the peripheral area of the associated vehicle.

22. The method according to claim 21, further comprising:

determining, by a blind spot overlap region determining unit operatively coupled with the processor, at the first time in accordance with the intrinsic blind spot data and the movement data;

a first portion of the first image in the blind spot region at the first time; and

a first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, and

wherein the current blind spot region restoration unit is configured to generate restored first image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, with ii) the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time, wherein the generated restored first image data is representative of a restored first image of the peripheral area of the associated vehicle at the first time unobstructed by the blind spot region.

23. The method according to claim 21, further comprising:

determining, by a blind spot overlap region determining unit operatively coupled with the processor in accordance with the intrinsic blind spot data and the movement data, a first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time,

wherein the current blind spot region restoration unit is configured to generate the restored second image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data at a first portion of the second image data corresponding to the first portion of the second image in the blind spot in accordance with the intrinsic blind spot data and the movement data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.

24. A method in an associated imaging system for filling in a blind spot region in a peripheral area of an associated vehicle, the method comprising:

storing intrinsic blind spot data representative of a blind spot region of a peripheral area of the associated vehicle in a non-transient memory operatively coupled with a processor of the associated imaging system;

obtaining, at a first time, first image data using an image obtaining unit operatively coupled with the processor of the associated imaging system, the first image data being representative of a first image of the peripheral area of the associated vehicle captured at a first time;

obtaining, at the first time, auxiliary image data using the image obtaining unit operatively coupled with the processor of the associated imaging system, the auxiliary image data being representative of an auxiliary image of the peripheral area of the associated vehicle captured at the first time;

determining, by a blind spot overlap region determining unit operatively coupled with the processor and in accordance with the intrinsic blind spot data;

a first portion of the first image in the blind spot region at the first time; and

a first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time; and

generating, by a current blind spot region restoration unit operatively coupled with the processor, restored first image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, with ii) the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time, wherein the generated restored first image data is representative of a restored first image of the peripheral area of the associated vehicle at the first time unobstructed by the blind spot region.

25. The method according to claim 24, further comprising:

generating, by an image output unit operatively coupled with the processor of the associated imaging system, a visual representation of the restored second image on an associated human readable display device in accordance with the generated restored second image data.

26. The method according to claim 24, further comprising:

capturing, by a first camera from a first perspective relative to the associated vehicle, the first image data representative of the first image of the peripheral area of the associated vehicle; and

capturing, by a second camera from a second perspective relative to the associated vehicle different than the first perspective, the auxiliary image data representative of the auxiliary image of the peripheral area of the associated vehicle.

27. The method according to claim 24, further comprising:

determining, by a planarity determination unit operatively coupled with the processor, in accordance with a planarity determination model, a planarity measure of the first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time; and

determining, by a planarity assessment unit operatively coupled with the processor, a planarity conformance level of the planarity measure relative to a predetermined planarity metric; and

selectively generating, by the planarity assessment unit, a planarity conformance signal in accordance with the determined planarity conformance level.

28. The method according to claim 27, wherein:

the determining the planarity measure comprises determining the planarity measure in accordance with a planarity assessment algorithm; and

the merging the of the first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot comprises selectively merging, by the current blind spot region restoration, default blank image data representative of a predetermined neutral image with the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time in accordance with a logical level of the planarity conformance signal.

29. A non-transitory computer readable storage medium storing one or more sequences of instructions for filling in a blind spot region in a peripheral area of an associated vehicle, wherein said instructions, when executed by one or more processors, cause the one or more processors to execute steps comprising:

storing intrinsic blind spot data representative of a blind spot region of a peripheral area of the associated vehicle in a non-transient memory operatively coupled with a processor of the associated imaging system;

storing movement data representative of a speed of movement of the associated vehicle in the non-transient memory;

obtaining, at a first time, first image data using an image obtaining unit operatively coupled with the processor of the associated imaging system, the first image data being representative of a first image of the peripheral area of the associated vehicle at the first time;

obtaining, at a second time after the first time, second image data using the image obtaining unit, the second image data being representative of a second image of the peripheral area of the associated vehicle at a second time after the first time;

determining, at the first time by a predicted blind spot region determining unit operatively coupled with the processor of the associated imaging system, a portion of the first image predicted to be in the blind spot region at the second time in accordance with the intrinsic blind spot data and the movement data; and

generating, at the second time by a current blind spot region restoration unit operatively coupled with the processor of the associated imaging system, restored second image data by merging;

i) a selected portion of the first image data corresponding to the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.

30. The non-transitory computer readable storage medium of claim 29, when executed causing the one or more processors to execute further steps comprising:

capturing, at the first time by a first camera from a first perspective relative to the associated vehicle, the first image data representative of the first image of the peripheral area of the associated vehicle;

capturing, at the second time by the first camera from the first perspective, the second image data representative of the second image of the peripheral area of the associated vehicle at the second time; and

capturing, at the first time by a second camera from a second perspective relative to the associated vehicle different than the first perspective, auxiliary image data representative of an auxiliary image of the peripheral area of the associated vehicle.

31. The non-transitory computer readable storage medium of claim 30, when executed causing the one or more processors to execute further steps comprising:

determining, by a blind spot overlap region determining unit operatively coupled with the processor, at the first time in accordance with the intrinsic blind spot data and the movement data;

a first portion of the first image in the blind spot region at the first time; and

a first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, and

wherein the current blind spot region restoration unit is configured to generate restored first image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the first portion of the first image in the blind spot region at the first time, with ii) the first image data at a first portion of the first image data corresponding to the first portion of the first image in the blind spot region at the first time, wherein the generated restored first image data is representative of a restored first image of the peripheral area of the associated vehicle at the first time unobstructed by the blind spot region.

32. The non-transitory computer readable storage medium of claim 30, when executed causing the one or more processors to execute further steps comprising:

determining, by a blind spot overlap region determining unit operatively coupled with the processor in accordance with the intrinsic blind spot data and the movement data, a first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time,

wherein the current blind spot region restoration unit is configured to generate the restored second image data by merging;

i) a first portion of the auxiliary image data corresponding to the first portion of the auxiliary image overlapping the portion of the first image predicted to be in the blind spot region at the second time, with ii) the second image data at a first portion of the second image data corresponding to the first portion of the second image in the blind spot in accordance with the intrinsic blind spot data and the movement data, wherein the generated restored second image data is representative of a restored second image of the peripheral area of the associated vehicle at the second time unobstructed by the blind spot region.