OBJECT HIGHLIGHTING AND SENSING IN VEHICLE IMAGE DISPLAY SYSTEMS

• US 20150042799A1
• Filed: 10/22/2013
• Published: 02/12/2015
• Est. Priority Date: 08/07/2013
• Status: Abandoned Application

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Wang, Jinsong, Litkouhi, Bakhtiar B., Kazensky, Dennis B., Piasecki, Jeffrey S., Green, Charles A., Frakes, Ryan M.
• Application Number:
  US14/059,729
• Publication Number:
  US 20150042799A1
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method of displaying a captured image on a display device of a driven vehicle. A scene exterior of the driven vehicle is captured by an at least one vision-based imaging device mounted on the driven vehicle. Objects in a vicinity of the driven vehicle are sensed. An image of the captured scene is generated by a processor. The image is dynamically expanded to include sensed objects in the image. The sensed objects are highlighted in the dynamically expanded image. The highlighted objects identify vehicles proximate to the driven vehicle that are potential collisions to the driven vehicle. The dynamically expanded image is displayed with highlighted objects in the display device.

Claims

1. A method of displaying a captured image on a display device of a driven vehicle comprising the steps of:

capturing a scene exterior of the driven vehicle by an at least one vision-based imaging device mounted on the driven vehicle;

sensing objects in a vicinity of the driven vehicle;

generating an image of the captured scene by a processor, the image being dynamically expanded to include sensed objects in the image;

highlighting sensed objects in the dynamically expanded image, the highlighted objects identifying objects proximate to the driven vehicle that are potential collisions to the driven vehicle; and

displaying the dynamically expanded image with highlighted objects in the display device.

2. The method of claim 1 further comprising the step of:

generating an interior component image overlay, the interior component image overlay including a replication of interior components of the driven vehicle as would be seen by a driver viewing a reflection through a rearview mirror;

displaying the interior component image overlay on the display device.

3. The method of claim 1 wherein highlighting detected objects in the dynamically expanded image includes overlaying an alert symbol on the object in the dynamically expanded image, the alert symbol identifying the object having a potential to collide with the driven vehicle.

4. The method of claim 1 wherein highlighting sensed objects in the dynamically expanded image includes overlaying an object overlay on the object for identifying captured vehicles proximate to the driven vehicle, the object overlay identifying an awareness condition of a vehicle relative to the driven vehicle.

5. The method of claim 4 wherein the object overlay identifying an awareness condition includes generating an object overlay boundary around the vehicle that represents a size of the vehicle in the dynamically expanded image.

6. The method of claim 5 wherein highlighting detected objects in the dynamically expanded image further includes overlaying an alert symbol on the vehicle having a potential to collide with the driven vehicle, the alert symbol providing a redundant warning to the driver.

7. The method of claim 6 further comprising the steps of:

determining a time-to-collision warning relating the highlighted object; and

displaying the time-to-collision warning on the display device.

8. The method of claim 7 wherein determining the time-to-collision further comprises the steps of:

detecting an object at a first instance of time and at a second instance of time;

determining a size of the object at the first instance of time and the second instance of time;

determining a change in the distance from the driven vehicle to the object as a function of the determined size of the object at the first and second instances of time;

determining a velocity of the object as a function of the change in the distance over time; and

calculating the time-to-collision as a function of an estimated distance between object and the driven vehicle and a determined velocity of the object.

9. The method of claim 8 wherein determining the size of the object further comprises the step of defining the object size as an object detection window, wherein the object detection window at time t is represented by the following formula:

win_t^det;

(uW_t,vH_t,vB_t);

where uW_t is the detected window width;

vH_t is the detected window height; and

, vB_t is the detected window bottom.

10. The method of claim 9 wherein an observed object size and distance of an object to a driven vehicle is represented by the following formula:

x_t=(w_t^o,h_t^o,d_t^o)

where w_t^o is an observed object width, h_t^o is an observed object height, and d_t^o is an observed object distance at time t.

11. The method of claim 10 wherein the observed object size and distance based on a camera calibration is determined utilizing an in-vehicle window size and location and is represented by the following equation:

12. The method of claim 11 further comprising the step of estimating output parameters of the object as a function of the observed object size and distance parameters and is represented by the following formula:

def;

Y_t=(w_t^e,h_t^e,d_t^e,v_t)

where w_t^e is an estimated object size of the object at time t, h_t^e is an estimated distance of the object at time t, d_t^e is an estimated distance of the object at time t, and v_t is a relative speed of the object at time t.

13. The method of claim 12 wherein the estimated object size of the object at time t is determined by the following formula:

14. The method of claim 13 wherein the estimated object distance of the object at time t is determined by the following formula:

estimated object distance;

d_t^e=d_t^o.

15. The method of claim 14 wherein the estimated object speed relative to the vehicle is represented by the following formula:

16. The method of claim 15 wherein the time-to-collision of the object is represented by the following formula:

TTC;

TTC_t=d_t^e/v_t.

17. The method of claim 6 wherein determining the time-to-collision further comprises the following steps:

detecting an object at a first instance of time and at a second instance of time;

determining a size of the object at the first instance of time and at the second instance of time;

determining a change in the object size between the first and second instances of time;

determining an occupancy of the object in the captured images at the first and the second instances of time; and

calculating the time-to-collision as a function of the determined change in size of the object between the captured image and the occupancy of the object at the first and second instances of time.

18. The method of claim 17 wherein determining the change in the object size comprises the following steps:

identifying the object overlay boundary that includes identifying a height boundary, a width boundary, and corner points of the object overlay boundary; and

determining a change in height, width, and corner points of the object overlay boundary.

22. The method of claim 1 further comprising the steps of:

detecting objects using at least one additional sensing device; and

applying sensor fusion of the objects sensed by the additional sensing device and the at least one vision-based imaging device mounted on the driven vehicle for cooperatively identifying objects for highlighting.

23. The method of claim 1 wherein objects are sensed by the at least one vision-based imaging device.

24. The method of claim 23 wherein objects are sensed by a vehicle-based sensing system.

25. The method of claim 24 wherein a plurality of vehicle based sensing systems are cooperatively used to identify objects exterior of the vehicle, wherein the sensed objects are highlighted in the displayed image, wherein highlighting the sensed objects includes generating a warning symbol overlay on the object in the display device.

26. The method of claim 24 wherein a plurality of vehicle based sensing systems are cooperatively used to identify objects exterior of the vehicle, wherein the sensed objects are highlighted in the displayed image, wherein highlighting the sensed objects includes generating a boundary overlay on the objects in the display device.

27. The method of claim 24 wherein a plurality of vehicle based sensing systems are cooperatively used to identify objects exterior of the vehicle, wherein the sensed objects are highlighted in the displayed image, wherein highlighting the sensed objects includes generating a warning symbol and a boundary overlay on the objects in the display device.

28. The method of claim 1 wherein the dynamically expanded image is displayed on a rearview mirror display device.

19. The method of claim 19 wherein determining the change in height, width, and corner points of the object overlay boundary is represented by the following equations:

Δ

w_t=w_t−

w<sub>t−

1</sub>,

Δ

h_t=hw_t−

h<sub>t−

1</sub>,

Δ

x(p_tⁱ)=x(p_tⁱ)−

x(p<sub>t−

1</sub>ⁱ),Δ

y(p_tⁱ)−

y(p_tⁱ)−

y(p<sub>t−

1</sub>ⁱ)

where

w_t=0.5*(x(p_t¹)−

x(p_t²))+0.5*(x(p_t³)−

x(p_t⁴)),

h_t=0.5*(y(p_t²)−

y(p_t⁴))+0.5*(y(p_t³)−

y(p_t¹)),

and where w_t is the object width at time t, h_t is the object height at time t, and p_tⁱ is the corner points, i=1, 2, 3, or 4, at time t.

20. The method of claim 19 further comprising the steps of estimating changes to the object size and location at a next instance of time, wherein the changes to the object size and location at the next instance of time is represented by the following formula:

Δ

w_t+1=ƒ

_w(Δ

w_t,Δ

w<sub>t−

1</sub>,Δ

w<sub>t−

2</sub>, . . . ),

Δ

h_t+1=ƒ

_h(Δ

h_t,Δ

h<sub>t−

1</sub>,Δ

h<sub>t−

2</sub>, . . . ),

Δ

x_t+1=ƒ

_x(Δ

x_t,Δ

x<sub>t−

1</sub>,Δ

x<sub>t−

2</sub>, . . . ),

Δ

y_t+1=ƒ

_x(Δ

y_t,Δ

y<sub>t−

1</sub>,Δ

y<sub>t−

2</sub>, . . . )

21. The method of claim 20 wherein determining the time-to-collision is determined by the following formula:

TTC_t+1=ƒ

_TCC(Δ

w_t+1,Δ

h_t+1,Δ

x_t+1,Δ

y_t+1 . . . )