Apparatus and method for estimating and using a predicted vehicle speed in an indirect vision driving task

• US 9,434,309 B1
• Filed: 02/02/2015
• Issued: 09/06/2016
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: The United States Of America As Represented By The Secretary Of The Army
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Smyth, Christopher C.
• Primary Examiner(s): Beck, Leron
• Application Number:
  US14/611,905
• Publication Number:
  US 20160236617A1
• Time in Patent Office: 582 Days
• Field of Search: 348/121
• US Class Current: 1/1
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60W 30/06 Automatic manoeuvring for parking
• CPC Class Codes: B60W 40/072 Curvature of the road
• CPC Class Codes: G01P 3/38 using photographic means
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method and apparatus for predicting vehicle speed during an indirect vision driving task. A further method and apparatus for optimizing the display of a camera return during an indirect vision driving task based on operator perceived vehicle speed as set by the display characteristics and the field-of-view of the camera. A further method and apparatus for using the perceived speed as a driving task aid, in particular, as an electronic aider for optimizing the driving scene display characteristics of scene compression and camera field-of view. In this manner, the invention adjusts the perceived speed in order to match the operator'"'"'s cognitive flow to the control dynamics needed from the operator for the task. The invention has application to autonomous driving where manual intervention is incorporated during critical events for particular tasks; and with limited display space within the vehicle, the display format is adjusted by the invention according to the operator'"'"'s task needs.

Claims

1. A method for aiding an operator during performance of indirect vision driving task in a vehicle, while viewing a display of at least one video camera return of a driving scene via a display device, comprising:

determining parameters for a display being viewed during an indirect vision driving task and of a camera providing a camera return that is being displayed;

determining an actual speed of the vehicle during the indirect vision driving task;

calculating an estimated operator-perceived driving performance for the operator of the vehicle as would be expected to be visually perceived and mentally interpreted by the operator as being the operator'"'"'s actual driving performance during performance of an indirect vision driving task while viewing scene display on the display device based on one or more of the display parameters, camera parameters, driving course, and the actual vehicle speed; and

adaptively controlling, via a processor, display characteristics of scene compression and/or field-of view for the display of the driving scene on the display device to the operator in a manner so as to mitigate differences between the estimated operator-perceived driving performance and the operator'"'"'s actual driving performance during the performance of the indirect vision driving task.

2. The method of claim 1, wherein calculating the estimated operator-perceived driving performance comprises calculating an estimated operator-perceived vehicle speed as would be expected to be visually perceived by the operator and mentally interpreted as being the actual vehicle speed during performance of an indirect vision driving task while viewing scene display on the display device.

3. The method of claim 2, wherein the estimated operator-perceived vehicle speed is calculated for a plurality of sections of the driving course in the driving scene being displayed, from parameters for both of the display and of the video camera, and from the actual speed of the vehicle during the indirect vision driving task, based on an optic flow locus point seen on the said display for the vehicle by the operator, at a camera viewing distance and look-down angle determined by the display and camera parameters, and the driving course geometrical characteristics.

4. The method of claim 3, wherein the estimated operator-perceived vehicle speed (V_p) comprises expressions for course sections which include:

a. an expression for a straight course, that is given by;

V_p=V_M*α

^+1/3, a function of the actual vehicle speed (V_M) and a display scene compression ratio (α

) relative to that of the camera;

b. an expression for a circular course with unlimited camera field-of-view (FOV), that is given by;

V_P=V_M*sqrt(1+(η

/(R*sin θ

′

_c))²)*α

^+1/3, a function of the radius of curvature (R), where θ

′

_c=a sin(η

*α

^+2/3/ρ

), and where ρ

=η

/sin θ

_c, is the camera viewing distance to the locus point, where θ

_c is the camera viewing angle to the locus point, and η

is the camera height above ground level; and

c. an expression for a circular course with a limited camera horizontal field-of view (FOV_L), that is given by;

V_P=V_M*sqrt(1+(η

/(R*sin θ

_L))²)*sin²(FOV_c/2)*α

^+1/3/sin²(FOV_L), where FOV_L<

FOV_c=2a sin(η

/(2R*tan θ

′

_c)), twice the horizontal viewing angle at the camera position to the locus point, and θ

_L=a tan(η

/(2R*sin(FOV_L/2))), the camera look-down angle to the ground as seen at the camera-viewing limit.

5. The method of claim 1, further comprising:

calculating one or more actual driving task performance offsets of the vehicle from a reference path in the driving course seen by the driving scene camera for display in the driving scene.

6. The method of claim 5, wherein the reference path is defined as a circular arc segment.

7. The method of claim 6, wherein calculations of the offsets are based on the arc center position (Po:

[xo,yo]) and radius (Ro), and on the vehicle position (Pv;

[xv,yv]) and heading (θ

v) in the terrain course coordinates, such that;

a. the heading angular offset (θ

e) is expressed as the difference in heading between that of the vehicle and that of the reference arc tangent (θ

t) at the intersection point of the arc radius extended to the vehicle position, such that;

θ

e=θ

t−

θ

v;

where θ

t=θ

r+Sr*η

/2, and θ

r=a tan((yv−

yo)/(xv−

xo)), where Sr=+1 for a counterclockwise turn and Sr=−

1 for a clockwise turn;

b. the position lateral offset (γ

e) is expressed as;

γ

e=Sr*γ

em, where γ

em is the lateral offset magnitude, γ

em=−

Ro+(yv−

yo)*sin(θ

r)+(xr−

xo)*cos(θ

r); and

c. the curvature offset (Ce) is expressed as the difference between the arc path curvature and that of the vehicle path;

Ce=1/Ro−

1/Rv, with curvature expressed as the reciprocal of the path radius, where in simplification of vehicle mechanics, the vehicle path radius is;

Rv=abs(L/sin(θ

w)), with L the wheel base length and θ

w the vehicle tire wheel angle.

8. The method of claim 5, wherein the reference path is defined as a straight line segment.

9. The method of claim 8, wherein the calculations of the offsets are based on an origin position (Pto:

[xto,yto]) on the segment and segment heading (θ

t), and on the vehicle position (Pv;

[xv,yv]) and heading (θ

v) in the terrain course coordinates, such that;

a. the heading angular offset (θ

e) is expressed as the difference in heading between that of the vehicle and that of the reference line segment (θ

t), such that;

θ

e=θ

t−

θ

v;

b. the position lateral offset (γ

e) is expressed as;

γ

e=Sr*γ

em, where γ

em is the lateral offset magnitude, γ

em=sqrt(Rti^2+Rtv^2), where Rtv is the straight line distance from the origin point to the vehicle and Rti is the distance along the reference line segment from the segment origin point to the intersection point (Pi;

[xi,yi]) of the line with a normal to the line from the vehicle, such that Rti=(xv−

xto)*cos(θ

t)+(yv−

yto)*sin(θ

t), and the coordinates of the intersection point are;

xi=Rti*cos(θ

t)+xto, yi=Rti*sin(θ

t)+yto;

here, Sr=+1 for the vehicle to the right of the reference line and Sr=−

1 for the vehicle to the left;

c. the curvature offset (Ce) is expressed as that for the vehicle path;

Ce=−

1/Rv, with curvature expressed as the reciprocal of the path radius, where in simplification of vehicle mechanics, the vehicle path radius is;

Rv=abs(L/sin(θ

w)), with L the wheel base length and θ

w the vehicle tire wheel angle; and

d. the time offset for the time to start and end on the reference path.

10. The method of claim 1, wherein calculating the estimated operator-perceived driving performance comprises calculating an estimated operator-perceived vehicle reference path as would be expected to be visually perceived by the operator and mentally interpreted as being the actual vehicle path to be followed during performance of an indirect vision driving task while viewing scene display on the display device.

11. The method of claim 10, wherein the estimated operator-perceived vehicle reference path is used for pursuit tracking control by the operator of the vehicle from estimations of path curvature by visual fixations on target points of inflection of the reference path perceived by the operator from the driving scene display.

12. The method of claim 11, where as a first order approximation, the estimated operator-perceived vehicle reference path is parallel to the line of travel, and includes calculations of:

a. an estimated operator-perceived angular size of a target point expressed as;

Φ

f=Φ

/a, where Φ

is the angular size as seen in unity-display of the terrain;

b. estimated operator-perceived coordinates of the location of the target point for the x-coordinate direction;

xf=α

*xo*sin(φ

/α

)*sin(φ

), and for the z-coordinate direction;

zf=a*xo*cos(φ

/α

)*cos(φ

);

c. an estimated operator-perceived speed of approach to the said target point, expressed as;

Vf=−

Vo*sqrt((α

^2−

1)*cos(φ

)^2+1), with

the speed component along the x-direction, expressed as;

Vfx=Vo*(α

*sin(φ

/α

)*cos(φ

)−

cos(φ

/α

)*sin(φ

)), and with

the speed component along the z-direction, expressed as;

Vfz=−

Vo*(α

*cos(φ

/α

)*cos(φ

)+sin(φ

/α

)*sin(φ

)); and

d. an estimated operator-perceived curvature at the location of said target point, expressed as;

Cf=(α

^2−

1)*sin(φ

)^4/(α

*xo*((α

^2−

1)*cos(φ

)^3+1)^1.5);

where;

α

is the ratio to the camera scene FOV to that of display FOV as seen by the operator, xo is the lateral offset magnitude of the vehicle from the reference path in the terrain course coordinates, φ

is the bearing from the camera to said visual fixation target point on the reference path Pf;

[xo,zo], such that φ

=a tan(xo/zo), Vo is the actual speed of the vehicle, and where the x-coordinate axis of the display lies along the lateral direction to the vehicle travel and the z-coordinate axis lies along the direction of vehicle travel originating from the camera position.

13. The method of claim 10, wherein the estimated operator-perceived vehicle reference path is used for compensatory control by the operator of the vehicle from control offset errors, where the offsets are estimated by the operator from a reference path perceived by the operator from the optical flow as seen on the driving scene display.

14. The method of claim 13, where as a first order approximation, the estimated operator-perceived vehicle reference path is parallel to the line of travel, and includes calculations of:

a. an estimated operator-perceived heading angular offset (θ

c), expressed as the arctangent of the ratio of the estimated operator-perceived speed in the x-direction to that in the z-direction;

θ

c=a tan((α

*sin(ψ

/α

)*cos(ψ

)−

cos(ψ

/α

)*sin(ψ

))/(α

*cos(ψ

/α

)*cos(ψ

)+sin(ψ

/α

)*sin(ψ

)));

b. an estimated operator-perceived position lateral offset (γ

c), expressed as;

γ

c=α

*xo*sin(ψ

/a)/sin(ψ

); and

c. and estimated operator-perceived curvature offset (Cc), expressed as;

Cc=(α

^2−

1)*sin(ψ

)^4/(α

*xo*((α

^2−

1)*cos(ψ

)^3+1)^1.5);

where;

α

is the ratio to the camera scene FOV to that of display FOV as seen by the operator, xo is the lateral offset magnitude of the vehicle from the reference path in the terrain course coordinates, and w is the bearing from the camera to a focal point for the optical flow on the reference path, here estimated as at Pc;

[xo,zo], with zo now at the optic flow locus origin point that seen by the operator on the display is at a camera viewing distance and look-down angle determined by the display and camera parameters, and by the driving course characteristics, such that ψ

=a tan(xo/zo), and where the lateral and heading offsets of the reference path in the terrain are such that the position lateral offset γ

e=xo, and the heading angular offset θ

e=0.

15. The method of claim 14, wherein the compensatory control uses signals which are weighted by error gains for the estimated operator-perceived heading curvature, heading and lateral offsets.

16. The method of claim 1, wherein controlling the display of the driving scene on the display device to the operator based on the estimated operator-perceived driving performance comprises using a model of information processing which specifies task rules and corresponding feature sets from a knowledge database indexed by operator task attention states and in evaluation sets up the rules for activation, a rules processor activates the rules directing control, and a procedural processor controls the task execution where the task rules are functions of cognitive loading.

17. The method of claim 1, wherein controlling the display of the driving scene on the display device to the operator based on the estimated operator-perceived driving performance comprises using the estimated operator-perceived driving performance as a metric of the task cognitive loading so as to control the display of the driving scene.

18. The method of claim 1, wherein the driving scene that is displayed is controlled in accordance with a control strategy that includes adjustments of one or more of:

a. parameters of the driving scene camera;

b. format parameters of the display of the said camera return; and

c. the actual vehicle speed.

19. The method of claim 18, wherein the adjustments are made in a manner so as to generate a cognitive flow rate for the operator that is optimal for a task.

20. The method of claim 18, wherein the control strategy computes associated costs for the estimated operator-perceived driving performance, and selects the minimum cost adjustment.

24. The method of claim 1, wherein the estimated operator-perceived driving performance for the operator of the vehicle in a driving task with indirect vision differs from what would be the expected performance of the same driving task with direct vision.

25. The method of claim 1, further comprising:

determining characteristics of one or more sections of a course in the driving scene being displayed to the operator via the display device.

21. Apparatus for aiding an operator during performance of an indirect vision driving task in a vehicle by a vehicle operator while viewing a display of a video camera return of a driving scene via a display device, comprising:

a camera for generating a video camera return signal representative of an driving scene;

a video signal processor for applying the video camera return signal to a display for viewing of the driving scene by the vehicle operator;

a prediction modeler for predicting operator-perceived driving performance for the operator of the vehicle as would be expected to be visually perceived and mentally interpreted by the operator as being the operator'"'"'s actual driving performance during performance of an indirect vision driving task while viewing scene display on the display device based on one or more of the display parameters, camera parameters, driving course, and the actual vehicle speed; and

an adapter aider for adaptively controlling display characteristics of scene compression and/or field-of view for the display of the driving scene on the display device to the operator in a manner so as to mitigate differences between the predicted operator-perceived driving performance and the operator'"'"'s actual driving performance during performance of the indirect vision driving task.

22. The apparatus of claim 21, wherein the adapter aider controls the display of the driving scene in accordance with a control strategy that includes adjustments of one or more of:

a. parameters of the camera;

b. parameters of the display of the camera return; and

c. the vehicle speed.

23. The apparatus of claim 21, wherein the adaptive aider determines the control strategy for the scheduling of the adjustments, uses a computational process for the scheduling of the adjustments with access to a database of task cost elements, computes by the information processing model associated cost variables as strategy costs for the predicted operator-perceived vehicle speeds of the adjustment combinations, and selects the minimum cost adjustment schedule, where the task cost elements are composed of corresponding subtask times and cognitive workload indexed by task attention states of the operator.