Virtual pan/tilt camera system and method for vehicles

• US 7,436,429 B2
• Filed: 11/24/2003
• Issued: 10/14/2008
• Est. Priority Date: 11/24/2003
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Tillotson, Brian J.
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US10/722,148
• Publication Number:
  US 20050110869A1
• Time in Patent Office: 1,786 Days
• Field of Search: 348 10-200
• US Class Current: 348/36
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

The present disclosure is directed to a virtual pan/tilt camera system and method for use with vehicles, and particularly ground vehicles (MGVs). The disclosure utilizes autonomous navigation systems (ANSs) used with pan/tilt cameras, but eliminates the pan/tilt cameras substituting a plurality of video cameras. Each video camera is mounted in a fixed orientation on the vehicle and covers a selected angular range of lateral view. Commands from the ANS are mapped to selected addresses where the video data from the video cameras are stored, and appropriately transformed data from the selected addresses are input to the ANS. Computers and software in the MGV receive video data from the cameras and stitch the imagery together into a single panoramic view. Video data from cameras with overlapping fields of view are used to simulate the view of stereo cameras.

Claims

1. A method of automatically navigating a ground vehicle using a plurality of fixed video cameras mounted to the vehicle and using an autonomous navigation system (ANS) programmed to operate with pan/tilt cameras, said method comprising the steps of:

generating video data from said fixed video cameras;

generating commands at the output of said ANS for controlling the positions of pan/tilt cameras;

converting said commands into signals representing azimuths and elevations that said pan/tilt cameras would view;

mapping the azimuth and elevation signals to selected addresses containing the video data;

reformatting the video data from said selected addresses; and

inputting the reformatted video data to the ANS as video streams, said ANS processing said video streams.

2. The method of claim 1 further comprising the steps of:

combining video data from said selected addresses to stitch together images represented by said video data; and

inputting the stitched and reformatted data to the ANS as video streams.

3. The method of claim 1 further comprising the step of:

displaying images from said video cameras.

4. The method of claim 3 wherein said video cameras view a panoramic image, and each display displays a selected portion of said image.

5. The method of claim 1 wherein said reformatting step further comprises the step of:

transforming the image from each fixed camera to the image that would be seen by a pan/tilt camera pointing in the direction as determined by said commands generated by said ANS.

6. The method of claim 1 further comprising the steps of:

stitching together the video data generated from the video cameras, said stitched together data representing a composite image from said video cameras;

storing said stitched together data; and

providing selected portions of said composite image of the ANS by selectively addressing the stored video data.

7. The method of claim 1 further comprising the steps of:

mounting each of the video cameras on the vehicle such that each camera is mounted in fixed position to point radially outwardly from a common center axis; and

providing a sufficient number of said video cameras such that adjacent ones of the cameras have overlapping fields of view.

8. The method of claim 7 wherein pairs of said cameras have overlapping views such that each portion of said image is viewed by at least two video cameras.

9. The method of claim 8 wherein said cameras are equally circumferentially spaced about said common center axis.

10. The method of claim 9 wherein said providing step further comprises:

providing eight such cameras, each camera having approximately a 90°

field of panoramic view.

11. The method of claim 6 further comprising the step of:

selecting portions of the image to be displayed to simulate the effect of a pan/tilt camera that pans in azimuth and elevation.

12. A method of automatically navigating a ground vehicle using a plurality of fixed cameras mounted to the vehicle and using an autonomous navigation system (ANS) programmed to operate with pan/tilt cameras, said method comprising the steps of:

providing a plurality of video cameras on said vehicle, each video camera mounted in a fixed position to view a selected portion of a selected image, said plurality of video cameras collectively viewing a panoramic image encompassing at least 360°

about said vehicle, pairs of said cameras having overlapping views such that each portion of said image is viewed by at least two video cameras;

generating video data from said plurality of cameras, the video data from each camera representing the image portion viewed by said camera;

inputting the video data to buffers;

converting signals at the outputs of said ANS, that would be used to control the positions of pan/tilt cameras, to signals representing azimuths and elevations that said pan/tilt cameras would view;

mapping said azimuths and elevations signals to selected addresses in said buffers; and

generating video streams in response to said selected video data for input to said ANS.

13. The method of claim 12 further comprising the steps of:

reformatting said selected video data; and

generating said video streams in response to said reformatted video data.

14. The method of claim 13 further comprising the steps of:

stitching together images as represented by the video data from the selected addresses of said buffers; and

inputting the stitched and reformatted data to the ANS as video streams.

15. A system for automatically navigating a ground vehicle using a plurality of fixed video cameras mounted to the vehicle and using an autonomous navigation system (ANS) programmed to operate with pan/tilt cameras, said system comprising:

a plurality of video cameras for generating video data, each video camera generating video data representing a selected portion of an image;

an ANS for generating commands for controlling the positions of pan/tilt cameras;

a conversion component for converting said commands into signals representing azimuths and elevations that said pan/tilt cameras would view;

a translation component for mapping the azimuth and elevation signals to selected addresses containing said video data generated by said video cameras;

a reformatting component for reformatting the video data from said selected addresses; and

inputting the reformatted video data to the ANS as video streams, said ANS processing said video streams.

16. The system of claim 15 wherein said reformatting component stitches together images as represented by said video data from the selected addresses, and inputs the stitched and reformatted data to the ANS as video streams.

17. The system of claim 15 further comprising displays for displaying images from said video cameras.

18. The system of claim 17 wherein said video cameras view a panoramic image, and each display displays a selected portion of said image.

19. The system of claim 15 wherein said reformatting component transforms the image from each fixed camera to the image that would be seen by a pan/tilt camera pointing in the direction as determined by said commands generated by said ANS.

20. The system of claim 15 further comprising:

a stitching component for stitching together the video data generated from the video cameras, said stitched together data representing a composite image from said video cameras; and

a selection component for selecting portions of said composite image for display.

21. The system of claim 15 wherein each of the video cameras is mounted on said vehicle such that each camera is in a fixed position to point radially outwardly from a common center axis, and there being a sufficient number of said video cameras such that adjacent ones of the cameras have overlapping fields of view.

22. The system of claim 21 wherein pairs of said cameras have overlapping views such that each portion of said image is viewed by at least two video cameras.

23. The system of claim 22 wherein said cameras are equally circumferentially spaced about said common center axis.

24. The system of claim 23 wherein there are eight such cameras, each camera having approximately a 90°

field of panoramic view.

25. A system for automatically navigating a ground vehicle using a plurality of fixed cameras mounted to the vehicle and using an autonomous navigation system (ANS) programmed to operate with pan/tilt cameras, said system comprising:

a plurality of video cameras on said vehicle, each video camera mounted in a fixed position to view a selected portion of a selected image, said plurality of Video cameras collectively viewing a panoramic image encompassing at least 360°

about said vehicle, pairs of said cameras having overlapping views such that each portion of said image is viewed by at least two video cameras, said video cameras generating video data, the video data from each camera representing the image portion viewed by said camera;

buffers for storing the video data generated by the video cameras;

an ANS for generating commands that would control the positions of pan/tilt cameras;

a conversion component for converting said commands generated by said ANS, to signals representing azimuths and elevations that said pan/tilt cameras would view;

a translation component for mapping said azimuths and elevations signals to selected addresses in said buffers to select the video data stored at said selected addresses; and

a reformatting component for generating video streams in response to said selected video data for input to said ANS.

26. The system of claim 25 further composing:

a stitching component for stitching together images as represented by the video data from the selected addresses of said buffers, said stitched and reformatted data being input to the ANS as video streams.

27. A vehicle capable of autonomous operation, said vehicle comprising:

a plurality of cameras mounted to said vehicle for generating a panoramic scene;

an autonomous navigation system (ANS) for generating azimuth and elevation commands;

a translator for translating azimuth and elevation commands from said ANS to select sub-sets of the panoramic scene such that the cameras function as a virtual pan/tilt camera system; and

a vehicle propulsion system for moving and guiding the vehicle under the direction of said ANS.

28. A method of autonomously navigating a vehicle, said method comprising the steps of:

generating video data from fixed video cameras mounted to said vehicle representing a panoramic image;

selecting video data representing a portion of said panoramic image; and

autonomously navigating said vehicle at least partially in response to said selected data.

29. The method of claim 28 further comprising the step of:

providing terrain data representing a map of the terrain in which said vehicle is operating, and wherein said navigating step further comprises navigating said vehicle at least partially in response to said terrain data and said selected data.

30. The method of claim 29 further comprising the step of:

displaying images from said video cameras.

31. The method of claim 30 further comprising the steps of:

stitching together the video data generated from the video cameras, said stitched together data representing a composite image from said video cameras; and

displaying selected portions of said composite image.

32. A method of automatically navigating a vehicle, said method comprising the steps of:

generating a panoramic image using fixed video cameras;

reformatting the panoramic image;

inputting the reformatted panoramic image to an autonomous navigation system designed to operate with pan/tilt cameras, said autonomous navigation system processing the transformed image to navigate the vehicle.

33. A method of automatically navigating a vehicle, said method comprising the steps of:

generating a panoramic image using fixed video cameras;

reformatting the panoramic image;

inputting the reformatted panoramic image to an autonomous navigation system designed to operate with pan/tilt cameras, said autonomous navigation system processing the transformed image to navigate the vehicle,

said reformatting step further comprising the steps of;

converting pixel indices for the focal planes of the fixed cameras to physical locations on the cameras'"'"' focal planes,

converting the image from intensity as a function of x and y locations on the fixed cameras'"'"' focal planes, to intensity as a function of Az-EI coordinates relative to the fixed cameras'"'"' optical axes,

rotating the Az-EI coordinates to Az-EI coordinates relative to the commanded optical axes of virtual pan/tilt cameras, thereby producing a resultant Az-EI image,

converting the resultant Az-EI image to intensity as a function of x and y locations on the virtual pan/tilt cameras'"'"' focal planes,

converting the x-y locations to pixel values, and interpolating the pixel values.

34. A method of automatically navigating a vehicle, said method comprising the steps of:

generating a panoramic image using fixed video cameras;

reformatting the panoramic image;

inputting the reformatted panoramic image to an autonomous navigation system designed to operate with pan/tilt cameras, said autonomous navigation system processing the transformed image to navigate the vehicle,

said reformatting step further comprising the steps of;

converting pixel indices for the focal planes of the fixed cameras to physical locations on the cameras'"'"' focal planes,

converting the image from intensity as a function of x and y locations on the fixed cameras'"'"' focal planes, to intensity as a function of Az-EI coordinates relative to the fixed cameras'"'"' optical axes,

rotating the Az-EI coordinates to Az-EI coordinates relative to the commanded optical axes of virtual pan/tilt cameras, thereby producing a resultant Az-EI image,

converting the resultant Az-EI image to intensity as a function of x and y locations on the virtual pan/tilt cameras'"'"' focal planes,

converting the x-y locations to pixel values, and interpolating the pixel values.