Panoramic video imaging and display system

• US 8,106,936 B2
• Filed: 03/14/2008
• Issued: 01/31/2012
• Est. Priority Date: 03/16/2007
• Status: Active Grant

• Current Assignee: L3 Technologies, Inc. (L3Harris Technologies, Inc.)
• Original Assignee: Kollmorgen Corporation (Regal Rexnord Corp.)
• Inventors: Strzempko, Thaddeus J., Chamberlain, Jesse David, LaValley, Daniel Lawrence, Rose, Michael
• Primary Examiner(s): Harrell, Robert B
• Application Number:
  US12/049,068
• Publication Number:
  US 20090058988A1
• Time in Patent Office: 1,418 Days
• Field of Search: 348/36, 348/37, 348/38, 348/39, 352/69, 352/70, 352/71
• US Class Current: 348/36
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

A system for low-latency, high-resolution, continuous motion panoramic video imaging includes a plurality of high-resolution video cameras, each video camera generating about at least 500 kilopixel near-real time video. The cameras can be supported for positioning the plurality of cameras at predetermined angular locations to generate a full 360 degree field of view. The system can also include an image processor for processing video image signals in parallel and providing panoramic images. In one embodiment, the system can include a display to provide seamless panoramic images.

Claims

1. A low-latency, high-resolution, continuous motion full 360 degree panoramic video imaging and display system, comprising:

a plurality of high-resolution video cameras, each video camera generating an at least 500 kilopixel video camera image signal as a digital stream of individual pixels representative of images in the field of view of the respective video camera;

a support for positioning the plurality of video cameras at predetermined angular locations around the support to enable the plurality of video cameras to together generate digital pixel stream video camera image signals encompassing a full 360 degree field of view around the plurality of video cameras;

a communication link between the plurality of video cameras and an image processor for communicating the digital pixel stream video camera image signals from the plurality of video cameras to the image processor;

the image processor receiving the digital pixel stream video camera image signals from each of the plurality of video cameras and processing the digital pixel stream video camera image signals from each of the plurality of video cameras in parallel with the digital pixel stream video camera image signals from others of the plurality of video cameras to generate combined video signals, with the image processor being configured to perform;

image correction of the digital pixel stream video camera image signals by which pixel corrections are executed and completed as the digital pixel stream video camera image signals are received and prior to a full frame of pixels being received for image correction;

cylindrical wrapping of the digital pixel stream video camera image signals by which cylindrical wrapping of the pixels is executed and completed as the digital pixel stream video camera image signals are received and prior to a full frame of pixels being received for cylindrical wrapping;

image blending of the digital pixel stream video camera image signals by which the blending of pixels is executed and completed as the digital pixel stream video camera image signals are received and prior to a full frame of pixels being received for image blending;

pixel binning of the digital pixel stream video camera image signals by which the binning of pixels is executed and completed as the digital pixel stream video camera image signals are received and prior to a full frame of pixels being received for pixel binning; and

image cropping of the digital pixel stream video camera image signals by which the cropping of pixels is executed and completed as the digital pixel stream video camera image signals are received and prior to a full frame of pixels being received for image cropping;

a communication link between the image processor and a display device for communicating the combined video signals to the display device; and

the display device displaying the combined video signals as full, distortion-corrected and seamless combined 360 degree panoramic images in the field of view around the plurality of video cameras, with each combined 360 degree panoramic image being displayed from the digital pixel stream video camera image signals within not more than 100 milliseconds from the imaging event and with each of the video camera images that together comprise the combined 360 degree panoramic image reflecting the same instant in time.

2. A video imaging and display system as set forth in claim 1, further comprising an operator controlled device for identifying an area of interest in the panoramic images and controlling the image processor to provide additional image signal information to display a magnified view of the area of interest.

3. A video imaging and display system as set forth in claim 2, wherein one or more areas of interest may be separately and independently processed by the video image processing system for display.

4. A video imaging and display system as set forth in claim 3, wherein the area of interest relates to the direction of platform motion associated with an operators orientation.

5. A video imaging and display system as set forth in claim 3, wherein one or more sensors of the video imaging system may gather data associated with the one or more areas of interest.

6. A video imaging and display system as set forth in claim 3, wherein platform coordinates associated with the area of interest are automatically transmitted to command and direct an external subsystem of the video imaging system.

7. A video imaging and display system as set forth in claim 1, wherein the image processor includes motion compensation circuitry configured to execute a motion compensation algorithm for stabilization of the panoramic images.

8. A video imaging and display system as set forth in claim 7, wherein the support is carried on a platform subject to movement and the motion compensation algorithm modifies the video signals to adjust for such movement.

9. A video imaging and display system as set forth in claim 8, wherein the motion compensation circuitry further comprises stabilization circuitry.

10. A video imaging and display system as set forth in claim 8, further comprising a motion sensor on the support, wherein the sensor generates a signal representative of relative movement of the support, and a communication link between the sensor and the motion compensation circuitry.

11. A video imaging and display system as set forth in claim 8, wherein the motion compensation algorithm utilizes a predetermined object in the field of vision of one of the plurality of video cameras to adjust the digital pixel stream video camera image signals from the others of the plurality of video cameras relative to the digital pixel stream video camera image signals from said one of the plurality of video cameras.

12. A video imaging and display system as set forth in claim 1, further comprising at least one sensor operable in the non-visual spectrum to gather non-visual data in the field of vision of the plurality of video cameras, and a communication link for communicating the non-visual data from the at least one sensor to the image processor, with the image processor associating the non-visual data with the panoramic images generated from the digital pixel stream video camera image signals gathered at the same time as the non-visual data.

13. A video imaging and display system as set forth in claim 12, wherein the at least one sensor is an infra-red sensor.

14. A video imaging and display system as set forth in claim 12, further comprising a global positioning sensor providing global positioning data to the image processor, and the image processor associating the global positioning data with the panoramic images generated from the digital pixel stream video camera image signals gathered at the same time as the non-visual data.

15. A video imaging and display system as set forth in claim 1, wherein the image processor is positioned proximate to the plurality of video cameras.

16. A video imaging and display system as set forth in claim 15, further comprising a system center for communication with image processors of a plurality of video imaging systems and for controlling operation of the plurality of video imaging systems from a location remote from the video imaging systems.

17. A video imaging and display system as set forth in claim 1, wherein the image processor includes field programmable gate arrays for parallel processing of the digital pixel stream video camera image signals.

18. A video imaging and display system as set forth in claim 1, further comprising detection circuitry for detecting movement of an object in the field of vision of at least one of the plurality of video cameras.

19. A video imaging and display system as set forth in claim 1, wherein the combined video signals representative of panoramic images generated by the image processor relate to seamless panoramic images.

20. A video imaging and display system as set forth in claim 1, wherein the plurality of high-resolution video cameras are mounted in a housing configured to arrange the high-resolution video cameras to provide the 360 degree field of vision and to protect the video cameras from the environment.

21. A video imaging and display system as set forth in claim 1, wherein the plurality of high-resolution video cameras are mounted to a quadrant of a platform subject to movement.

22. A video imaging and display system as set forth in claim 1, wherein the plurality of high-resolution video cameras are separately mounted at spaced locations around the support in separate housings, one housing for each video camera for protecting the video cameras from the environment.

23. A low-latency, high-resolution, continuous motion full 360 degree panoramic video imaging and display system for mounting on a platform, the system comprising:

a plurality of high-resolution video cameras, each video camera generating an at least 500 kilopixel video camera image signal as a digital stream of individual pixels representative of images in the field of view of the respective video camera;

a support adapted to be mounted on the platform for positioning the plurality of video cameras at predetermined angular locations around the support to enable the plurality of video cameras to together generate digital pixel stream video camera image signals encompassing a full 360 degree field of view around the platform;

a communication link between the plurality of video cameras and an image processor for communicating the digital pixel stream video camera image signals from the plurality of video cameras to the image processor;

the image processor being configured to receive the digital pixel stream video camera image signals from each of the plurality of video cameras and to process the digital pixel stream video camera image signals from each of the plurality of video cameras in parallel with the digital pixel stream video camera image signals from others of the plurality of video cameras to generate combined video signals, wherein the processing includes dynamic motion compensation such that the images displayed from the combined video signals remain stationary even as at least one of the plurality of video cameras is subjected to motion;

a communication link between the image processor and a display device for communicating the combined video signals to the display device; and

the display device displaying the combined video signals as full, distortion-corrected and seamless combined 360 degree panoramic images, with each combined 360 degree panoramic image being displayed from the digital pixel stream video camera image signals within not more than 100 milliseconds from the imaging event and with each of the video camera images that together comprise the combined 360 degree panoramic image of the field of view around the platform reflecting the same instant in time.

24. A video imaging and display system as set forth in claim 23, wherein the platform is a vehicle.

25. A video imaging and display system as set forth in claim 24, wherein the vehicle is selected from the group consisting of land vehicles, marine vehicles and air vehicles.