Modular parallel-pipelined vision system for real-time video processing

• US 6,188,381 B1
• Filed: 12/31/1997
• Issued: 02/13/2001
• Est. Priority Date: 09/08/1997
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hansen, Michael Wade, Piacentino, Michael Raymond, Brehm, Frederic William, van der Wal, Gooitzen Siemen
• Primary Examiner(s): Faile, Andrew I.
• Assistant Examiner(s): BROWN, RUEBEN M
• Application Number:
  US09/002,265
• Time in Patent Office: 1,140 Days
• Field of Search: 348/721, 315/676, 345/118, 345/112, 345/18, 711/151, 712/11
• US Class Current: 345/581
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining

Abstract

A real-time modular video processing system (VPS) which can be scaled smoothly from relatively small systems with modest amounts of hardware to very large, very powerful systems with significantly more hardware. The modular video processing system includes a processing module containing at least one general purpose microprocessor which controls hardware and software operation of the video processing system using control data and which also facilitates communications with external devices. One or more video processing modules are also provided, each containing parallel pipelined video hardware which is programmable by the control data to provide different video processing operations on an input stream of video data. Each video processing module also contains one or more connections for accepting one or more daughterboards which each perform a particular image processing task. A global video bus routes video data between the processing module and each video processing module and between respective processing modules, while a global control bus provides the control data to/from the processing module from/to the video processing modules separate from the video data on the global video bus. A hardware control library loaded on the processing module provides an application programming interface including high level C-callable functions which allow programming of the video hardware as components are added and subtracted from the video processing system for different applications.

Claims

1. A modular video processing system comprising:

a processing module containing at least one general purpose microprocessor, which controls hardware and software operation of said video processing system using configuration data, and which processes video data;

at least one video processing module which contains a plurality of parallel pipelined video hardware components, the video processing module being responsive to said configuration data to perform different video processing operations on the video data;

a global video bus which establishes a direct connection between the processing module and said at least one video processing module to route the video data between said processing module and said at least one video processing module; and

a global control bus which provides said configuration data to/from said processing module from/to said at least one video processing module, said global control bus being separate from said global video bus.

2. The system of claim 1, wherein the plurality of parallel pipelined video hardware components are configured to perform the different video processing operations asynchronously.

3. The system of claim 2, wherein said processing module comprises at least two microprocessors each of which has an associated random access memory which is not shared with any other microprocessor, said processing module further comprising a shared memory which is accessible by each microprocessor of said processor module through an arbitrated control bus which arbitrates requests for access to said shared memory from each microprocessor.

4. The system of claim 3, wherein said processing module includes means for using the configuration data to program the at least one video processing module to perform at least one of the different video processing operations on the video data and means for providing a synchronous start signal to begin the at least one video processing operation.

5. The system of claim 3, wherein said processing module further comprises a communications interface for communicating with external devices, said communications interface being coupled to said arbitrated control bus for access by each of said at least two microprocessors.

6. The system of claim 3, wherein said random access memory associated with each microprocessor is connected to said global video bus to store video data for transmission to, and video data received from said global video bus.

7. The system of claim 3, wherein said processor module further comprises a semaphore register coupled to said arbitrated control bus, said semaphore register storing semaphores so as to facilitate coordination of mutually exclusive operations by said at least two microprocessors.

8. The system of claim 2, further comprising a hardware control library loaded on a general purpose microprocessor of said processing module, said hardware control library comprising a set of functions for programming the plurality of parallel pipelined video hardware components of said at least one video processing module to perform respective ones of said different video processing operations concurrently.

9. The system of claim 8, wherein said processing module comprises at least two general purpose microprocessors and said hardware control library further comprises a set of functions for coordinating concurrent multitask processing operations of said at least two general purpose microprocessors.

10. The system of claim 8, wherein said configuration data comprises respective control signals for each hardware component of said video processing system, wherein said functions of said hardware control library manipulate said control signals to program said hardware components for each of said different video processing operations.

11. A modular video processing system comprising:

a processing module containing at least one general purpose microprocessor, which controls hardware and software operation of said video processing system using configuration data, and which processes video data;

at least one video processing module which contains a plurality of parallel pipelined video hardware components, the video processing module being responsive to said configuration data to perform different video processing operations on the video data;

a global video bus which establishes a direct connection between the processing module and said at least one video processing module to route the video data between said processing module and said at least one video processing module; and

a global control bus which provides said configuration data to/from said processing module from/to said at least one video processing module, said global control bus being separate from said global video bus;

wherein said video data is coupled with associated video timing signals synchronized to a system clock signal, and each video processing module comprises a crosspoint switch coupled to the global video bus, the cross point switch routing said video data and its associated video timing signals to/from respective ones of the plurality of parallel pipelined video hardware components, said timing signals indicating when the video data represents active video information.

12. The system of claim 11, wherein each video processing module further comprises a crosspoint switch state machine which monitors transfers of video data over each data path of said crosspoint switch and facilitates allocating paths for transferring the video data among the parallel pipelined video hardware components.

13. The system of claim 11, wherein the plurality of parallel pipelined video hardware components comprises a configurable arithmetic logic unit (CALU) responsive to said video data and its associated video timing signals so as to automatically compensate for differences in input video timing between respective images and to provide image accumulations and pointwise video processing operations on a pair of input images.

14. The system of claim 11, wherein the plurality of parallel pipelined video hardware components includes at least one pyramid filtering processor which generates spatially filtered representations of the video data at respectively different resolutions so as to facilitate real-time processing of said video data.

15. The system of claim 11, wherein the modular video processing system includes a plurality of video processing modules and each video processing module comprises a connection for at least one daughterboard the connection including means for routing at least a portion of the configuration data between the processing module and the daughterboard and means for coupling the daughterboard to the crosspoint switch.

16. The system of claim 15, further including a daughterboard comprising a display processor card coupled to the daughterboard connection of one of the plurality of video processing modules, the display processor card including video input ports connected to said crosspoint switch and an encoder which converts video data received at at least one of the input ports into a video signal of a predetermined format for display.

17. The system of claim 15, further including a daughterboard comprising a digitizer card coupled to the daughterboard connection of one of the plurality of video processing modules, the digitizer card including video output ports connected to said crosspoint switch and a decoder which decodes and digitizes video data received at at least one input port into a video signal of a predetermined format and provides the digitized video data to at least one of the output ports.

18. The system of claim 15, further including a daughterboard comprising a correlator card coupled to the daughterboard connection of one of the plurality of video processing modules, the correlator card including a plurality of video input and output ports connected to said crosspoint switch and a plurality of video processing channels, each channel comprising a configurable arithmetic logic unit, a pyramid filtering processor, and a look up table.

19. The system of claim 17, further including a daughterboard comprising a warper card coupled to the daughterboard connection of one of the plurality of video processing modules, the warper card including video input and output ports connected to said crosspoint switch, a pair of memory banks, an address generator, for controlling memory read operations and memory write operations in the memory banks, and a bilinear interpolator.

20. The system of claim 19, wherein said address generator and said bilinear interpolator are implemented as at least one field programmable gate array.

21. A method of creating a modular video processing system, comprising the steps of:

providing video data via a global video bus;

providing a global control bus separate from the global video bus;

connecting a processing module containing at least one general purpose microprocessor to said global video bus and said global control bus said microprocessor controlling hardware and software operations of said video processing system using configuration data and processing said video data;

connecting said global control bus and said global video bus to at least one video processing module which contains parallel pipelined video hardware that is responsive to said configuration data to provide respectively different video processing operations on the video data, the video processing module being configured to perform the different video processing operations concurrently;

said processing module detecting each video processing module connected to said global control bus; and

passing said configuration data directly to each detected video processing module over said global control bus to program said parallel pipelined video hardware to perform at least one of said video processing operations; and

associating video timing signals synchronized to a system clock signal with the video data, wherein each video processing module routes the video data and associated timing signals from any one of the multiple concurrent video processing operations to any one or more of the multiple concurrent video processing operations said timing signals indicating when the video data represents active video information.

22. The method of claim 21, comprising the additional step of providing at least one synchronous start signal via said global control bus said synchronous start signal being coupled to the processing module and to the at least one video processing module to signal the start of the at least one video processing operation.

23. The method of claim 22, comprising the additional step of coupling a semaphore register to the processing module said semaphore register being configured to coordinate multitask processing operations when said processing module comprises at least two general purpose microprocessors.

24. The method of claim 22, comprising the steps of configuring the video processing module to automatically compensate for differences in input video timing between two respective input images, including respective combinations of video data and video timing signals provided by said crosspoint switch and to provide pointwise video processing operations using the two input images and image accumulations of said respective images.

25. The method of claim 21, comprising the additional steps of connecting a crosspoint state machine to said crosspoint switch, said crosspoint state machine monitoring transfers of video data over each data path of said crosspoint switch and allocating paths for transferring the video data among the parallel pipelined video hardware components.

26. The method of claim 21, wherein said configuration data comprises respective control signals for each hardware component of said video processing system, the method including the step of coupling said processing module to manipulate said control signals to program said hardware components for each of said different video processing operations.

27. A modular processing system comprising:

at least one specialized processing module which contains a plurality of parallel pipelined hardware components that are programmable to provide respectively different, asynchronous specialized processing operations on an input stream of data;

a general processing module containing a general purpose microprocessor which controls hardware and software operation of said specialized processing module and said general processing module, using a hardware control library loaded on said general purpose microprocessor, said hardware control library comprising a set of functions for programming said parallel pipelined hardware of said at least one specialized processing module and said microprocessor of said general processing module to perform predetermined specialized processing operations on the input stream of data; and

a global control bus which provides control data to/from said hardware control library of said general processing module from/to said at least one specialized processing module separate from said stream of input data to be processed by said general processing module and said at least one specialized processing module.

28. The system of claim 27, wherein said stream of input data is coupled with an associated timing signal synchronized to a system clock signal, and each specialized processing module comprises a crosspoint switch which routes said stream of input data and its associated timing signal among the parallel pipelined hardware components, said timing signal indicating when the input data represents active information.

29. The system of claim 27, wherein said general processing module comprises at least two general purpose microprocessors and said hardware control library further comprises a set of functions for coordinating concurrent multitask processing operations of said at least two general purpose microprocessors.

30. The system of claim 27, wherein said hardware control library includes control signals for each hardware component of said specialized processing system, wherein said functions of said hardware control library manipulate said control signals to program said hardware components for each of said different specialized processing operations.