Real-Time Video Frame Pre-Processing Hardware

• US 20120008002A1
• Filed: 12/02/2010
• Published: 01/12/2012
• Est. Priority Date: 07/07/2010
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Tessera Technologies Ireland Limited (Adeia Inc.)
• Inventors: Bigioi, Petronel, Florea, Corneliu, Corcoran, Peter
• Granted Patent: US 9,053,681 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06F 15/8046 Systolic arrays
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primatives from the image frame, and to store the primatives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame.

Claims

1. A dynamically reconfigurable heterogeneous systolic array configured to process a first image frame, and to generate a plurality of image processing primatives from said image frame, and to store said primatives and the corresponding image frame in a memory store, wherein based on a determined characteristic of said image frame, the dynamically reconfigurable heterogeneous systolic array is reconfigurable to process a following image frame.

2. The dynamically reconfigurable heterogeneous systolic array of claim 1, wherein the determining of said at least one characteristic is performed external to the systolic array.

3. The dynamically reconfigurable heterogeneous systolic array of claim 1, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.

5. The device of claim 1, wherein the determining of said at least one characteristic is performed external to the dynamically reconfigurable heterogeneous systolic array.

6. The device of claim 1, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.

4. An image acquisition and processing device comprising:

a processor;

a lens and image sensor for acquiring digital image frames;

a dynamically reconfigurable heterogeneous systolic array configured to process a first image frame, and to generate a plurality of image processing primatives from said image frame, and to store said primatives and the corresponding image frame in a memory store, and

wherein the device is configured to determine at least one characteristic of said image frame and, based on the at least one characteristic, to reconfigure said dynamically reconfigurable heterogeneous systolic array to process a following image frame.

7. A method of image processing using a dynamically reconfigurable heterogeneous systolic array, comprising:

acquiring and processing a first image frame,

generating a plurality of image processing primatives from said image frame,

storing said primatives and the corresponding image frame in a memory store,

determining at least one characteristic of said image frame, and

based on the at least one characteristic, reconfiguring said array to process a following image frame.

8. The method of claim 7, wherein the determining of said at least one characteristic is performed external to the systolic array.

9. The method of claim 7, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.

10. An digital image acquisition and processing device, comprising:

a lens and image sensor for acquiring digital images;

an image processing pipeline (IPP) module for receiving raw image data of an image frame and distributing corresponding formatted image data to a memory and a hardware module configured to perform image processing;

the hardware module for receiving the formatted image data of the image frame from the IPP module and providing scene processing primatives to the memory;

the memory for storing therein the scene processing primatives provided by the hardware module;

a processing unit configured to process a following image frame based on the scene processing primatives received from the memory and provided by the hardware module.

11. The device of claim 10, wherein the scene processing primatives comprise processed image maps or other processed hardware module data, or both.

12. The device of claim 10, wherein the hardware module is also configured to provide formatted image data to the memory.

13. The device of claim 10, wherein the processed image maps comprise threshold maps or integral image maps, or both.

14. The device of claim 10, wherein the scene processing primatives comprise regional primatives or frame data primatives, or both.

15. The device of claim 10, wherein the processing unit also receives formatted image data from the memory.

16. The device of claim 10, further comprising a transcode module configured to receive processed image data from the processing unit and to provide to or receive from the memory compressed image or video data, or both.

17. The device of claim 10, wherein the transcode module is further configured to output selected subsets of the image data or video data, or both, to a display.

18. The device of claim 10, wherein the hardware module is configured to process image data within a single clock cycle.

19. The device of claim 10, wherein the processing unit is further configured to program the hardware module or the IPP module, or both.

20. A method of acquiring and processing digital images, comprising:

acquiring digital images;

receiving raw image data of an image frame and distributing corresponding formatted image data to a memory and a hardware-based image processing module;

receiving at the hardware-based image processing module the formatted image data of the image frame and providing scene processing primatives to a memory;

storing in the memory the scene processing primatives provided by the hardware-based image processing module; and

processing a following image frame based on the scene processing primatives received from the memory and provided by the hardware-based image processing module.

21. The method of claim 20, wherein the scene processing primatives comprise processed image maps or other processed hardware module data, or both.

22. The method of claim 20, comprising providing formatted image data to the memory from the hardware-based image processing module.

23. The method of claim 20, comprising processing image maps including threshold maps or integral image maps, or both.

24. The method of claim 20, wherein the scene processing primatives comprise regional primatives or frame data primatives, or both.

25. The method of claim 20, comprising receiving formatted image data from the memory.

26. The method of claim 20, comprising transmitting processed image data to a transcode module and communicating compressed image or video data to or from the transcode module, or combinations thereof.

27. The method of claim 20, comprising outputting selected subsets of the image data or video data, or both, to a display.

28. The method of claim 20, comprising processing image data within a single clock cycle with the hardware-based image processing module.

29. The method of claim 20, comprising programming the hardware-based image processing module.

30. A hardware-based image processing module for operating within an image acquisition and processing device that acquires raw image data and distributes corresponding formatted image data to the hardware-based image processing module which is configured to generate and provide scene processing primatives to a memory based on the formatted image data to be stored in the memory for processing a following image frame based on the scene processing primatives received from the memory and provided by the hardware-based image processing module.

31. The hardware-based image processing module of claim 30, wherein the scene processing primatives comprise processed image maps or other processed hardware module data, or both.

32. The hardware-based image processing module of claim 30, wherein the module is further configured to provide formatted image data to the memory.

33. The hardware-based image processing module of claim 30, wherein the module is further configured to process image maps including threshold maps or integral image maps, or both.

34. The hardware-based image processing module of claim 30, wherein the scene processing primatives comprise regional primatives or frame data primatives, or both.

35. The hardware-based image processing module of claim 30, wherein the module is further configured to receive and process formatted image data from the memory.

36. The hardware-based image processing module of claim 30, wherein the module is further configured to transmit processed image data to a transcode module and communicate compressed image or video data to or from the transcode module, or combinations thereof.

37. The hardware-based image processing module of claim 30, wherein the module is further configured to output selected subsets of the image data or video data, or both, to a display.

38. The hardware-based image processing module of claim 30, wherein the module is further configured to process image data within a single clock cycle with the hardware-based image processing module.

39. The hardware-based image processing module of claim 30, wherein the module is programmable.

40. A dynamically reconfigurable heterogeneous systolic array configured to:

process a first image frame;

generate a plurality of image processing primatives from said image frame;

identify the location of at least one subregion of said image frame and generate at least one

image processing primative from said subregion; and

store said primatives along with the image frame and subregion location in a memory store, and

wherein the dynamically reconfigurable heterogeneous systolic array is reconfigurable to process a following image frame based on one or more determined characteristics of the image frame or subregion, or both,

41. The dynamically reconfigurable heterogeneous systolic array of claim 40, wherein the determining of said at least one characteristic is performed external to the systolic array.

42. The dynamically reconfigurable heterogeneous systolic array of claim 40, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.

43. An image acquisition and processing device comprising:

a processor;

a lens and image sensor for acquiring digital image frames;

a dynamically reconfigurable heterogeneous systolic array configured to;

process a first image frame;

generate a plurality of image processing primatives from said image frame;

identify the location of at least one subregion of said image frame and generate at least one image processing primative from said subregion;

store said primatives along with the image frame and subregion location in a memory store,

wherein the device is configured to determine at least one characteristic of the image frame or subregion, or both, and to reconfigure the array based on said at least one characteristic to process a following image frame.

44. The image acquisition and processing device of claim 43, wherein the determining of said at least one characteristic is performed in part external to the dynamically reconfigurable heterogeneous systolic array.

45. The image acquisition and processing device of claim 43, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.

46. A method of image processing using a dynamically reconfigurable heterogeneous systolic array, comprising:

acquiring and processing a first image frame;

generating a plurality of image processing primatives from said image frame;

identifying the location of at least one subregion of said image frame and generate at least one image processing primative from said subregion;

storing said primatives, along with the image frame and subregion location in a memory store,

determining at least one characteristic of the image frame or subregion, or both; and

based on said characteristic, reconfiguring said array to process in a following image frame.

47. The method of claim 46, wherein the determining of said at least one characteristic is performed in part external to the dynamically reconfigurable heterogeneous systolic array.

48. The method of claim 46, wherein the determining of the at least one characteristic is based on processing at least a portion of the image frame and at least one of said image processing primatives.