Real-time video frame pre-processing hardware

• US 9,053,681 B2
• Filed: 12/02/2010
• Issued: 06/09/2015
• Est. Priority Date: 07/07/2010
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Bigioi, Petronel, Zaharia, Corneliu, Corcoran, Peter
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US12/959,281
• Publication Number:
  US 20120008002A1
• Time in Patent Office: 1,650 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 15/8046 Systolic arrays
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primatives from the image frame, and to store the primatives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame.

Claims

1. An apparatus comprising one or more hardware processors, memory, and one or more modules configured to:

process, in a dynamically reconfigurable heterogeneous systolic array, a first image frame to generate a plurality of image processing primitives from the first image frame;

wherein the plurality of image processing primitives comprises at least one image map;

store the plurality of image processing primitives and the first image frame in a memory store;

based, at least in part, on the plurality of image processing primitives, determine one or more characteristics of the first image frame;

based on the one or more characteristics determined for the first image frame, reconfigure the dynamically reconfigurable heterogeneous systolic array; and

process, using the reconfigured dynamically reconfigurable heterogeneous systolic array, a second image frame based on the one or more characteristics.

2. The apparatus of claim 1, wherein an image processing primitive, of the plurality of image processing primitives, is generated by identifying a sub-region of the first image frame and generating the image processing primitive from m the sub-region.

3. The apparatus of claim 1,

wherein the first image frame and the second image frame are video frames.

4. The apparatus of claim 1, wherein the plurality of image processing primitives comprises processed image maps or other processed hardware module data, or both.

5. The apparatus of claim 1, wherein the apparatus is also configured to provide formatted image data to a memory.

6. The apparatus of claim 4, wherein the processed image maps comprise threshold maps or integral image maps, or both.

7. The apparatus of claim 1, wherein the plurality of image processing primitives comprises regional primitives or frame data primitives, or both.

8. The apparatus of claim 1, wherein the apparatus is further configured to receive formatted image data from a memory.

9. The apparatus of claim 1, wherein the apparatus further comprises a transcode module configured to receive processed image data and to provide to or receive from a memory compressed image or video data, or both.

10. The apparatus of claim 9, wherein the transcode module is further configured to output selected subsets of the image data or video data, or both, to a display.

11. The apparatus of claim 1, wherein the apparatus is configured to process image data within a single clock cycle.

12. The apparatus of claim 1, wherein the apparatus is further configured to program a hardware module or an IPP module, or both.

13. An image acquisition and processing device comprising:

a processor;

a lens and image sensor for acquiring digital image frames;

a dynamically reconfigurable heterogeneous systolic array configured to;

process, in a dynamically reconfigurable heterogeneous systolic array, a first image frame to generate a plurality of image processing primitives from the first image frame;

wherein an image processing primitive, of the plurality of image processing primitives, is generated by identifying a sub-region of the first image frame and generating the image processing primitive from the sub-region;

wherein the plurality of image processing primitives comprises at least one image map;

store the plurality of image processing primitives and the first image frame in a memory store;

based, at least in part, on the plurality of image processing primitives, determine one or more characteristics of the first image frame;

based on the one or more characteristics determined for the first image frame, reconfigure the dynamically reconfigurable heterogeneous systolic array; and

process, using the reconfigured dynamically reconfigurable heterogeneous systolic array, a second image frame based on the one or more characteristics.

14. The device of claim 13, wherein an image processing primitive, of the plurality of image processing primitives, is generated by identifying a sub-region of the first image frame and generating the image processing primitive from the sub-region.

15. The device of claim 13,

wherein the first image frame and the second image frame are video frames.

16. A method, comprising:

processing, in a dynamically reconfigurable heterogeneous systolic array, a first image frame to generate a plurality of image processing primitives from the first image frame;

wherein the plurality of image processing primitives comprises at least one image map;

storing the plurality of image processing primitives and the first image frame in a memory store;

based, at least in part, on the plurality of image processing primitives, determining one or more characteristics of the first image frame;

based on the one or more characteristics determined for the first image frame, reconfiguring the dynamically reconfigurable heterogeneous systolic array; and

processing, using the reconfigured dynamically reconfigurable heterogeneous systolic array, a second image frame based on the one or more characteristics;

wherein the method is performed using one or more computing devices.

17. The method of claim 16, wherein an image processing primitive, of the plurality of image processing primitives, is generated by identifying a sub-region of the first image frame and generating the image processing primitive from the sub-region.

18. The method of claim 16,

wherein the first image frame and the second image frame are video frames.

19. The method of claim 16, wherein the plurality of image processing primitives comprises processed image maps or other processed hardware module data, or both.

20. The method of claim 16, further comprising providing formatted image data to a memory from a hardware-based image processing module.

21. The method of claim 16, further comprising processing image maps, including threshold maps or integral image maps, or both.

22. The method of claim 16, wherein the plurality of image processing primitives comprises regional primitives or frame data primitives, or both.

23. The method of claim 16, further comprising receiving formatted image data from a memory.

24. The method of claim 16, further comprising transmitting processed image data to a transcode module and communicating compressed image or video data to or from the transcode module, or combinations thereof.

25. The method of claim 16, further comprising outputting selected subsets of image data or video data, or both, to a display.

26. The method of claim 16, further comprising processing image data within a single clock cycle with a hardware-based image processing module.

27. The method of claim 16, further comprising programming a hardware-based image processing module.

28. A non-transitory computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the processors to perform:

processing in a dynamically reconfigurable heterogeneous systolic array, a first image frame to generate a plurality of image processing primitives from the first image frame;

wherein the plurality of image processing primitives comprises at least one image map;

storing the plurality of image processing primitives and the first image frame in a memory store;

based, at least in part, on the plurality of image processing primitives, determining one or more characteristics of the first image frame;

based on the one or more characteristics determined for the first image frame, reconfiguring the dynamically reconfigurable heterogeneous systolic array; and

processing, using the reconfigured dynamically reconfigurable heterogeneous systolic array, a second image frame based on the one or more characteristics.

29. The non-transitory computer-readable storage medium of claim 28, wherein the plurality of image processing primitives comprises processed image maps or other processed hardware module data, or both.

30. The non-transitory computer-readable storage medium of claim 28 storing additional instructions which, when executed by the one or more processors, cause the processors to perform providing formatted image data to a memory.

31. The non-transitory computer-readable storage medium of claim 28, storing additional instructions which, when executed by the one or more processors, cause the processors to perform processing image maps including threshold maps or integral image maps, or both.

32. The non-transitory computer-readable storage medium of claim 28, wherein the plurality of image processing primitives comprises regional primitives or frame data primitives, or both.

33. The non-transitory computer-readable storage medium of claim 28, storing additional instructions which, when executed by the one or more processors, cause the processors to perform receiving and processing formatted image data from a memory.

34. The non-transitory computer-readable storage medium of claim 28, storing additional instructions which, when executed by the one or more processors, cause the processors to perform transmitting processed image data to a transcode module and communicating compressed image or video data to or from the transcode module, or combinations thereof.

35. The non-transitory computer-readable storage medium of claim 28, storing additional instructions which, when executed by the one or more processors, cause the processors to perform outputting selected subsets of image data or video data, or both, to a display.

36. The non-transitory computer-readable storage medium of claim 28, storing additional instructions which, when executed by the one or more processors, cause the processors to perform processing image data within a single clock cycle with a hardware-based image processing module.

37. The non-transitory computer-readable storage medium of claim 36, wherein the hardware-based image processing module is programmable.