Real-time video frame pre-processing hardware

• US 10,418,001 B2
• Filed: 03/24/2017
• Issued: 09/17/2019
• Est. Priority Date: 07/07/2010
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Zaharia, Corneliu, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): Chang, Jon
• Application Number:
  US15/469,392
• Publication Number:
  US 20170256239A1
• Time in Patent Office: 907 Days
• CPC Class Codes: G06F 15/8046 Systolic arrays
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primitives from the image frame, and to store the primitives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame.

Claims

1. A system for processing a sequence of frames of video data based on data acquired for a plurality of pixels with an imaging sensor, comprising:

image processing circuitry providing one or more functions taken from the group consisting of adjusting exposure or white balance, demosaicing, color alteration, gamma correction, a converting between color-encoded signals and downsampling;

advanced hardware circuitry providing additional image processing functions, configured to operate in conjunction with said image processing circuitry, comprising a plurality of image processing modules, each including a processor, with multiple ones of the image processing modules arranged into processing chains according to a systolic architecture, where the image processing modules are interconnected and configured to perform a sequence of operations on data that flows between them, the image processing modules including one or more data processing units taken from the group consisting of a plurality of pixel processing modules, one or more frame processing modules, one or more region processing modules and one or more kernel processing modules;

a CPU connected to receive and process RGB data; and

data storage including an image and data cache and a long-term data store, the image and data cache configured to receive raw data from the sensor and RGB data from said image processing circuitry, and the long term data store configured to store processed RGB data received from the CPU and image data, based on the processed RGB data, in a MPEG format or in a JPEG format, the additional image processing functions of the advanced hardware circuitry providing in the data storage one or more scene processing primitives where the advanced hardware circuitry provides a primitive including a direct pixel to map-pixel mapping which includes a color thresholding, based on multiple thresholds, with data values of one or more of the thresholds used for determining how close a particular pixel value is to a predetermined color space value indicative of how close an image pixel is to skin color.

2. The system of claim 1 where the advanced hardware circuitry includes ports through which data streams can enter and leave an image processing module in the array according to one or more data counters by which each image processing module shares computed information with its neighbours immediately after processing according to alternating computation and communication phases.

3. The system of claim 1 configured to process one image frame on the CPU while a second image frame is being acquired and pre-processed by said image processing circuitry or the advanced hardware circuitry.

4. The system of claim 1 where the CPU is programmable to reconfigure the advanced hardware circuitry.

5. The system of claim 1 where the advanced hardware circuitry is programmed to process a pixel on every clock cycle and thereby not delay transfer of image pixels from the sensor.

6. The system of claim 1 where the data values of the thresholds, relative to one or more of pixel values, are used to differentiate different skin areas of a facial region.

7. A system for processing a sequence of frames of video data based on data acquired for a plurality of pixels with an imaging sensor, comprising:

image processing circuitry providing one or more functions taken from the group consisting of adjusting exposure or white balance, demosaicing, color alteration, gamma correction, a converting between color-encoded signals and downsampling;

advanced hardware circuitry providing additional image processing functions, configured to operate in conjunction with said image processing circuitry, comprising a plurality of image processing modules, each including a processor, with multiple ones of the image processing modules arranged into processing chains according to a systolic architecture, where the image processing modules are interconnected and configured to perform a sequence of operations on data that flows between them, the image processing modules including one or more data processing units taken from the group consisting of a plurality of pixel processing modules, one or more frame processing modules, one or more region processing modules and one or more kernel processing modules;

a CPU connected to receive and process RGB data; and

data storage including an image and data cache and a long-term data store, the image and data cache configured to receive raw data from the sensor and RGB data from said image processing circuitry, and the long term data store configured to store processed RGB data received from the CPU and image data, based on the processed RGB data, in a MPEG format or in a JPEG format, the additional image processing functions of the advanced hardware circuitry providing in the data storage one or more scene processing primitives taken from the group consisting of regional primitives, frame data primitives, threshold maps and integral image maps, said system applied to perform a method of determining how many pixels in a particular image frame fall within individual histogram bins defined for a set of skin-color values.

8. The system of claim 7 where the method is applied to determine whether to adjust skin color thresholds for a next image frame in the sequence.

9. A system for processing a sequence of frames of video data based on data acquired for a plurality of pixels with an imaging sensor, comprising:

image processing circuitry providing one or more functions taken from the group consisting of adjusting exposure or white balance, demosaicing, color alteration, gamma correction, a converting between color-encoded signals and downsampling;

advanced hardware circuitry providing additional image processing functions, configured to operate in conjunction with said image processing circuitry, comprising an image and data cache and a plurality of image processing modules, each including a processor, with multiple ones of the image processing modules arranged into processing chains according to a systolic architecture, where the image processing modules are interconnected and configured to perform a sequence of operations on data that flows between them, the image processing modules including one or more data processing units taken from the group consisting of a plurality of pixel processing modules, one or more frame processing modules, one or more region processing modules and one or more kernel processing modules;

one or more computer processing units connected to (i) program said image processing circuitry or the advanced hardware circuitry and (ii) receive and process the RGB data;

data storage including an image and data cache and a long-term data storage, the image and data cache configured to receive raw data from the sensor and RGB data from said image processing circuitry, and the long term data store configured to store processed RGB data received from the CPU and image data, based on the processed RGB data, in a MPEG format or in a JPEG format; and

a non-transitory computer-readable storage medium storing a set of computer instructions which, when executed, cause the one or more computer processing units to (i) program said image processing circuitry or the advanced hardware circuitry and (ii) provide processed RGB data to the image and data cache, where the computer instructions, when executed, also cause the one or more computer processing units to;

receive frame data of a first image frame;

generate, based on the frame data of the first image frame, an image processing primitive, wherein;

the image processing primitive is generated by identifying a sub-region of the first image frame and generating the image processing primitive from data of the sub-region; and

the image processing primitive comprises an image map; and

generate, based on the image map, an integral image (II) primitive by computing for a first pixel of the II primitive a sum of luminance values of all pixels located above and to the left from the first pixel in the image map; and

store the II primitive and the first image frame in the data storage.