Real-time video frame pre-processing hardware

• US 9,607,585 B2
• Filed: 10/27/2014
• Issued: 03/28/2017
• Est. Priority Date: 07/07/2010
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Zaharia, Corneliu, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US14/524,877
• Publication Number:
  US 20150042670A1
• Time in Patent Office: 883 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 15/8046 Systolic arrays
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primatives from the image frame, and to store the primatives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame.

Claims

1. An image processing hardware unit comprising:

a frame receiving component configured to;

receive, in a dynamically reconfigurable heterogeneous systolic array, frame data of a first image frame;

a frame processing component configured to;

in the dynamically reconfigurable heterogeneous systolic array, generate, based on the frame data of the first image frame, an integral image primitive;

wherein the integral image primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing, for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the sub-region of the first image frame;

store the integral image primitive and the first image frame in a memory store;

based, at least in part, on the integral image primitive, determine at least one characteristic;

based, at least in part, on the at least one characteristic, reconfigure the dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame.

2. The image processing hardware unit of claim 1, further comprising an integral image accumulator, wherein the integral image accumulator is further configured to:

generate, based on the integral image primitive, an integral image square (primitive as a product of the integral image primitive multiplied by the integral image primitive; and

store the integral image square primitive in the memory store.

3. The image processing hardware unit of claim 2, wherein the integral image accumulator is further configured to:

for each pixel of the sub-region, determine a similarity score indicating a similarity between a color of the pixel and a skin color;

determine a skin integral image primitive based on the similarity scores determined for the sub-region; and

store the skin integral image primitive in the memory store.

4. The image processing hardware unit of claim 3,

wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;

wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and

wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.

5. The image processing hardware unit of claim 4, wherein the frame processing component is further configured to:

determine whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and

determine whether the variance is high, and if so, determine that the sub-region probably depicts the face.

6. The image processing hardware unit of claim 5, wherein the first image frame is a video frame.

7. The image processing hardware unit of claim 6, wherein the frame processing component is further configured to:

generate, based on the frame data of the first image frame, a skin-map primitive;

wherein the skin-map primitive is a bit-map; and

store the skin-map primitive in the memory store.

8. An image processing method comprising:

receiving frame data of a first image frame;

generating, based on the frame data of the first image frame, an integral image primitive;

wherein the image processing primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the sub-region of the first image frame;

storing the integral image primitive and the first image frame in a memory store;

based, at least in part, on the integral image primitive, determining at least one characteristic,

based, at least in part, on the at least one characteristic, reconfiguring a dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame;

wherein the method is performed using one or more computing devices.

9. The image processing method of claim 8, further comprising:

generating, based on the integral image primitive, an integral image square primitive as a product of the integral image primitive multiplied by the integral image primitive; and

storing the integral image square primitive in the memory store.

10. The image processing method of claim 9, further comprising:

for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color;

determining a skin integral image primitive based on the similarity scores determined for the sub-region; and

storing the skin integral image primitive in the memory store.

11. The image processing method of claim 10,

wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;

wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and

wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.

12. The image processing method of claim 11, further comprising:

determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and

determining whether the variance is high, and if so, determine that the sub-region probably depicts the face.

13. The image processing method of claim 12, wherein the first image frame is a video frame.

14. The image processing method of claim 13, further comprising:

generating, based on the frame data of the first image frame, a skin-map primitive;

wherein the skin-map primitive is a bit-map; and

storing the skin-map primitive in the memory store.

15. A non-transitory computer-readable storage medium storing one or more computer instructions which, when executed, cause one or more processors to perform:

receiving frame data of a first image frame;

generating, based on the frame data of the first image frame, an integral image primitive;

wherein the integral image primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the first image frame; and

storing the integral image primitive and the first image frame in a memory store;

based, at least in part, on the integral image primitive, determine at least one characteristic;

based, at last in part, on the at least one characteristic, reconfiguring a dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame.

16. The non-transitory computer-readable storage medium of claim 15, further comprising additional instructions which, when executed, cause the one or more processors to perform:

generating, based on the integral image primitive, an integral image square primitive as a product of the integral image primitive multiplied by the integral image primitive; and

storing the integral image primitive in the memory store.

17. The non-transitory computer-readable storage medium of claim 16, further comprising additional instructions which, when executed, cause the one or more processors to perform:

for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color;

determining a skin integral image primitive based on the similarity scores determined for the sub-region; and

storing the skin integral image primitive in the memory store.

18. The non-transitory computer-readable storage medium of claim 17,

wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;

wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and

wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.

19. The non-transitory computer-readable storage medium of claim 18, further comprising additional instructions which, when executed, cause the one or more processors to perform:

determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and

determining whether the variance is high, and if so, determine that the sub-region probably depicts the face.

20. The non-transitory computer-readable storage medium of claim 19, wherein the first image frame is a video frame.