Systems and methods for sampling an image sensor

• US 7,483,059 B2
• Filed: 04/30/2004
• Issued: 01/27/2009
• Est. Priority Date: 04/30/2004
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Obrador, Pere
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): Hsu; Amy
• Application Number:
  US10/836,745
• Publication Number:
  US 20050243183A1
• Time in Patent Office: 1,733 Days
• Field of Search: 348/297, 348/222.1, 348/572, 382/118, 382/274, 345/426
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 25/445 by skipping some contiguous pixels within the read portion of the array

Abstract

Systems and methods according to the present invention provide techniques which adjust the sampling scheme associated with an image sensor in order to provide high quality renditions, e.g., printed or displayed outputs. The sampling scheme can be varied on a frame by frame basis (or less frequently) to output more samples from the image sensor for regions which are predicted to contain more perceptually significant information and to output fewer samples for regions which are predicted to be less perceptually significant. Various parameters can be used as proxies for perceptual significance. For example, in images which contain people, faces can be characterized as more perceptually significant and regions which are predicted to contain faces can be sampled more densely than regions which are not predicted to contain faces. Alternatively, the spatial frequency content of data captured by an image sensor can be measured to identify regions of higher and lower spatial frequency content. Regions with higher spatial frequency content will typically contain perceptually more significant image features, e.g., edges and textures, than regions having lower spatial frequency content. Accordingly, the image sensor can be programmed to sample the former more densely.

Claims

1. A method for sampling an image sensor comprising the steps of:

evaluating a first captured image;

determining a sampling allocation for said image sensor based upon said step of evaluating;

capturing a second image by sampling a first region of said image sensor at a first sampling density and a second region of said image sensor at a second sampling density based upon said determined sampling allocation; and

wherein the sampling provides image data of the second image comprising both a first portion having the first sampling density and a second portion having the second sampling density according to the determining.

2. The method of claim 1, wherein said step of evaluating further comprises the step of:

processing said captured image to determine spatial frequency content associated with pixel data captured by said image sensor.

3. The method of claim 2, wherein said step of determining a sampling allocation further comprises the step of:

assigning said first sampling density to said first region of said image sensor which has a lower spatial frequency content than that associated with said second region, wherein said first sampling density is less than said second sampling density.

4. The method of claim 1, wherein said step of evaluating further comprises the step of:

processing said captured image to determine a region containing a face in said captured image.

5. The method of claim 4, wherein said first region includes said face and said second region includes a remainder and wherein said first sampling density is greater than said second sampling density.

6. The method of claim 1, wherein said step of evaluating further comprises the step of:

evaluating said captured image to determine a plurality of regions, each of said plurality of regions having a different perceived quality value; and

wherein said step of determining further comprises the step of;

allocating more samples per unit area to those of said plurality of regions having a higher perceived quality value and allocating fewer samples per unit area to those of said plurality of regions having a lower perceived quality value.

7. The method of claim 1, wherein said step of determining said sample allocation further comprises the step of:

determining said sample allocation as a function of a bandwidth associated with a data transfer mechanism used to output samples from said image sensor and quantitative quality values output as a result of said evaluating step.

8. The method of claim 7, wherein said function is a Lagrangian optimization.

23. The method of claim 1 further comprising capturing the first image at a first moment in time using the image sensor and wherein capturing of the second image comprises capturing the second image at a second moment in time subsequent to the first moment in time.

24. The method of claim 1 wherein the evaluating comprises identifying portions of the captured image as comprising regions having high and low frequency content, and the sampling comprises sampling to provide the image data comprising the first and second sampling densities corresponding to the first and second portions of the second image.

25. The method of claim 1 further comprising prior to the evaluating, capturing the first image using all pixels of the image sensor at a first moment in time and wherein the sampling provides the image data of the single second image using only a portion of the pixels comprising less than an entirety of the pixels of the image sensor at a second moment in time and according to the determining, the second moment in time occurring subsequent to the first moment in time.

26. The method of claim 25 wherein the evaluating comprises identifying portions of the captured image as comprising regions having high and low frequency content, and the sampling comprises sampling to provide the image data comprising the first and second sampling densities corresponding to the first and second portions of the second image.

33. The method of claim 23 wherein the capturing of the first image at the first moment in time comprises exposing the image sensor to first light and the capturing of the second image at the second moment in time comprises exposing the image sensor to second light, the second light being different than the first light.

9. A computer-readable medium encoded with a computer program that performs the steps of:

evaluating a captured image;

determining a sampling allocation for an image sensor based upon said step of evaluating, the image sensor being configured to produce electrical signals representing light to which the image sensor is exposed; and

controlling the image sensor to provide sampling of a first region of said image sensor at a first sampling density and a second region of said image sensor at a second sampling density based upon said determined sampling allocation to provide another captured image.

10. The computer-readable medium of claim 9, wherein said step of evaluating further comprises the step of:

processing said captured image to determine spatial frequency content associated with pixel data captured by said image sensor.

11. The computer-readable medium of claim 10, wherein said step of determining a sampling allocation further comprises the step of:

assigning said first sampling density to said first region of said image sensor which has a lower spatial frequency content than that associated with said second region, wherein said first sampling density is less than said second sampling density.

12. The computer-readable medium of claim 9, wherein said step of evaluating further comprises the step of:

evaluating said captured image to determine a plurality of regions, each of said plurality of regions having a different perceived quality value; and

wherein said step of determining further comprises the step of;

allocating more samples per unit area to those of said plurality of regions having a higher perceived quality value and allocating fewer samples per unit area to those of said plurality of regions having a lower perceived quality value.

13. The computer-readable medium of claim 9, wherein said step of determining said sample allocation further comprises the step of:

determining said sample allocation as a function of a bandwidth associated with a data transfer mechanism used to output samples from said image sensor and quantitative quality values output as a result of said evaluating step.

14. The computer-readable medium of claim 13, wherein said function is a Lagrangian optimization.

27. The computer-readable medium of claim 9 wherein the captured image comprises a first image and the program further performs, using the sampling, capturing a single second image comprising both a first portion having the first sampling density and a second portion having the second sampling density.

28. The computer-readable medium of claim 9 wherein the captured image comprises a first image and the program further performs:

prior to the evaluating and using the image sensor, capturing first digital data representing the first image from a first number of pixels of the image sensor; and

using the sampling, capturing second digital data representing a second image, wherein the second digital data corresponds to a second number of pixels of the image sensor less than the first number of pixels.

15. An image processing device comprising:

an image sensor for capturing an image, wherein said image sensor includes an array of image sensing pixels, wherein said image sensor is controllable to vary a sampling density for at least two regions across said array of image sensing pixels;

a processor for evaluating a captured image and for allocating a first sampling density to a first region of said image sensor and a second sampling density to a second region of said image sensor based on said evaluation of said captured image; and

a data transfer mechanism for transferring samples associated with a next image from said image sensor to said processor, wherein said samples are generated based on said first and second sampling densities.

16. The image processing device of claim 15, wherein said processor evaluates said captured image to determine spatial frequency content associated with pixel data captured by said image sensor.

17. The image processing device of claim 16, wherein said processor assigns said first sampling density to said first region of said image sensor which has a lower spatial frequency content than that associated with said second region, wherein said first sampling density is less than said second sampling density.

18. The image processing device of claim 15, wherein said processor evaluates said captured image to determine a region containing a face in said captured image.

19. The image processing device of claim 15, wherein said processor evaluates said captured image to determine a plurality of regions, each of said plurality of regions having a different perceived quality value and allocates more samples per unit area of said image sensor to those of said plurality of regions having a higher perceived quality value and allocates fewer samples per unit area of said image sensor to those of said plurality of regions having a lower perceived quality value.

20. The image processing device of claim 15, wherein said processor determines said sample allocation as a function of a bandwidth associated with said data transfer mechanism and quantitative quality values output as a result of said evaluating step.

21. The image processing device of claim 20, wherein said function is a Lagrangian optimization.

29. The image processing device of claim 15 wherein the captured image comprises a first captured image and the next image comprises a second captured image, the second captured image being captured subsequent to the first captured image and the second captured image comprising both a first portion having the first sampling density and a second portion having the second sampling density.

30. The image processing device of claim 15 wherein:

the image sensor is configured to produce electrical signals representing light to which the image sensor is exposed; and

the processor is configured to utilize the evaluation of the captured image to control the image sensor to generate the samples for the next image having the first and second densities corresponding to the first and second regions of the image sensor.

31. The device of claim 15 wherein the captured image is captured at a first moment in time and the next image is captured at a second moment in time, the second moment in time being subsequent to the first moment in time.

32. The device of claim 15 wherein the image sensor is configured to produce electrical signals representing light to which the image sensor is exposed.

22. An image processing system comprising:

means for evaluating a first captured image;

means for determining a sampling allocation for an image sensor based upon said step of evaluating; and

means for capturing a second image by sampling a first region of said image sensor at a first sampling density and a second region of said image sensor at a second sampling density based upon said determined sampling allocation.