Sensor noise profile

• US 9,892,517 B2
• Filed: 12/18/2015
• Issued: 02/13/2018
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Arm Ltd. (United Kingdom) (SoftBank Group Corp.)
• Original Assignee: Apical Limited (SoftBank Group Corp.)
• Inventors: Romanenko, Ilya
• Primary Examiner(s): Saini, Amandeep
• Application Number:
  US14/973,937
• Publication Number:
  US 20160180541A1
• Time in Patent Office: 788 Days
• Field of Search: 382103
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

The invention relates to feature extraction technique based on edge extraction. It can be used in computer vision systems, including image/facial/object recognition systems, scene interpretation, classification and captioning systems. A model or profile of the noise in the sensor is used to improve feature extraction or object detection on an image from a sensor.

Claims

1. A method of extracting a feature from an image, the method comprising performing, in a processor or circuitry, steps of:

receiving a digital image from a sensor;

demosaicing the digital image;

prior to demosaicing the digital image, taking an input image from the digital image for feature extraction; and

performing feature extraction on the input image, wherein the step of performing feature extraction comprises;

using a model or profile of noise comprising a variance of the noise in the sensor in defined areas of the input image to normalize a response of a feature extraction algorithm;

applying the feature extraction algorithm to the input image; and

for each pixel of the input image;

calculating an edge response for an orientation, and

normalizing the edge response for the orientation by taking into account the variance in the noise in the sensor.

2. The method of claim 1, wherein the feature is an edge.

3. The method of claim 2, wherein the feature extraction is based on detection of the edge.

4. The method of claim 1, wherein the feature is a local binary pattern.

5. The method of claim 1, wherein the method is implemented in a system that is not part of, does not use, or is not downstream of, an image processing pipeline.

6. The method of claim 1, wherein the method operates operating in a RAW domain with linear data.

7. The method of claim 1, wherein the edge response is calculated by convoluting a filter kernel with an intensity of the input image.

8. The method of claim 7, wherein the filter kernel is a Gabor filter kernel or a Convolutional Neural Network filter kernel.

9. The method of claim 1, in which the normalizing of the edge response for an orientation ∝

is calculated for each pixel (x,y) in the input image from;

10. The method of claim 1, wherein the input image is RAW image sensor data.

11. The method of claim 1, wherein the edge response is fed into a linear classifier comprising a Support Vector Machine or into a classification layer of a Convolutional Neural Network.

12. The method of claim 1, wherein the method is implemented in real-time.

13. The method of claim 1, wherein the method operates as a part of a computer vision system, applied to posture detection, people detection, object detection in the digital image.

14. The method of claim 1, wherein the method is utilized in one of the following:

Smartphone;

Computer vision systems;

Objects recognition systems;

Human detection systems;

Facial recognition systems;

Scene interpretation systems;

Image classification systems;

Image captioning systems;

Autonomous vehicle computer vision systems;

Robotics computer vision systems.

15. The method of claim 1, wherein the method is implemented in an embedded hardware block.

16. Image processing hardware configured to:

receive a digital image from a sensor;

demosaic the digital image;

prior to demosaicing the digital image, take an input image from the digital image for feature extraction; and

perform feature extraction on the input image, wherein the step of performing feature extraction comprises;

using a model or profile of noise comprising a variance of the noise in the sensor in defined areas of the input image to normalize a response of a feature extraction algorithm;

applying the feature extraction algorithm to the input image; and

for each pixel of the input image;

calculating an edge response for an orientation, and

normalizing the edge response for the orientation by taking into account the variance in the noise in the sensor.

17. A device including image processing hardware configured to:

receive a digital image from a sensor;

demosaic the digital image;

prior to demosaicing the digital image, take an input image from the digital image for feature extraction; and

perform feature extraction on the input image, wherein the step of performing feature extraction comprises;

using a model or profile of noise in the sensor to normalize a response of a feature extraction algorithm;

applying the feature extraction algorithm to the input image; and

for each pixel of the input image;

calculating an edge response for an orientation, and

normalizing the edge response for the orientation by taking into account the variance in the noise in the sensor.

18. The device of claim 17, wherein the device comprises or is a part of one of the following:

Smartphone;

Computer vision system;

Objects recognition system;

Human detection system;

Facial recognition system;

Scene interpretation system;

Image classification system;

Image captioning system;

Autonomous vehicle computer vision system;

Robotics computer vision system.