KEYPOINT UNWARPING

• US 20130279751A1
• Filed: 04/24/2013
• Published: 10/24/2013
• Est. Priority Date: 04/24/2012
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Bruna, Arcangelo Ranieri, Pau, Danilo Pietro
• Granted Patent: US 10,579,904 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 30/18143 Extracting features based on salient regional features, e.g. scale invariant feature transform [SIFT] keypoints
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape

Abstract

Apparatus and methods to unwarp at least portions of distorted, electronically-captured images are described. Keypoints, instead of an entire image, may be unwarped and used in various machine-vision algorithms, such as object recognition, image matching, and 3D reconstruction algorithms. When using unwarped keypoints, the machine-vision algorithms may perform reliably irrespective of distortions that may be introduced by one or more image capture systems.

Claims

1. An image processing system having at least one processor adapted to unwarp keypoints, the system comprising:

a keypoint extractor configured to receive image data representative of a first image and to identify a plurality of keypoints within the image data and generate first keypoint data;

a keypoint transformer configured to transform the first keypoint data to second keypoint data based upon an image deformation model; and

a descriptor configured to produce, from the image data that has not been transformed based upon the image deformation model, descriptor data for at least some of the plurality of keypoints.

2. The image processing system of claim 1, wherein at least one of the keypoint extractor, the keypoint transformer, or the descriptor comprises a field-programmable gate array.

3. The image processing system of claim 1, further comprising a multiplexor configured to combine at least the descriptor data and second keypoint data into a data stream.

4. The image processing system of claim 1, wherein the first keypoint data comprises first spatial coordinates of the plurality of keypoints and the second keypoint data comprises second spatial coordinates that are transformations of the first spatial coordinates according to the image deformation model.

5. The image processing system of claim 4, further comprising:

a descriptor compressor configured to compress the descriptor data; and

a keypoint data compressor configured to compress the second keypoint data, wherein the descriptor compressor and keypoint data compressor provide respective compressed data to the multiplexor.

6. The image processing system of claim 4, wherein the second keypoint data further comprises image rotation and/or image magnification information.

7. The image processing system of claim 1, further comprising an image processor configured to execute a machine-vision algorithm using at least the second keypoint data and the descriptor data.

8. The image processing system of claim 1, further comprising a feature matcher configured to compare features of the first image with features of a second image based at least upon the second keypoint data and the descriptor data.

9. The image processing system of claim 1, further comprising an image capture device disposed in a smart phone, mobile phone, or personal digital assistant.

10. The image processing system of claim 1, wherein the image deformation model is representative of an image distortion introduced into the image by an image-capture device or is representative of an operation to remove image distortion introduced into the image by an image-capture device.

11. The image processing system of claim 1, wherein information about the image deformation model is received with the image data.

12. The image processing system of claim 1, further comprising image distortion detection apparatus configured to enable or disable the keypoint transformer based upon a type of image distortion detected.

13. An image processing method for electronically unwarping keypoints by at least one processor, the method comprising:

receiving, by a keypoint extractor, image data representative of a first image;

identifying, by a keypoint extractor, a plurality of keypoints within the image data;

generating, by the keypoint extractor, first keypoint data corresponding to the plurality of identified keypoints;

transforming, by a keypoint transformer, the first keypoint data to second keypoint data based upon an image deformation model; and

producing, by a descriptor, from the image data that has not been transformed based upon the image deformation model, descriptor data for at least some of the plurality of keypoints.

14. The image processing method of claim 13, wherein at least one of the keypoint extractor, the keypoint transformer, or the descriptor comprises a field-programmable gate array.

15. The image processing method of claim 13, further comprising multiplexing, by a multiplexor, at least the descriptor data and second keypoint data into a data stream.

16. The image processing method of claim 13, wherein transforming the first keypoint data comprises transforming spatial coordinates of the first keypoint data and not transforming spatial coordinates of the full image.

17. The image processing method of claim 13, further comprising compressing the second keypoint data and descriptor data.

18. The image processing method of claim 13, further comprising comparing features of the first image with features of a second image based at least upon the second keypoint data and the descriptor data.

19. An image processing system having at least one processor adapted to unwarp keypoints, the system comprising:

a matching model generator configured to receive image deformation data associated with a distorted image and to identify a matching model to a feature matcher; and

the feature matcher configured to receive keypoint data and descriptor data obtained from a distorted image, the matching model, and feature data associated with at least one comparison image, wherein the feature matcher unwarps keypoints in the keypoint data in accordance with the matching model.

20. The image processing system of claim 19, wherein at least one of the matching model generator or the feature matcher comprises a field-programmable gate array.

21. The image processing system of claim 19, wherein the image deformation data is representative of an image distortion introduced into the image by an image-capture device or is representative of an operation to remove image distortion introduced into the image by an image-capture device.

22. The image processing system of claim 19, further comprising a demultiplexor that is configured to parse the keypoint data, the descriptor data, and the image deformation data from received data.

23. The image processing system of claim 19, wherein the identified matching model is selected or generated based upon the received image deformation data.

24. The image processing system of claim 19, further comprising image distortion detection apparatus configured to enable or disable the matching model generator based upon a type of image distortion detected in the received image deformation data and a type of image distortion detected in the feature data.

25. An image processing method for electronically unwarping keypoints by at least one processor, the method comprising:

receiving, by a matching model generator, image deformation data associated with a distorted image identifying, by the matching model generator, a matching model to a feature matcher;

receiving, by the feature matcher, keypoint data obtained from a distorted image; and

unwarping, by the feature matcher, the keypoint data in accordance with the identified matching model.

26. The image processing method of claim 25, wherein at least one of the matching model generator or the feature matcher comprises a field-programmable gate array.

27. The image processing method of claim 25, further comprising receiving, by the feature matcher, descriptor data associated with the keypoint data.

28. The image processing method of claim 25, wherein the identifying comprises selecting or generating the matching model based upon the received image deformation data.

29. The image processing method of claim 25, further comprising:

receiving, by the feature matcher, feature data associated with at least one image;

detecting a first type of image distortion in the received image deformation data and a second type of image distortion in the feature data; and

making a determination to enable or disable the matching model generator based upon the first and second types of image distortion detected.