Keypoint unwarping for machine vision applications

• US 10,579,904 B2
• Filed: 04/24/2013
• Issued: 03/03/2020
• Est. Priority Date: 04/24/2012
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Bruna, Arcangelo Ranieri, Pau, Danilo Pietro
• Primary Examiner(s): Bella, Matthew C
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US13/869,639
• Publication Number:
  US 20130279751A1
• Time in Patent Office: 2,505 Days
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 30/18143 Extracting features based on salient regional features, e.g. scale invariant feature transform [SIFT] keypoints
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape

Abstract

Apparatus and methods to unwarp at least portions of distorted, electronically-captured images are described. Keypoints, instead of an entire image, may be unwarped and used in various machine-vision algorithms, such as object recognition, image matching, and 3D reconstruction algorithms. When using unwarped keypoints, the machine-vision algorithms may perform reliably irrespective of distortions that may be introduced by one or more image capture systems.

Claims

1. An image processing system, comprising:

one or more memories, which, in operation, store image data; and

image processing circuitry, which, in operation;

identifies a plurality of keypoints within received image data representative of a first image;

generates, using the received image data representative of the first image, descriptor data for at least some keypoints within the received image data;

transforms a subset of keypoints of the identified plurality of keypoints within the received image data representative of the first image, wherein the subset of keypoints does not include all of the keypoints of the identified plurality of keypoints, the transforming the subset of keypoints including;

generating, for the identified plurality of keypoints within the received image data, first keypoint data; and

transforming, using an image deformation model, first keypoint data of only the subset of keypoints of the identified plurality of keypoints, producing second keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints;

generates output image data based on the second keypoint data and the descriptor data; and

determines whether one or more features of the first image match one or more features of at least one comparison image based on the output image data.

2. The image processing system of claim 1 wherein the image processing circuitry comprises a field-programmable gate array.

3. The image processing system of claim 1, comprising a multiplexor, which, in operation combines at least the descriptor data and second keypoint data into a data stream.

4. The image processing system of claim 1, wherein the first keypoint data comprises first spatial coordinates of the plurality of keypoints and the second keypoint data comprises second spatial coordinates that are transformations of the first spatial coordinates of the subset of keypoints of the identified plurality of keypoints according to the image deformation model.

5. The image processing system of claim 1 wherein the image processing circuitry, in operation,

compresses the descriptor data;

compresses the second keypoint data; and

combines the compressed descriptor data and the compressed second keypoint data into a data stream.

6. The image processing system of claim 4, wherein the second keypoint data further comprises image rotation and/or image magnification information.

7. The image processing system of claim 1 wherein the image processing circuitry, in operation, executes a machine-vision algorithm using the generated image output data.

8. The image processing system of claim 1 wherein the image processing circuitry, in operation, determines whether one or more features of the first image match one or more features of the at least one comparison image based on second keypoint data included in the output image data.

9. The image processing system of claim 1, further comprising an image sensor disposed in a smart phone, mobile phone, or personal digital assistant.

10. The image processing system of claim 1, wherein the image deformation model is representative of an image distortion introduced into the image by an image-capture device or is representative of an operation to remove image distortion introduced into the image by an image-capture device.

11. The image processing system of claim 1, wherein information about the image deformation model is received with the received image data.

12. The image processing system of claim 1 wherein the image processing circuitry, in operation, enables or disables the transforming based upon a type of image distortion detected.

13. The image processing system of claim 1 wherein the image processing circuitry, in operation, tracks one or more objects in an image based on the generated output image data.

14. The image processing system of claim 1 wherein the first keypoint data of the subset of keypoints comprises a fraction of the first keypoint data.

15. The image processing system of claim 1 wherein the transforming comprises unwarping the first keypoint data of the subset of keypoints and determining whether one or more features of the first image match one or more features of the at least one comparison image is based on unwarped first keypoint data included in the output image data.

16. An image processing method, comprising:

receiving, by image processing circuitry, image data representative of a first image;

identifying, by the image processing circuitry, a plurality of keypoints within the received image data;

generating, by the image processing circuitry and using the received image data representative of the first image, descriptor data for at least some of the plurality of keypoints within the received image data;

transforming a subset of keypoints of the identified plurality of keypoints within the received image data representative of the first image, wherein the subset of keypoints does not include all of the keypoints of the identified plurality of keypoints, the transforming the subset of keypoints including;

generating, by the image processing circuitry, first keypoint data corresponding to the plurality of identified keypoints; and

transforming, by the image processing circuitry and using an image deformation model, first keypoint data corresponding to only the subset of keypoints of the identified plurality of keypoints, producing second keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints;

generating, by the image processing circuitry, output image data based on the second keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints and the descriptor data; and

determining, by the image processing circuitry, whether one or more features of the first image match one or more features of at least one comparison image based on the output image data.

17. The image processing method of claim 16 wherein the image processing circuitry comprises a field-programmable gate array.

18. The image processing method of claim 16 wherein the generating output image data comprises multiplexing at least the descriptor data and second keypoint data into a data stream.

19. The image processing method of claim 16, wherein transforming the first keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints comprises transforming spatial coordinates of the first keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints.

20. The image processing method of claim 16, further comprising compressing the second keypoint data and descriptor data.

21. The image processing method of claim 16, wherein the determining whether one or more features of the first image match one or more features of the at least one comparison image is based on second keypoint data included in the generated output image data.

22. The image processing method of claim 16, comprising tracking one or more objects based on the generated output image data.

23. A non-transitory computer-readable medium whose contents configure image processing circuitry to perform a method, the method comprising:

identifying a plurality of keypoints within image data representing a first image;

generating, using the image data representing the first image, descriptor data for at least some of the identified plurality of keypoints within the image data representing the first image;

transforming a subset of keypoints of the identified plurality of keypoints within image data representative of the first image, wherein the subset of keypoints does not include all of the keypoints of the identified plurality of keypoints, the transforming the subset of keypoints including;

generating first keypoint data corresponding to the plurality of identified keypoints; and

transforming, using an image deformation model, first keypoint data of only the subset of keypoints of the identified plurality of keypoints, producing second keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints;

generating output image data based on the second keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints and the descriptor data; and

determining whether one or more features of the first image match one or more features of at least one comparison image based on the output image data.

24. The medium of claim 23 wherein the generating output image data comprises multiplexing at least the descriptor data and second keypoint data into a data stream.

25. The medium of claim 23 wherein transforming the first keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints comprises transforming spatial coordinates of the first keypoint data corresponding to the subset of keypoints of the identified plurality of keypoints.

26. The medium of claim 23 wherein the method comprises tracking one or more objects in an image based on the generated output image data.

27. The medium of claim 23 wherein the transforming comprises unwarping the subset of keypoints of the identified plurality of keypoints.

28. An image processing system, comprising:

one or more memories, which, in operation, store image data; and

digital image processing circuitry, which, in operation;

identifies a plurality of keypoints within received image data representative of a first image;

generates, using the image data representative of the first image, descriptor data for at least some of the identified plurality of keypoints;

transforms a subset of first keypoint data of the identified plurality of keypoints within the image data representative of the first image, wherein the subset does not include first keypoint data of all of the keypoints of the identified plurality of keypoints, the transforming the subset including;

generating, for the identified plurality of keypoints within the received image data, the first keypoint data; and

transforming, using an image deformation model, only the subset of the first keypoint data to produce second keypoint data, the subset of the first keypoint data corresponding to a subset of the identified plurality of keypoints;

generates output image data based on the second keypoint data and the descriptor data; and

determines whether one or more features of the first image match one or more features of at least one comparison image based on the output image data.

29. The image processing system of claim 28, wherein the first keypoint data comprises first spatial coordinates of the plurality of keypoints and the second keypoint data comprises second spatial coordinates that are transformations of the first spatial coordinates of keypoints of the subset of the identified plurality of keypoints.

30. The image processing system of claim 29, wherein the second keypoint data further comprises image rotation and/or image magnification information.

31. The image processing system of claim 28 wherein the image processing circuitry, in operation,

compresses the descriptor data;

compresses the second keypoint data; and

combines the compressed descriptor data and the compressed second keypoint data into a data stream.

32. The image processing system of claim 28 wherein the generating of the descriptor data is independent of the transforming of the keypoint data.

33. An image processing system, comprising:

one or more memories, which, in operation, store image data; and

digital image processing circuitry, which, in operation;

identifies a plurality of keypoints within received image data representative of a first image;

generates, for the identified plurality of keypoints within the received image data, first keypoint data;

transforms keypoint data of at least a portion of the identified plurality of keypoints within the image data representative of the first image, the transforming the keypoint data of the at least a portion of the identified plurality of keypoints including;

transforming, using an image deformation model, the first keypoint data of the at least a portion of the identified plurality of keypoints to produce second keypoint data;

generates, using the image data representative of the first image, descriptor data for at least some of the identified plurality of keypoints, wherein the generating of the descriptor data is independent of the transforming of the first keypoint data of the at least a portion of the identified plurality of keypoints;

generates output image data based on the second keypoint data and the descriptor data; and

determines whether one or more features of the first image match one or more features of at least one comparison image based on the output image data.

34. The image processing system of claim 33 wherein the first keypoint data comprises first spatial coordinates of the identified plurality of keypoints and the second keypoint data comprises second spatial coordinates that are transformations of the first spatial coordinates of keypoints of the at least a portion of the identified plurality of keypoints.

35. The image processing system of claim 33 wherein the image processing circuitry, in operation,

compresses the descriptor data;

compresses the second keypoint data; and

combines the compressed descriptor data and the compressed second keypoint data into a data stream.

36. The image processing system of claim 33 wherein the second keypoint data comprises at least one of image rotation and image magnification information.