Image registration

• US 10,346,949 B1
• Filed: 05/30/2017
• Issued: 07/09/2019
• Est. Priority Date: 05/27/2016
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Augmented Pixels, Inc. (Qualcomm, Inc.)
• Inventors: Raitarovskyi, Maksym, Norel, Mykhaylo, Goncharuk, Vitaliy
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US15/608,338
• Time in Patent Office: 770 Days
• Field of Search: 382128, 382151, 382174, 382190, 382197, 382201, 382253, 382276, 382277, 382285, 382286, 382294
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/35 using statistical methods
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Systems and methods for computationally efficient and precise transformation recovery using two image frames are provided. A key image frame and a tracked image frame are selected from a series of image frames, a plurality of points on the key image frame are selected, and the plurality of points are then projected onto the tracked image frame to produce a number of corresponding points. Then, for each corresponding point, a patch of pixels is defined centered on the corresponding point. Each pixel of the patch is reprojected back onto the key image frame to identify a reprojected pixel in that image frame. An intensity value for each reprojected pixel is determined and each patch is parameterized. A normal equation for each patch is determined and then solved for the transformation.

Claims

1. A method for determining a transformation in SE(3), a special Euclidean Group for 3 dimensions which represents a group of rigid transformations in 3D space and that can be represented by linear transformations on homogeneous four-vectors, the method comprising:

receiving, by a host computing system, a video stream comprising a series of image frames and selecting a key image frame and a tracked image frame from the series of image frames;

selecting a plurality of points on the key image frame;

projecting the plurality of points onto the tracked image frame to produce a number of corresponding points on the tracked image frame;

defining, for each corresponding point in the tracked image frame, an array of pixels centered on the corresponding point;

for each corresponding point, reprojecting each pixel of the array thereof back onto the key image frame, wherein the reprojection of each said pixel of the array identifies a reprojected pixel in the key image frame;

determining an intensity value for each reprojected pixel;

parameterizing each array of pixels to determine, for each array of pixels, an outer product of the gradient vector for each array of pixels;

determining a normal equation for each array of pixels; and

solving the normal equation for SE(3).

2. The method of claim 1 wherein the step of selecting the plurality of points on the key image frame includes selecting the plurality of points according to a grid.

3. The method of claim 1 wherein the step of selecting the plurality of points on the key image frame includes employing a feature extraction method to select the plurality of points to correspond to features in the key image frame.

4. The method of claim 1 wherein the step of selecting the plurality of points on the key image frame further includes assigning random depth values to the points.

5. The method of claim 1 wherein the step of selecting the plurality of points on the key image frame further includes mapping depth values from a previous key frame.

6. The method of claim 1 wherein the step of selecting the plurality of points on the key image frame further includes extrapolating a depth value for a point from depth values of neighboring points.

7. The method of claim 1 wherein the step of projecting the plurality of points onto the tracked image frame includes rigidly transforming each point in SE(3) and then projecting a result onto the tracked image frame using a perspective matrix.

8. The method of claim 1 wherein the step of, for each corresponding point, reprojecting each pixel of the array back onto the key image frame includes finding the positions of the reprojected pixels in the key image frame.

9. The method of claim 8 wherein the step of, for each corresponding point, reprojecting each pixel of the array back onto the key image frame further includes finding the intensities of the reprojected pixels.

10. The method of claim 8 wherein finding the positions of the reprojected pixels in the key image frame includes reprojecting each pixel onto the key image frame using an homography.

11. The method of claim 1 wherein the step of solving the normal equation for SE(3) includes determining a patch intensity difference.

12. The method of claim 11 wherein the patch intensity difference is greater than a threshold and the method further includes

defining, for each corresponding point in the tracked image frame, a second array of pixels centered on the corresponding point;

for each corresponding point, reprojecting each pixel of the second array thereof back onto the key image frame, wherein the reprojection of each said pixel of the array identifies a reprojected pixel in the key image frame;

determining a second normal equation for each second array of pixels; and

solving the second normal equation for SE(3), wherein an intensity value for each

reprojected pixel corresponding to the second array is not determined nor are each second array of pixels parameterized.

13. A system for determining a transformation in SE(3), a special Euclidean Group for 3 dimensions which represents a group of rigid transformations in 3D space and that can be represented by linear transformations on homogeneous four-vectors, the system comprising:

a host computer including

logic configured to receive a video stream comprising a series of image frames and selecting a key image frame and a tracked image frame from the series of image frames;

logic configured to select a plurality of points on the key image frame;

logic configured to project the plurality of points onto the tracked image frame to produce a number of corresponding points on the tracked image frame;

logic configured to define, for each corresponding point in the tracked image frame, an array of pixels centered on the corresponding point;

logic configured to, for each corresponding point, reproject each pixel of the array thereof back onto the key image frame, wherein the reprojection of each said pixel of the array identifies a reprojected pixel in the key image frame;

logic configured to determine an intensity value for each reprojected pixel;

logic configured to parameterize each array of pixels to determine, for each array of pixels, an outer product of the gradient vector for each array of pixels;

logic configured to determine a normal equation for each array of pixels; and

logic configured to solve the normal equation for SE(3).