System and method of pose estimation

• US 9,607,388 B2
• Filed: 09/19/2014
• Issued: 03/28/2017
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lin, Kuen-Han, Sundaresan, Sairam, Sarkis, Michel Adib
• Primary Examiner(s): Li, Tracy Y
• Application Number:
  US14/491,857
• Publication Number:
  US 20160086336A1
• Time in Patent Office: 921 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

A method includes receiving, from an image capture device, a first image frame of a sequence of image frames. The method also includes estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame. The second image frame precedes the first image frame in the sequence of image frames. The method further includes estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to a keyframe. The keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

Claims

1. A method comprising:

receiving, from an image capture device, a first image frame of a sequence of image frames;

estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;

selecting a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each of the plurality of keyframes; and

estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to the first keyframe.

2. The method of claim 1, wherein the camera pose corresponding to the first image frame is estimated as part of a first processing thread of the processor, wherein the refined camera pose corresponding to the first image frame is estimated as part of a second processing thread of the processor.

3. The method of claim 2, wherein the processor is configured to execute processing threads in parallel, and wherein the first processing thread and the second processing thread are executed in parallel.

4. The method of claim 1, wherein estimating the camera pose corresponding to the first image frame comprises:

generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the second image frame;

determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the second image frame; and

determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the second image frame, wherein the depth error is determined based on normal values of the one or more pixels of the second image frame, and wherein a normal value of each pixel includes a vector that indicates a direction perpendicular to a tangential plane of each pixel.

5. The method of claim 1, wherein estimating the refined camera pose corresponding to the first image frame comprises:

generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the first keyframe;

determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the first keyframe; and

determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the first keyframe, wherein the depth error is determined based on normal values of the one or more pixels of the first keyframe, and wherein a normal value of each pixel includes a vector that indicates a direction perpendicular to a tangential plane of each pixel.

6. The method of claim 1, further comprising:

determining, at the processor, an overlap ratio between the first image frame and the first keyframe, wherein the overlap ratio is based on a pixel-by-pixel comparison of first pixel values of the first image frame and second pixel values of the first keyframe;

comparing the overlap ratio to an overlap ratio threshold to generate a comparison result; and

generating a new keyframe using the first image frame based on the comparison result.

7. The method of claim 6, wherein generating the new keyframe comprises creating a copy image frame corresponding to the first image frame in memory, and wherein the copy image frame is designated as the new keyframe.

8. The method of claim 6, wherein the first pixel values include depth values, photometric values, or a combination thereof.

9. The method of claim 1, further comprising generating a three-dimensional point cloud using the refined camera pose and the sequence of image frames.

10. The method of claim 9, further comprising:

receiving, from the image capture device, one or more additional image frames of the sequence of image frames; and

updating the three-dimensional point cloud based on the one or more additional image frames, wherein a rate of updating the three-dimensional point cloud is substantially equal to a rate of receiving the one or more additional image frames.

11. The method of claim 1, wherein the camera pose corresponding to the first image frame indicates an approximation of a position of the image capture device at a particular time when the first image frame is captured, wherein the camera pose corresponding to the first image frame includes position coordinates in a 3D space and an orientation corresponding to the image capture device at a time when the first image frame is captured, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose of the first keyframe is based on at least one of:

a difference between a relative angle of the camera pose corresponding to the first image frame and a relative angle of the camera pose of the first keyframe;

a relative distance threshold for the first image frame; and

a relative position threshold for the first image frame.

12. The method of claim 1, wherein the first image frame has a first resolution, wherein the keyframe has a keyframe resolution, and wherein the first resolution is different from the keyframe resolution.

13. An apparatus comprising

a processor;

and

a memory storing instruction that are executable by the processor to cause the processor to;

receive, from an image capture device, a first image frame of a sequence of image frames;

estimate a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;

select a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each keyframe of the plurality of keyframes; and

estimate a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to the first keyframe.

14. The apparatus of claim 13, wherein the camera pose corresponding to the first image frame is estimated as part of a first processing thread of the processor, wherein the refined camera pose corresponding to the first image frame is estimated as part of a second processing thread of the processor, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe satisfies a relative distance threshold for the first image frame.

15. The apparatus of claim 14, wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe satisfies a relative position threshold for the first image frame.

16. The apparatus of claim 13, wherein the instructions are further executable to cause the processor to transform the first image frame to generate a warped frame, wherein the first image frame is transformed based on an estimate of relative motion between the image capture device and position coordinates in a 3D space corresponding to the second image frame, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe is based on:

a difference between position coordinates of the camera pose corresponding to the first image frame and position coordinates corresponding to the camera pose corresponding to the first keyframe.

17. The apparatus of claim 16, wherein the instructions are further executable to cause the processor to determine a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the second image frame.

18. The apparatus of claim 16, wherein the instructions are further executable to cause the processor to determine a depth error, wherein the depth error indicates a difference of depth between one or more pixels of the warped frame and one or more pixels of the second image frame, wherein the depth error is determined based on normal values of the one or more pixels of the second image frame, and wherein a normal value of each pixel includes a vector that indicates a direction perpendicular to a tangential plane of each pixel.

19. The apparatus of claim 13, wherein the instructions are further executable to cause the processor to determine an overlap ratio between the first image frame and the first keyframe, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose of the first keyframe satisfies a relative angle threshold for the first image frame.

20. The apparatus of claim 19, wherein the overlap ratio is based on a pixel-by-pixel comparison of first pixel values of the first image frame and second pixel values of the first keyframe.

21. The apparatus of claim 19, wherein the instructions are further executable to cause the processor to generate a result based on a comparison of the overlap ratio to an overlap ratio threshold.

22. The apparatus of claim 21, wherein the instructions are further executable to cause the processor to generate a new keyframe using the first image frame based on the result indicating that the first image frame is not similar to the first keyframe.

23. A non-transitory computer-readable storage device storing instructions that, when executed by a processor, cause the processor to:

receive, from an image capture device, a first image frame of a sequence of image frame;

estimate a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;

select a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each keyframe of the plurality of keyframes; and

estimate a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to the first keyframe.

24. The non-transitory computer-readable storage device of claim 23, wherein the camera pose corresponding to the first image frame is estimated as part of a first processing thread of the processor, wherein the refined camera pose corresponding to the first image frame is estimated as part of a second processing thread of the processor, wherein the first processing thread and the second processing thread are executed in parallel, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe satisfies a relative position threshold for the first image frame.

25. The non-transitory computer-readable storage device of claim 23, wherein the instructions, when executed by the processor, further cause the processor to generate a three-dimensional point cloud based on the sequence of image frames, wherein a rate of generating the three-dimensional point cloud is substantially equal to a rate of receiving the sequence of image frames.

26. An apparatus comprising:

means for receiving, from an image capture device, a first image frame of a sequence of image frames;

means for estimating a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;

means for selecting a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each keyframe of the plurality of keyframes; and

means for estimating a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to the first keyframe.

27. The apparatus of claim 26, further comprising:

means for performing the first comparison of the first image frame to the second image frame; and

means for performing the second comparison of the first image frame to the first keyframe.

28. The apparatus of claim 26, wherein the similarity of the camera pose of the first image frame to the camera pose of the first keyframe satisfies a relative position threshold for the first image frame.

29. The apparatus of claim 26, further comprising means for generating a new keyframe based on the first image frame.

30. The apparatus of claim 26, further comprising means for generating a three-dimensional point cloud using the refined camera pose and the sequence of image frames.