SYSTEM AND METHOD OF POSE ESTIMATION

• US 20160086336A1
• Filed: 09/19/2014
• Published: 03/24/2016
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lin, Kuen-Han, Sundaresan, Sairam, Sarkis, Michel Adib
• Granted Patent: US 9,607,388 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

A method includes receiving, from an image capture device, a first image frame of a sequence of image frames. The method also includes estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame. The second image frame precedes the first image frame in the sequence of image frames. The method further includes estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to a keyframe. The keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

Claims

1. A method comprising:

receiving, from an image capture device, a first image frame of a sequence of image frames;

estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames; and

estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to a keyframe, wherein the keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

2. The method of claim 1, wherein the camera pose is estimated as part of a first processing thread of the processor, and wherein the refined camera pose is estimated as part of a second processing thread of the processor.

3. The method of claim 2, wherein the processor is configured to execute processing threads in parallel, and wherein the first processing thread and the second processing thread are executed in parallel.

4. The method of claim 1, wherein estimating the camera pose corresponding to the first image frame comprises:

generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the second image frame;

determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the second image frame; and

determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the second image frame, and wherein the depth error is determined based on normal values of the one or more pixels of the second image frame.

5. The method of claim 1, wherein estimating the refined camera pose corresponding to the first image frame comprises:

generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the keyframe;

determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the keyframe; and

determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the keyframe, and wherein the depth error is determined based on normal values of the one or more pixels of the keyframe.

6. The method of claim 1, further comprising:

determining, at the processor, an overlap ratio between the first image frame and the keyframe, wherein the overlap ratio is based on a pixel-by-pixel comparison of first pixel values of the first image frame and second pixel values of the keyframe;

comparing the overlap ratio to an overlap ratio threshold to generate a comparison result; and

generating a new keyframe using the first image frame based on the comparison result indicating that the first image frame and the keyframe are dissimilar.

7. The method of claim 6, wherein generating the new keyframe comprises creating a copy image frame corresponding to the first image frame in memory, and wherein the copy image frame is designated as the new keyframe.

8. The method of claim 6, wherein the first pixel values include depth values, photometric values, or a combination thereof.

9. The method of claim 1, further comprising generating a three-dimensional point cloud using the refined camera pose and the sequence of image frames.

10. The method of claim 9, further comprising:

receiving, from the image capture device, one or more additional image frames of the sequence of image frames; and

updating the three-dimensional point cloud based on the one or more additional image frames, wherein a rate of updating the three-dimensional point cloud is substantially equal to a rate of receiving the one or more additional image frames.

11. The method of claim 1, wherein the camera pose indicates an approximation of a position of the image capture device at a particular time when the first image frame is captured.

12. The method of claim 1, wherein the first image frame has a first resolution, wherein the keyframe has a keyframe resolution, and wherein the first resolution is different from the keyframe resolution.

13. An apparatus comprising

a processor;

and

a memory storing instruction that are executable by the processor to cause the processor to;

receive, from an image capture device, a first image frame of a sequence of image frames;

estimate a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames; and

estimate a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to a keyframe, wherein the keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

14. The apparatus of claim 13, wherein the camera pose is estimated as part of a first processing thread of the processor, and wherein the refined camera pose is estimated as part of a second processing thread of the processor.

15. The apparatus of claim 14, wherein the instructions are further executable to cause the processor to select the key frame from a set of keyframes, wherein the keyframe is selected based on the estimated camera pose.

16. The apparatus of claim 13, wherein the instructions are further executable to cause the processor to transform the first image frame to generate a warped frame, wherein the first image frame is transformed based on an estimate of relative motion between the image capture device and coordinates corresponding to the second image frame.

17. The apparatus of claim 16, wherein the instructions are further executable to cause the processor to determine a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the second image frame.

18. The apparatus of claim 16, wherein the instructions are further executable to cause the processor to determine a depth error, wherein the depth error indicates a difference of depth between one or more pixels of the warped frame and one or more pixels of the second image frame, and wherein the depth error is determined based on normal values of the one or more pixels of the second image frame.

19. The apparatus of claim 13, wherein the instructions are further executable to cause the processor to determine an overlap ratio between the first image frame and a keyframe.

20. The apparatus of claim 19, wherein the overlap ratio is based on a pixel-by-pixel comparison of first pixel values of the first image frame and second pixel values of the keyframe.

21. The apparatus of claim 19, wherein the instructions are further executable to cause the processor to generate a result based on a comparison of the overlap ratio to an overlap ratio threshold.

22. The apparatus of claim 21, wherein the instructions are further executable to cause the processor to generate a new keyframe using the first image frame based on the result indicating that the first image frame is not similar to the keyframe.

23. A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to:

receive, from an image capture device, a first image frame of a sequence of image frames;

estimate a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames; and

estimate a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to a keyframe, wherein the keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

24. The computer-readable storage device of claim 23, wherein the camera pose is estimated as part of a first processing thread of the processor, wherein the refined camera pose is estimated as part of a second processing thread of the processor, and wherein the first processing thread and the second processing thread are executed in parallel.

25. The computer-readable storage device of claim 23, wherein the instructions, when executed by the processor, further cause the processor to generate a three-dimensional point cloud based on the sequence of image frames, wherein a rate of generating the three-dimensional point cloud is substantially equal to a rate of receiving the sequence of image frames.

26. An apparatus comprising:

means for receiving, from an image capture device, a first image frame of a sequence of image frames;

means for estimating a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames; and

means for estimating a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to a keyframe, wherein the keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

27. The apparatus of claim 26, further comprising:

means for performing the first comparison of the first image frame to the second image frame; and

means for performing the second comparison of the first image frame to the keyframe.

28. The apparatus of claim 26, further comprising means for selecting the keyframe from a set of keyframes.

29. The apparatus of claim 26, further comprising means for generating a new keyframe based on the first image frame.

30. The apparatus of claim 26, further comprising means for generating a three-dimensional point cloud using the refined camera pose and the sequence of image frames.