Fast initialization for monocular visual SLAM

• US 9,576,183 B2
• Filed: 03/14/2013
• Issued: 02/21/2017
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Reitmayr, Gerhard, Mulloni, Alessandro
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US13/831,405
• Publication Number:
  US 20140126769A1
• Time in Patent Office: 1,440 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Apparatuses and methods for fast visual simultaneous localization and mapping are described. In one embodiment, a three-dimensional (3D) target is initialized immediately from a first reference image and prior to processing a subsequent image. In one embodiment, one or more subsequent reference images are processed, and the 3D target is tracked in six degrees of freedom. In one embodiment, the 3D target is refined based on the processed the one or more subsequent images.

Claims

1. A processor-implemented method for visual simultaneous localization and mapping, the method comprising:

initializing a three-dimensional (3D) target based on a first set of target 3D points obtained from a first reference image, wherein the target 3D points in the first set are initialized along a plane at a predetermined initial depth value prior to processing a subsequent image;

processing one or more subsequent images;

tracking the 3D target in six degrees of freedom; and

refining the 3D target based on subsequent sets of target 3D points obtained from the processing of the one or more subsequent images, each subsequent set of target 3D points corresponding to one of the one or more subsequent images.

2. The processor-implemented method of claim 1, further comprising:

displaying a correctly aligned, positioned, and distanced augmented reality representation of the 3D target upon initializing the 3D target and prior to the processing of the one or more subsequent images.

3. The processor-implemented method of claim 2, further comprising:

updating the augmented reality representation of the 3D target while tracking the 3D target in six degrees of freedom.

4. The processor-implemented method of claim 1, wherein the processing one or more subsequent images further comprises:

extracting 2D interest points from the one or more subsequent images.

5. The processor-implemented method of claim 1, wherein initializing the 3D target further comprises:

extracting a reference set of 2D interest points from the first reference image;

determining the first set of target 3D points for the first reference image, each target 3D point in the first set corresponding to a 2D interest point in the reference set of 2D interest points; and

assigning the predetermined initial depth value, as a corresponding initial depth value, to each target 3D point.

6. The processor-implemented method of claim 5, wherein the tracking the 3D target further comprises comparing a reference location corresponding to at least one of the target 3D points in the reference set to a corresponding updated 2D location of the at least one target 3D points extracted from the one or more subsequent images.

7. The processor-implemented method of claim 5, wherein refining the 3D target further comprises:

determining an updated depth value for one or more of the target 3D points in the first set based on corresponding target 3D points in a subsequent set; and

replacing the assigned corresponding initial depth value for each of the one or more target 3D points in the first set with the corresponding updated depth value.

8. The processor-implemented method of claim 7, further comprising:

determining when a threshold number of the target 3D points having updated depth values is met, wherein the threshold number of target 3D points is obtained from a corresponding subsequent image of the one or more subsequent images; and

assigning the respective subsequent image as a second reference image.

9. The processor-implemented method of claim 8, wherein the refining of the 3D target further comprises performing a further refinement of the 3D target by triangulating the target 3D points with a plurality of 2D interest points extracted from the second reference image.

10. A computer readable non-transitory storage medium containing executable program instructions which cause a data processing device to perform a method for visual simultaneous localization and mapping, the method comprising:

initializing a three-dimensional (3D) target based on a first set of target 3D points obtained from a first reference image, wherein the target 3D points in the first set are initialized along a plane at a predetermined initial depth value prior to processing a subsequent image;

processing one or more subsequent images;

tracking the 3D target in six degrees of freedom; and

refining the 3D target based on the processing of subsequent sets of target 3D points obtained from the one or more subsequent images, each subsequent set of target 3D points corresponding to one of the one or more subsequent images.

11. The medium of claim 10, further comprising:

displaying a correctly aligned, positioned, and distanced augmented reality representation of the 3D target upon initializing the 3D target and prior to the processing of the one or more subsequent images.

12. The medium of claim 11, further comprising:

updating the augmented reality representation of the 3D target while tracking the 3D target in six degrees of freedom.

13. The medium of claim 10, wherein the processing one or more subsequent images further comprises:

extracting 2D interest points from the one or more subsequent images.

14. The medium of claim 10, wherein initializing the 3D target further comprises:

extracting a reference set of 2D interest points from the first reference image;

determining the first set of target 3D points, each target 3D point in the first set corresponding to a 2D interest point in the reference set of 2D interest points; and

assigning the predetermined initial depth value, as a corresponding initial depth value, to each target 3D point.

15. The medium of claim 14, wherein the tracking the 3D target further comprises comparing a reference location corresponding to at least one of the target 3D points in the reference set to a corresponding updated 2D location of the at least one target 3D points extracted from the one or more subsequent images.

16. The medium of claim 14, wherein refining the 3D target further comprises:

determining an updated depth value for one or more of the target 3D points in the first set based on corresponding target 3D points in a subsequent set; and

replacing the assigned corresponding initial depth value for each of the one or more target 3D points in the first set with the corresponding updated depth value.

17. The medium of claim 16, further comprising:

determining when a threshold number of the target 3D points having updated depth values is met, wherein the threshold number of target 3D points is obtained from a corresponding subsequent image of the one or more subsequent images; and

assigning the respective subsequent image as a second reference image.

18. The medium of claim 17, wherein the refining of the 3D target further comprises performing a further refinement of the 3D target by triangulating the target 3D points with a plurality of 2D interest points extracted from the second reference image.

19. A data processing device for visual simultaneous localization and mapping comprising:

a processor; and

a storage device coupled to the processor and configurable for storing instructions, which, when executed by the processor cause the processor to;

initialize a three-dimensional (3D) target based on a first set of target 3D points obtained from a first reference image, wherein the target 3D points in the first set are initialized along a plane at a predetermined initial depth value prior to processing a subsequent image;

process one or more subsequent images;

track the 3D target in six degrees of freedom; and

refine the 3D target based on the processing of subsequent sets of target 3D points obtained from the one or more subsequent images, each subsequent set of target 3D points corresponding to one of the one or more subsequent images.

20. The data processing device of claim 19, further comprising instructions to:

display a correctly aligned, positioned, and distanced augmented reality representation of the 3D target upon initializing the 3D target and prior to the processing of the one or more subsequent images.

21. The data processing device of claim 20, further comprising instructions to:

update the augmented reality representation of the 3D target while tracking the 3D target in six degrees of freedom.

22. The data processing device of claim 19, wherein the processing one or more subsequent images further comprises instructions to:

extract 2D interest points from the one or more subsequent images.

23. The data processing device of claim 19, wherein initializing the 3D target further comprises instructions to:

extract a reference set of 2D interest points from the first reference image;

determine the first set of target 3D points, each target 3D point in the first set corresponding to a 2D interest point in the reference set of 2D interest points; and

assign the predetermined initial depth value, as a corresponding initial depth value, to each target 3D point.

24. The data processing device of claim 23, wherein tracking the 3D target further comprises comparing a reference location corresponding to at least one of the target 3D points in the reference set to a corresponding updated 2D location extracted from the one or more subsequent images.

25. The data processing device of claim 23, wherein refining the 3D target further comprises instructions to:

determine an updated depth value for one or more of the target 3D points in the first set based on corresponding target 3D points in a subsequent set; and

replace the assigned corresponding initial depth value for each of the one or more target 3D points in the first set with the corresponding updated depth value.

26. The data processing device of claim 25, further comprising instructions to:

determine when a threshold number of the target 3D points having updated depth values is met, wherein the threshold number of target 3D points is obtained from a corresponding subsequent image of the one or more subsequent images; and

assign the respective subsequent image as a second reference image.

27. The data processing device of claim 26, wherein the refining of the 3D target further comprises instructions to perform a further refinement of the 3D target by triangulating the target 3D points with a plurality of 2D interest points extracted from the second reference image.

28. An apparatus for visual simultaneous localization and mapping comprising:

means for initializing a three-dimensional (3D) target based on a first set of target 3D points obtained from a first reference image, wherein the target 3D points in the first set are initialized along a plane at a predetermined initial depth value prior to processing a subsequent image;

means for processing one or more subsequent images;

means for tracking the 3D target in six degrees of freedom; and

means for refining the 3D target based on the processing of subsequent sets of target 3D points obtained from the one or more subsequent images.

29. The apparatus of claim 28, further comprising:

means for displaying a correctly aligned, positioned, and distanced augmented reality representation of the 3D target upon initializing the 3D target and prior to the processing of the one or more subsequent images.

30. The apparatus of claim 29, further comprising:

means for updating the augmented reality representation of the target while tracking the 3D target in six degrees of freedom.

31. The apparatus of claim 28, wherein the processing one or more subsequent images further comprises:

means for extracting 2D interest points from the one or more subsequent images.

32. The apparatus of claim 28, wherein initializing the 3D target further comprises:

means for extracting a reference set of 2D interest points from the first reference image;

means for determining the first set of target 3D points, each target 3D point corresponding to a 2D interest point in the reference set of 2D interest points; and

means for assigning the predetermined initial depth value, as a corresponding initial depth value, to each target 3D point.

33. The apparatus of claim 32, wherein the tracking the 3D target further comprises means for comparing a reference location corresponding to at least one of the target 3D points in the reference set to a corresponding updated 2D location of the at least one target 3D points extracted from the one or more subsequent images.

34. The apparatus of claim 32, wherein refining the 3D target further comprises:

means for determining an updated depth value for one or more of the target 3D points in the first set based on corresponding target 3D points in a subsequent set; and

means for replacing the assigned corresponding initial depth value for each of the one or more target 3D points in the first set with the corresponding updated depth value.

35. The apparatus of claim 34, further comprising:

means for determining when a threshold number of the target 3D points having updated depth values is met, wherein the threshold number of target 3D points is obtained from a corresponding subsequent image of the one or more subsequent images; and

means for assigning the respective subsequent image as a second reference image.

36. The apparatus of claim 35, wherein the refining of the 3D target further comprises means for performing a further refinement of the 3D target by triangulating the target 3D points with a plurality of 2D interest points extracted from the second reference image.