Accelerated geometric shape detection and accurate pose tracking

• US 9,224,205 B2
• Filed: 06/14/2012
• Issued: 12/29/2015
• Est. Priority Date: 06/14/2012
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Tsin, Yanghai, Choudhury, Prasun, Chari, Murali Ramaswamy
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Rahman, Mohammad J
• Application Number:
  US13/523,635
• Publication Number:
  US 20130335575A1
• Time in Patent Office: 1,293 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models

Abstract

A reference in an unknown environment is generated on the fly for positioning and tracking. The reference is produced in a top down process by capturing an image of a planar object with a predefined geometric shape, detecting edge pixels of the planar object, then detecting a plurality of line segments from the edge pixels. The plurality of line segments may then be used to detect the planar object in the image based on the predefined geometric shape. An initial pose of the camera with respect to the planar object is determined and tracked using the edges of the planar object.

Claims

1. A method comprising:

capturing an image of a planar object with a camera, the planar object having a predefined geometric shape;

detecting edge pixels of the planar object in the image, wherein detecting edge pixels comprises;

calculating a gradient magnitude and a gradient orientation of each pixel in the image on a sparse grid; and

performing a non-maximum suppression by comparing the gradient magnitude of pixels on the sparse grid to gradient magnitudes of nearest neighboring pixels along the gradient orientation and that are not on the sparse grid;

detecting a plurality of line segments from the edge pixels;

using the plurality of line segments to detect the planar object in the image based on the predefined geometric shape;

determining an initial pose of the camera using the planar object; and

tracking a pose of the camera with respect to the planar object in subsequently captured images using the initial pose and edges of the planar object.

2. The method of claim 1, further comprising qualifying the planar object for tracking before determining the initial pose of the camera.

3. The method of claim 2, wherein qualifying the planar object for tracking comprises at least one of manual input, a short baseline structure from motion analysis or a long baseline structure from motion analysis.

4. The method of claim 1, wherein detecting edge pixels further comprises comparing the gradient magnitude of pixels on the sparse grid to a threshold.

5. The method of claim 1, wherein detecting the plurality of line segments comprises:

selecting two edge pixels with gradient orientations that are within a predefined tolerance from each other;

finding pixels on a line between the two edge pixels that are edge pixels; and

comparing a percentage of edge pixels on the line between the two edge pixels to a threshold to determine if the two edge pixels are on a line segment.

6. The method of claim 1, wherein detecting the plurality of line segments comprises:

selecting two edge pixels with different gradient orientations;

defining a junction at an intersection of hypothesized lines extending through the two edge pixels, wherein each hypothesized line extends in a direction orthogonal to a gradient orientation of one of the two edge pixels;

finding pixels on the hypothesized lines between the two edge pixels that are edge pixels; and

comparing a percentage of edge pixels on the hypothesized lines between the two edge pixels to a threshold to determine if the two edge pixels and the junction are on line segments.

7. The method of claim 6, further comprising discarding a junction determined to be on a T-junction or cross junction.

8. The method of claim 1, wherein using the plurality of line segments to detect the planar object comprises:

sampling the plurality of line segments to find line segments that satisfy a shape grammar for the predefined geometric shape.

9. The method of claim 8, further comprising:

placing each of the plurality of line segments in a Hough transform bin based on gradient orientation and intercept, wherein a probability of sampling a line segment is proportional to a total length of all line segments in the Hough transform bin that contains the line segment.

10. The method of claim 1, wherein using the plurality of line segments to detect the planar object based on the predefined geometric shape comprises:

generating a plurality of hypothesized objects in the image using the plurality of line segments; and

determining at least one hypothesized object that has the predefined geometric shape.

11. The method of claim 10, wherein there are a plurality of hypothesized objects that have the predefined geometric shape, the method further comprising manually or automatically selecting one of the plurality of hypothesized objects as the planar object.

12. The method of claim 1, wherein the predefined geometric shape is a convex quadrilateral.

13. The method of claim 1, wherein the predefined geometric shape comprises a regular polygon.

14. The method of claim 1, wherein determining the initial pose of the camera is with respect to the planar object in the image comprises reconstructing a three dimensional shape from the planar object in the image.

15. The method of claim 1, wherein tracking the pose of the camera using the edges of the planar object comprises extracting features within the planar object and using the extracted features in addition to the planar object with the edges of the planar object.

16. The method of claim 1, wherein tracking the pose of the camera comprises down sampling the image and using patch based tracking on a coarse resolution level and using edge based tracking based on results of the patch based tracking on a fine resolution level.

17. A mobile device comprising:

a camera for capturing an image of a planar object, the planar object having a predefined geometric shape; and

a processor coupled to the camera for receiving the image of the planar object, the processor is adapted to detect edge pixels of the planar object in the image;

detect a plurality of line segments from the edge pixels;

use the plurality of line segments to detect the planar object in the image based on the predefined geometric shape;

determine an initial pose of the camera using the planar object; and

track a pose of the camera with respect to the planar object in subsequently captured images using the initial pose and edges of the planar object, wherein the processor is adapted to detect edge pixels by being adapted to;

calculate a gradient magnitude and a gradient orientation of each pixel in the image on a sparse grid; and

perform a non-maximum suppression by comparing the gradient magnitude of pixels on the sparse grid to gradient magnitudes of neighboring pixels along the gradient orientation and that are not on the sparse grid.

18. The mobile device of claim 17, wherein the processor is further adapted to qualify the planar object for tracking before the initial pose of the camera is determined.

19. The mobile device of claim 18, wherein the processor is adapted to qualify the planar object with at least one of manual input, a short baseline structure from motion analysis or a long baseline structure from motion analysis.

20. The mobile device of claim 17, wherein the processor is adapted to detect edge pixels by being adapted to compare the gradient magnitude of pixels on the sparse grid to a threshold.

21. The mobile device of claim 17, wherein the processor is adapted to detect the plurality of line segments by being adapted to:

select two edge pixels with gradient orientations that are within a predefined tolerance from each other;

find pixels on a line between the two edge pixels that are edge pixels; and

compare a percentage of edge pixels on the line between the two edge pixels to a threshold to determine if the two edge pixels are on a line segment.

22. The mobile device of claim 17, wherein the processor is adapted to detect the plurality of line segments by being adapted to:

select two edge pixels with different gradient orientations;

define a junction at an intersection of hypothesized lines extending through the two edge pixels, wherein each hypothesized line extends in a direction orthogonal to a gradient orientation of one of the two edge pixels;

find pixels on the hypothesized lines between the two edge pixels that are edge pixels; and

compare a percentage of edge pixels on the hypothesized lines between the two edge pixels to a threshold to determine if the two edge pixels and the junction are on line segments.

23. The mobile device of claim 22, wherein the processor is further adapted to discard a junction determined to be on a T-junction or cross junction.

24. The mobile device of claim 17, wherein the processor is adapted to use the plurality of line segments to detect the planar object by being adapted to:

sample the plurality of line segments to find line segments that satisfy a shape grammar for the predefined geometric shape.

25. The mobile device of claim 24, wherein the processor is further adapted to:

place each of the plurality of line segments in a Hough transform bin based on gradient orientation and intercept, wherein a probability of sampling a line segment is proportional to a total length of all line segments in the Hough transform bin that contains the line segment.

26. The mobile device of claim 17, wherein the processor is adapted to use the plurality of line segments to detect the planar object by being adapted to:

generate a plurality of hypothesized objects in the image using the plurality of line segments; and

determine at least one hypothesized object that has the predefined geometric shape.

27. The mobile device of claim 26, wherein there are a plurality of hypothesized objects that have the predefined geometric shape, the processor being further adapted to select one of the plurality of hypothesized objects as the planar object automatically or by user input.

28. The mobile device of claim 17, wherein the predefined geometric shape is a convex quadrilateral.

29. The mobile device of claim 17, wherein the predefined geometric shape comprises a regular polygon.

30. The mobile device of claim 17, wherein the processor is adapted to determine the initial pose of the camera with respect to the planar object in the image based on the predefined geometric shape.

31. The mobile device of claim 17, wherein the processor is adapted to track the pose of the camera by being adapted to extract features within the planar object and use the extracted features in addition to the planar object with the edges of the planar object.

32. The mobile device of claim 17, wherein the processor is adapted to track the pose of the camera by being adapted to down sample the image and use patch based tracking on a coarse resolution level and use edge based tracking based on results of the patch based tracking on a fine resolution level.

33. A mobile device comprising:

means for capturing an image of a planar object with a camera, the planar object having a predefined geometric shape;

means for detecting edge pixels of the planar object in the image, wherein the means for detecting edge pixels calculates a gradient magnitude and a gradient orientation of each pixel in the image on a sparse grid and performs a non-maximum suppression by comparing the gradient magnitude of pixels on the sparse grid to gradient magnitudes of neighboring pixels along the gradient orientation and that are not on the sparse grid;

means for detecting a plurality of line segments from the edge pixels;

means for using the plurality of line segments to detect the planar object in the image based on the predefined geometric shape;

means for determining an initial pose of the camera using the planar object; and

means for tracking a pose of the camera with respect to the planar object in subsequently captured images using the initial pose and edges of the planar object.

34. The mobile device of claim 33, wherein the means for detecting the plurality of line segments selects two edge pixels with gradient orientations that are within a predefined tolerance from each other;

finds pixels on a line between the two edge pixels that are edge pixels; and

compares a percentage of edge pixels on the line between the two edge pixels to a threshold to determine if the two edge pixels are on a line segment.

35. The mobile device of claim 33, wherein the means for detecting the plurality of line segments selects two edge pixels with different gradient orientations;

defines a junction at an intersection of hypothesized lines extending through the two edge pixels, wherein each hypothesized line extends in a direction orthogonal to a gradient orientation of one of the two edge pixels;

finds pixels on the hypothesized lines between the two edge pixels that are edge pixels; and

compares a percentage of edge pixels on the hypothesized lines between the two edge pixels to a threshold to determine if the two edge pixels and the junction are on line segments.

36. The mobile device of claim 33, wherein the means for using the plurality of line segments to detect the planar object places each of the plurality of line segments in a Hough transform bin based on gradient orientation and intercept;

and samples the plurality of line segments to find line segments that satisfy a shape grammar for the predefined geometric shape, wherein a probability of sampling a line segment is proportional to a total length of all line segments in the Hough transform bin that contains the line segment.

37. The mobile device of claim 33, wherein the means for tracking the pose of the camera extracts features within the planar object and uses the extracted features in addition to the planar object with the edges of the planar object.

38. The mobile device of claim 33, wherein the means for tracking the pose of the camera down samples the image and uses patch based tracking on a coarse resolution level and uses edge based tracking based on results of the patch based tracking on a fine resolution level.

39. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to detect edge pixels of a planar object in an image captured with a camera, the planar object having a predefined geometric shape, wherein the program code to detect edge pixels calculates a gradient magnitude and a gradient orientation of each pixel in the image on a sparse grid; and

performs a non-maximum suppression by comparing the gradient magnitude of pixels on the sparse grid to gradient magnitudes of neighboring pixels along the gradient orientation and that are not on the sparse grid;

program code to detect a plurality of line segments from the edge pixels;

program code to use the plurality of line segments to detect the planar object in the image based on the predefined geometric shape;

program code to determine an initial pose of the camera using the planar object; and

program code to track a pose of the camera with respect to the planar object in subsequently captured images using the initial pose and edges of the planar object.

40. The non-transitory computer-readable medium of claim 39, wherein the program code to detect the plurality of line segments selects two edge pixels with gradient orientations that are within a predefined tolerance from each other;

finds pixels on a line between the two edge pixels that are edge pixels; and

compares a percentage of edge pixels on the line between the two edge pixels to a threshold to determine if the two edge pixels are on a line segment.

41. The non-transitory computer-readable medium of claim 39, wherein the program code to detect the plurality of line segments selects two edge pixels with different gradient orientations;

defines a junction at an intersection of hypothesized lines extending through the two edge pixels, wherein each hypothesized line extends in a direction orthogonal to a gradient orientation of one of the two edge pixels;

finds pixels on the hypothesized lines between the two edge pixels that are edge pixels; and

compares a percentage of edge pixels on the hypothesized lines between the two edge pixels to a threshold to determine if the two edge pixels and the junction are on line segments.

42. The non-transitory computer-readable medium of claim 39, wherein the program code to use the plurality of line segments to detect the planar object places each of the plurality of line segments in a Hough transform bin based on gradient orientation and intercept;

and samples the plurality of line segments to find line segments that satisfy a shape grammar for the predefined geometric shape, wherein a probability of sampling a line segment is proportional to a total length of all line segments in the Hough transform bin that contains the line segment.

43. The non-transitory computer-readable medium of claim 39, wherein the program code to track the pose of the camera extracts features within the planar object and uses the extracted features in addition to the planar object with the edges of the planar object.

44. The non-transitory computer-readable medium of claim 39, wherein the program code to track the pose of the camera down samples the image and uses patch based tracking on a coarse resolution level and uses edge based tracking based on results of the patch based tracking on a fine resolution level.