FAST EDGE-BASED OBJECT RELOCALIZATION AND DETECTION USING CONTEXTUAL FILTERING

• US 20140270362A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Najafi Shoushtari, Seyed Hesameddin, Tsin, Yanghai, Chari, Murali Ramaswamy, Diaz Spindola, Serafin
• Application Number:
  US13/844,692
• Publication Number:
  US 20140270362A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 10/752 Contour matching

Abstract

Embodiments include detection or relocalization of an object in a current image from a reference image, such as using a simple and relatively fast and invariant edge orientation based edge feature extraction, then a weak initial matching combined with a strong contextual filtering framework, and then a pose estimation framework based on edge segments. Embodiments include fast edge-based object detection using instant learning with a sufficiently large coverage area for object re-localization. Embodiments provide a good trade-off between computational efficiency of the extraction and matching processes.

Claims

1. A machine implemented method to perform detection or relocalization of an object in a current frame (CF) image from a reference frame image (RF), comprising:

calculating CF and RF edge profile illumination intensity gradients of CF and RF image data within a predetermined radius from a location of each of a plurality of CF and RF edge features of a CF and a RF, and along a normal direction of each of the plurality of CF and RF edge features;

selecting CF and RF edge features having at least one extrema of the CF and RF profile gradients within the predetermined radius; and

identifying at least one CF and RF patch for each selected CF and RF edge feature based on at least one distance between the CF and RF edge feature and at least one location of a local extrema of the CF and RF profile gradients.

2. The method of claim 1, further comprising:

identifying a changed location of each CF and RF edge feature to a center location between the CF and RF edge feature and the local extrema; and

defining a scale of each at least one CF and RF patch as a distance between the changed location of each CF and RF edge feature and the location of the local extrema for each patch.

3. The method of claim 1, further comprising:

calculating CF and RF patch binary descriptors for each of the at least one CF and RF patch, wherein each at least one CF and RF patch binary descriptor include a binary data stream having a bit for each pair compared and having a same length as each other descriptor;

comparing d random locations within each at least one CF patch binary descriptor with each at least one RF patch binary descriptors to identify a number of similar bits; and

selecting N possible RF patch binary descriptor matches for each at least one CF patch binary descriptors based on the N possible matches have the most similar bits of any RF patch binary descriptor as compared to the CF descriptor.

4. The method of claim 3, further comprising:

identifying N possible RF edge feature matches for each at least one CF edge feature based on the selecting;

comparing a first sequence of each at least one RF patch binary descriptor of each N possible RF edge feature matches to a second sequence of each at least one CF patch binary descriptor of each at least one CF edge feature; and

identifying as compatible ones of the N possible RF edge feature matches to each at least one CF edge feature, each of the N possible RF edge feature matches having the first sequence sequentially similar to the second sequence based on dynamic programming.

5. The method of claim 4, further comprising:

calculating an angular difference between the normal of each of the compatible ones of the N possible RF edge feature matches and the normal of each at least one CF edge feature;

calculating an illumination gradient magnitude difference between the illumination gradient magnitude of each of the compatible ones of the N possible RF edge feature matches and the illumination gradient magnitudes of each at least one CF edge feature;

performing a Hough Transform based on the angular difference and the illumination gradient magnitude difference for each of the compatible ones of the N possible RF edge feature matches and each at least one CF edge feature; and

identifying a set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches having their Hough Transform greater than a NM threshold.

6. The method of claim 5, further comprising:

identifying Homography and affine transformation matrix projected line segments of the RF having up to a predetermined number of adjacently located and similar normaled ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches;

identifying line segments of the CF having up to a predetermined number of adjacently located and similar normaled ones of the each at least one CF edge features;

calculating differences in the distances between locations of two line segment ends of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge feature;

calculating differences in the angles between the line segment directions of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge features.

identifying a set of strong RF edge feature matches as ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches that are part of projected line segments of the RF having differences in the distances below a first threshold and differences in the angles below a second threshold; and

calculating 3D pose using the strong RF edge feature matches and the each at least one CF edge feature.

7. The method of claim 6, further comprising displaying the strong RF edge feature matches and the each at least one CF edge feature on a display.

8. A device comprising:

an object detection or relocalization computer module to perform detection or relocalization of an object in a current frame (CF) image from a reference frame image (RF), the module configured to;

calculate CF and RF edge profile illumination intensity gradients of CF and RF image data within a predetermined radius from a location of each of a plurality of CF and RF edge features of a CF and a RF, and along a normal direction of each of the plurality of CF and RF edge features;

select CF and RF edge features having at least one extrema of the CF and RF profile gradients within the predetermined radius; and

identify at least one CF and RF patch for each selected CF and RF edge feature based on at least one distance between the CF and RF edge feature and at least one location of a local extrema of the CF and RF profile gradients.

9. The device of claim 8, the object detection or relocalization computer module further configured to:

identify a changed location of each CF and RF edge feature to a center location between the CF and RF edge feature and the local extrema; and

define a scale of each at least one CF and RF patch as a distance between the changed location of each CF and RF edge feature and the location of the local extrema for each patch.

10. The device of claim 8, the object detection or relocalization computer module further configured to:

calculate CF and RF patch binary descriptors for each of the at least one CF and RF patch, wherein each at least one CF and RF patch binary descriptor include a binary data stream having a bit for each pair compared and having a same length as each other descriptor;

compare d random locations within each at least one CF patch binary descriptor with each at least one RF patch binary descriptors to identify a number of similar bits; and

select N possible RF patch binary descriptor matches for each at least one CF patch binary descriptors based on the N possible matches have the most similar bits of any RF patch binary descriptor as compared to the CF descriptor.

11. The device of claim 10, the object detection or relocalization computer module further configured to:

identify N possible RF edge feature matches for each at least one CF edge feature based on the selecting;

compare a first sequence of each at least one RF patch binary descriptor of each N possible RF edge feature matches to a second sequence of each at least one CF patch binary descriptor of each at least one CF edge feature; and

identify as compatible ones of the N possible RF edge feature matches to each at least one CF edge feature, each of the N possible RF edge feature matches having the first sequence sequentially similar to the second sequence based on dynamic programming.

12. The device of claim 11, the object detection or relocalization computer module further configured to:

calculate an angular difference between the normal of each of the compatible ones of the N possible RF edge feature matches and the normal of each at least one CF edge feature;

calculate an illumination gradient magnitude difference between the illumination gradient magnitude of each of the compatible ones of the N possible RF edge feature matches and the illumination gradient magnitudes of each at least one CF edge feature;

perform a Hough Transform based on the angular difference and the illumination gradient magnitude difference for each of the compatible ones of the N possible RF edge feature matches and each at least one CF edge feature; and

identify a set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches having their Hough Transform greater than a NM threshold.

13. The device of claim 12, the object detection or relocalization computer module further configured to:

identify Homography and affine transformation matrix projected line segments of the RF having up to a predetermined number of adjacently located and similar normaled ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches;

identify line segments of the CF having up to a predetermined number of adjacently located and similar normaled ones of the each at least one CF edge features;

calculate differences in the distances between locations of two line segment ends of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge feature;

calculate differences in the angles between the line segment directions of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge features;

identify a set of strong RF edge feature matches as ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches that are part of projected line segments of the RF having differences in the distances below a first threshold and differences in the angles below a second threshold;

and further comprising a pose calculation module to calculate 3D pose using the strong RF edge feature matches and the each at least one CF edge feature.

14. The device of claim 13, further comprising a display to display the strong RF edge feature matches and the each at least one CF edge feature.

15. A computer program product comprising a computer-readable medium comprising code to perform detection or relocalization of an object in a current frame (CF) image from a reference frame image (RF), the code for:

calculating CF and RF edge profile illumination intensity gradients of CF and RF image data within a predetermined radius from a location of each of a plurality of CF and RF edge features of a CF and a RF, and along a normal direction of each of the plurality of CF and RF edge features;

selecting CF and RF edge features having at least one extrema of the CF and RF profile gradients within the predetermined radius; and

identifying at least one CF and RF patch for each selected CF and RF edge feature based on at least one distance between the CF and RF edge feature and at least one location of a local extrema of the CF and RF profile gradients.

16. The computer program product of claim 15, further comprising code for:

identifying a changed location of each CF and RF edge feature to a center location between the CF and RF edge feature and the local extrema; and

defining a scale of each at least one CF and RF patch as a distance between the changed location of each CF and RF edge feature and the location of the local extrema for each patch.

17. The computer program product of claim 15, further comprising code for:

calculating CF and RF patch binary descriptors for each of the at least one CF and RF patch, wherein each at least one CF and RF patch binary descriptor include a binary data stream having a bit for each pair compared and having a same length as each other descriptor;

comparing d random locations within each at least one CF patch binary descriptor with each at least one RF patch binary descriptors to identify a number of similar bits; and

selecting N possible RF patch binary descriptor matches for each at least one CF patch binary descriptors based on the N possible matches have the most similar bits of any RF patch binary descriptor as compared to the CF descriptor.

18. The computer program product of claim 17, further comprising code for:

identifying N possible RF edge feature matches for each at least one CF edge feature based on the selecting;

comparing a first sequence of each at least one RF patch binary descriptor of each N possible RF edge feature matches to a second sequence of each at least one CF patch binary descriptor of each at least one CF edge feature; and

identifying as compatible ones of the N possible RF edge feature matches to each at least one CF edge feature, each of the N possible RF edge feature matches having the first sequence sequentially similar to the second sequence based on dynamic programming.

19. The computer program product of claim 18, further comprising code for:

calculating an angular difference between the normal of each of the compatible ones of the N possible RF edge feature matches and the normal of each at least one CF edge feature;

calculating an illumination gradient magnitude difference between the illumination gradient magnitude of each of the compatible ones of the N possible RF edge feature matches and the illumination gradient magnitudes of each at least one CF edge feature;

performing a Hough Transform based on the angular difference and the illumination gradient magnitude difference for each of the compatible ones of the N possible RF edge feature matches and each at least one CF edge feature; and

identifying a set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches having their Hough Transform greater than a NM threshold.

20. The computer program product of claim 19, further comprising code for:

identifying Homography and affine transformation matrix projected line segments of the RF having up to a predetermined number of adjacently located and similar normaled ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches;

identifying line segments of the CF having up to a predetermined number of adjacently located and similar normaled ones of the each at least one CF edge features;

calculating differences in the distances between locations of two line segment ends of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge feature;

calculating differences in the angles between the line segment directions of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge features;

identifying a set of strong RF edge feature matches as ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches that are part of projected line segments of the RF having differences in the distances below a first threshold and differences in the angles below a second threshold; and

calculating 3D pose using the strong RF edge feature matches and the each at least one CF edge feature.

21. The computer program product of claim 20, further comprising code for displaying the strong RF edge feature matches and the each at least one CF edge feature.

23. The computing device of claim 21, further comprising:

a means for identifying a changed location of each CF and RF edge feature to a center location between the CF and RF edge feature and the local extrema; and

a means for defining a scale of each at least one CF and RF patch as a distance between the changed location of each CF and RF edge feature and the location of the local extrema for each patch.

24. The computing device of claim 21, further comprising:

a means for calculating CF and RF patch binary descriptors for each of the at least one CF and RF patch, wherein each at least one CF and RF patch binary descriptor include a binary data stream having a bit for each pair compared and having a same length as each other descriptor;

a means for comparing d random locations within each at least one CF patch binary descriptor with each at least one RF patch binary descriptors to identify a number of similar bits; and

a means for selecting N possible RF patch binary descriptor matches for each at least one CF patch binary descriptors based on the N possible matches have the most similar bits of any RF patch binary descriptor as compared to the CF descriptor.

25. The computing device of claim 24, further comprising:

a means for identifying N possible RF edge feature matches for each at least one CF edge feature based on the selecting;

a means for comparing a first sequence of each at least one RF patch binary descriptor of each N possible RF edge feature matches to a second sequence of each at least one CF patch binary descriptor of each at least one CF edge feature; and

a means for identifying as compatible ones of the N possible RF edge feature matches to each at least one CF edge feature, each of the N possible RF edge feature matches having the first sequence sequentially similar to the second sequence based on dynamic programming.

26. The computing device of claim 25, further comprising:

a means for calculating an angular difference between the normal of each of the compatible ones of the N possible RF edge feature matches and the normal of each at least one CF edge feature;

a means for calculating an illumination gradient magnitude difference between the illumination gradient magnitude of each of the compatible ones of the N possible RF edge feature matches and the illumination gradient magnitudes of each at least one CF edge feature;

a means for performing a Hough Transform based on the angular difference and the illumination gradient magnitude difference for each of the compatible ones of the N possible RF edge feature matches and each at least one CF edge feature; and

a means for identifying a set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches having their Hough Transform greater than a NM threshold.

27. The computing device of claim 26, further comprising:

a means for identifying Homography and affine transformation matrix projected line segments of the RF having up to a predetermined number of adjacently located and similar normaled ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches;

a means for identifying line segments of the CF having up to a predetermined number of adjacently located and similar normaled ones of the each at least one CF edge features;

a means for calculating differences in the distances between locations of two line segment ends of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge feature;

a means for calculating differences in the angles between the line segment directions of (1) the projected line segments of the RF of the ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches, and (2) the line segments of the CF of the each at least one CF edge features;

a means for identifying a set of strong RF edge feature matches as ones of the set of normal and magnitude filtered compatible ones of the N possible RF edge feature matches that are part of projected line segments of the RF having differences in the distances below a first threshold and differences in the angles below a second threshold; and

a means for calculating 3D pose using the strong RF edge feature matches and the each at least one CF edge feature.

28. The computing device of claim 27, further comprising a means for displaying the strong RF edge feature matches and the each at least one CF edge feature on a display.

22. A computing device to perform detection or relocalization of an object in a current frame (CF) image from a reference frame image (RF), comprising:

a means for calculating CF and RF edge profile illumination intensity gradients of CF and RF image data within a predetermined radius from a location of each of a plurality of CF and RF edge features of a CF and a RF, and along a normal direction of each of the plurality of CF and RF edge features;

a means for selecting CF and RF edge features having at least one extrema of the CF and RF profile gradients within the predetermined radius; and

a means for identifying at least one CF and RF patch for each selected CF and RF edge feature based on at least one distance between the CF and RF edge feature and at least one location of a local extrema of the CF and RF profile gradients.