IMAGE PROCESSING INCLUDING ADJOIN FEATURE BASED OBJECT DETECTION, AND/OR BILATERAL SYMMETRIC OBJECT SEGMENTATION

• US 20140369608A1
• Filed: 06/14/2013
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Anbang Yao, Jianguo Li, Lianrui Fu, Qiang Li, Tao Wang, Ya-Ti Peng, Yimin Zhang, Yurong Chen
• Original Assignee: Ya-Ti Peng
• Inventors: Wang, Tao, Li, Qiang, Li, Jianguo, Chen, Yurong, Zhang, Yimin, Fu, Lianrui, Peng, Ya-ti, Yao, Anbang
• Granted Patent: US 10,074,034 B2
• Time in Patent Office: Days
• US Class Current: 382/201
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

Apparatuses, methods and storage medium associated with processing an image are disclosed herein. In embodiments, a method for processing one or more images may include generating a plurality of pairs of keypoint features for a pair of images. Each pair of keypoint features may include a keypoint feature from each image. Further, for each pair of keypoint features, corresponding adjoin features may be generated. Additionally, for each pair of keypoint features, whether the adjoin features are similar may be determined. Whether the pair of images have at least one similar object may also be determined, based at least in part on a result of the determination of similarity between the corresponding adjoin features. Other embodiments may be described and claimed.

Claims

1. A method for processing one or more images, comprising:

generating, by a computing device, a plurality of pairs of keypoint features, with each pair of keypoint features including a first keypoint feature from a first of the one or more images, and a second keypoint feature from a second of the one or more images;

generating, by the computing device, for each pair of keypoint features, first one or more adjoin features for the first keypoint feature of the keypoint feature pair, and corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair;

determining, by the computing device, for each pair of keypoint features, whether an adjoin feature of the first keypoint feature of the keypoint feature pair is similar to the corresponding adjoin feature of the second keypoint feature of the keypoint feature pair; and

determining, by the computing device, whether the first and the second image have at least one similar object, based at least in part on a result of the determination of similarity between the corresponding adjoin features of the first and second keypoint features of the keypoint feature pairs.

2. The method of claim 1, wherein generating a plurality of pairs of keypoint features comprises:

identifying a first plurality of keypoint features of the first image;

identifying a second plurality of keypoint features of the second image; and

pairing the first and second plurality of keypoint features of the first and second images.

3. The method of claim 1, wherein generating, for each pair of keypoint features, first one or more adjoin features for the first keypoint feature of the keypoint feature pair, and corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair, comprises:

for each pair of keypoint features,

identifying the first one or more adjoin features for the first keypoint feature of the keypoint feature pair; and

identifying the corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair.

4. The method of claim 1, wherein determining whether the first and second images have at least one similar object, comprises:

generating a set of similar keypoint feature pairs, wherein the keypoint features of a similar keypoint feature pair are determined to be similar based at least in part on a result of the determination of similarity between the corresponding adjoin features of the keypoint features of the similar keypoint feature pairs; and

outputting the set of similar keypoint feature pairs.

5. The method of claim 4, further comprising highlighting, by the computing device, at least an object in the first image based at least in part on some of the keypoint features of the first image that are members of the set of similar keypoint feature pairs.

6. The method of claim 5, wherein highlighting comprises bounding the object with a geometric primitive.

7-24. -24. (canceled)

25. At least one non-transitory storage medium comprising a plurality of instructions configured to cause an apparatus, in response to execution of the instructions by an apparatus, to perform processing of a plurality of images, wherein performance of the processing includes:

generation of a plurality of pairs of keypoint features, with each pair of keypoint features including a first keypoint feature from a first of the one or more images, and a second keypoint feature from a second of the one or more images;

for each pair of keypoint features, generation of first one or more adjoin features for the first keypoint feature of the keypoint feature pair, and corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair;

for each pair of keypoint features, determination of whether an adjoin feature of the first keypoint feature of the keypoint feature pair is similar to the corresponding adjoin feature of the second keypoint feature of the keypoint feature pair; and

determination of whether the first and the second image have at least one similar object, based at least in part on a result of the determination of similarity between the corresponding adjoin features of the first and second keypoint features of the keypoint feature pairs.

26. The storage medium of claim 25, wherein generation of a plurality of pairs of keypoint features comprises:

identification of a first plurality of keypoint features of the first image;

identification of a second plurality of keypoint features of the second image; and

performance of pairing of the first and second plurality of keypoint features of the first and second images.

27. The storage medium of claim 25, wherein for each pair of keypoint features, generation of first one or more adjoin features for the first keypoint feature of the keypoint feature pair, and corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair, comprises:

for each pair of keypoint features,

identification of the first one or more adjoin features for the first keypoint feature of the keypoint feature pair; and

identification of the corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair.

28. The storage medium of claim 25, wherein determination of whether the first and the second image have at least one similar object, comprises

generation a set of similar keypoint feature pairs, wherein the keypoint features of a similar keypoint feature pair are determined to be similar based at least in part on a result of the determination of similarity between the corresponding adjoin features of the keypoint features of the similar keypoint feature pairs;

and

output the set of similar keypoint feature pairs.

29. The storage medium of claim 28, wherein process one or more images further includes highlight at least an object in the first image based at least in part on some of the keypoint features of the first image that are members of the set of similar keypoint feature pairs.

30. The storage medium of claim 29, wherein highlight at least an object comprises bound the object with a geometric primitive.

31. At least one non-transitory storage medium comprising a plurality of instructions configured to cause an apparatus, in response to execution of the instructions by the apparatus, to perform processing of a plurality of images, including:

performance of pairwise matching of local features of an image to generate an initial plurality of point pairs for the image;

scale consistency check the point pairs to eliminate falsely matched point pairs from the initial plurality of point pairs, resulting with a first intermediate plurality of scale consistency checked point pairs;

content consistency check the first intermediate plurality of scale consistency checked point pairs to discard dominant point pairs in background of the image, resulting in a second intermediate plurality of content consistency checked point pairs;

performance of discriminative foreground and background modeling of the image based at least in part on the second intermediate plurality of content consistency checked point pairs; and

segmentation of a bilateral symmetric object in the image based at least in part on geometric properties of the discriminative foreground and background modeled image.

32. The storage medium of claim 31, wherein performance of pairwise matching comprises detection of symmetry in the image;

and generation of symmetric constellations of features, wherein generation of symmetric constellations of features includes generation of two sets of points {P_i} and {P_j} that respectively include image coordinates, orientations, scales and local feature vectors of the points {P_i} and {P_j}.

33. The storage medium of claim 31, wherein scale consistency check comprises computation of a scale ratio for each point pair, and discard of a point pair if the scale ratio of a point pair is less than a threshold.

34. The storage medium of claim 31, wherein content consistency check comprises grouping of matched point pairs that share a bilateral symmetry axis.

35. The storage medium of claim 31, wherein content consistency check a point pair comprises computation of an average color difference for regions correspondingly comprising the points of the point pair, and discard the point pair if the average color difference of the point pair is larger than a threshold.

36. The storage medium of claim 31, wherein performance of discriminative foreground and background modeling of the image comprises:

obtain a rectangular region that bounds the dominant point pairs;

temporarily designate the rectangular region as foreground of the image, and remainder of the image as background of the image; and

generate a plurality of histograms to model features of the foreground and the background based at least in part on color values of pixels of the image.

37. The storage medium of claim 36, wherein segmentation comprises:

obtain a binary image based at least in part on the histograms;

performance of morphological operations to eliminate noises in the binary image;

label connect components sharing largest overlap with the foreground as a segmented portion of the bilateral symmetric object; and

mirror the segmented portion across an axis to identify the other segmented portion of the bilateral symmetric object.

38. An apparatus for processing one or more images, comprising

one or more processors;

and

storage medium coupled to the one or more processors, and having a plurality of modules configured to cause the apparatus, in response to operation by the one or more processors, to perform processing of a plurality of images, wherein performance of the processing includes;

generation of a plurality of pairs of keypoint features, with each pair of keypoint features including a first keypoint feature from a first of the one or more images, and a second keypoint feature from a second of the one or more images;

for each pair of keypoint features, generation of first one or more adjoin features for the first keypoint feature of the keypoint feature pair, and corresponding second one or more adjoin features for the second keypoint feature of the keypoint feature pair;

for each pair of keypoint features, determination of whether an adjoin feature of the first keypoint feature of the keypoint feature pair is similar to the corresponding adjoin feature of the second keypoint feature of the keypoint feature pair; and

determination of whether the first and the second image have at least one similar object, based at least in part on a result of the determination of similarity between the corresponding adjoin features of the first and second keypoint features of the keypoint feature pairs.

39. The apparatus of claim 38, wherein performance of processing one or more images further includes highlight at least an object in the first image based at least in part on some of the keypoint features of the first image that are members of the set of similar keypoint feature pairs.

40. The apparatus of claim 38, wherein the plurality of modules are first plurality of modules, and the storage medium further comprises second plurality of modules configured to cause the apparatus, in response to operation by the one or more processors, to perform processing of a second image, wherein performance of processing of the second image includes:

performance of pairwise matching of local features of the second image to generate an initial plurality of point pairs for the second image;

scale consistency check the point pairs to eliminate falsely matched point pairs from the initial plurality of point pairs, resulting with a first intermediate plurality of scale consistency checked point pairs;

content consistency check the first intermediate plurality of scale consistency checked point pairs to discard dominant point pairs in background of the second image, resulting in a second intermediate plurality of content consistency checked point pairs;

performance of discriminative foreground and background modeling of the second image based at least in part on the second intermediate plurality of content consistency checked point pairs; and

segmentation of a bilateral symmetric object in the second image based at least in part on geometric properties of the discriminative foreground and background modeled second image.

41. The apparatus of claim 38, wherein the storage medium further comprises a third plurality of modules configured to cause the apparatus, in response to operation by the one or more processors, to perform processing of a third image, wherein performance of processing of the third image includes:

detection of bilateral symmetric point pair groups to locate a plurality of butterfly candidates in the third image;

identification of poses of the plurality of butterfly candidates;

elimination of false positives of the butterfly candidates, based at least in part on the poses, and a plurality of templates;

generation of descriptors representing remaining butterfly candidates; and

identification of species of the remaining butterfly candidates through spectral regression, using the generated descriptors.

42. The apparatus of claim 41, wherein identification of poses comprises estimation of poses and rectification of estimated poses.

43. The apparatus of claim 38, wherein the apparatus is a selected one of a set-top box, a smartphone, a computing tablet, an ultrabook, a laptop computer, a desktop computer, a server computer, a game console, or a media player.