Image retrieval method, real-time drawing prompting method, and devices thereof

• US 9,183,228 B2
• Filed: 11/08/2013
• Issued: 11/10/2015
• Est. Priority Date: 12/18/2012
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Wang, Lin, Xiong, Junjun
• Primary Examiner(s): Couso, Jose
• Application Number:
  US14/075,411
• Publication Number:
  US 20140169683A1
• Time in Patent Office: 732 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis

Abstract

An image retrieval method is provided. The method includes detecting an outline of an image and obtaining an outline feature of the image, generating an index list in an image database according to the outline feature, and obtaining a sketch input by a user and retrieving images containing the sketch from the index list. The various embodiments of the present disclosure also provide an image retrieval device and a real-time drawing prompting method and device. The various embodiments of the present disclosure can rapidly and precisely retrieve image information based on contents, and the user can rapidly retrieve desired images by inputting a sketch. The real-time drawing prompting method and device can provide the limner with real-time guidance based on the retrieval of a sketch map.

Claims

1. An image retrieval method, the method comprising:

detecting an outline of an image and obtaining an outline feature of the image;

generating an index list in an image database according to the outline feature;

obtaining a sketch input by a user; and

retrieving images containing the sketch from the index list,

wherein the detecting of the outline of the image comprises;

obtaining feature information of respective edges in the image; and

normalizing intensities of the edges.

2. The method according to claim 1, wherein the detecting of the outline of the image comprises:

detecting a long edge in the image and obtaining feature information of the long edge.

3. The method according to claim 2, wherein the detecting of the outline of the image comprises:

implementing a steerable filtering to obtain the feature information, wherein the feature information of the edges comprises edge position, edge orientation, and edge intensity;

making the intensities of the edges uniform; and

connecting the normalized intensities of the edges according to edge orientations to obtain the long edge.

4. The method according to claim 3, wherein the generating of the index list in the image database according to the outline features comprises:

generating the index list for the feature information of the long edge in an inverted order.

5. The method according to claim 4, wherein the generating of the index list for the feature information of the long edge in the inverted order comprises:

generating an index entry (x, y, θ

) according to the obtained feature information of the long edge;

making statistics of gradient orientation of each pixel in each image sequentially and storing an index number of each image in a corresponding index entry (x, y, θ

); and

generating the index list for the index entry (x, y, θ

) in the inverted order.

6. The method according to claim 5, wherein the obtaining of the sketch and the retrieving of the images containing the sketch from the index list according to the sketch comprises:

detecting the outline feature of the sketch;

determining whether the index list contains a long edge similar to the outline of the sketch according to accumulated hits; and

obtaining images containing the long edge based on the accumulated hits.

7. The method according to claim 6, wherein the determining of whether the index list contains the long edge similar to the outline of the sketch according to accumulated hits comprises:

detecting the feature information of each pixel in the outline of the sketch, wherein the feature information of the pixel comprises a position feature and an orientation feature; and

determining whether the index list contains pixels in the long edge that are similar to the pixel in the outline of the sketch.

8. The method according to claim 6, wherein the obtaining of the images containing the long edges based on the accumulated hits comprises:

performing a preliminary retrieval from the index list and generating a candidate image set for the images containing the long edge; and

when a new sketch is detected, performing the retrieval again and updating the candidate image set.

9. The method according to claim 8, wherein the generating of the candidate image set for the images containing the long edge comprises:

sorting the images in the candidate image set according to rankings of similarity thereof after determining the index list contains the long edge similar to the outline of the sketch.

10. The method according to claim 9, wherein the obtaining of the sketch input by the user and the retrieving of the images containing the sketch from the index list further comprises:

displaying the images in the candidate image set underlying the sketch input by the user in a superimposed manner according to the rankings of the similarity thereof.

11. The method according to claim 10, wherein the superimposed manner comprises presenting part of the long edges of the image not overlapped with the outline of the sketch as a background image.

12. An image retrieval device, the device comprising:

an image outline detection module configured to detect an outline of an image and obtain an outline feature of the image;

an outline feature management module comprising an index list generated in an image database according to the outline feature; and

a retrieval module configured to obtain a sketch input by a user and retrieve images containing the sketch from the index list,

wherein the image outline detection module is configured to obtain feature information of respective edges in the image and to normalize intensities of the edges.

13. The device according to claim 12, wherein the image outline detection module is configured to detect a long edge of the image and obtain feature information of the long edge.

14. The device according to claim 13, wherein the image outline detection module comprises:

a steerable filtering unit configured to implement a steerable filtering to obtain feature information, wherein the feature information of the edges comprises edge position, edge orientation, and edge intensity;

a normalization unit configured to make the intensities of the edge uniform; and

a long edge obtaining unit configured to connect the normalized intensities of the edges according to the edge orientations to obtain the long edge.

15. The device according to claim 14, wherein the outline feature management module generates the index list for the feature information of the long edge in an inverted order.

16. The device according to claim 15, wherein the index list comprises:

an index entry (x, y, θ

) generated according to the obtained feature information of the long edge;

a gradient orientation of each pixel in each image and the index number of each image stored in the corresponding index entry (x, y, θ

); and

an inverted sorting unit configured to generate the index list for the index entry (x, y, θ

) in an inverted order.

17. The device according to claim 16, wherein the retrieval module comprises:

an outline feature extraction unit configured to detect the outline feature of the sketch;

a determination unit configured to determine whether the index list contains a long edge similar to the outline of the sketch; and

an image retrieval unit configured to retrieve images containing the long edge.

18. The device according to claim 17, wherein the outline feature obtaining unit is configured to:

detect the feature information of each pixel in the outline of the sketch, wherein the feature information of the pixel comprises a position feature and an orientation feature; and

determine whether the index list contains a pixel in the long edge similar to the pixel in the outline of the sketch.

19. The device according to claim 17, wherein the image retrieval unit is configured to:

perform a preliminary retrieval from the index list and generate a candidate image set for the images containing the long edge; and

perform the retrieval again when a new input sketch is detected and update the candidate image set.

20. The device according to claim 19, wherein the image retrieval unit is configured to:

sort the images in the candidate image set according to rankings of the similarity thereof after determining the index list contains the long edge similar to the outline of the sketch.

21. The device according to claim 20, further comprising:

a display module configured to display the images in the candidate image set underlying the sketch in a superimposed manner according to the rankings of the similarity thereof.

22. The device according to claim 21, wherein the superimposed manner comprises presenting part of the long edge of the image not overlapped with the outline of the sketch as a background image.

23. A real-time drawing prompting method, the method comprising:

detecting and displaying a sketch input by a user on a drawing area;

obtaining the sketch;

retrieving images containing the sketch; and

displaying the retrieved images underlying the sketch as a background image in a superimposed manner with weights,

wherein the obtaining of the sketch comprises;

obtaining feature information of respective edges from an outline of the sketch; and

normalizing intensities of the edges.

24. The method according to claim 23, further comprising:

when the sketch changes, retrieving images containing the sketch again and displaying the updated background images in real time.

25. The method according to claim 24, wherein the changes of the sketch comprise at least one of:

a new sketch sequentially input by the user is detected;

a sketch that the user traces the background image is detected; and

a new sketch input by the user is detected after erasing an existing sketch.

26. The method according to claim 23, wherein the displaying of the retrieved images underlying the sketch as a background image in a superimposed manner with weights comprises:

differentially displaying the retrieved images containing the sketch according to rankings of similarity thereof.

27. The method according to claim 26, wherein the differential displaying of the retrieved images comprises at least one of:

a correspondingly deep color is displayed according to a degree of similarity between the retrieved images and the sketch; and

a correspondingly bright luminance is displayed according to the degree of similarity between the retrieved images and the sketch.

28. The method according to claim 23, further comprising:

when the images containing the sketch are retrieved, generating a candidate image box and displaying the images for user selection in the candidate image box when the similarity reaches a threshold.

29. The method according to claim 28, wherein the displaying of the images in the candidate image box when the similarity reaches the threshold comprises:

sorting the images containing the sketch according to the rankings of the similarity thereof.

30. The method according to claim 28, wherein the displaying of the images in the candidate image box when the similarity reaches the threshold comprises:

when the user selecting an image displayed in the candidate image box is detected, displaying an image superimposed with a weight as a background image with highlight or in deep color corresponding to the selected image.

31. A real-time drawing prompting device, the device comprising:

a sketch detection module configured to detect and display a sketch input by a user on a drawing area and obtain the sketch;

an image retrieval module configured to retrieve images containing the sketch; and

a display module configured to displaying the retrieved images underlying the sketch as a background image in a superimposed manner with weights,

wherein the sketch detection module is configured to obtain feature information of respective edges from an outline of the sketch and to normalize intensities of the edges.

32. The device according to claim 31, wherein the image retrieval module is further configured to:

retrieve images containing the sketch again and sending the retrieved images to the display module when the sketch changes, so that the display module displays the updated background image in real time.

33. The device according to claim 31, wherein the sketch detection module is further configured to:

detect a new sketch sequentially input by the user;

detect a sketch that the user traces the background image; and

detect a new sketch input by the user after erasing an existing sketch.

34. The device according to claim 31, wherein the display module is further configured to:

differentially display the retrieved images containing the sketch according to rankings of similarity thereof.

35. The device according to claim 34, wherein the display module is further configured to:

display a retrieved image in a correspondingly deep color when the retrieved image has a threshold similarity with the sketch;

or

display a retrieved image in a correspondingly bright luminance when the retrieved image has the threshold similarity with the sketch.

36. The device according to claim 31, further comprising:

a candidate image box configured to display the images therein which the similarity reaches a threshold for the user to select when the image retrieval module retrieves images containing the sketch.

37. The device according to claim 36, wherein the candidate image box is further configured to:

sort the images containing the sketch according to rankings of the similarity thereof.

38. The device according to claim 36, wherein the display module is configured to:

when the user selecting an image displayed in the candidate image box is detected, display an image superimposed with a weight as a background image with highlight or in deep color corresponding to the selected image.