Semantic visual search engine

• US 20070073749A1
• Filed: 09/28/2005
• Published: 03/29/2007
• Est. Priority Date: 09/28/2005
• Status: Active Grant

• Current Assignee: Conversant Wireless Licensing S.à.r.l. (f/k/a Core Wireless Licensing S.a.r.l.) (MOSAID Technologies Inc. (f/k/a Conversant Intellectual Property Management))
• Original Assignee: Nokia Corporation
• Inventors: Fan, Lixin
• Granted Patent: US 7,865,492 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour

Abstract

An improved method, device and computer program product for enabling a system to learn, categorize and search items such as images and video clips according to their semantic meanings. According to the present invention, prominent features can be separated from low-level features in an item using supervised learning approaches. Prominent features are used to categorize and annotate new target items. Users can then use key words and/or template items for searching through the respective database.

Claims

1. A method of categorizing a plurality of items on a mobile electronic device, comprising:

converting a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in the respective item; and

using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories.

2. The method of claim 1, wherein the supervised learning approach comprises the AdaBoosting learning algorithm.

3. The method of claim 1, wherein the supervised learning approach comprises the use of Bayesian statistics.

4. The method of claim 1, wherein the supervised learning approach comprises SVM pattern recognition.

5. The method of claim 1, wherein the plurality of items comprise videos.

6. The method of claim 1, wherein the plurality of items comprise images.

7. The method of claim 1, further comprising:

converting a target item into a plurality of multi-scale local features; and

for each of the plurality of multi-scale local features, matching the multi-scale local feature with the prominent low-level features using a probabilistic model.

8. The method of claim 7, wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.

9. The method of claim 8, wherein the predetermined threshold is determined through the use of the supervised learning approach.

10. The method of claim 7, further comprising:

receiving an input from a user; and

returning at least one item to the user that share certain similarities with the input.

11. The method of claim 10, wherein the input comprises a keyword, and wherein the at least one item includes an annotation that is similar to the keyword.

12. The method of claim 10, wherein the input comprises a template item, and wherein low-level features in the input are compared to the prominent low-level features to identify the at least one item.

13. The method of claim 10, wherein the input comprises a template item and the keyword, and the returning of the at least one item comprises:

pre-filtering low-level features in the template item using probabilistic models of interested categories; and

matching the pre-filtered low-level features with target images in the same category, the category being identified by the keyword.

14. A computer program product for categorizing a plurality of items on a mobile electronic device, comprising:

computer code for converting a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in the respective item; and

computer code for using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories.

15. The computer program product of claim 14, further comprising:

computer code for converting a target item into a plurality of multi-scale local features; and

computer code for, for each of the plurality of multi-scale local features, matching the multi-scale local feature with the prominent low-level features using a probabilistic model.

16. The computer program product of claim 15, wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.

17. The computer program product of claim 16, wherein the predetermined threshold is determined through the use of the supervised learning approach.

18. The computer program product of claim 15, further comprising:

computer code for receiving an input from a user; and

computer code for returning at least one item to the user that share certain similarities with the input.

19. The computer program product of claim 18, wherein the input comprises a keyword, and wherein the at least one item includes an annotation that is similar to the keyword.

20. The computer program product of claim 18, wherein the input comprises a template item, and wherein low-level features in the input are compared to the prominent low-level features to identify the at least one item.

21. The computer program product of claim 18, wherein the input comprises a template item and the keyword, and the returning of the at least one item comprises:

computer code for pre-filtering low-level features in the template item using probabilistic models of interested categories; and

computer code for matching the pre-filtered low-level features with target images in the same category, the category being identified by the keyword.

22. An electronic device, comprising:

a processor, and

a memory unit operatively connected to the processor and including;

computer code for converting a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in the respective item; and

computer code for using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories.

23. The electronic device of claim 22, wherein the memory unit further comprises:

computer code for converting a target item into a plurality of multi-scale local features; and

computer code for, for each of the plurality of multi-scale local features, matching the multi-scale local feature with the prominent low-level features using a probabilistic model.

24. The electronic device of claim 23, wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.

25. The electronic device of claim 24, wherein the predetermined threshold is determined through the use of the supervised learning approach.

26. The electronic device of claim 25, wherein the memory unit further comprises:

computer code for receiving an input from a user; and

computer code for returning at least one item to the user that share certain similarities with the input.