SYSTEMS AND METHODS FOR IMAGE RECOGNITION USING MOBILE DEVICES

• US 20100260426A1
• Filed: 08/07/2009
• Published: 10/14/2010
• Est. Priority Date: 04/14/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: YONG, Chang, LI, Hsiang-Tsun, DOS SANTOS, Jose Ricardo, HUANG, Joseph Jyh-Huei, YAMAKAWA, Devender Akira
• Granted Patent: US 9,195,898 B2
• Time in Patent Office: Days
• US Class Current: 382/218
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Implementations relate to systems and methods for real-time image recognition and mobile visual searching. A mobile device, such as a cellular phone, acquires an image and pre-processes the acquired image to generate a visual search query based on objects detected in the acquired image. The visual search query includes the acquired image or a query image extracted therefrom and metadata associated with the detected objects. The mobile device wirelessly communicates the visual search query to a remote server, and in response to the visual search query, the remote server recognizes an object in the query image based on the associated metadata. The remote server then generates information content based on the recognized object and communicates the information content to the mobile device to be presented via the mobile device.

Claims

1. A mobile device, comprising:

a wireless interface to a server; and

a processor, communicating with the wireless interface, the processor being configured to initiate processing comprising;

acquiring an image;

detecting one or more objects in the acquired image;

generating metadata associated with at least one of the detected objects;

extracting a query image from the acquired image based on the at least one object;

generating a visual search query, wherein the visual search query includes the query image and the metadata;

communicating the visual search query to the server via the wireless interface; and

receiving and presenting, in response to the visual search query, information content associated with the at least one object.

2. The mobile device of claim 1, wherein generating metadata further comprises:

categorizing one or more of the detected objects as being in an object category;

highlighting the categorized objects;

receiving input that indicates a selected one of the categorized objects; and

generating the metadata based on the object category of the selected object.

3. The mobile device of claim 2, wherein categorizing one or more of the detected objects further comprises:

generating a feature vector based on at least one of the detected objects;

comparing the feature vector with a set of image coefficients of trained images in the object category to determine a matching trained image in the object category that matches the at least one detected object, wherein the set of image coefficients is stored in the mobile device; and

categorizing the detected object based on the matching trained image.

4. The mobile device of claim 3, wherein the processor performs, in real-time, the operations of acquiring, detecting, highlighting, and categorizing.

5. The mobile device of claim 3, wherein the object category includes a logo, a design, a face, a landmark, apparel, a sign, a natural object, or a man-made object.

6. The mobile device of claim 1, wherein generating metadata further comprises:

highlighting the detected objects;

receiving input that indicates a selected one of the detected objects; and

generating the metadata based on the selected object.

7. The mobile device of claim 6, wherein generating the metadata based on the selected objects further comprises:

categorizing the selected object as being in an object category; and

generating the metadata based on the object category of the selected object.

8. The mobile device of claim 7, wherein communicating the visual search query to the server further comprises:

generating a destination address for the visual search query based on the object category of the selected object; and

communicating the visual search query to the server according to the destination address.

9. The mobile device of claim 1, wherein extracting a query image from the acquired image further comprises cropping the acquired image, compressing the acquired image, scaling the acquired image, or converting the acquired image to grayscale.

10. The mobile device of claim 1, wherein generating metadata further comprises:

generating a feature vector based on at least one of the detected objects;

comparing the feature vector with a set of image coefficients of trained images to determine a matching trained image that matches the detected object, wherein the set of image coefficients is stored in the mobile device;

recognizing the detected object based on the matching trained image; and

generating the metadata associated with the recognized object based on the matching trained image.

11. The mobile device of claim 1, wherein the mobile device comprises a contextual data sensor, and wherein acquiring an image further comprises:

acquiring contextual data associated with the acquired image via the contextual data sensor, wherein the visual search query further includes the contextual data.

12. The mobile device of claim 11, wherein the contextual data comprises a Global Positioning System (GPS) location fix, an Assisted Global Positioning System (A-GPS) location fix, a Galileo system location fix, a tower trilateration fix, textual information, auditory information, an accelerometer reading, a gyroscope reading, or a temperature reading.

13. The mobile device of claim 1, wherein the information content includes a name, a price, a manufacturer, a review, a coupon, or an advertisement.

14. A method of performing image recognition, comprising:

acquiring an image by a mobile device;

detecting one or more objects in the acquired image;

generating metadata associated with at least one of the detected objects;

extracting a query image from the acquired image based on the at least one object;

generating a visual search query, wherein the visual search query includes the query image and the metadata;

wirelessly communicating the visual search query; and

receiving and presenting, in response to the visual search query, information content associated with the at least one object.

15. The method of claim 14, wherein generating metadata further comprises:

categorizing one or more of the detected objects as being in an object category;

highlighting the categorized objects;

receiving input that indicates a selected one of the categorized objects; and

generating the metadata based on the object category of the selected object.

16. The method of claim 15, wherein categorizing one or more of the detected objects further comprises:

generating a feature vector based on at least one of the detected objects;

comparing the feature vector with a set of image coefficients of trained images in the object category to determine a matching trained image in the object category that matches the at least one detected object, wherein the set of image coefficients is stored in the mobile device; and

categorizing the detected object based on the matching trained image.

17. The method of claim 16, wherein the mobile device performs, in real-time, the operations of acquiring, detecting, highlighting, or categorizing.

18. The method of claim 14, wherein generating metadata further comprises:

highlighting the detected objects;

receiving input that indicates a selected one of the detected objects; and

generating the metadata based on the selected object.

19. The method of claim 18, wherein generating the metadata based on the selected objects further comprises:

categorizing the selected object as being in an object category; and

generating the metadata based on the object category of the selected object.

20. The method of claim 19, wherein wirelessly communicating the visual search query further comprises:

generating a destination address for the visual search query based on the object category of the selected object; and

communicating the visual search query to a destination according to the destination address.

21. The method of claim 14, wherein acquiring an image further comprises:

acquiring contextual data associated with the acquired image via a contextual data sensor of the mobile device, the visual search query further including the contextual data.

22. A system for performing image recognition, comprising:

means for acquiring an image by a mobile device;

means for detecting one or more objects in the acquired image;

means for generating metadata associated with at least one of the detected objects;

means for extracting a query image from the acquired image based on the at least one object;

means for generating a visual search query, wherein the visual search query includes the query image and the metadata;

means for wirelessly communicating the visual search query; and

means for receiving and presenting, in response to the visual search query, information content associated with the at least one object.

23. The system of claim 22, wherein the means for generating metadata further comprises:

means for categorizing one or more of the detected objects as being in an object category;

means for highlighting the categorized objects;

means for receiving input that indicates a selected one of the categorized objects; and

means for generating the metadata based on the object category of the selected object.

24. The system of claim 23, wherein the means for categorizing one or more of the detected objects further comprises:

means for generating a feature vector based on at least one of the detected objects;

means for comparing the feature vector with a set of image coefficients of trained images in the object category to determine a matching trained image in the object category that matches the at least one detected object, wherein the set of image coefficients is stored in the mobile device; and

means for categorizing the detected object based on the matching trained image.

25. The system of claim 24, wherein the means for acquiring, detecting, highlighting, and categorizing perform in real-time.

26. The system of claim 22, wherein the means for generating metadata further comprises:

means for highlighting the detected objects;

means for receiving input that indicates a selected one of the detected objects; and

means for generating the metadata based on the selected object.

27. The system of claim 26, wherein the means for generating the metadata based on the selected objects further comprises:

means for categorizing the selected object as being in an object category; and

means for generating the metadata based on the object category of the selected object.

28. The system of claim 27, wherein the means for wirelessly communicating the visual search query further comprises:

means for generating a destination address for the visual search query based on the object category of the selected object; and

means for communicating the visual search query to a destination according to the destination address.

29. The system of claim 22, wherein the means for acquiring an image further comprises:

means for acquiring contextual data associated with the acquired image via a contextual data sensor of the mobile device, the visual search query further including the contextual data.

30. A computer program product, comprising:

computer-readable medium storing instructions that cause a computer to perform image recognition when executed by the computer, comprising;

at least one instruction for causing the computer to acquire an image;

at least one instruction for causing the computer to detect one or more objects in the acquired image;

at least one instruction for causing the computer to generate metadata associated with at least one of the detected objects;

at least one instruction for causing the computer to extract a query image from the acquired image based on the at least one object;

at least one instruction for causing the computer to generate a visual search query, wherein the visual search query includes the query image and the metadata;

at least one instruction for causing the computer to wirelessly communicate the visual search query; and

at least one instruction for causing the computer to receive and present, in response to the visual search query, information content associated with the at least one object.

31. The computer program product of claim 30, wherein the at least one instruction for causing the computer to generate metadata further comprises:

at least one instruction for causing the computer to categorize one or more of the detected objects as being in an object category;

at least one instruction for causing the computer to highlight the categorized objects;

at least one instruction for causing the computer to receive input that indicates a selected one of the categorized objects; and

at least one instruction for causing the computer to generate the metadata based on the selected object category of the selected object.

32. The computer program product of claim 31, wherein the at least one instruction for causing the computer to categorize one or more of the detected objects further comprises:

at least one instruction for causing the computer to generate a feature vector based on at least one of the detected objects;

at least one instruction for causing the computer to compare the feature vector with a set of image coefficients of trained images in the object category to determine a matching trained image in the object category that matches the at least one detected object, wherein the set of image coefficients is stored in the computer; and

at least one instruction for causing the computer to categorize the detected object based on the matching trained image.

33. The computer program product of claim 32, wherein the at least one instruction for causing the computer to acquire, detect, highlight, and categorize causes the computer to acquire, detect, highlight, and categorize in real-time.

34. The computer program product of claim 30, wherein the at least one instruction for causing the computer to generate metadata further comprises:

at least one instruction for causing the computer to highlight the detected objects;

at least one instruction for causing the computer to receive input that indicates a selected one of the highlighted object; and

at least one instruction for causing the computer to generate the metadata based on the selected object.

35. The computer program product of claim 34, wherein the at least one instruction for causing the computer to generate the metadata based on the selected objects further comprises:

at least one instruction for causing the computer to categorize the selected object as being in an object category; and

at least one instruction for causing the computer to generate the metadata based on the object category of the selected object.

36. The computer program product of claim 35, wherein the at least one instruction for causing the computer to wirelessly communicate the visual search query further comprises:

at least one instruction for causing the computer to generate a destination address for the visual search query based on the object category of the selected object; and

at least one instruction for causing the computer to communicate the visual search query to a destination according to the destination address.

37. The computer program product of claim 30, wherein the at least one instruction for causing the computer to acquire an image further comprises:

at least one instruction for causing the computer to acquire contextual data associated with the acquired image via a contextual data sensor of the mobile device, the visual search query further including the contextual data.

38. A system for performing image recognition, comprising:

a server configured to;

receive a visual search query from a mobile device, wherein the visual search query includes an image and metadata associated with at least one object in the image,

recognize an object in the image associated with the at least one object based on the metadata,

generate information content based on the recognized object, and

communicate the information content in response to the visual search query.

39. The system of claim 38, wherein

the visual search query further includes contextual data associated with the image;

and

further wherein the server is configured to generate the information content based on the recognized object and the contextual data.

40. The system of claim 39, wherein the contextual data comprises a Global Positioning System (GPS) location fix, an Assisted Global Positioning System (A-GPS) location fix, a Galileo system location fix, a tower trilateration fix, textual information, auditory information, an accelerometer reading, a gyroscope reading, or a temperature reading.

41. The system of claim 38, wherein the server is further configured to:

compare the image with trained images to determine a matching trained image that matches the image, wherein the trained images are selected based on the metadata, and

recognize the object in the image based on the matching trained image.

42. The system of claim 38, wherein the server is further configured to:

detect the object in the image based on the metadata,

generate a feature vector of the object,

compare the feature vector with image coefficients of trained images to determine a matching trained image that matches the object, wherein the image coefficients are selected based on the metadata, and

recognize the object based on the matching trained image.

43. The system of claim 38, wherein the object includes a logo, a design, a face, a landmark, apparel, a sign, a natural object, or a man-made object.

44. The system of claim 38, wherein the information content includes a name, a price, a manufacturer, a review, a coupon, or an advertisement.

45. The system of claim 38, wherein the server is further configured to:

store the visual search query, and

associate the information content with the visual search query.

46. A method of performing image recognition, comprising:

receiving a visual search query from a mobile device, wherein the visual search query includes an image and metadata associated with at least one object in the image;

recognizing an object in the image associated with the at least one object based on the metadata;

generating information content based on the recognized object; and

communicating the information content in response to the visual search query.

47. The method of claim 46, wherein the visual search query further includes contextual data associated with the image, and further wherein generating information content comprises generating the information content based on the recognized object and the contextual data.

48. The method of claim 47, wherein the contextual data comprises a Global Positioning System (GPS) location fix, an Assisted Global Positioning System (A-GPS) location fix, a Galileo system location fix, a tower trilateration fix, textual information, auditory information, an accelerometer reading, a gyroscope reading, or a temperature reading.

49. The method of claim 46, wherein recognizing an object in the image further comprises:

comparing the image with a set of trained images to determine a matching trained image that matches the image, wherein the set of trained images is selected based on the metadata; and

recognizing the object in the image based on the matching trained image.

50. A system of performing image recognition, comprising:

means for receiving a visual search query from a mobile device, wherein the visual search query includes an image and metadata associated with at least one object in the image;

means for recognizing an object in the image associated with the at least one object based on the metadata;

means for generating information content based on the recognized object; and

means for communicating the information content in response to the visual search query.

51. The system of claim 50, wherein the visual search query further includes contextual data associated with the image, and further wherein the means for generating information content comprises means for generating the information content based on the recognized object and the contextual data.

52. The system of claim 51, wherein the contextual data comprises a Global Positioning System (GPS) location fix, an Assisted Global Positioning System (A-GPS) location fix, a Galileo system location fix, a tower trilateration fix, textual information, auditory information, an accelerometer reading, a gyroscope reading, or a temperature reading.

53. The system of claim 50, wherein the means for recognizing an object in the image further comprises:

means for comparing the image with a set of trained images to determine a matching trained image that matches the image, wherein the set of trained images is selected based on the metadata; and

means for recognizing the object in the image based on the matching trained image.

54. A computer program product, comprising:

computer-readable medium storing instructions that cause a computer to perform image recognition when executed by the computer, comprising;

at least one instruction for causing the computer to receive a visual search query from a mobile device, wherein the visual search query includes an image and metadata associated with at least one object in the image;

at least one instruction for causing the computer to recognize an object in the image associated with the at least one object based on the metadata;

at least one instruction for causing the computer to generate information content based on the recognized object; and

at least one instruction for causing the computer to communicate the information content in response to the visual search query.

55. The computer program product of claim 54, wherein the visual search query further includes contextual data associated with the image, and further wherein the at least one instruction for causing the computer to generate information content comprises at least one instruction for causing the computer to generate the information content based on the recognized object and the contextual data.

56. The computer program product of claim 55, wherein the contextual data comprises a Global Positioning System (GPS) location fix, an Assisted Global Positioning System (A-GPS) location fix, a Galileo system location fix, a tower trilateration fix, textual information, auditory information, an accelerometer reading, a gyroscope reading, or a temperature reading.

57. The computer program product of claim 54, wherein the at least one instruction for causing the computer to recognize an object in the image further comprises:

at least one instruction for causing the computer to compare the image with a set of trained images to determine a matching trained image that matches the image, wherein the set of trained images is selected based on the metadata; and

at least one instruction for causing the computer to recognize the object in the image based on the matching trained image.