AUTOMATIC IMAGE RECTIFICATION FOR VISUAL SEARCH

• US 20140233845A1
• Filed: 02/21/2013
• Published: 08/21/2014
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: WAGNER, Daniel, Pan, Qi
• Granted Patent: US 9,058,683 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

Disclosed is a computing device that can perform automatic image rectification for a visual search. A method implemented at a computing device includes receiving one or more images from an image capture device, storing the one or more images with the computing device, building a three dimensional (3D) geometric model for one or more potential objects of interest within an environment based on at least one image of the one or more images, and automatically creating at least one rectified image having at least one potential object of interest for a visual search.

Claims

1. A method implemented at a computing device, comprising:

receiving one or more images from an image capture device;

storing the one or more images with the computing device;

building a three dimensional (3D) geometric model for one or more potential objects of interest within an environment based on at least one image of the one or more images; and

automatically creating at least one rectified image having at least one potential object of interest for a visual search.

2. The method of claim 1, further comprising:

automatically uploading the at least one rectified image to a server for the visual search.

3. The method of claim 1, further comprising:

uploading the at least one rectified image to a server for the visual search.

4. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

5. The method of claim 1, wherein the one or more images are automatically captured with an image capture device that is associated with the computing device during a background operation without receiving a user input.

6. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device during a time period with the image capture device moving for at least a portion of the time period.

7. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

8. The method of claim 1, further comprising:

searching the 3D geometric model for at least one planar structure associated with the one or more potential objects of interest within the environment.

9. The method of claim 1, wherein the 3D geometric model is built using a structure-from-motion system.

10. The method of claim 1, further comprising:

searching the 3D geometric model for at least two planar structures associated with the one or more potential objects of interest within the environment

11. The method of claim 1, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

12. The method of claim 1, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

13. A computing device comprising:

a storage medium to store one or more received images; and

a processing circuit coupled to the storage medium, the processing circuit is configured to execute instructions to build a three dimensional (3D) geometric model for one or more potential objects of interest within an environment based on at least one image of the one or more stored images and to automatically create at least one rectified image having at least one potential object of interest for a visual search.

14. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to automatically upload the at least one rectified image to a server for the visual search.

15. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to upload the at least one rectified image to a server for the visual search.

16. The computing device of claim 13, further comprising:

an image capture device coupled to the storage medium, the image capture device to capture the one or more images, wherein the storage medium to receive the one or more images from the image capture device.

17. The computing device of claim 16, wherein the processing circuit is configured to execute instructions to automatically create at least one rectified image having at least one potential object of interest for the visual search without receiving a user input.

18. The computing device of claim 16, wherein the images are automatically captured with the image capture device during a background operation without receiving a user input.

19. The computing device of claim 16, wherein the one or more images are captured with the image capture device during a time period in which the image capture device is moving for at least a portion of the time period.

20. The computing device of claim 16, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

21. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to search the 3D geometric model for at least one planar structure associated with the one or more potential objects of interest within the environment.

22. The computing device of claim 13, wherein the 3D geometric model is built using a structure-from-motion system.

23. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to search the 3D geometric model for at least two planar structures associated with the one or more potential objects of interest within the environment.

24. The computing device of claim 13, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

25. The computing device of claim 13, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

26. A computer program product executed at a computing device comprising:

a computer-readable medium comprising code for;

storing one or more images with the computing device;

building a three dimensional (3D) geometric model with the computing device for one or more potential objects of interest within an environment based on at least one image of the one or more images; and

automatically creating at least one rectified image having at least one potential object of interest for a visual search.

27. The computer program product of claim 26, further comprising code for automatically uploading the at least one rectified image to a server for the visual search.

28. The computer program product of claim 26, further comprising code for uploading the at least one rectified image to a server for the visual search.

29. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

30. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device during a background operation without receiving a user input.

31. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device during a time period with the image capture device moving for at least a portion of the time period.

32. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

33. The computer program product of claim 26, further comprising code for searching the 3D geometric model for at least one planar structure associated with the one or more potential objects of interest within the environment.

34. The computer program product of claim 26, wherein the 3D geometric model is built using a structure-from-motion system.

35. The computer program product of claim 26, further comprising code for searching the 3D geometric model for at least two planar structures associated with the one or more potential objects of interest within the environment.

36. The computer program product of claim 26, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

37. The computer program product of claim 26, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

38. A server comprising:

a storage medium to store a plurality of images; and

a processing circuit configured to execute instructions to receive at least one rectified image having at least one potential object of interest from a computing device for a visual search and to extract descriptors representing features of the at least one rectified image, wherein the extracted descriptors of the at least one rectified image are designed to be invariant to rotation, scale, and lighting without needing to be invariant to perspective or affine distortion.

39. The server of claim 38, wherein the processing circuit is further configured to execute instructions to match the extracted descriptors of the at least one rectified image with descriptors of the plurality of images stored in the database.

40. The server of claim 39, wherein the processing circuit is further configured to execute instructions to transmit information associated with at least one image stored in the storage medium that has matching descriptors.

41. The server of claim 38, wherein the extracted descriptors of the at least one rectified image are not invariant to perspective or affine distortion.

42. A method implemented at a server comprising:

storing a plurality of images with the server;

receiving at least one rectified image having at least one potential object of interest from a computing device for a visual search; and

extracting descriptors representing features of the at least one rectified image, wherein the extracted descriptors of the at least one rectified image are designed to be invariant to rotation, scale, and lighting without needing to be invariant to perspective or affine distortion.

43. The method of claim 42, further comprising:

matching the extracted descriptors of the at least one rectified image with descriptors of the plurality of images.

44. The method of claim 43, further comprising:

transmitting information associated with at least one image stored with the server that has matching descriptors.

45. The method of claim 42, wherein the extracted descriptors of the at least one rectified image are not invariant to perspective or affine distortion.

46. A computer program product executed at a server comprising:

a computer-readable medium comprising code for;

storing a plurality of images;

receiving at least one rectified image having at least one potential object of interest from a computing device for a visual search; and

extracting descriptors representing features of the at least one rectified image, wherein the extracted descriptors of the at least one rectified image are designed to be invariant to rotation, scale, and lighting without needing to be invariant to perspective or affine distortion.

47. The computer program product of claim 46, further comprising code for matching the extracted descriptors of the at least one rectified image with descriptors of the plurality of images.

48. The computer program product of claim 47, further comprising code for transmitting information associated with at least one image stored in a database that has matching descriptors.

49. The computer program product of claim 46, wherein the extracted descriptors of the at least one rectified image are not invariant to perspective or affine distortion.

50. An apparatus comprising:

means for storing one or more received images;

means for building a three dimensional (3D) geometric model for one or more potential objects of interest within an environment based on at least one image of the one or more stored images; and

means for automatically creating at least one rectified image having at least one potential object of interest for a visual search.

51. The apparatus of claim 50, further comprising:

means for automatically uploading the at least one rectified image to a server for the visual search.

52. The apparatus of claim 50, further comprising:

means for uploading the at least one rectified image to a server for the visual search.

53. The apparatus of claim 50, further comprising:

means for capturing the one or more images, wherein the means for storing to receive the one or more images from the means for capturing.

54. The apparatus of claim 53, wherein the means for automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

55. The apparatus of claim 53, wherein the one or more images are captured automatically during a background operation without receiving a user input.

56. The apparatus of claim 53, wherein the one or more images are captured during a time period in which the means for capturing the one or more images is moving for at least a portion of the time period.

57. The apparatus of claim 53, wherein the means for capturing is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the means for capturing is not required to move.

58. The apparatus of claim 50, further comprising:

means for searching the 3D geometric model for at least one planar structure associated with the one or more potential objects of interest within the environment.

59. The apparatus of claim 50, further comprising:

means for searching the 3D geometric model for at least two planar structures associated with the one or more potential objects of interest within the environment.

60. The apparatus of claim 50, wherein the means for automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

61. The apparatus of claim 50, wherein the means for automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

62. The apparatus of claim 50, wherein the three dimensional (3D) geometric model is built using a structure-from-motion system.