Automatic image rectification for visual search

• US 9,058,683 B2
• Filed: 02/21/2013
• Issued: 06/16/2015
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wagner, Daniel, Pan, Qi
• Primary Examiner(s): Cunningham, Gregory F
• Application Number:
  US13/772,640
• Publication Number:
  US 20140233845A1
• Time in Patent Office: 845 Days
• Field of Search: 345/653, 345/664, 345/679, 345/953, 348/580, 382/154, 382/275, 715/757, 715/852
• US Class Current: 1/1
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

Disclosed is a computing device that can perform automatic image rectification for a visual search. A method implemented at a computing device includes receiving one or more images from an image capture device, storing the one or more images with the computing device, building a three dimensional (3D) geometric model for one or more potential objects of interest within an environment based on at least one image of the one or more images, and automatically creating at least one rectified image having at least one potential object of interest for a visual search.

Claims

1. A method implemented at a computing device, comprising:

receiving one or more images from an image capture device, the one or more images including an environment and at least a portion of one or more potential objects of interest within the environment;

storing the one or more images with the computing device;

building at least a partial three dimensional (3D) geometric model of the environment included in the one or more images, the at least partial 3D geometric model including at least the portion of the one or more potential objects of interest within the environment, wherein the at least partial 3D geometric model is built using at least one image of the one or more images; and

automatically creating, using the at least partial 3D geometric model, at least one rectified image having at least one potential object of interest for a visual search, the at least one rectified image including a modified perspective view of the at least one potential object of interest relative to a view of the at least one potential object of interest in the one or more images.

2. The method of claim 1, further comprising:

automatically uploading the at least one rectified image to a server for the visual search.

3. The method of claim 1, further comprising:

uploading the at least one rectified image to a server for the visual search.

4. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

5. The method of claim 1, wherein the one or more images are automatically captured with an image capture device that is associated with the computing device during a background operation without receiving a user input.

6. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device during a time period with the image capture device moving for at least a portion of the time period.

7. The method of claim 1, wherein the one or more images are captured with an image capture device that is associated with the computing device, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

8. The method of claim 1, further comprising:

searching the at least partial 3D geometric model of the environment for at least one planar structure associated with at least the portion of the one or more potential objects of interest within the environment.

9. The method of claim 1, wherein the at least partial 3D geometric model of the environment is built using a structure-from-motion system.

10. The method of claim 1, further comprising:

searching the at least partial 3D geometric model of the environment for at least two planar structures associated with at least the portion of the one or more potential objects of interest within the environment.

11. The method of claim 1, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

12. The method of claim 1, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

13. A computing device comprising:

a storage medium to store one or more received images, the one or more received images including an environment and at least a portion of one or more potential objects of interest within the environment; and

a processing circuit coupled to the storage medium, the processing circuit is configured to execute instructions to build at least a partial three dimensional (3D) geometric model of the environment included in the one or more images, the at least partial 3D geometric model including at least the portion of the one or more potential objects of interest within the environment, wherein the at least partial 3D geometric model is built using at least one image of the one or more stored images and to automatically create, using the at least partial 3D geometric model, at least one rectified image having at least one potential object of interest for a visual search, the at least one rectified image including a modified perspective view of the at least one potential object of interest relative to a view of the at least one potential object of interest in the one or more images.

14. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to automatically upload the at least one rectified image to a server for the visual search.

15. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to upload the at least one rectified image to a server for the visual search.

16. The computing device of claim 13, further comprising:

an image capture device coupled to the storage medium, the image capture device to capture the one or more images, wherein the storage medium to receive the one or more images from the image capture device.

17. The computing device of claim 16, wherein the processing circuit is configured to execute instructions to automatically create at least one rectified image having at least one potential object of interest for the visual search without receiving a user input.

18. The computing device of claim 16, wherein the images are automatically captured with the image capture device during a background operation without receiving a user input.

19. The computing device of claim 16, wherein the one or more images are captured with the image capture device during a time period in which the image capture device is moving for at least a portion of the time period.

20. The computing device of claim 16, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

21. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to search the at least partial 3D geometric model of the environment for at least one planar structure associated with at least the portion of the one or more potential objects of interest within the environment.

22. The computing device of claim 13, wherein the at least partial 3D geometric model of the environment is built using a structure-from-motion system.

23. The computing device of claim 13, wherein the processing circuit is further configured to execute instructions to search the at least partial 3D geometric model of the environment for at least two planar structures associated with at least the portion of the one or more potential objects of interest within the environment.

24. The computing device of claim 13, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

25. The computing device of claim 13, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

26. A computer program product executed at a computing device comprising:

a non-transitory computer-readable medium comprising code for;

storing one or more images with the computing device, the one or more images including an environment and at least a portion of one or more potential objects of interest within the environment;

building at least a partial three dimensional (3D) geometric model with the computing device of the environment included in the one or more images, the at least partial 3D geometric model including at least the portion of the one or more potential objects of interest within the environment, wherein the at least partial 3D geometric model is built using at least one image of the one or more images; and

automatically creating, using the at least partial 3D geometric model, at least one rectified image having at least one potential object of interest for a visual search, the at least one rectified image including a modified perspective view of the at least one potential object of interest relative to a view of the at least one potential object of interest in the one or more images.

27. The computer program product of claim 26, further comprising code for automatically uploading the at least one rectified image to a server for the visual search.

28. The computer program product of claim 26, further comprising code for uploading the at least one rectified image to a server for the visual search.

29. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

30. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device during a background operation without receiving a user input.

31. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device during a time period with the image capture device moving for at least a portion of the time period.

32. The computer program product of claim 26, wherein the one or more images are captured by an image capture device that is associated with the computing device, wherein the image capture device is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the image capture device is not required to move.

33. The computer program product of claim 26, further comprising code for searching the at least partial 3D geometric model of the environment for at least one planar structure associated with at least the portion of the one or more potential objects of interest within the environment.

34. The computer program product of claim 26, wherein the at least partial 3D geometric model of the environment is built using a structure-from-motion system.

35. The computer program product of claim 26, further comprising code for searching the at least partial 3D geometric model of the environment for at least two planar structures associated with at least the portion of the one or more potential objects of interest within the environment.

36. The computer program product of claim 26, wherein automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

37. The computer program product of claim 26, wherein automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

38. An apparatus comprising:

means for storing one or more received images, the one or more received images including an environment and at least a portion of one or more potential objects of interest within the environment;

means for building at least a partial three dimensional (3D) geometric model of the environment included in the one or more images, the at least partial 3D geometric model including at least the portion of the one or more potential objects of interest within the environment, wherein the at least partial 3D geometric model is built using at least one image of the one or more stored images; and

means for automatically creating, using the at least partial 3D geometric model, at least one rectified image having at least one potential object of interest for a visual search, the at least one rectified image including a modified perspective view of the at least one potential object of interest relative to an initial view of the at least one potential object of interest.

39. The apparatus of claim 38, further comprising:

means for automatically uploading the at least one rectified image to a server for the visual search.

40. The apparatus of claim 38, further comprising:

means for uploading the at least one rectified image to a server for the visual search.

41. The apparatus of claim 38, further comprising:

means for capturing the one or more images, wherein the means for storing to receive the one or more images from the means for capturing.

42. The apparatus of claim 41, wherein the means for automatically creating at least one rectified image having at least one potential object of interest for the visual search occurs without receiving a user input.

43. The apparatus of claim 41, wherein the one or more images are captured automatically during a background operation without receiving a user input.

44. The apparatus of claim 41, wherein the one or more images are captured during a time period in which the means for capturing the one or more images is moving for at least a portion of the time period.

45. The apparatus of claim 41, wherein the means for capturing is capable of providing depth information for the at least one potential object of interest and to capture the one or more images while the means for capturing is not required to move.

46. The apparatus of claim 38, further comprising:

means for searching the at least partial 3D geometric model of the environment for at least one planar structure associated with at least the portion of the one or more potential objects of interest within the environment.

47. The apparatus of claim 38, further comprising:

means for searching the at least partial 3D geometric model of the environment for at least two planar structures associated with at least the portion of the one or more potential objects of interest within the environment.

48. The apparatus of claim 38, wherein the means for automatically creating at least one rectified image having at least one potential object of interest for the visual search comprises unwarping one planar structure into a frontal view for each rectified image.

49. The apparatus of claim 38, wherein the means for automatically creating at least one rectified image having at least one potential object of interest comprises rectifying portions of the at least one potential object of interest from multiple input images to create the at least one rectified image.

50. The apparatus of claim 38, wherein the at least partial 3D geometric model of the environment is built using a structure-from-motion system.