Method and apparatus for enabling virtual tags

• US 8,661,053 B2
• Filed: 11/12/2012
• Issued: 02/25/2014
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Flynn, John, Anguelov, Dragomir, Neven, Hartmut, Cummins, Mark, Philbin, James, Spring, Rafael, Adam, Hartwig, Pillai, Anand
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US13/674,483
• Publication Number:
  US 20130066878A1
• Time in Patent Office: 470 Days
• Field of Search: 707/769, 707/809, 707/915, 707/999.201, 709/203, 705/301, 455/557
• US Class Current: 707/769
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location

Abstract

A method and apparatus for enabling virtual tags is described. The method may include receiving a first digital image data and virtual tag data to be associated with a real-world object in the first digital image data, wherein the first digital image data is captured by a first mobile device, and the virtual tag data includes metadata received from a user of the first mobile device. The method may also include generating a first digital signature from the first digital image data that describes the real-world object, and in response to the generation, inserting in substantially real-time the first digital signature into a searchable index of digital images. The method may also include storing, in a tag database, the virtual tag data and an association between the virtual tag data and the first digital signature inserted into the index of digital images.

Claims

1. A computer-implemented method comprising:

receiving, at a server computer system, a first digital image data and virtual tag data to be associated with a real-world object in the first digital image data, wherein the first digital image data is captured by a first mobile device, and the virtual tag data includes metadata received from a user of the first mobile device;

generating a first digital signature from the first digital image data that describes the real-world object;

in response to the generation, inserting in substantially real-time the first digital signature into a searchable index of digital images;

storing, in a tag database, the virtual tag data and an association between the virtual tag data and the first digital signature inserted into the index of digital images;

receiving a second signature generated from a second digital image data captured by a second mobile device, the second digital image data includes one or more real-world objects; and

transmitting the virtual tag data, received from the user of the first mobile device and associated with the real-world object in the first digital image data, to the second mobile device when a search of the index of digital images based on the second digital signature results in a match with the first digital signature.

2. The computer-implemented method of claim 1, wherein the virtual tag data received from the user of the first mobile device includes a virtual sticker selected by the user from among a plurality of different virtual stickers, each virtual sticker to convey different information, and the selected virtual sticker to be rendered over digital image data of the real-world object.

3. The computer-implemented method of claim 1, wherein generating the first digital signature and inserting in substantially real-time the first digital signature further comprises:

generating a set of feature vectors from the first digital image data, each feature vector describing a neighborhood of pixels that surround an interest point for the real-world object in the first digital image data; and

inserting in real-time the set of feature vectors into a searchable k-dimensional tree structure of feature vectors for a plurality of real-world objects as the set of feature vectors for the first digital image data are extracted.

4. The method of claim 1, further comprising:

in response to storage of the virtual tag data in the tag database, transmitting a confirmation to the first mobile device that images of the real-world object are searchable, the confirmation further indicating storage of the virtual tag data was successful.

5. The method of claim 1, further comprising:

in response to a determination that the second digital signature matches the first digital signature, querying a tag database to locate additional virtual tag data associated with the real-world object;

obtaining one or more additional tags associated with the real-world object, each of the one or more additional tags includes user-created content; and

transmitting the additional virtual tag data to the second mobile device.

6. The method of claim 5, further comprising:

filtering the obtained one or more additional tags prior to transmission of the additional virtual tag data to limit a number of virtual tags that are returned to the mobile device.

7. The method of claim 5, wherein the filtering is performed to limit the virtual tags that are returned to the mobile device to virtual tags created by social networking friends of the user of the mobile device.

8. The method of claim 1, wherein the virtual tag data includes a collection of one or more photographs that are relevant to the real-world object and supplied by the user of the first mobile device.

9. The method of claim 1, wherein the real-world object is a consumer product and the virtual tag data includes a coupon that is redeemable when purchasing the consumer product.

10. The method of claim 1, wherein the first mobile device is a cellular telephone and the second mobile device is a user-wearable computing device in the form of eyeglasses.

11. The method of claim 1, wherein the second digital image data is digital video data captured by the second mobile device and the real-world object is contained within the digital video data.

12. The method of claim 11, wherein the digital video data is a live video feed being captured by the second mobile device, and wherein the virtual tag data is transmitted to the second mobile device to enable the second mobile device to render the virtual tag data over the live video feed.

13. The method of claim 1, further comprising:

receiving global positioning system (GPS) data from the first mobile device;

storing an association between the GPS data and the virtual tag data in the tag database that indicates a real-world location where the virtual tag is located; and

sending a notification to the second mobile device that one or more virtual tags are nearby when the second mobile device is within a predetermined distance of the GPS data associated with the virtual tag data.

14. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform a method comprising:

receiving, at a server computer system, a first digital image data and virtual tag data to be associated with a real-world object in the first digital image data, wherein the first digital image data is captured by a first mobile device, and the virtual tag data includes metadata received from a user of the first mobile device;

generating a first digital signature from the first digital image data that describes the real-world object;

in response to the generation, inserting in substantially real-time the first digital signature into a searchable index of digital images;

storing, in a tag database, the virtual tag data and an association between the virtual tag data and the first digital signature inserted into the index of digital images;

receiving a second signature generated from a second digital image data captured by a second mobile device, the second digital image data includes one or more real-world objects; and

transmitting the virtual tag data, received from the user of the first mobile device and associated with the real-world object in the first digital image data, to the second mobile device when a search of the index of digital images based on the second digital signature results in a match with the first digital signature.

15. The computer readable storage medium of claim 14, wherein the virtual tag data received from the user of the first mobile device includes a virtual sticker selected by the user from among a plurality of different virtual stickers, each virtual sticker to convey different information, and the selected virtual sticker to be rendered over digital image data of the real-world object.

16. The computer readable storage medium of claim 14, wherein generating the first digital signature and inserting in substantially real-time the first digital signature further comprises:

generating a set of feature vectors from the first digital image data, each feature vector describing a neighborhood of pixels that surround an interest point for the real-world object in the first digital image data; and

inserting in real-time the set of feature vectors into a searchable k-dimensional tree structure of feature vectors for a plurality of real-world objects as the set of feature vectors for the first digital image data are extracted.

17. The computer readable storage medium of claim 14, in response to a determination that the second digital signature matches the first digital signature, querying a tag database to locate additional virtual tag data associated with the real-world object;

obtaining one or more additional tags associated with the real-world object, each of the one or more additional tags includes user-created content; and

transmitting the additional virtual tag data to the second mobile device.

18. The computer readable storage medium of claim 14, wherein the first mobile device is a cellular telephone and the second mobile device is a user-wearable computing device in the form of eyeglasses.

19. A system comprising:

a memory; and

a processor coupled with the memory to

receive a first digital image data and virtual tag data to be associated with a real-world object in the first digital image data, wherein the first digital image data is captured by a first mobile device, and the virtual tag data includes metadata received from a user of the first mobile device,

generate a first digital signature from the first digital image data that describes the real-world object,

in response to the generation, insert in substantially real-time the first digital signature into a searchable index of digital images,

store, in a tag database, the virtual tag data and an association between the virtual tag data and the first digital signature inserted into the index of digital images,

receive a second signature generated from a second digital image data captured by a second mobile device, the second digital image data includes one or more real-world objects, and

transmit the virtual tag data, received from the user of the first mobile device and associated with the real-world object in the first digital image data, to the second mobile device when a search of the index of digital images based on the second digital signature results in a match with the first digital signature.

20. The system of claim 19, wherein the virtual tag data received from the user of the first mobile device includes a virtual sticker selected by the user from among a plurality of different virtual stickers, each virtual sticker to convey different information, and the selected virtual sticker to be rendered over digital image data of the real-world object.