AUGMENTING A LIVE VIEW

• US 20120299961A1
• Filed: 05/27/2011
• Published: 11/29/2012
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Ramkumar, Gurumurthy D., Stasior, William F., Feldman, Bryan E., Dhua, Arnab S., Senthamil, Nalin Pradeep
• Granted Patent: US 9,547,938 B2
• Time in Patent Office: Days
• US Class Current: 345/632
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/12 Bounding box

Abstract

Techniques for augmenting an image of an object captured and displayed in real time with associated content are disclosed. In one embodiment, the method for augmenting the image includes receiving information defining a sampled frame of a video being captured by an electronic device in substantially real time, determining information representative of an object captured in the sampled frame based on the received information, causing the determined information to match stored information defining a plurality of items to locate an item matched to the captured object, retrieving content associated with the matched item, and providing the retrieved content for display with the captured image on the electronic device. The retrieved content may be rendered in an overlay element that overlays the captured image displayed on the electronic device. The rendered content is configured to enable a user to interact with the content.

Claims

1. A computer-implemented method for augmenting an image or a stream of images of an object captured and displayed in substantially real time with associated content, comprising:

under the control of one or more computer systems configured with executable instructions,

capturing at least one image in a stream of images of an object using an image capturing element of an electronic device operated by a user;

displaying the captured at least one image on a display element of the electronic device;

determining one or more features defining the object, the one or more features being derived from the at least one image of the object;

matching the determined one or more features defining the object to features defining a plurality of items stored in an item repository;

when a match is found, retrieving content associated with the matched item from the item repository, the content including content elements selectable by the user; and

displaying, in substantially real time, the retrieved content in an overlay element on the display element of the electronic device such that the overlay element augments the image displayed on the display element of the electronic device,

wherein the user is able to interact with the retrieved content displayed in the overlay element by accessing the content elements selectable by the user.

2. The computer-implemented method of claim 1, wherein the overlay element augments the image by overlaying the image, wherein the overlay element is selected from one of:

a box, a button, a three-dimensional (3D) structure, an animation, audio, video, Web page, or interactive user interface.

3. The computer-implemented method of claim 1, wherein image capturing element of the electronic device is selected from one of a video camera or a photo camera.

4. A computer-implemented method for augmenting an image of an object captured and displayed in substantially real time with associated content, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving information defining a sampled frame of a video being captured by an electronic device in substantially real time;

determining information representative of an object captured in the sampled frame based at least in part on the received information;

causing the determined information to match stored information defining a plurality of items to locate an item matched to the captured object;

retrieving content associated with the matched item; and

providing the retrieved content for display with the captured image on the electronic device,

wherein the retrieved content is rendered in an overlay element that overlays the captured image displayed on the electronic device.

5. The computer-implemented method of claim 4, wherein the content includes content elements selectable by the user, the content elements including at least one of:

a button or a hyperlink.

6. The computer-implemented method of claim 4, wherein the content includes information associated with the item comprising at least one of an item price, description of the item, customer rating of the item, customer review of the item, special offers and promotions related to the item, information related to products related to the item, or at least one content element selectable by the user that enables the user to purchase the item.

7. The computer-implemented method of claim 4, wherein the determined information is matched to stored information at the electronic device and wherein information defining a plurality of items is stored in a local repository associated with the electronic device.

8. The computer-implemented method of claim 4, wherein the determined information is matched to stored information remote from the electronic device and wherein information defining a plurality of items is stored in a remote repository.

9. The computer-implemented method of claim 4, further comprising:

determining a number of matched items to each object when multiple matches are found between one or more object and stored items.

10. The computer-implemented method of claim 4, further comprising:

tagging the matched item with at least one of a current location identifier or a timestamp.

25. The computer-implemented method of claim 10, further comprising:

querying the tagged item to determine a current location of the item.

11. A computer-implemented method for augmenting an image of an object captured and displayed in real time with associated content, comprising:

under the control of one or more computer systems configured with executable instructions,

capturing at least one image of an object with an electronic device, the captured image being displayed by the electronic device;

determining information representative of the object based at least in part on the at least one captured image of the object;

matching the determined information to stored information defining each item of a plurality of items;

retrieving content associated with the matched item from the repository;

overlaying the captured image with the retrieved content rendered in an overlay element, the retrieved content including at least one selectable content element; and

in response to a user selecting the at least one selectable content element, overlaying the captured image with content associated with the content element.

12. The computer-implemented method of claim 11, further comprising:

enabling the user to enter a second overlay element and to tag the object with additional content pertaining to the object within the second overlay element.

13. The computer-implemented method of claim 11, further comprising:

in response to the user entering a search query in the overlay element, retrieving search results corresponding to the search query.

14. The computer-implemented method of claim 11, wherein the overlay element comprises a box.

15. A computer system for augmenting an image of an object captured and displayed in real time with associated content, comprising:

a processor; and

a memory having computer-executable instructions that, when executed on the processor, cause the processor to;

receive information defining a sampled frame of a video being captured by an electronic device in substantially real time;

determine information representative of an object captured in the sampled frame based at least in part on the received information;

cause the determined information to match stored information defining a plurality of items to locate an item matched to the captured object;

retrieve content associated with the matched item;

provide the retrieved content for display with the captured image on the electronic device; and

cause the retrieved content to be rendered in an overlay element that overlays the captured image displayed on the electronic device.

16. The computer system of claim 15, wherein the content includes content elements that are selectable by the user, the content elements including at least one of:

a button or a hyperlink.

17. The computer system of claim 15, wherein the content includes information associated with the item comprising at least one of an item price, description of the item, customer rating of the item, customer review of the item, special offers and promotions related to the item, information related to products related to the item, or at least one content element selectable by the user that enables the user to purchase the item.

18. The computer system of claim 15, wherein the computer-executable instructions further cause the processor to tag the matched object with at least one of a current location identifier or a timestamp.

19. The computer system of claim 16, wherein the computer-executable instructions further cause the processor to, in response to the user selecting a content element, retrieve and provide information associated with the content element, the information overlaying the captured image.

20. A computer-readable storage medium having computer-executable instructions for augmenting an image of an object captured and displayed in real time with associated content that, when executed by a computer, cause the computer to:

receive information defining a sampled frame of a video being captured by an electronic device in substantially real time;

determine information representative of an object captured in the sampled frame based at least in part on the received information;

cause the determined information to match stored information defining a plurality of items to locate an item matched to the captured object;

retrieve content associated with the matched item;

provide the retrieved content for display with the captured image on the electronic device; and

cause the retrieved content to be rendered in an overlay element that overlays the captured image displayed on the electronic device.

21. The computer-readable storage medium of claim 20, wherein the content includes content elements that are selectable by the user, the content elements including at least one of:

a button or a hyperlink.

22. The computer-readable storage medium of claim 20, wherein the content includes information associated with the item comprising at least one of an item price, description of the item, customer rating of the item, customer review of the item, special offers and promotions related to the item, information related to products related to the item, or at least one content element selectable by the user that enables the user to purchase the item.

23. The computer-readable storage medium of claim 20, wherein the overlay element is a box.

24. The computer-readable storage medium of claim 21, wherein the computer-executable instructions further cause the computer to, in response to the user selecting a content element, retrieve and provide information associated with the content element, the information overlaying the captured image.