Image-based item location identification

• US 9,665,960 B1
• Filed: 12/22/2014
• Issued: 05/30/2017
• Est. Priority Date: 12/22/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Masters, Nathan Eugene, Hasan, Shiblee Imtiaz, Johnson, Joseph Edwin
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US14/579,522
• Time in Patent Office: 890 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Systems and methods are provided for augmenting or annotating image data to help a user locate an item among a number of items in a physical environment. The systems disclosed herein can access an image of a location that includes a number of items. The systems may then identify the items and the location of the items within the image. Further, the systems may create an interactive alternative visualization of the identified items to facilitate a user locating a particular item. For example, the alternative visualization may include presenting an ordered set of thumbnails corresponding to the items. The user can select the desired item using the alternative visualization and the systems can annotate a location within the image where the item can be located, thereby enabling the user to locate the item in the physical environment.

Claims

1. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific computer-executable instructions,

accessing a first image that depicts at least a plurality of items located within a physical area, wherein the first image is one of a continuous stream of images that are received over a time period from a user computing device comprising an image capturing device configured to capture the continuous stream of images;

analyzing the first image to identify the plurality of items depicted in the first image;

determining a visual organization of the plurality of items based at least in part on the identification of the plurality of items;

generating a visual representation of the plurality of items based at least in part on the determined visual organization of the plurality of items, wherein the visual representation of the plurality of items differs from a depiction of the plurality of items within the first image and differs from a presentation of the items within the physical area, the visual representation of the plurality of items comprising selectable elements that enable a user to select a desired item to be located within the first image, wherein the visual representation of the plurality of items comprises a scrollable user interface element configured to present a first subset of the plurality of items, and wherein the scrollable user interface element is further configured to present a second subset of the plurality of items in response to interaction with the scrollable user interface by the user;

providing the visual representation of the plurality of items to the user computing device for output on a display of the user computing device;

receiving a selection of an item from the plurality of items included in the visual representation, wherein the selection occurs via interaction of the user with the visual representation of the plurality of items;

determining a location of the item in the first image;

applying an annotation to the first image corresponding to the location of the item in the first image to obtain an annotated image that includes the annotation, wherein the annotation visually distinguishes a portion of the annotated image depicting the item;

providing the annotated image to the user computing device for output on the display of the user computing device, thereby providing the user with information to locate the item in the physical area; and

repeatedly updating the annotated image based at least in part on changes detected between the first image and subsequent images of the continuous stream of images, wherein the repeatedly updating comprises maintaining a position of the annotation relative to the location of the item as the location of the item changes in the continuous stream of images.

2. The computer-implemented method of claim 1, wherein the visual representation comprises a user interface element configured to receive the selection of the item included in the visual representation.

3. The computer-implemented method of claim 1, wherein the computer-implemented method further comprises, for each item of the plurality of items:

determining a location of the item within the first image; and

storing the location of the item within the first image at a data repository,

wherein determining the location of the item of the selection comprises accessing the data repository to determine the location of the item of the selection.

4. The computer-implemented method of claim 1, wherein determining the location of the item within the first image comprises identifying a relative location of the item with respect to a reference item included in the first image.

5. The computer-implemented method of claim 4, wherein causing output of the annotated image comprises causing output of directional information for the item based at least in part on the relative location of the item with respect to the reference item.

6. The computer-implemented method of claim 1, wherein the visual representation of the plurality of items comprises an alternative view of a given item from the plurality of items, the alternative view comprising a depiction of the given item that differs from a depiction of the given item in the first image.

7. The computer-implemented method of claim 1, wherein the annotation comprises at least one of the following:

a static graphical annotation or an animation.

8. A system comprising:

an electronic data store configured to at least store specific computer-executable instructions; and

an interactive computing system comprising computer hardware in communication with the electronic data store, the interactive computing system configured to execute the specific computer-executable instructions to at least;

access an image, from a stream of images, depicting at least a plurality of items, the stream of images received from an image capturing device;

generate an alternative visual representation of the plurality of items, wherein the alternative visual representation of the plurality of items includes one or more alternative depictions of individual items of the plurality of items that differ from a presentation of the items within a physical area, the alternative visual representation enabling a user to select a desired item to be annotated within the first image, wherein the alternative visual representation of the plurality of items comprises a selectable user interface element configured to present a first subset of the plurality of items, and wherein the selectable user interface element is further configured to present a second subset of the plurality of items in response to an interaction with the selectable user interface;

receive a selection of an item from the plurality of items, wherein the selection occurs with respect to the alternative visual representation of the plurality of items;

determine a location of the item in the image;

modify the image by adding an annotation based at least in part on the location of the item in the image to obtain an annotated image; and

cause presentation of a modified stream of images, wherein causing presentation of the modified stream of images comprises repeatedly updating a displayed annotation position with respect to the location of the item in subsequent images of the stream of images as changes in the location of the item are detected in the subsequent images.

9. The system of claim 8, wherein the alternative visual representation of the plurality of items depicts a given item from the plurality of items differently than depicted in the image.

10. The system of claim 8, wherein the interactive computing system is further configured to execute the specific computer-executable instructions at least:

identify complementary content associated with the item; and

output the complementary content for presentation in response to receiving the selection of the item.

11. The system of claim 10, wherein the complementary content comprises a preview of content included in the item.

12. The system of claim 8, wherein the alternative visual representation of the plurality of items comprises an ordered set of thumbnail images representative of the plurality of items.

13. The system of claim 8, wherein the interactive computing system is further configured to execute the specific computer-executable instructions to at least:

identify an item type for each item of at least some items in the plurality of items; and

generate the alternative visual representation of the plurality of items based at least in part on the item type for the at least some items.

14. The system of claim 8, wherein the interactive computing system is further configured to execute the specific computer-executable instructions to at least animate a portion of the annotated image depicting the item.

15. The system of claim 8, wherein the interactive computing system is further configured to execute the specific computer-executable instructions to at least determine the location of the item relative to a reference item in the image.

16. A computer-readable, non-transitory storage medium storing computer executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

accessing a stream of images depicting a plurality of items received from an image capturing device configured to capture the stream of images;

generating an interactive element that includes a representation of an item from the plurality of items, the representation including an alternative depiction of the item that differs from a presentation of the item in a physical area and differs from a depiction of the item within the stream of images, wherein the interactive element is configured to present a first subset of the plurality of items, and wherein the interactive element is further configured to present a second subset of the plurality of items in response to interaction with the scrollable user interface;

causing output of the interactive element;

receiving a selection of the item from the plurality of items based at least in part on an interaction with the interactive element; and

causing output of a presentation of an annotated stream of images comprising at least a portion of image data from the accessed stream of images and an annotation identifying the item within the annotated stream of images, wherein a location of the annotation in the presentation is repeatedly updated based at least in part on a position of the item that differs between two or more images of the stream of images.

17. The computer-readable, non-transitory storage medium of claim 16, wherein the interactive element comprises at least one of:

an audio representation, a text representation, a video representation, an animated representation, or an image representation.

18. The computer-readable, non-transitory storage medium of claim 16, wherein the operations further comprise:

determining an item type for the item; and

aggregating the item with additional items of the same item type from the plurality of items,

wherein the generating the interactive element comprises grouping the item with the additional items as an item group.

19. The computer-readable, non-transitory storage medium of claim 18, wherein the operations further comprise causing output of a representation of items in the item group separately in response to a selection of the item group.

20. The computer-readable, non-transitory storage medium of claim 16, wherein the operations further comprise causing output of a recommendation of another item based at least in part on the item.