Image-based item location identification

• US 10,083,357 B2
• Filed: 05/01/2017
• Issued: 09/25/2018
• Est. Priority Date: 12/22/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Masters, Nathan Eugene, Hasan, Shiblee Imtiaz, Johnson, Joseph Edwin
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US15/583,744
• Publication Number:
  US 20180157913A1
• Time in Patent Office: 512 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Systems and methods are provided for augmenting or annotating image data to help a user locate an item among a number of items in a physical environment. The systems disclosed herein can access an image of a location that includes a number of items. The systems may then identify the items and the location of the items within the image. Further, the systems may create an interactive alternative visualization of the identified items to facilitate a user locating a particular item. For example, the alternative visualization may include presenting an ordered set of thumbnails corresponding to the items. The user can select the desired item using the alternative visualization and the systems can annotate a location within the image where the item can be located, thereby enabling the user to locate the item in the physical environment.

Claims

1. A computer-implemented method comprising:

as implemented by a user computing device comprising one or more hardware processors and configured with specific computer-executable instructions,

obtaining a continuous stream of images that depict at least a plurality of items located within a physical area, wherein the continuous stream of images are obtained from an image capturing device configured to capture the continuous stream of images;

identifying the plurality of items depicted in the continuous stream of images;

generating an interactive user interface element comprising a representation of the plurality of items that differs from a presentation of the plurality of items within the continuous stream of images and differs from a presentation of the plurality of items within the physical area;

outputting for display to a user the continuous stream of images and the interactive user interface element;

receiving, in response to interaction by a user with a portion of the interactive user interface element, a selection of a representation of an item from the plurality of items included in the representation of the plurality of items; and

for at least a subset of images of the continuous stream of images output for display, animating at least a portion of each of the subset of images, the portion including a depiction of an item corresponding to the representation of the item selected from the interactive user interface element.

2. The computer-implemented method of claim 1, wherein said outputting for display comprises outputting the continuous stream of images and the interactive user interface element on a display of a device that includes the image capturing device.

3. The computer-implemented method of claim 2, wherein the user computing device includes one of or both of the display and the image capturing device.

4. The computer-implemented method of claim 1, wherein, in response to receiving the selection of the representation of the item, the method further comprises:

identifying complementary content associated with the item; and

outputting the complementary content on a display.

5. The computer-implemented method of claim 1, wherein said animating comprises moving the depiction of the item within the subset of images.

6. The computer-implemented method of claim 1, wherein said animating comprises presenting different depictions of the item within different images of the subset of images.

7. The computer-implemented method of claim 1, wherein said animating comprises animating different portions of each of the subset of images to reflect changes in a relative position of the depiction of the item within the continuous stream of images as the image capturing device moves.

8. The computer-implemented method of claim 1, further comprising annotating the depiction of the item within the continuous stream of images output for display.

9. A system comprising:

an electronic data store configured to at least store specific computer-executable instructions;

an image capturing device configured to capture a continuous stream of images; and

a computing device comprising computer hardware in communication with the electronic data store and the image capturing device, the computing device configured to execute the specific computer-executable instructions to at least;

obtain the continuous stream of images from the image capturing device, the continuous stream of images depicting at least a plurality of items located within a physical area;

identify the plurality of items depicted in the continuous stream of images;

generate an interactive user interface element comprising a representation of the plurality of items that differs from a presentation of the plurality of items within the continuous stream of images and differs from a presentation of the plurality of items within the physical area;

output for display the continuous stream of images and the interactive user interface element;

receive, in response to interaction with the interactive user interface element, a selection of a representation of an item from the plurality of items included in the representation of the plurality of items; and

for at least a subset of images of the continuous stream of images output for display, animate at least a portion of each of the subset of images, the portion including a depiction of an item corresponding to the representation of the item selected from the interactive user interface element.

10. The system of claim 9, wherein the computing device comprises one or both of the image capturing device and the electronic data store.

11. The system of claim 9, wherein the computing device is further configured to output the continuous stream of images and the interactive user interface element on a display of the computing device.

12. The system of claim 9, wherein, in response to receiving the selection of the representation of the item, the computing device is further configured to execute the specific computer-executable instructions to at least:

identify complementary content associated with the item; and

output the complementary content on a display.

13. The system of claim 9, wherein the computing device is further configured to animate the at least a portion of each of the subset of images by at least moving a location of the depiction of the item within at least some of the subset of images.

14. The system of claim 9, wherein the computing device is further configured to animate the at least a portion of each of the subset of images by at least presenting different depictions of the item within different images of the subset of images.

15. The system of claim 9, wherein the computing device is further configured to animate the at least a portion of each of the subset of images by at least animating different portions of each of the subset of images to reflect changes in a relative position of the depiction of the item within the continuous stream of images as the image capturing device moves.

16. The system of claim 9, wherein the computing device is further configured to execute the specific computer-executable instructions to at least annotate the depiction of the item within the continuous stream of images output for display.

17. A computer-readable, non-transitory storage medium storing computer executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

receiving a continuous stream of images from an image capturing device, the continuous stream of images depicting at least a plurality of items located within a physical area;

identifying the plurality of items depicted in the continuous stream of images;

generating an interactive user interface element comprising a representation of the plurality of items that differs from a presentation of the plurality of items within the continuous stream of images;

outputting for display the continuous stream of images and the interactive user interface element;

receiving, in response to interaction with the interactive user interface element, a selection of a representation of an item from the plurality of items included in the representation of the plurality of items; and

for at least a subset of images of the continuous stream of images output for display, animating at least a portion of each of the subset of images, the portion including a depiction of an item corresponding to the representation of the item selected from the interactive user interface element.

18. The computer-readable, non-transitory storage medium of claim 17, wherein the operations further comprise:

identifying complementary content associated with the item; and

outputting the complementary content on a display.

19. The computer-readable, non-transitory storage medium of claim 17, wherein said animating comprises moving a location of the depiction of the item within at least some of the subset of images.

20. The computer-readable, non-transitory storage medium of claim 17, wherein the continuous stream of images depicts at least one additional item other than the plurality of items.