Augmented reality enhanced interaction system

• US 10,282,696 B1
• Filed: 06/06/2014
• Issued: 05/07/2019
• Est. Priority Date: 06/06/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bettis, David William, McNamara, Alexander Michael, Hollis, Benjamin Ralph, Pageau, Felix Joseph Etienne, Boyapati, Sridhar, Smith, Korwin Jon, Jones, Jack Bradley
• Primary Examiner(s): Iwarere, Oluseye
• Application Number:
  US14/298,761
• Time in Patent Office: 1,796 Days
• Field of Search: 705 28, 705 23, 345 8, 345633, 709206, 342147
• CPC Class Codes: G06Q 10/08355 Routing methods
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

Described are systems and techniques configured to present information to a user of a materials handling facility storing items in inventory locations. A device such as a head-mounted display or a tablet computer is configured to provide an augmented reality user interface in which computer-generated data overlays objects in the real-world. The user interface may present information associated with assisting other users of the facility, operation of the facility, and so forth.

Claims

1. A system comprising:

a wearable computing device comprising a head-mounted display device configured to present visual information to a first user in a materials handling facility (facility);

a plurality of cameras external to the wearable computing device and configured to acquire images within the facility;

a wireless interface configured to communicate with the wearable computing device;

a memory, storing computer-executable instructions; and

a hardware processor in communication with the plurality of cameras, the wireless interface, and the memory, wherein the hardware processor is configured to execute the computer-executable instructions to;

access operational data stored in the memory that is associated with a second user in the facility, wherein the operational data is determined using image data acquired from the images, wherein the operational data comprises information related to a request for assistance submitted by the second user and relationship data indicative of a relationship between the second user and a third user existing prior to the second user submitting the request for assistance;

generate, using the hardware processor, user interface data configured to present a user interface using the head-mounted display device, wherein the user interface data is indicative of the operational data associated with the second user; and

send, using the wireless interface, the user interface data to the wearable computing device, wherein the user interface based on the user interface data is presented to the first user to indicate the request for assistance by the second user and the relationship between the second user and the third user.

2. The system of claim 1, wherein the hardware processor is further configured to execute the computer-executable instructions to:

detect the second user in one or more of the images;

analyze the one or more of the images to determine an identity of the second user using one or more of;

facial recognition comprising;

detection of a face of the second user in the one or more images,

calculation of one or more eigenvectors of the face of the second user, and

comparison of the one or more eigenvectors with previously stored data;

clothing recognition comprising;

detection of one or more articles of clothing of the second user in the one or more images,

classification of the one or more articles of clothing of the second user, and

comparison of the classification of the one or more articles of clothing of the second user with the previously stored data;

or

gait recognition comprising;

determination of one or more limbs of the second user in the one or more images,

determination of a change in position over time of the one or more limbs of the second user, and

comparison of the change in position over time with the previously stored data;

retrieve the operational data using the identity of the second user; and

present, in the user interface, a portion of the one or more of the images depicting the face of the second user.

3. The system of claim 1, further comprising:

a radio frequency identification (RFID) reader in communication with the hardware processor, the RFID reader configured to retrieve a value of a radio frequency (RF) tag;

a weight sensor in communication with the hardware processor, the weight sensor configured to determine a weight of a tote; and

wherein the hardware processor is further configured to execute the computer-executable instructions to;

receive RFID tag data indicative of a RFID tag associated with the second user;

determine an identity of the second user using the RFID tag data;

based at least in part on the weight of the tote, determine one or more items in the tote; and

wherein at least a portion of the operational data associated with the second user in the facility is generated using the identity of the second user and the one or more items in the tote.

4. The system of claim 1, wherein the hardware processor is further configured to execute the computer-executable instructions to:

access a first set of operational data when the first user and the second user are at a first distance from one another;

access a second set of operational data when the first user and the second user are at a second distance from one another, further wherein the second distance is less than the first distance and the second set of operational data comprises more information than the first set of operational data; and

wherein the user interface data changes from a first time using the first set of operational data to a second time using the second set of operational data.

10. The system of claim 1, wherein the information related to the request for assistance submitted by the second user comprises information regarding one or more items selected for purchase by at least one of the second user or the third user.

11. The system of claim 1, wherein the hardware processor is further configured to execute the computer-executable instructions to:

retrieve the relationship data indicative of the relationship between the second user and the third user; and

wherein the relationship data is used to prevent the second user and the third user from crosschecking a pick of an item.

12. The system of claim 1, wherein the hardware processor is further configured to execute the computer-executable instructions to:

retrieve access permission data associated with the first user, wherein the access permission data specifies one or more of information accessible to or inaccessible to the first user; and

generate the operational data based on the access permission data of the first user to include information accessible to the first user in the user interface data and omit from the user interface data information inaccessible to the first user.

13. The system of claim 1, wherein the hardware processor is further configured to execute the computer-executable instructions to:

detect the second user in one or more of the images acquired within the facility; and

determine an identity of the second user using at least one or more of;

facial recognition,

clothing recognition, or

gait recognition; and

wherein the operational data includes the identity of the second user.

14. The system of claim 1, wherein the operational data is indicative of one or more of:

a distance between the first user and the second user,

a pick status of an order being picked by the second user,

a total value or quantity of items picked by the second user,

an elapsed time since initiation of the request for assistance by the second user, or

deviation from a navigation path of the second user through the facility.

5. A system comprising:

a wearable computing device configured to present visual information to a first user;

a plurality of cameras external to the wearable computing device configured to acquire images within a materials handling facility (facility);

a communication interface configured to communicate with the wearable computing device;

a memory, storing computer-executable instructions; and

a hardware processor in communication with the plurality of cameras, the communication interface, and the memory, wherein the hardware processor is configured to execute the computer-executable instructions to;

access, from the memory, the images;

based at least in part on the images, determine operational data of the facility using the hardware processor, wherein the operational data comprises information related to a request for assistance submitted by a second user and relationship data indicative of a pre-existing relationship between the second user and a third user;

generate, using the hardware processor, user interface data comprising one or more instructions to present a user interface on the wearable computing device at the facility, wherein the user interface presents information based at least in part on the operational data of the facility as an overlay on real-world objects; and

send the user interface data to the wearable computing device using the communication interface, wherein the user interface based on the user interface data is presented to the first user to indicate the request for assistance by the second user and the pre-existing relationship between the second user and the third user.

6. The system of claim 5, the hardware processor further configured to execute the computer-executable instructions to:

access user data associated with one or more users of the facility; and

wherein the user interface presents information based at least in part on the user data.

7. The system of claim 5, wherein the operational data comprises at least one or more of:

a distance between a location of the first user and a location of the second user,

the request for assistance associated with the second user,

an inventory location of an inventory item, or

an inventory level of an item.

8. The system of claim 5, wherein the wearable computing device comprises:

a second communication interface configured to communicate with the communication interface;

a head-mounted display device;

a second memory, storing second computer-executable instructions; and

a second hardware processor in communication with the second communication interface, the head-mounted display device, and the second memory, wherein the second hardware processor is configured to execute the second computer-executable instructions to;

receive the user interface data using the second communication interface; and

present, based at least in part on the user interface data, the user interface using the head-mounted display device, wherein the user interface comprises one or more of graphical representations or text presented as overlaying the real-world objects within a field of view.

9. The system of claim 5, further comprising:

a second plurality of sensors configured to acquire sensor data associated with operation of the facility; and

wherein the determination of the operational data is based at least in part on the sensor data.

15. A method comprising:

presenting visual information to a first user using a wearable computing device at a materials handling facility (facility);

acquiring, at the facility, sensor data from a plurality of sensors, wherein the plurality of sensors are separate from the wearable computing device;

determining presence of a second user in the facility;

generating, using at least the sensor data, operational data associated with the second user, wherein the operational data comprises information related to a request for assistance submitted by the second user and relationship data indicative of a relationship between the second user and a third user; and

generating, using at least the operational data, user interface data comprising one or more instructions to present a user interface with the wearable computing device worn by the first user, wherein the user interface comprises one or more elements which overlay real-world objects, and wherein the user interface based on the user interface data is presented to the first user to indicate the request for assistance by the second user and the relationship between the second user and the third user.

16. The method of claim 15, further comprising:

retrieving facility data indicative of operation of the facility, the facility data including at least one or more of;

emergency exit location,

evacuation path,

alarm status, or

equipment status; and

generating the user interface data based on the facility data.

17. The method of claim 15, further comprising:

retrieving data associated with the first user, wherein the data associated with the first user comprises one or more of;

a message,

a task list, or

a navigation path; and

generating the user interface data based at least in part on the data associated with the first user.

18. The method of claim 15, further comprising:

scanning the second user at an entry portal for one or more machine readable tags, wherein the one or more machine readable tags comprise one or more of a radio frequency identification tag, an optical tag, or a magnetic tag.

19. The method of claim 15, wherein the operational data comprises information indicative of a distance between the first user and the second user, and further wherein the distance is one or more of a straight-line distance or a travel path distance.

20. The method of claim 15, further comprising:

determining that the third user entered the facility with the second user, based on one or more of;

analyzing second sensor data comprising images acquired at an entry portal of the facility to determine the second user was proximate to the third user at the entry portal;

or

analyzing third sensor data comprising tag data received from scanning tags at the entry portal to determine a tag associated with the second user was scanned at the entry portal of the facility within a threshold time of tags associated with the third user; and

generating the relationship data.

21. A method comprising:

receiving user interface data at a wearable computing device worn by a first user at a materials facility using a communication interface; and

presenting a user interface to the first user on a display device of the wearable computing device, wherein

the user interface is presented using the user interface data,

the user interface is indicative of a request for assistance submitted by a second user and relationship data between the second user and a third user, and

the user interface presents one or more elements as graphical representation or text overlaid on real-world objects within a field of view.