Visual task feedback for workstations in materials handling facilities

• US 9,299,013 B1
• Filed: 03/27/2014
• Issued: 03/29/2016
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Curlander, James Christopher, Colburn, Robert Alexander
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/228,127
• Time in Patent Office: 733 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Visual task feedback for workstations in a materials handling facility may be implemented. Image data of a workstation surface may be obtained from image sensors. The image data may be evaluated with regard to the performance of an item-handling task at the workstation. The evaluation of the image data may identify items located on the workstation surface, determine a current state of the item-handling task, or recognize an agent gesture at the workstation. Based, at least in part on the evaluation, one or more visual task cues may be selected to project onto the workstation surface. The projection of the selected visual task cues onto the workstation surface may then be directed.

Claims

1. A materials handling facility, comprising:

one or more imaging sensors, wherein the one or more imaging sensors include a sensor configured to capture geometric information of a workstation surface that corresponds to image data of the workstation surface;

one or more visual projectors;

a control system, configured to;

receive, from the one or more imaging sensors, image data of the workstation surface in the materials handling facility, wherein the geometric information corresponding to the image data is received along with the image data from the one or more imaging sensors;

evaluate the image data to identify one or more items located on the workstation surface associated with an item-handling task performed at the workstation, wherein to evaluate the image data to identify the one or more items located on the workstation surface, the control system is further configured to;

access a data store maintaining item description information regarding items handled in the materials handling facility; and

perform a comparative analysis based at least in part on the image data of the workstation surface of the image data and the item description information to identify the one or more items located on the workstation surface;

based, at least in part, on the one or more identified items, select one or more visual task cues to project onto the workstation surface, wherein the one or more visual task cues indicate a current state of the item-handling task associated with the one or more identified items; and

direct the one or more visual projectors to project the one or more visual task cues onto the workstation surface.

2. The materials handling facility of claim 1, wherein the selected one or more visual task cues include at least one of:

an indication that an item associated with the item-handling task is missing from the workstation surface;

an indication that a particular one of the identified one or more items located on the workstation surface is incorrect for performance of the item handling task;

an indication that a particular one of the one or more items located on the workstation surface requires manual identification;

or

an indication that the identified one or more items are complete for performance of the item handling task.

3. The system of claim 1, wherein the materials handling facility includes a plurality of different workstations including the workstation, wherein different ones of the plurality of different workstations perform at least one different respective item-handling task, wherein the control system is configured to perform said receiving, said evaluating, said selecting, and said directing for at least some of the different ones of the plurality of different workstations in the materials handling facility.

4. A method, comprising:

performing, by one or more computing devices;

obtaining, via one or more imaging sensors, image data of a surface of a workstation in a materials handling facility;

evaluating the image data of the workstation surface with regard to a performance of an item-handling task performed at the workstation;

receiving location information for an agent performing the item-handling task at the workstation;

based, at least in part, on the location information, determining a viewing perspective for the agent;

based, at least in part, on the evaluation of the image data with regard to the performance of the item-handling task, determining one or more visual task cues to project onto the workstation surface; and

directing the projection of the one or more visual task cues onto the workstation surface, wherein said directing the projection of the one or more visual task cues onto the workstation surface is performed such that the one or more visual task cues are projected according to the viewing perspective of the agent.

5. The method of claim 4, wherein said evaluation comprises performing at least one of:

identifying one or more items located on the workstation surface for the performance of the item-handling task;

determining a current state of the performance of the item-handling task;

or

recognizing an agent gesture at the workstation associated with the performance of the item-handling task.

6. The method of claim 5, wherein said identifying the one or more items located on the workstation surface associated with the item handling task performed at the workstation, comprises:

accessing a data store maintaining item description information regarding items handled in the materials handling facility; and

comparing the image data and the item description information to identify the one or more items located on the workstation surface.

7. The method of claim 6, further comprising:

obtaining, via the one or more imaging sensors, geometric information of the workstation surface corresponding to the image data;

based, at least in part on the geometric information and the image data, calculating one or more item dimensions for the one or more items; and

wherein said comparing the image data and the item description information to identify the one or more items located on the workstation surface includes comparing the one or more item dimensions as part of said comparison.

8. The method of claim 5, wherein said recognizing the agent gesture at the workstation associated with the performance of the item-handling task, comprises:

detecting an agent gesture with regard to a user interface projected on the workstation surface as an input action for the user interface.

9. The method of claim 4, further comprising:

obtaining, via the one or more imaging sensors, additional image data of the workstation surface and geometric information of the workstation surface corresponding to the additional image data;

based, at least in part, on the additional image data of the workstation surface and the geometric information corresponding to the additional image data obtained from the one or more imaging sensors, detecting a surface change of the workstation surface that interferes with at least one of the projected one or more visual task cues;

determining one or more projection modifications for the at least one visual task cue; and

updating the at least one visual task cue projected onto the workstation according to the one or more projection modifications.

10. The method of claim 4, wherein the one or more visual task cues are projected in an order indicating a sequence of actions for the performance of the item handling task.

11. The method of claim 4, further comprising performing a respective one or more audio task cues in coordination with said projection of the determined one or more visual task cues.

12. The method of claim 4, wherein the materials handling facility includes a plurality of different types of workstations each performing at least one different respective item-handling task, wherein the one or more computing devices perform said obtaining, said evaluating, said determining, and said directing for at least some of the plurality of different types of workstations in the materials handling facility.

13. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to at least implement:

obtaining, via one or more imaging sensors, image data of a surface of a workstation in a materials handling facility;

evaluating the image data of the workstation surface with regard to a performance of an item-handling task at the workstation based, at least in part on the image data of the workstation surface, wherein in said evaluating the image data of the workstation surface with regard to the performance of the item-handling task at the workstation, the program instructions cause the one or more computing devices to at least implement;

identifying one or more items located on the workstation surface associated with the item handling task performed at the workstation, comprising;

accessing a data store maintaining item description information regarding items handled in the materials handling facility, and

comparing the image data and the item description information to identify the one or more items located on the workstation surface;

based, at least in part, on the evaluation of the image data of the workstation surface, determining one or more visual task cues to project onto the workstation surface; and

directing the projection of the one or more visual task cues onto the workstation surface.

14. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computing devices to at least implement:

obtaining, via the one or more imaging sensors, geometric information of the workstation surface corresponding to the image data;

based, at least in part on the geometric information and the image data, calculating one or more item dimensions for the one or more items; and

wherein said comparing the image data and the item description information to identify the one or more items located on the workstation surface includes comparing the one or more item dimensions as part of said comparison.

15. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computing devices to at least implement:

obtaining, via a scale configured to weigh objects on the workstation surface, weight information for the workstation surface;

wherein said comparing the image data and the item description information to identify the one or more items located on the workstation surface includes the weight information of the workstation surface as part of said comparison.

16. The non-transitory, computer-readable storage medium of claim 13, wherein said identifying the one or more items located on the workstation surface associated with the item handling task performed at the workstation is performed without receiving item information for the one or more items via a manual item scan.

17. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computing devices to at least implement:

obtaining, via the one or more imaging sensors, additional image data and geometric information of the workstation surface corresponding to the additional image data;

based, at least in part, on the additional image data of the workstation surface and the additional geometric information, detecting a surface change of the workstation surface that interferes with at least one of the projected one or more visual task cues;

determining one or more projection modifications for the at least one visual task cue; and

updating the at least one visual task cue projected onto the workstation according to the one or more projection modifications.

18. The non-transitory, computer-readable storage medium of claim 13, wherein in said evaluating the image data of the workstation surface with regard to the performance of the item-handling task at the workstation, the program instructions cause the one or more computing devices to at least implement:

recognizing an agent gesture at the workstation associated with the performance of the item-handling task;

wherein the determined one or more visual task cues indicate correctness of the agent gesture for performing the item-handling task.

19. The non-transitory, computer-readable storage medium of claim 13, wherein the materials handling facility includes a plurality of different types of workstations each performing at least one different respective item-handling task, wherein the one or more computing devices perform said obtaining, said evaluating, said determining, and said directing for at least some of the plurality of different types of workstations in the materials handling facility.

20. The non-transitory, computer-readable storage medium of claim 13, wherein the workstation is a packing workstation, wherein the item-handling task is a packing process to pack one or more items together into a shipping container, and wherein the program instructions further cause the one or more computing devices to at least implement:

based, at least in part, on the image data of the workstation surface, selecting the shipping container from a plurality of different shipping containers; and

wherein the one or more visual task cues include an indication to identify the selected shipping container for the one or more items.