Performing Automated Tasks Based on Visual Cues

• US 20150379347A1
• Filed: 06/26/2014
• Published: 12/31/2015
• Est. Priority Date: 06/26/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Graybill, Jules Cook, Guan, Danny, Mishra, Pragyana K.
• Granted Patent: US 9,846,806 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation

Abstract

Visual cues may be provided for the purpose of triggering the performance of one or more actions associated with a given task. A visual cue may include one or more characters or other markings on any relevant surfaces, as well as naturally appearing edges, contours or outlines within an environment in which the visual cue is provided. When one or more visual cues appear within a field of view of an imaging device, and the visual cues are recognized as such, a task relating to the visual cues may be identified, and instructions for performing the task may be provided. The task may be dependent upon a context in which the visual cues are provided, such that a single visual cue may be associated with multiple tasks in different contexts, and the performance of the task may be confirmed based on imaging data or other relevant information.

Claims

1. An automated task performing system comprising:

an imaging device;

at least one data store;

an automated machine; and

a computing device in communication with the imaging device and the automated machine,

wherein the computing device is configured to implement one or more services, and

wherein the one or more services are configured to;

capture an image of an object using the imaging device;

recognize a visual cue expressed within the image, wherein the visual cue is associated with at least a portion of the object;

determine an association between the visual cue and at least one task to be performed upon the object based at least in part on information regarding a plurality of tasks maintained in the at least one data store;

identify at least one action required for executing the at least one task; and

provide an electronic instruction for performing the at least one task to the automated machine.

2. The automated task performing system of claim 1, wherein the one or more services are further configured to:

identify a context associated with the visual cue, wherein the association is determined based at least in part on the context.

3. The automated task performing system of claim 1, wherein the visual cue comprises at least one of:

an alphanumeric character;

a symbol;

an icon;

or

an outline of at least the portion of the object.

4. A computer-implemented method comprising:

identifying a first set of imaging data regarding an object, wherein the first set of imaging data comprises information regarding at least one visual cue associated with the object;

identifying, by at least one computer processor, the at least one visual cue based at least in part on the first set of imaging data;

identifying, by the at least one computer processor, a first automated task associated with the object based at least in part on the at least one visual cue; and

providing, by the at least one computer processor, an instruction for performing at least one action associated with the first automated task to a first automated machine.

5. The computer-implemented method of claim 4, wherein identifying the first set of imaging data regarding the object comprises:

capturing the first set of imaging data using at least one imaging device.

6. The computer-implemented method of claim 5, wherein the first automated machine is the at least one imaging device.

7. The computer-implemented method of claim 5, wherein the first set of imaging data comprises at least one image of the object.

wherein the at least one visual cue is expressed within the at least one image of the object.

8. The computer-implemented method of claim 7, wherein at least one of the object or the at least one visual cue is presented within the at least one image by at least one human.

9. The computer-implemented method of claim 4, wherein identifying the at least one visual cue comprises:

recognizing at least one of an edge, a contour or an outline of the object based at least in part on the imaging data; and

identifying the at least one visual cue based at least in part on the edge, the contour or the outline.

10. The computer-implemented method of claim 4, wherein identifying the first automated task associated with the object comprises:

identifying a plurality of records stored in at least one data store, wherein each of the records comprises an association between one of a plurality of visual cues and at least one of a plurality of automated tasks;

selecting at least one of the plurality of records corresponding to the at least one visual cue; and

identifying the first automated task based at least in part on the selected at least one of the plurality of records.

11. The computer-implemented method of claim 4, wherein identifying the first automated task associated with the object comprises:

determining a context associated with at least one of the at least one visual cue, the object or an environment in which the object is provided,

wherein the first automated task is identified based at least in part on the context.

12. The computer-implemented method of claim 4, further comprising:

identifying a desired condition of an environment in which the object is provided, wherein the desired condition of the environment is associated with the first automated task;

capturing a second set of imaging data regarding the object using at least one imaging device;

determining a condition of the environment in which the object is provided based at least in part on the second set of imaging data;

determining whether the condition of the environment is consistent with the desired condition of the environment; and

upon determining that the condition of the environment is consistent with the desired condition of the environment,

storing an indication that the first automated task has been performed.

13. The computer-implemented method of claim 12, further comprising:

identifying, by the at least one computer processor, a second automated task associated with the object based at least in part on the at least one visual cue; and

providing, by the at least one computer processor, an instruction for performing at least one action associated with the second automated task to the first automated machine,

wherein the second automated task is identified based at least in part on the indication that the first automated task has been performed.

14. The computer-implemented method of claim 12, further comprising:

identifying, by the at least one computer processor, a second automated task associated with the object based at least in part on the at least one visual cue; and

providing, by the at least one computer processor, an instruction for performing at least one action associated with the second automated task to a second automated machine,

wherein the second automated task is identified based at least in part on the indication that the first automated task has been performed.

15. The computer-implemented method of claim 12, further comprising:

upon determining that the condition of the environment is not consistent with the desired condition of the environment,

storing an indication that the first automated task has not been performed;

identifying, by the at least one computer processor, a second automated task associated with the object based at least in part on the at least one visual cue; and

providing, by the at least one computer processor, an instruction for performing at least one action associated with the second automated task to the first automated machine,

wherein the second automated task is identified based at least in part on the indication that the first automated task has not been performed.

16. The computer-implemented method of claim 4, wherein identifying the at least one visual cue based at least in part on the first set of imaging data comprises:

identifying information regarding a visual pattern based at least in part on the first set of imaging data, wherein the visual pattern comprises the at least one visual cue associated with the object,

wherein the first automated task associated with the object is identified based at least in part on the visual pattern.

17. The computer-implemented method of claim 16, wherein the visual pattern comprises a first visual cue associated with the object and a second visual cue associated with the object, and

wherein identifying the visual pattern based at least in part on the first set of imaging data comprises:

identifying information regarding the first visual cue at a first time based at least in part on the first set of imaging data;

identifying information regarding the second visual cue at a second time based at least in part on the first set of imaging data; and

defining the visual pattern based at least in part on the first visual cue, the first time, the second visual cue and the second time.

18. The computer-implemented method of claim 16, wherein the visual pattern comprises a first visual cue provided on the object and a second visual cue provided on the object, and

wherein identifying the visual pattern based at least in part on the first set of imaging data comprises:

identifying information regarding the first visual cue based at least in part on the first set of imaging data, wherein the information comprises a first location of the first visual cue on the object;

identifying information regarding the second visual cue at a second time based at least in part on the first set of imaging data, wherein the information comprises a second location of the second visual cue on the object; and

defining the visual pattern based at least in part on the first visual cue, the first location, the second visual cue and the second location.

19. A non-transitory computer-readable medium having computer-executable instructions stored thereon,

wherein the instructions, when executed, cause a computer system having at least one computer processor and at least one data store to perform a method comprising:

capturing a first image of at least a portion of an environment using an imaging device;

identifying a visual cue expressed within the first image;

selecting a first action to be performed upon the environment based at least in part on the visual cue expressed within the first image;

transmitting an instruction for performing at least a portion of the first action to an automated machine;

capturing a second image of at least the portion of the environment using the imaging device;

determining whether the automated machine performed the first action based at least in part on the second image;

upon determining that the automated machine performed the first action,

selecting a second action to be performed upon the environment based at least in part on the visual cue expressed within the first image; and

transmitting an instruction for performing at least a portion of the second action to the automated machine.

20. The non-transitory computer-readable medium of claim 19, wherein the method further comprises:

identifying a context of the visual cue based at least in part on the first image,

wherein the first action is selected based at least in part on the context of the visual cue.