Systems and methods for distributed autonomous robot interfacing using live image feeds

• US 10,189,642 B2
• Filed: 01/26/2018
• Issued: 01/29/2019
• Est. Priority Date: 01/30/2017
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: High, Donald, Winkle, David, McHale, Brian Gerard, Mattingly, Todd Davenport
• Primary Examiner(s): Cumbess, Yolanda R
• Application Number:
  US15/880,713
• Publication Number:
  US 20180215546A1
• Time in Patent Office: 368 Days
• Field of Search: None
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/06 characterised by multi-articulated arms
• CPC Class Codes: B65G 1/1375 the orders being assembled on a commissioning stacker-crane or truck
• CPC Class Codes: B65G 1/1378 the orders being assembled on fixed commissioning areas remote from the storage areas
• CPC Class Codes: G05B 19/124 using tapes, cards or discs with optically sensed marks or codes G05B19/128, G05B19/14 take precedence
• CPC Class Codes: G05B 2219/23363 Barcode
• CPC Class Codes: G05B 2219/32388 Autonomous flexible system, cells and agv autonomous
• CPC Class Codes: G05B 2219/36412 Fine, autonomous movement of end effector by using camera
• CPC Class Codes: G05B 2219/40507 Distributed planning, offline trajectory, online motion, avoid collision
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0234 using optical markers or beacons optical beacons per se G01S1/70
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

Described in detail herein is an autonomous fulfillment system. The system includes the first computing system, with an interactive display. The first computing system can transmit a request for physical objects from a facility. A second computing system can transmit instructions autonomous robot devices to retrieve the physical objects from the facility. The second computing system can control the image capturing device of the autonomous robot device to capture a live image feed of the at least one physical object picked up by the at least autonomous robot device. The second computing system can switch an input feed of the first computing system to display the live image feed on the display of the first computing system. The second computing system, instruct the autonomous device to discard the physical objects picked up by the at least one autonomous robot device and to pick up a replacement physical object.

Claims

1. An autonomous system including autonomous robot devices, the system comprising:

a first computing system including an interactive display configured to transmit a first request for a plurality of physical objects from a facility;

a plurality of autonomous robot devices including a controller, a drive motor, an articulated arm, a reader, an inertial navigation system and an image capturing device, the plurality of autonomous robot devices configured to navigate autonomously through the facility;

a database in communication with the plurality of robots, configured to store information associated with the plurality of physical objects;

a second computing system in communication with the first computing system, the database, and the plurality of autonomous robot devices, the second computing system configured to;

receive the first request for the plurality of physical objects from the first computing system;

transmit instructions the at least one of the plurality of autonomous robot devices to retrieve the plurality of physical objects from the facility;

detect the at least one autonomous robot device has picked up at least one of the plurality of physical objects;

control the image capturing device of the at least autonomous robot device to capture a live image feed of the at least one physical object picked up by the at least autonomous robot device;

switch an input feed of the first computing system to display the live image feed on the display of the first computing system;

receive a second request from first computing system to discard the at least one of the physical objects picked up by the at least one autonomous robot device and to pick up a replacement physical object; and

transmit instructions to the at least one autonomous device to discard the at least one of the physical objects picked up by the at least one autonomous robot device and to pick up the replacement physical object, the instructions include one or more identifiers for the plurality of physical objects,

wherein the at least one of the autonomous robot further devices configured to;

query the database using the one or more identifiers for the plurality of physical objects to retrieve the locations at which the plurality of physical objects are disposed;

navigate autonomously through the facility to the locations in response to operation of the drive motor by the controller;

locate and scan one or more machine readable elements encoded with the one or more identifiers;

detect, via at least one image captured by the image capture device, that the plurality of physical objects are disposed at the locations;

pick up a first quantity of each of the plurality of physical objects using the articulated arm.

2. The system in claim 1, wherein the at least one of the autonomous robot further devices configured to:

carry and navigate with the first quantity of each of the plurality of physical objects to a specified location.

3. The system in claim 2, wherein the first computing system is disposed in the facility and the specified location is a location of the first computing system within the facility.

4. The system in claim 2, wherein the at least one autonomous robot device is further configured to navigate to a storage container within the facility, deposit the first quantity of each of the plurality of physical object in the storage container, pick up the storage container and carry and navigate the storage container to the specified location.

5. The system in claim 4, wherein the specified location is a parking lot in which a plurality of automobiles are disposed.

6. The system in claim 5, wherein the first request received by the second computing system includes an identification number associated with at least one of the plurality of automobile and a tag number associated with the at least one automobile.

7. The system in claim 5, wherein the at least one autonomous robotic device is further configured to:

carry the storage container to the parking lot;

detect using the image capturing device, the identification number of the at least one automobile;

navigate to the at least one automobile;

autonomously open an access point of the at least one automobile using the tag number; and

deposit the storage container inside the at least one automobile, via the access point.

8. The system in claim 2, further comprising a plurality of sensors disposed at the locations of each of the plurality of physical objects, the plurality of sensors configured to determine a set of attributes associated with each of the plurality of physical objects retrieved by the at least one autonomous robot device.

9. The system in claim 8, wherein the second computing system is further configured to transmit an alert to be displayed on the display of the first computing system based on the detected set of attributes for the at least one of the plurality of physical objects retrieved by the at least one autonomous robot device.

10. The system in claim 1, wherein the first computing system is a mobile device or a kiosk.

11. An autonomous method including autonomous robot devices, the method comprising:

transmitting, via a first computing system including an interactive display, a first request for a plurality of physical objects from a facility;

navigating, via a plurality of autonomous robot devices including a controller, a drive motor, an articulated arm, a reader, an inertial navigation system and an image capturing device, autonomously through the facility;

receiving, via a second computing system in communication with the first computing system and the plurality of autonomous robot devices, the first request for the plurality of physical objects from the first computing system;

transmitting, via the second computing system, instructions the at least one of the plurality of autonomous robot devices to retrieve the plurality of physical objects from the facility;

detecting, via the second computing system, the at least one autonomous robot device has picked up at least one of the plurality of physical objects;

controlling, via the second computing system, the image capturing device of the at least autonomous robot device to capture a live image feed of the at least one physical object picked up by the at least autonomous robot device;

switching, via the second computing system, an input feed of the first computing system to display the live image feed on the display of the first computing system;

receiving, via the second computing system, a second request from first computing system to discard the at least one of the physical objects picked up by the at least one autonomous robot device and to pick up a replacement physical object;

transmitting, via the second computing system, instructions to the at least one autonomous device to discard the at least one of the physical objects picked up by the at least one autonomous robot device and to pick up the replacement physical object, the instructions from the second computing system include one or more identifiers for the plurality of physical objects;

querying, via the at least one of the autonomous robot devices, a database operatively coupled to the second computing system and the plurality of the autonomous robot devices using the one or more identifiers for the plurality of physical objects;

navigating, via the at least one autonomous robot device, autonomously through the facility to the first set of object locations in response to operation of the drive motor by the controller;

locating and scanning, via the at least one autonomous robot device, one or more machine readable elements encoded with the one or more identifiers;

detecting, via at least one image captured by the image capture device of the at least one autonomous robot device, that the first group of physical objects are disposed at the first set of locations; and

picking up, via the at least one autonomous robot device, a first quantity of physical objects in the first group using the articulated arm.

12. The method in claim 11, further comprising:

carrying and navigating, via the at least one autonomous robot device, with the first quantity of the first quantity of physical objects in the first group to the storage containers located at a specified location.

13. The method in claim 12, wherein the first computing system is disposed in the facility and the specified location is a location of the first computing system within the facility.

14. The method in claim 12, further comprising navigating, via the at least autonomous robot device, to a storage container within the facility, deposit the first quantity of each of the plurality of physical object in the storage container, pick up the storage container and carry and navigate the storage container to the specified location.

15. The method in claim 14, wherein the specified location is a parking lot in which a plurality of automobiles are disposed.

16. The method in claim 15, wherein the first request received by the second computing system includes an identification number associated with at least one of the plurality of automobile and a tag number associated with the at least one automobile.

17. The method in claim 16, further comprising:

carrying, via the at least one autonomous robotic device, the storage container to the parking lot;

detecting, via the at least one autonomous robotic device, using the image capturing device, the identification number of the at least one automobile;

navigating, via the at least one autonomous robotic device, to the at least one automobile;

autonomously opening, via the at least one autonomous robotic device, an access point of the at least one automobile using the tag number; and

depositing, via the at least one autonomous robotic device, the storage container inside the at least one automobile, via the access point.

18. The method in claim 11, further comprising determining, via a plurality of sensors disposed at the locations of each of the plurality of physical objects, a set of attributes associated with each of the plurality of physical objects retrieved by the at least one autonomous robot device.

19. The method in claim 18, further comprising transmitting, via the second computing system, an alert to be displayed on the display of the first computing system based on the detected set of attributes for the at least one of the plurality of physical objects retrieved by the at least one autonomous robot device.

20. The method in claim 11, wherein the first computing system is a mobile device or a kiosk.