Remotely operating a mobile robot

• US 9,592,604 B2
• Filed: 03/14/2016
• Issued: 03/14/2017
• Est. Priority Date: 01/07/2014
• Status: Active Grant

• Current Assignee: Flir Detection, Inc. (Teledyne Technologies Incorporated)
• Original Assignee: iRobot Defense Holdings, Inc. (Teledyne Technologies Incorporated)
• Inventors: Hoffman, Orin P. F., Keefe, Peter, Smith, Eric, Wang, John, Labrecque, Andrew, Ponsler, Brett, Macchia, Susan, Madge, Brian J.
• Primary Examiner(s): Jen, Ian
• Application Number:
  US15/069,523
• Publication Number:
  US 20160243698A1
• Time in Patent Office: 365 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: B25J 13/025 comprising haptic means
• CPC Class Codes: B25J 13/06 Control stands, e.g. consoles, switchboards
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 19/04 Viewing devices
• CPC Class Codes: B25J 5/005 mounted on endless tracks or belts
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A method of operating a robot includes electronically receiving images and augmenting the images by overlaying a representation of the robot on the images. The robot representation includes user-selectable portions. The method includes electronically displaying the augmented images and receiving an indication of a selection of at least one user-selectable portion of the robot representation. The method also includes electronically displaying an intent to command the selected at least one user-selectable portion of the robot representation, receiving an input representative of a user interaction with at least one user-selectable portion, and issuing a command to the robot based on the user interaction.

Claims

1. A method comprising:

identifying multiple robots in communication with a network;

for each identified robot, electronically displaying a robot identity, a wireless communication signal strength, and a communication position within any series of robots relaying communications to the network;

establishing control of one or more of the robots; and

for each controlled robot;

electronically receiving images from the one or more controlled robots;

augmenting the images by overlaying a representation of the corresponding robot on the images, the robot representation comprising user-selectable portions corresponding to movable portions of the corresponding robot; and

electronically displaying the augmented images.

2. The method of claim 1, further comprising, for at least one controlled robot:

receiving an indication of a selection of at least one user-selectable portion of the corresponding robot representation of the at least one controlled robot;

electronically displaying an intent to command the selected at least one user-selectable portion of the corresponding robot representation;

receiving an input representative of a user interaction with at least one user-selectable portion;

determining at least one movement parameter of at least one movable portion of the corresponding robot using inverse kinematics based on the received input; and

issuing a command to the corresponding robot based on the at least one movement parameter.

3. The method of claim 2, wherein the representation of the at least one controlled robot comprises at least one of a representation of a vehicle body, a representation of a gripper, a representation of a link, or a representation of an actuator.

4. The method of claim 3, wherein electronically displaying the intent to command the selected at least one user-selectable robot representation portion comprises modifying an appearance of the robot representation.

5. The method of claim 2, wherein the input representative of the user interaction is received from a touch display and comprises at least one of a linear finger swipe, a curved finger swipe, a multi-finger swipe, a multi-finger gesture, a tap, or a prolonged press.

6. The method of claim 2, wherein receiving the input representative of the user interaction with at least one user-selectable portion comprises:

receiving a first input representative of a selection of a displayed object; and

receiving a second input representative of a selection of a robot behavior, the robot behavior associated with the object.

7. The method of claim 6, wherein the robot behavior comprises navigating the at least one controlled robot towards the object.

8. The method of claim 7, further comprising receiving an indication of a selection of an alternate approach direction, the robot behavior determining a drive path using odometry and/or inertial measurement signals from an inertial measurement unit of the robot to navigate the at least one controlled robot from a current location and a current approach direction to approach the object from the alternate approach direction.

9. The method of claim 7, wherein the robot behavior comprises grasping the object with a manipulator of the robot.

10. The method of claim 6, further comprising identifying in the images a plane of a ground surface supporting the at least one controlled robot and a location of the object with respect to the ground surface plane.

11. The method of claim 2, further comprising:

receiving an indication of a selection of a reverse-out behavior; and

executing the reverse-out behavior, the reverse-out behavior;

determining at least one reverse-movement parameter of the at least one movable portion of the robot using inverse kinematics to move the at least one movable portion of the at least one controlled robot in an opposite direction along a path moved according to the issued command; and

commanding the at least one movable portion of the at least one controlled robot based on the at least one reverse-movement movement parameter.

12. The method of claim 2, further comprising providing haptic feedback in response to the received input.

13. The method of claim 2, further comprising, when the determined at least one movement parameter of the at least one movable portion of the robot based on the received input violates a movement policy or is unexecutable, issuing a negative feedback response.

14. The method of claim 13, wherein the negative feedback response comprises at least one of a haptic feedback response, an audio feedback response, or a visual feedback response, the visual feedback response comprising displaying an indicator at or near any portions of the robot representation corresponding to any unmovable portions of the robot based on the received input.

15. The method of claim 2, further comprising:

receiving an indication of a selection of a series of robots in communication with the network and an order of deployment of a communication repeater deployment carried by each of the selected robots; and

issuing repeater deployment instructions to each of the selected robots, each repeater instruction indicating a repeater deployment order of the corresponding robot with respect to the other selected robots, each selected robot deploying its carried communication repeater when the robot maneuvers to or near an edge of a wireless communications reception area of the network.

16. The method of claim 15, wherein each repeater instruction includes a repeater deployment location.

17. The method of claim 2, further comprising communicating with and using a remote scalable computing resource to execute commands demanding a threshold computing capacity.

18. A system comprising:

a network;

multiple robots in wireless communication with the network; and

multiple operator control units in communication with the network, each operator controller unit comprising;

a screen; and

a processor in communication with the screen, the processor configured to;

identify the multiple robots in communication with the network;

for each identified robot, electronically display a robot identity, a wireless communication signal strength, and a communication position within any series of robots relaying communications to the network;

establish control of one or more of the robots; and

for each controlled robot;

electronically receive images from the controlled robot;

augment the images by overlaying a representation of the corresponding robot on the images, the robot representation comprising user-selectable portions corresponding to movable portions of the corresponding robot; and

electronically display the augmented images.

19. The system of claim 18, wherein the processor is configured to, for at least one controlled robot:

receive an indication of a selection of at least one user-selectable portion of the corresponding robot representation of the at least one controlled robot;

electronically display an intent to command the selected at least one user-selectable portion of the corresponding robot representation;

receive an input representative of a user interaction with at least one user-selectable portion;

determine at least one movement parameter of at least one movable portion of the corresponding robot using inverse kinematics based on the received input; and

issue a command to the corresponding robot based on the at least one movement parameter.

20. The method of claim 19, wherein the representation of the at least one controlled robot comprises at least one of a representation of a vehicle body, a representation of a gripper, a representation of a link, or a representation of an actuator.

21. The method of claim 20, wherein electronically displaying the intent to command the selected at least one user-selectable robot representation portion comprises modifying an appearance of the robot representation.

22. The method of claim 19, wherein the input representative of the user interaction is received from a touch display and comprises at least one of a linear finger swipe, a curved finger swipe, a multi-finger swipe, a multi-finger gesture, a tap, or a prolonged press.