Remotely operating a mobile robot

• US 9,789,612 B2
• Filed: 03/13/2017
• Issued: 10/17/2017
• Est. Priority Date: 01/07/2014
• Status: Active Grant

• Current Assignee: Flir Detection, Inc. (Teledyne Technologies Incorporated)
• Original Assignee: iRobot Defense Holdings, Inc. (Teledyne Technologies Incorporated)
• Inventors: Hoffman, Orin P. F., Keefe, Peter, Smith, Eric, Wang, John, Labrecque, Andrew, Ponsler, Brett, Macchia, Susan, Madge, Brian J.
• Primary Examiner(s): Jen, Ian
• Application Number:
  US15/457,671
• Publication Number:
  US 20170217021A1
• Time in Patent Office: 218 Days
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: B25J 13/025 comprising haptic means
• CPC Class Codes: B25J 13/06 Control stands, e.g. consoles, switchboards
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 19/04 Viewing devices
• CPC Class Codes: B25J 5/005 mounted on endless tracks or belts
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A method of operating a robot includes electronically receiving images and augmenting the images by overlaying a representation of the robot on the images. The robot representation includes user-selectable portions. The method includes electronically displaying the augmented images and receiving an indication of a selection of at least one user-selectable portion of the robot representation. The method also includes electronically displaying an intent to command the selected at least one user-selectable portion of the robot representation, receiving an input representative of a user interaction with at least one user-selectable portion, and issuing a command to the robot based on the user interaction.

Claims

1. A method for controlling a plurality of different robots by an operator control unit, the method comprising:

for each robot of the plurality of different robots;

receiving, over a wireless communications connection with the robot, a video feed from an imaging device disposed on the robot;

displaying the video feed in a graphical user interface on a touchscreen display of the operator control unit;

augmenting the video feed as displayed on the touchscreen display by overlaying a representation of the robot on the video feed as displayed on the touchscreen display;

receiving a first touchscreen input selecting, on the touchscreen display, the representation of the robot on the video feed and, in response, altering the graphical user interface on the touchscreen display of the operator control unit;

after altering the graphical user interface, receiving a second touchscreen input and determining at least one movement parameter of the robot based on the second touchscreen input; and

issuing, over the wireless communications connection, a command to the robot based on the at least one movement parameter.

2. The method of claim 1, wherein receiving the first touchscreen input comprises receiving an indication of a selection of at least one user-selectable portion of the representation of the robot.

3. The method of claim 2, wherein altering the graphical user interface comprises electronically displaying an intent to command the at least one user-selectable portion of the representation of the robot.

4. The method of claim 3, wherein receiving the second touchscreen input comprises receiving an input representative of a user interaction with the at least one user-selectable portion.

5. The method of claim 4, wherein determining the at least one movement parameter comprises determining the at least one movement parameter for at least one movable portion of the robot using inverse kinematics based on the second touchscreen input.

6. The method of claim 1, wherein overlaying the representation of the robot comprises displaying at least one of a representation of a vehicle body, a representation of a gripper, a representation of a link, or a representation of an actuator.

7. The method of claim 1, wherein altering the graphical user interface comprises modifying an appearance of the representation of the robot.

8. The method of claim 1, wherein receiving the first touchscreen input comprises at least one of a linear finger swipe, a curved finger swipe, a multi-finger swipe, a multi-finger gesture, a tap, or a prolonged press.

9. The method of claim 1, wherein receiving the second touchscreen input comprises receiving a selection of a robot behavior associated with an object.

10. The method of claim 9, wherein the robot behavior comprises navigating the robot towards the object or grasping the object with a manipulator of the robot.

11. A operator control unit comprising:

a touchscreen display; and

a processor in communication with the touchscreen display, wherein the processor is programmed to control a plurality of different robots, including, for each robot;

receiving, over a wireless communications connection with the robot, a video feed from an imaging device disposed on the robot;

displaying the video feed in a graphical user interface on a touchscreen display of the operator control unit;

augmenting the video feed as displayed on the touchscreen display by overlaying a representation of the robot on the video feed as displayed on the touchscreen display;

receiving a first touchscreen input selecting, on the touchscreen display, the representation of the robot on the video feed and, in response, altering the graphical user interface on the touchscreen display of the operator control unit;

after altering the graphical user interface, receiving a second touchscreen input and determining at least one movement parameter of the robot based on the second touchscreen input; and

issuing, over the wireless communications connection, a command to the robot based on the at least one movement parameter.

12. The operator control unit of claim 11, wherein receiving the first touchscreen input comprises receiving an indication of a selection of at least one user-selectable portion of the representation of the robot.

13. The operator control unit of claim 12, wherein altering the graphical user interface comprises electronically displaying an intent to command the at least one user-selectable portion of the representation of the robot.

14. The operator control unit of claim 13, wherein receiving the second touchscreen input comprises receiving an input representative of a user interaction with the at least one user-selectable portion.

15. The operator control unit of claim 14, wherein determining the at least one movement parameter comprises determining the at least one movement parameter for at least one movable portion of the robot using inverse kinematics based on the second touchscreen input.

16. The operator control unit claim of 11, wherein overlaying the representation of the robot comprises displaying at least one of a representation of a vehicle body, a representation of a gripper, a representation of a link, or a representation of an actuator.

17. The operator control unit of claim 11, wherein altering the graphical user interface comprises modifying an appearance of the representation of the robot.

18. The operator control unit of claim 11, wherein receiving the first touchscreen input comprises at least one of a linear finger swipe, a curved finger swipe, a multi-finger swipe, a multi-finger gesture, a tap, or a prolonged press.

19. The operator control unit of claim 11, wherein receiving the second touchscreen input comprises receiving a selection of a robot behavior associated with an object.

20. The operator control unit of claim 19, wherein the robot behavior comprises navigating the robot towards the object or grasping the object with a manipulator of the robot.