System and method for mobile robot teleoperation

• US 9,684,305 B2
• Filed: 09/11/2015
• Issued: 06/20/2017
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Kratz, Sven, Vaughan, James L
• Primary Examiner(s): Marc, McDieunel
• Assistant Examiner(s): Stroud, James E
• Application Number:
  US14/852,246
• Publication Number:
  US 20170075348A1
• Time in Patent Office: 648 Days
• Field of Search: 701 2
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Example implementations include a system and method for remotely controlling the motion of a robot. In example implementations, the user is presented with a display having video from a camera on the robot augmented with abstract sensor data, and facilitates the user to draw motion path commands directly on the camera view. Analysis of the scene for obstacles is performed, so that the drawing of commands is interactive, preventing the user from drawing impossible paths. The path sketch is then transformed into the coordinate system of the robot and used to generate commands that will move the robot to the desired location. Sensor data is also used to augment the camera view, in particular for highlighting dangers and obstacles.

Claims

1. A first apparatus configured to remotely control a movable second apparatus, the first apparatus comprising:

a display; and

a processor, configured to;

receive a proposed navigation path for the movable second apparatus;

identify one or more obstacles from sensor data provided by the movable second apparatus;

determine that the proposed navigation path has at least one portion intersecting the identified one or more obstacles; and

provide to the display an alternate navigation path comprising one or more highlighted portions that differ from the proposed navigation path;

wherein the processor is configured to transmit instructions to the movable second apparatus, the instructions comprising navigating the proposed navigation path at a velocity based on a mapping function relating velocity to distance from the identified one or more obstacles.

2. The first apparatus of claim 1, wherein the display is configured to display the proposed navigation path overlaid on visual data received from the movable second apparatus, and wherein the processor is configured to highlight, on the display, one or more portions of the proposed navigation path that are within a proximity threshold of the identified one or more obstacles.

3. The first apparatus of claim 1, wherein the display is configured to display the proposed navigation path overlaid on visual data received from the movable second apparatus, and wherein the processor is configured to highlight, on the display, one or more portions of the proposed navigation path that intersect the identified one or more obstacles.

4. The first apparatus of claim 1, wherein the processor is configured to highlight the identified one or more obstacles on the display.

5. The first apparatus of claim 1, wherein the processor is configured to retrieve one or more overlays to overlay on visual data received from the movable second apparatus, based on a location of the movable second apparatus and the identified one or more obstacles.

6. A method, comprising:

receiving a proposed navigation path for a movable apparatus;

identifying one or more obstacles from sensor data provided by the movable apparatus;

determining that the proposed navigation path has at least one portion intersecting the identified one or more obstacles;

providing, on a display, an alternate navigation path comprising one or more highlighted portions that differ from the proposed navigation path; and

transmitting instructions to the movable apparatus, the instructions comprising navigating the proposed navigation path at a velocity based on a mapping function relating velocity to distance from the identified one or more obstacles.

7. The method of claim 6, further comprising overlaying, on the display, the proposed navigation path on visual data received from the movable apparatus, and highlighting, on the display, one or more portions of the proposed navigation path that are within a proximity threshold of the identified one or more obstacles.

8. The method of claim 6, further comprising overlaying, on the display, the proposed navigation path on visual data received from the movable apparatus, and highlighting, on the display, one or more portions of the proposed navigation path that intersect the identified one or more obstacles.

9. The method of claim 6, further comprising highlighting the identified one or more obstacles on the display.

10. The method of claim 6, further comprising retrieving one or more overlays to overlay on visual data from the movable apparatus, based on a location of the movable apparatus and the identified one or more obstacles.

11. A non-transitory computer readable medium, storing instructions for executing a process, the instructions comprising:

receiving a proposed navigation path for a movable apparatus;

identifying an obstacle from sensor data provided by the movable apparatus;

determining that the proposed navigation path has at least one portion intersecting the identified one or more obstacles;

providing, on a display, an alternate navigation path comprising one or more highlighted portions that differ from the proposed navigation path; and

transmitting instructions to the movable apparatus, the instructions comprising navigating the proposed navigation path at a velocity based on distance from the identified one or more obstacles.

12. The non-transitory computer readable medium of claim 11, wherein the instructions further comprise overlaying, on the display, the proposed navigation path on visual data received from the movable apparatus, and highlighting, on the display, one or more portions of the proposed navigation path that are within a proximity threshold of the identified one or more obstacles.

13. The non-transitory computer readable medium of claim 11, wherein the instructions further comprise overlaying, on the display, the proposed navigation path on visual data received from the movable apparatus, and highlighting, on the display, one or more portions of the proposed navigation path that intersect the identified one or more obstacles.

14. The non-transitory computer readable medium of claim 11, wherein the instructions comprise highlighting the identified one or more obstacles on the display.

15. The non-transitory computer readable medium of claim 11, wherein the instructions further comprise retrieving one or more overlays to overlay on visual data from the movable apparatus, based on a location of the movable apparatus and the identified one or more obstacles.

16. The non-transitory computer readable medium of claim 11, wherein the velocity based on a mapping function relating velocity to distance from the identified one or more obstacles is configured to conduct velocity reduction proportional to the distance from the identified one or more obstacles.

17. The non-transitory computer readable medium of claim 11, wherein the instructions further comprise overlaying the alternate navigation path on the proposed navigation path on the display.

18. The non-transitory computer readable medium of claim 11, wherein the velocity based on the distance is reduced as the movable apparatus is closer in proximity to the obstacle.