Method and system for remote control of mobile robot
First Claim
Patent Images
1. A method for tele-operating a robot in an environment, comprising the steps of:
- providing a user interface for controlling the tele-operation of the robot;
providing, via an imaging device associated with the robot, image information representative of the environment around the robot to the user interface;
using the image information to provide a user-perceptible image at the user interface representative of the environment around the robot;
designating a target anywhere in the user-perceptible image towards which the robot will move;
automatically converting the target designated in the user-perceptible image into a target location having x, y, and z coordinates in the environment of the robot at the user interface; and
providing real-time instructions to the robot from the user interface to move from the robot'"'"'s current location in the environment to the x, y, and z coordinates of the target location in the environment.
1 Assignment
0 Petitions
Accused Products
Abstract
Methods of remote control of a mobile robot and an intuitive user interface for remotely controlling a mobile robot are provided. Using a point-and-click device (405), the user is able to choose a target location (430) within a heads-up display (400) toward which to move a mobile robot. Additional graphical overlays (410 & 412) are provided to aid the user in navigating even in systems with asynchronous communication.
858 Citations
23 Claims
-
1. A method for tele-operating a robot in an environment, comprising the steps of:
-
providing a user interface for controlling the tele-operation of the robot;
providing, via an imaging device associated with the robot, image information representative of the environment around the robot to the user interface;
using the image information to provide a user-perceptible image at the user interface representative of the environment around the robot;
designating a target anywhere in the user-perceptible image towards which the robot will move;
automatically converting the target designated in the user-perceptible image into a target location having x, y, and z coordinates in the environment of the robot at the user interface; and
providing real-time instructions to the robot from the user interface to move from the robot'"'"'s current location in the environment to the x, y, and z coordinates of the target location in the environment. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
determining a current frame of reference of the robot in the environment;
determining a current frame of reference for the imaging device associated with the robot;
identifying one end of a click vector based upon the current frame of references of the robot and the imaging device;
determining the other end of the click vector based upon the view of the user-perceptible image and the designated target; and
projecting the click vector onto a three-dimensional global map of the robot'"'"'s environment to provide the x, y, and z coordinates of the target location.
-
-
3. The robot tele-operating method of claim 2 wherein the click vector determining step further comprises the steps of:
-
determining the fixed angle and zoom of the imaging device associated with the robot that provides the view of the user-perceptible image;
determining the pixel size of the user-perceptible image;
identifying a pixel location in the user-perceptible image corresponding to the designated target; and
calculating the other end of the click vector using the fixed angle and zoom of the imaging device, the pixel size of the user-perceptible image, and the pixel location corresponding to the designated target.
-
-
4. The robot tele-operating method of claim 1 wherein the target designating step further comprises the steps of:
-
moving a visible pointer to a location in the user-perceptible image corresponding to the designated target; and
activating a point-and-click selection device at the location to select the designated target.
-
-
5. The robot tele-operating method of claim 1 wherein the target designating step further comprises the steps of:
-
selecting an icon;
dragging the icon to a location in the user-perceptible image corresponding to the designated target; and
dropping the icon at the location in the user-perceptible image to select the designated target.
-
-
6. The robot tele-operating method of claim 1 wherein the user-perceptible image is provided on a computer monitor.
-
7. The robot tele-operating method of claim 1 wherein the user-perceptible image is provided on a portable liquid-crystal display.
-
8. The robot tele-operating method of claim 1 wherein the image information representative of the environment around the robot is provided by a video signal.
-
9. The robot tele-operating method of claim 8 wherein the video signal is transmitted via the Internet.
-
10. The robot tele-operating method of claim 1 wherein the imaging device is a camera.
-
11. A system for tele-operating a robot in an environment, comprising:
-
a user interface for controlling the tele-operation of the robot;
an imaging device associated with the robot for providing image information representative of the environment around the robot;
means for transmitting the image information to the user interface;
means for converting the image information to a user-perceptible image at the user interface;
means for designating a target in the user-perceptible image towards which the robot should move;
means for automatically converting the target designated in the user-perceptible image into a target location having x, y, and z coordinates in the environment of the robot; and
means for providing real-time instructions to the robot from the user interface to move from the robot'"'"'s current location in the environment to the x, y, and z coordinates of the target location in the environment. - View Dependent Claims (12, 13, 14, 15, 16, 17)
means for determining the current frame of reference of the robot in the environment;
means for determining the current frame of reference of the image device associated with the robot;
means for identifying one end of a click vector using the current frames of references of the robot and the imaging device;
means for determining the other end of the click vector based upon the view of the user-perceptible image and the designated target; and
means for projecting the click vector onto a three-dimensional global map of the robot'"'"'s environment to provide the x, y, and z coordinates of the target location.
-
-
13. The robot tele-operating system of claim 12 wherein the means for determining the other end of the click vector comprises:
-
means for determining the fixed angle and zoom of the imaging device associated with the robot that provides the view of the user-perceptible image;
means for determining the pixel size of the user-perceptible image;
means for identifying a pixel location in the user-perceptible image corresponding to the designated target; and
means for calculating the other end of the click vector using the fixed angle and zoom of the imaging device, the pixel size of the user-perceptible image, and the pixel location corresponding to the designated target.
-
-
14. The robot tele-operating system of claim 11 wherein the target designating means is a point-and-click selection device that includes a visible pointer movable within the user-perceptible image and means for activating the point-and-click selection device such that the location of the visible pointer within the user-perceptible image when the point-and-click selection device is activated designates the target towards which the robot should move.
-
15. The robot tele-operating system of claim 11 wherein the target designating means comprises:
-
means for selecting and dragging an icon to a location in the user-perceptible image corresponding to the designated target; and
means for dropping the icon at the location in the user-perceptible image to select the designated target.
-
-
16. The robot tele-operating system of claim 11 wherein the imaging device is a camera.
-
17. The robot tele-operating system of claim 11 wherein the image-information transmitting means is the Internet.
-
18. A graphical user interface for tele-operating a robot in an environment, comprising:
-
a display device for providing a user-perceptible image representative of the environment around the robot;
means for designating a target in the user-perceptible image towards which the robot thould move; and
means for superimposing at least one projection in the user-perceptible image in correlation with movement of the target designating means;
wherein the at least one projection represents an area to which the robot may move prior to target designation and represents the area to which the robot will move upon target designation. - View Dependent Claims (19, 20, 21)
-
-
22. A graphical user interface for tele-operating a robot in an environment, comprising:
-
a display device for providing a user-perceptible image representative of the environment around the robot;
means for designating a target in the user-perceptible image towards which the robot should move; and
means for overlaying a floor plan grid on top of the user-perceptible image to provide an indication of relative distance of objects within the environment of the robot based upon the robot'"'"'s current position. - View Dependent Claims (23)
means for overlaying at least one rotation tape on top of the user perceptible image to aid in tele-operating the robot.
-
Specification