Method and system for controlling a remote vehicle

• US 20080027591A1
• Filed: 07/16/2007
• Published: 01/31/2008
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Jones, Chris, Lenser, Scott
• Granted Patent: US 8,577,538 B2
• Time in Patent Office: Days
• US Class Current: 701/2
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

A system for controlling more than one remote vehicle. The system comprises an operator control unit allowing an operator to receive information from the remote vehicles and send commands to the remote vehicles via a touch-screen interface, the remote vehicles being capable of performing autonomous behaviors using information received from at least one sensor on each remote vehicle. The operator control unit sends commands to the remote vehicles to perform autonomous behaviors in a cooperative effort, such that high-level mission commands entered by the operator cause the remote vehicles to perform more than one autonomous behavior sequentially or concurrently. The system may perform a method for generating obstacle detection information from image data received from one of a time-of-flight sensor and a stereo vision camera sensor.

Claims

1. A portable system for controlling a remote vehicle, the system comprising:

an operator control unit allowing an operator to receive information from the remote vehicle and send commands to the remote vehicle via a touch-screen interface, the remote vehicle being capable of performing autonomous behaviors using information received from at least one sensor on the remote vehicle,

wherein the operator control unit sends commands to the remote vehicle to perform autonomous behaviors, such that high-level mission commands entered by the operator cause the remote vehicle to perform more than one autonomous behavior sequentially or concurrently, and low-level teleoperation commands entered by the operator cause non-autonomous operation of the remote vehicle,

wherein the operator control unit displays remote vehicle information to the operator, allowing the operator to monitor a status and progress of the remote vehicle,

wherein the at least one sensor comprises a range finding system and the remote vehicle performs simultaneous localization and mapping prior to beginning other autonomous behaviors,

wherein the remote vehicle sends data to the operator, including a map generated during simultaneous localization and mapping and its position on the map, and

wherein the remote vehicle sends information regarding its position on the map at predetermined intervals of time.

5. The system of claim 1, wherein the autonomous behaviors include automatic flipper deployment and one or more of:

a follow-path behavior;

an obstacle detection and avoidance behavior;

a follow-waypoints behavior;

a follow-street behavior; and

a follow-perimeter behavior.

2-4. -4. (canceled)

6. A portable system for controlling more than one remote vehicle, the system comprising:

an operator control unit allowing an operator to receive information from the remote vehicles and send commands to the remote vehicles via a touch-screen interface, the remote vehicles being capable of performing autonomous behaviors using information received from at least one sensor on each remote vehicle,

wherein the operator control unit sends commands to the remote vehicles to perform autonomous behaviors in a cooperative effort, such that high-level mission commands entered by the operator cause the remote vehicles to perform more than one autonomous behavior sequentially or concurrently, and

wherein the operator control unit displays remote vehicle information to the operator, allowing the operator to monitor a status and progress of each remote vehicle.

7. The system of claim 6, wherein the at least one sensor comprises a range-finding system and a remote vehicle having a range-finding system performs simultaneous localization and mapping prior to the remote vehicles beginning other autonomous behaviors,

wherein the remote vehicle having a range-finding system sends data to the operator control unit, the data comprising a map generated during simultaneous localization and mapping and the remote vehicle'"'"'s position on the map, and

wherein the remote vehicle or the operator control unit sends the map to each remote vehicle, and information regarding each remote vehicle'"'"'s position on the map is sent to the operator control unit at predetermined intervals of time.

8-10. -10. (canceled)

11. A method for controlling more than one remote vehicle, the method comprising:

starting a portable operator control unit and a remote vehicle controllable by the operator control unit;

teleoperating the remote vehicle with the operator control unit to perform simultaneous localization and mapping;

sending data from the remote vehicle to the operator control unit, the data including a map generated during simultaneous localization and mapping;

starting other remote vehicles and sending data to the started remote vehicles including the map generated during simultaneous localization and mapping, each of the started remote vehicles localizing itself relative to the map;

sending data from the started remote vehicles to the operator control unit, the data including each remote vehicle'"'"'s initial localization information and additional localization information that is updated at predetermined intervals;

displaying data sent to the operator control unit to an operator; and

inputting operator commands via a touch-screen interface of the operator control unit to control the started remote vehicles.

12. The method of claim 11, wherein the started remote vehicles are controlled using high-level mission commands entered by the operator, the high-level mission commands causing the remote vehicle to perform more than one autonomous behavior sequentially or concurrently.

13. The method of claim 12, wherein the autonomous behaviors include only mission-specific behaviors.

14. The method of claim 11, wherein the operator commands include both high-level mission commands and low-level teleoperation commands.

15. A system for controlling one or more remote vehicles, the system including a portable operator control unit with a touch-screen user interface comprising:

an initial screen including a map view window that facilitates operator entry of high-level mission commands to one or more remote vehicles;

a remote vehicle selection/detection window allowing the operator to see which remote vehicles have been detected by the operator control unit and select among those vehicles to display a detailed window for the selected remote vehicle, the detailed window including status information regarding the selected remote vehicle; and

a button or icon for launching a control application including the initial screen and the remote vehicle selection/detection window,

wherein the map view window displays a map of a remote vehicle environment.

16. The system of claim 15, wherein the detailed window allows the operator to teleoperate a selected remote vehicle.

17. The system of claim 15, wherein each detected remote vehicle appears as an icon on the map of the remote vehicle environment, the icon indicating the real time location of the remote vehicle within the remote vehicle environment.

18. The system of claim 17, wherein each remote vehicle icon is a different color.

19. The system of claim 18, wherein each remote vehicle is represented by a button in the remote vehicle selection/detection window, the button for each remote vehicle matching the icon color for that vehicle.

20. The system of claim 15, wherein the map view window allows the operator to set navigational goals for remote vehicles.

21. The system of claim 15, wherein the map view window shows an intended path for remote vehicles.

22. The system of claim 15, wherein the map view window shows a status halo for remote vehicles.

23. A method for generating an occupancy grid map from image data received from one of a time-of-flight sensor and a stereo vision camera sensor, the method comprising:

determining a direction for each pixel of the image data from the sensor and combining the direction with a detected-potential obstacle depth reading for that pixel;

using the combined direction and depth information to plot points, sequentially, for each column of the image data, each point representing a distance of a detected potential obstacle from the remote vehicle and a direction of the detected potential object;

creating one or more best-fit lines from a predetermined number of sequential plotted points;

determining the slope of each best-fit line, and determining the existence of an obstacle if the slope of the best-fit line is above a predetermined threshold slope;

creating a one-dimensional set of values, derived from the best-fit lines, representing the distance to any obstacle detected by the sensor and indexed by angle from the platform;

creating the occupancy grid map from the one-dimensional set of values by incrementing a cell of the occupancy grid map if an obstacle was detected within that cell; and

adding a location of the remote vehicle to the occupancy grid map.

24. The method of claim 23 wherein, when creating the one-dimensional set of values, a value of infinity is used when no obstacle is detected.

25. The method of claim 23, further comprising downsampling the one-dimensional set of values to a predetermined number of values.

26. The method of claim 23, wherein creating the occupancy grid map from the one-dimensional set of values further comprises decrementing a cell of the occupancy grid map if no obstacle was detected within that cell.

27. The method of claim 23, wherein more recent information is weighted to represent its greater importance in creating the occupancy grid map.

28. The method of claim 27, wherein the time-of-flight sensor is a 3D sensor that obtains a depth for each cell of a 2D image.

29. A method for generating obstacle detection information from image data received from one of a time-of-flight sensor and a stereo vision camera sensor, the method comprising:

determining a direction for each pixel of the image data from the sensor and combining the direction with a detected potential obstacle depth reading for that pixel;

using the combined direction and depth information to plot points, sequentially, for each column of the image data, each point representing a distance of a detected potential obstacle from the remote vehicle and a direction of the detected potential object;

creating one or more best-fit fines from a predetermined number of sequential plotted points;

determining the slope of each best-fit line, and determining the existence of an obstacle if the slope of the best-fit line is above a predetermined threshold slope; and

creating a one-dimensional set of values, derived from the best-fit lines, representing the distance to any obstacle detected by the sensor and indexed by angle from the platform.

31. The method of claim 29, further comprising downsampling the one-dimensional set of values to a predetermined number of values.

30. (canceled)

32. A method for controlling one or more remote vehicles with a portable device including a touching a screen, the method comprising:

launching a control application including an initial-screen allowing touch navigation to a map view window, a remote vehicle availability window, and one or more remote vehicle detail windows;

selecting at least one remote vehicle to control via touch-screen manipulation of the remote vehicle availability window;

issuing high-level commands for at least one selected remote vehicle via touch screen manipulation of the map view window; and

issuing low-level teleoperation commands for at least one selected remote vehicle via touch-screen manipulation of a remote vehicle detail window.

34. The method of claim 32, wherein the map view window includes a map of the remote vehicle'"'"'s environment and the remote vehicle'"'"'s position on the map, and at least some of the high-level commands are issued by touching areas of the map to which the remote vehicle should navigate, and

wherein the may view window additionally displays a path that the remote vehicle has been instructed to follow and a status of the remote vehicle.

36. The method of claim 34, wherein the remote vehicle'"'"'s position is indicated by an icon and the status of the remote vehicle is indicated by a colored halo surrounding the icon.

37. The method of claim 32, wherein the remote vehicle availability window lists remote vehicles controllable via the control application.

38. The method of claim 32, wherein the remote vehicle detail window includes status information regarding the selected remote vehicle and allows touch-screen control of the selected remote vehicle, and

wherein the remote vehicle detail window includes a representation of the remote vehicle'"'"'s view of its surrounding area and touch-screen control of the remote vehicle is performed by touching the representation of the remote vehicle'"'"'s view of its surrounding area.

33. (canceled)

35. (canceled)

39-48. -48. (canceled)