Remote control system

• US 6,633,800 B1
• Filed: 01/31/2001
• Issued: 10/14/2003
• Est. Priority Date: 01/31/2001
• Status: Expired due to Fees

• Current Assignee: Macdonald Dettwiler and Associates Limited
• Original Assignee: Ainsworth Inc. (GDI Integrated Facility Services, Inc.)
• Inventors: Ward, Robert S., Law, James M., Bloomquist, Leif A.
• Primary Examiner(s): ARTHUR JEANGLAUDE, GERTRUDE
• Application Number:
  US09/774,014
• Time in Patent Office: 986 Days
• Field of Search: 701/1, 701/2, 701/33, 701/36, 701/41, 180/167, 180/168, 180/169, 348/114, 348/118, 348/119
• US Class Current: 701/2
• CPC Class Codes: B60W 50/02 Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246

Abstract

An apparatus and method for a control unit which allows for autonomous, manual and tele-operation of mining vehicles. The control unit has a robust system design to withstand the harsh environment of underground mines. The control unit allows a tele-operator, in a remote tele-operator station, to use image and operational data, joysticks and foot pedals to remotely control the mining vehicle. In another aspect, the control unit provides safety features such as supervising its operation for operational errors and providing status, warning and error information to the tele-operator station.

Claims

1. A control unit, for mounting with a machine, for enabling control thereof from a tele-operator station via a communications channel, the control unit comprising:

(1) a control input for receiving control signals from the tele-operator station;

(2) a digital processing means connected to said control input for processing said control signals and generating commands;

(3) a control output for providing said commands to the machine;

(4) a second input connected to said digital processing means for receiving data from sensors on the machine indicative of the location and status of the machine; and

,

(5) a data output for providing visual data to the tele-operator station;

wherein, the control unit is adapted to operably switch between a manual mode in which the machine is controlled locally by controls on the machine, an autonomous mode in which the machine is controlled by data received through the second input, and a tele-operation mode in which the machine is controlled remotely by said control signals based at least in part on said visual data.

2. A control unit at claimed in claim 1, wherein the digital processing means comprises a main processor and a supervisory processor, the supervisory processor being connected to the main processor for control thereat, and for interrupting operation of the main processor and disabling the machine when an error is detected in the operation of the control unit.

3. A control unit as claimed in claim 2, which includes at least one of:

the main processor being adapted to transmit a status signal to the supervisory processor and the supervisory processor being adapted to send random status inquiry signals to the main processor, wherein the supervisory processor interrupts the operation of the main processor if the status signal is not received for a predetermined period of time.

4. A control unit as claimed in claim 2, which includes a device interface for communication with external devices.

5. A control unit as claimed in claim 2, for use with a mobile machine, said mobile machine having a drive unit, for driving the mobile machine, and brakes, wherein the output of the control unit includes a command signal for actuating the brakes on the mobile machine, and wherein each of the main processor and the supervisory processor include separate means for sending a brake command signal, to effect actuation of the brakes.

6. A control unit as claimed in claim 5, that further comprises a guidance module for guiding the mobile machine.

7. A control unit, as claimed in claim 6, wherein the output of the control unit includes commands for controlling the drive and transmission unit of the mobile machine and the steering of the mobile machine, whereby to control the direction and speed of said mobile machine.

8. A control unit as claimed in claim 7, wherein the guidance module is adapted to provide autonomous operation of said mobile machine, and includes a centering means, a position determining means, a route following means, a Y-switching means and a route learning means.

9. A control unit as claimed in claim 2, which includes a separate device input/output module, for permitting communication With additional devices on the machine.

10. A contol unit as claimed in claim 2, which includes a critical input/output module, for controlling critical input and output signals, the critical input/output module being connected to and controlled by the main processor and the supervisory processor, whereby the critical input/ouput module is disabled if either one of the main processor and the supervisory processor is inoperative.

11. A control unit as claimed in claim 10, in combination with a mobile machine including:

a drive unit;

a steering actuation means;

a brake; and

a plurality of machine sensors for indicating machine status;

wherein the critical input/output module and the device input/output module are connected to the plurality of machine sensors, the drive unit, the brakes and the steering actuation means for providing diagnostic information.

12. A control unit as claimed in claim 10, wherein the control unit is adapted to disable the machine in a safe location to allow maintenance by service personnel.

13. A control unit as claimed in claim 1, wherein said second input includes separate inputs from a plurality of vision sensors and at least one microphone.

14. A control unit as claimed in claim 13, which includes at least one of:

a forward looking sensor and a rearward looking sensor adapted to provide visual data of the imnmediate location of said mobile machine, respectively, in front of the mobile machine and behind the mobile machine;

at least two vision sensors mounted to provide visual data indicative of the position of the mobile machine, for autonomous operation of said mobile machine.

15. A control unit as claimed in claim 14, wherein each vision sensor comprises a video camera.

16. A control unit as claimed in claim 1, in combination with a tele-operator station including at least one display screen, wherein said data output further provides information regarding the status of the machine, and said at least one display screen includes at least one tele-operation screen and at least one computer graphics screen, said at least one tele-operation screen displaying a video overlay image for displaying said visual data and messages about the operational status of the machine, and said computer graphics screen is adapted to provide said operational information.

17. A control unit as claimed in claim 1, further comprising an enclosure for housing a plurality of modules, said modules being adapted for providing a reduced number of connections therebetween and for reducing the size of said enclosure.

18. A control unit as claimed in claim 17, wherein the enclosure includes a pressurization port, for presurizing the enclosure with dry gas to a pressure in excess or ambient pressure.

19. A control unit as claimed in claim 18, wherein the enclosure is pressurized above a minimum pressure of 3 psi gauge.

20. A control unit as claimed in claim 18, wherein the enclosure is pressurized to a pressure of at least 8 psi gauge.

21. A control unit as claimed in claim 17, wherein said enclosure is sealed and comprises connectors all adapted to prevent ingress of moisture.

22. A control unit as claimed in claim 17, wherein said enclosure comprises at least one flexible circuit board having an interconnect for providing external and internal connections, said interconnect being adapted to reduce the hardware complexity of said control unit and said flexible circuit board being sufficiently flexible to withstand vibrations.

23. A control unit as claimed in claim 17, wherein said modules comprise:

a) a main processor;

b) a vision module connected to said main processor for processing visual information regarding the location of said machine;

c) a guidance module connected to said main processor and said vision module for providing autonomous operation of said machine;

d) a device interface and an inpuvoutput module both connected to said main processor and to various devices on said machine for monitoring and control thereof;

e) a supervisory module connected to each module for monitoring operability of said control unit; and

,

f) a power distribution module connected to each module for providing and regulating power thereto,

whereby said modules provide an integrated control system housed within said enclosure.

24. A control unit as claimed in claim 23, wherein the connectors are military-style connectors, whereby the connectors are used to connect the control unit to external devices.

25. A control unit as claimed in claim 1, which includes shock mountings for mounting the control unit on the machine.

26. A control unit as claimed in claim 1, which is adapted to allow a tele-operator station to control two or more machines.

27. A control unit as claimed in claim 1, wherein the machine is a vehicle.

28. A control unit as claimed in claim 27, which is adapted to allow a tele-operator station to control two or more vehicles.

29. A control unit as claimed in claim 1, wherein said control unit further has an external interface that allows for the updating of software, within the control unit, which directs the activity of the control unit.

30. A control unit as claimed in claim 29, wherein the external interface is an ethernet interface and the software updating can be done remotely.

31. A control unit as claimed in claim 1, wherein the control unit is adapted to switch between said modes without stopping the operation of said machine.

32. A control unit as claimed in claim 1, wherein said tele-operation mode allows for both line of sight and non-line of sight remote control.

33. A control unit, for mounting on a machine, for enabling control thereof from a tele-operator station via a communications channel, the control unit comprising:

(1) a control input for receiving control signals from the tele-operator station;

(2) a digital processing means connected to said control input for processing said signals and generating commands, said digital processing means comprising a main processor and a supervisory processor connected to the main processor for monitoring the main processor to ensure safe operation of the machine;

(3) a control output for providing said commands to the machine;

(4) a second input for receiving data from sensors on the machine indicative of the status of the machine; and

,

(5) an enclosure adapted to prevent ingress of contaminants, said digital processing means being located in said enclosure.

34. A control unit as claimed in claim 33, which includes at least one of:

the main processor providing periodic status signals to the supervisory processor indicating correct operation of the main processor; and

the supervisory processor randomly querying the status of the main processor, to ensure correct operation of the main processor.

35. A control unit as claimed in claim 33 or 34, for use with the machine, the machine comprising a vehicle including a drive unit, a steering actuation means and a brake, wherein the supervisory processor includes a separate input for control commands from the tele-operator station for controlling operation of the drive unit, the steering actuation means and the brakes of the vehicle, wherein the supervisory processor ensures that the main processor correctly acts on these control signals and wherein the supervisory processor includes a separate brake means, for applying the brakes to the vehicle, if any error in operation of the main processor is detected.

36. A control unit as claimed in claim 33, wherein the control unit is adapted to operably switch between a manual mode in which the machine is controlled locally by controls on the machine, an autonomous mode in which the machine is controlled by data received through the second input, and a tele-operation mode in which the machine is controlled remotely from the tele-operator station by said control signals received through the control input, wherein said tele-operation mode allows for both line of sight and non-line of sight remote control of said machine.

37. A control unit as claimed in claim 33, wherein said control unit has a data output for providing visual data regarding the location of the machine and operational information regarding the status of the machine to the tele-operator station.

38. A control unit as claimed in claim 33, wherein said enclosure houses a plurality of modules, said modules being adapted for providing a reduced number of connections therebetween and for reducing the size of said enclosure.

39. A control unit as claimed in claim 38, wherein the enclosure includes a pressurization port, for pressurizing the enclosure with dry gas to a pressure in excess of ambient pressure.

40. A control unit as claimed in claim 39, wherein the enclosure is pressurized above a minimum pressure of 3 psi gauge.

41. A control unit as claimed in claim 39, wherein the enclosure is pressurized to a pressure of at least 8 psi gauge.

42. A control unit as claimed in claim 38, wherein said enclosure is sealed and comprises connectors all adapted to prevenit ingress of moisture.

43. A control unit as claimed in claim 42, wherein the connectors are military-style connectors, whereby the connectors are used to connect the control unit to external devices.

45. A control unit as claimed in claim 38, wherein said modules comprise:

a) a main processor;

b) a vision module connected to said main processor for processing visual information regarding the location of said machine;

c) a guidance module connected to said main processor and said vision module for providing autonomous operation of said machine;

d) a device interface and an inpuvoutput module both connected to said main processor and to various devices on said machine for monitoring and control thereof;

e) a supervisory module connected to each module for monitoring operability of said control unit; and

,

f) a power distribution module connected to each module for providing and regulating power thereto,

whereby said modules provide an integrated control system housed within said enclosure.

46. A control unit as claimed in claim 33, which includes shock mountings for mounting the control unit on the machine.

44. A control unit as claimed in claime 38, wherein said enclosure comprises at least one flexible circuit board having an interconnect for providing external and internal connections, said interconnect being adapted to reduce the hardware complexity of said control unit and said flexible circuit board being sufficiently flexible to withstand vibrations.

47. A method of controlling a machine from a remote tele-operator station via a communications channel, the method comprising:

(a) providing the machine with a control unit comprising a main processor and a supervisory processor connected to the main processor, the control unit providing modes of operation comprising manual operation, autonomous operation, and tele-operation;

(b) selecting a desired mode of operation;

(c) operating the machine in the desired mode of operation; and

,

(d) receiving visual data from the control unit at the tele-operator station enabling activity of the machine to be monitored in at least one the modes of operation and

(e) monitoring the main processor with the supervisory processor to ensure safe operation of the machine, and wherein at least one of the main processor provides periodic status signals to the supervisory processor to indicate correct operation of the main processor; and

the supervisory processor randomly queries the status of the main processor.

48. A method as claimed in claim 47, wherein said activity comprises remote control operation of said machine.

49. A method as claimed in claim 48, which includes, for selection of the tele-operation mode of operation, the further steps of:

(1) configuring the machine and the control unit to allow for tele-operation of the machine;

(2) sending a series of commands to the control unit from the tele-operator station, to direct activity of the machine; and

,

(3) monitoring said data to allow for the remote control of said machine.

50. A method as claimed in claim 48, wherein said visual data provides information about the immediate environment of the machine, data indicative of the operational status of the machine and messages about the condition of the machine.

51. A method as claimed in claim 50, which includes displaying the visual data on at least one video screen, providing the messages as an overlay image on the at least one video screen and providing the operational status data on at least one computer graphics screen.

52. A method as claimed in claim 51, in which the messages include at least one of an indication of the view shown, machine identification, warnings and error codes.

53. A method as claimed in claim 47, wherein the method further includes the step of updating software, within the control unit, which directs the activities of the control unit.

54. A method as in claim 47, wherein the machine is a vehicle.

55. A method as claimed in claim 54, wherein for selection of the autonomous mode operation, the method includes:

(1) selecting a path along which the vehicle is to travel;

(2) providing a guidance system for guiding the vehicle along the path; and

(3) guiding the venicle along said selected path.

56. A method as claimed in claim 55, which includes, providing for the vehicle to travel along a plurality of selected paths, and for each path, first causing the vehicle to traverse a path in the tele-operation mode and simultaneously recording operator inputs, and, in subsequent autonomous mode operation, providing for the control unit to use the operator inputs for travelling each path.

57. A method as claimed in claim 56, which includes continually monitoring the status of the main processor of the control unit via the supervisory processor, and, if an error is detected, sending an error signal to the tele-operator station and actuating brakes on the vehicle to stop the vehicle.

58. A method as claimed in claim 57, wherein the vehicle is disabled in a safe location to allow maintenance by service personnel.

59. A method as claimed in claim 47, which includes providing for placing the control unit in a shutdown mode when deactivated and for initially putting the control unit into a sleep mode when the control unit is turned on, and the method further including sending a control signal from the tele-operator station to the control unit to put the control unit into standby mode for operation.

60. A method as claimed in claim 59, which includes, for autonomous operation, switching the control unit into a guidance idle mode, and for tele-operation, switching the control unit into a tele-operation mode.

61. A method as claimed in claim 60, wherein the machine is a vehicle.

62. A method as claimed in claim 61, which includes enabling the control unit to be placed into a doze mode, to tempoarily disable the vehicle to enable personnel to access an area in which the vehicle is located.

63. A method as claimed in claim 60 or 62, which includes providing, for the autonomous mode of operation, the guidance idle mode, in which the control unit receives a guidance command and awaits further commands before causing the vehicle to move, and a guidance active mode in which an initial direction is selected, and in which the vehicle commences movement in the selected direction.

64. A control unit, for mounting on a machine, for enabling control thereof from a tele-operator station via a communications channel, the control unit comprising:

(1) a control input for receiving control signals from the tele-operator station;

(2) a digital processing means connected to said control input for procesesing said signals and generating commands;

(3) a control output for providing said commands to the machine; and

,

(4) a second input for receiving data from sensors on the machine indicative of the status of the machine,

wherein, said digital processing means comprises a main processor and a supervisory processor connected to the main processor for monitoring the main processor to ensure safe operation of the machine, and wherein at least one of;

the main processor provides periodic status signals to the supervisory processor to indicate correct operation of the main processor; and

the supervisory processor randomly queries the status of the main processor, to ensure correct operation of the main processor.