ADAPTIVE USER INTERFACE FOR SEMI-AUTOMATIC OPERATION

• US 20100305778A1
• Filed: 05/27/2009
• Published: 12/02/2010
• Est. Priority Date: 05/27/2009
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Rogers, William, Whitlow, Stephen, De Mers, Robert E., Dorneich, Michael Christian, Feigh, Karen
• Granted Patent: US 8,977,407 B2
• Time in Patent Office: Days
• US Class Current: 701/2
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

A method is provided for semi-automatic operation of a portable control device for a remote-controlled, unmanned vehicle. The method includes the steps of monitoring parameters of an operational environment of the portable control device, switching from a manual operation mode to a semi-automatic operation mode in response to occurrence of predetermined criteria within the operational environment, and presenting a semi-automatic operation graphical user interface to a user of the portable control device. The semi-automatic operation graphical user interface includes a reduced set of user interfaces for the semi-automatic operation mode presented by the portable control device.

Claims

1. A method for semi-automatic operation of a portable control device for an unmanned vehicle, the method comprising the steps of:

monitoring parameters of an operational environment of the portable control device;

switching from a manual operation mode to a semi-automatic operation mode in response to a measured and weighted combination of detected inputs and the monitored parameters within the operational environment; and

while in the semi-automatic operation mode, presenting a semi-automatic operation graphical user interface to a user of the portable control device, wherein the semi-automatic operation graphical user interface comprises a reduced set of user interfaces for the semi-automatic operation mode presented by the portable control device.

2. The method in accordance with claim 1 wherein the step of monitoring parameters of the operational environment includes the step of determining one of movement of the portable control device, noise within the operational environment of the portable control device, user body position, user operation of the portable control device within a predetermined period of time, or entry of a user semi-automatic mode input.

3. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting one or more predetermined user input devices to the user of the portable control device, wherein at least a portion of the one or more predetermined user input devices comprise one or more of low dexterity input devices and quick button input devices.

4. The method in accordance with claim 3 wherein the step of presenting the one or more predetermined user input devices comprises the step of presenting the one or more predetermined user input devices to the user of the portable control device, wherein the at least a portion of the one or more predetermined user input devices comprise one or more quick button input devices performing an input command selected from a set of limited action input commands comprising move now, stop now, follow user, follow an identified object, go to an identified location, travel in an identified direction, search an identified object, and search within an identified radius.

5. The method in accordance with claim 1 further comprising the step of the user of the portable control unit adjusting the weighted combination to provide means for calculating an adjusted weighted combination, and wherein the step of switching to the semi-automatic operation mode comprises the step of switching from the manual operation mode to the semi-automatic operation mode in response to the adjusted weighted combination of the detected inputs and the monitored parameters detected and monitored within the operational environment.

6. The method in accordance with claim 1 further comprising the step of adjusting the weighted combination in response to a learning function learning from the monitored parameters over time to provide means for calculating an adjusted weighted combination, and wherein the step of switching to the semi-automatic operation mode comprises the step of switching from the manual operation mode to the semi-automatic operation mode in response to the adjusted weighted combination of the detected inputs and the monitored parameters detected and monitored within the operational environment.

7. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting a graphical map view depicting at least a location of the unmanned vehicle, wherein the graphical map view is determined in response to the operational environment of the portable control device.

8. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting a display of video feed view generated in response to visual signals received from the unmanned vehicle, the video feed view displayed determined in response to the operational environment of the portable control device.

9. A portable control device for an unmanned vehicle, the portable control device comprising:

a graphic user interface (GUI) for presenting information to a user of the portable control device and for receiving user inputs thereby; and

a controller coupled to the GUI and controlling the operational mode of the portable control device, wherein the controller monitors an operational environment of the portable control device and switches from a manual operation mode to a semi-automatic operation mode in response to a measured and weighted combination of detected inputs and the monitored parameters within the operational environment of the portable control device, the controller further providing GUI operational signals to the GUI for presenting a semi-automatic operation graphical user interface to the user while in the semi-automatic operation mode, wherein the semi-automatic operation graphical user interface comprises a reduced set of user interfaces for the semi-automatic operation mode presented by the portable control device.

10. The portable control device in accordance with claim 9 further comprising an environmental sensor for generating a signal in response to detecting environmental changes within the operational environment of the portable control device, wherein the controller is coupled to the environmental sensor to monitor the environmental changes as one of the predetermined criteria within the operational environment of the portable control device in response to the signal from the environmental sensor.

11. The portable control device in accordance with claim 10 wherein the environmental sensor generates the signal in response to detecting environmental changes within the operational environment, the environmental changes selected from the group of environmental changes comprising a change in noise level within the operational environment, movement of the portable control unit, and a change in a body position of an operator of the portable control unit.

12. The portable control device in accordance with claim 9 further comprising a workload measuring module which generates a user workload signal in response to detection of user operation of the portable control device within a predetermined period of time, wherein the controller is coupled to the workload measuring module to monitor user workload as one of the predetermined criteria within the operational environment of the portable control device.

13. The portable control device in accordance with claim 9 wherein the controller switches from the manual operation mode to the semi-automatic operation mode in response to a user semi-automatic mode input received from the GUI.

14. The portable control device in accordance with claim 13 wherein the controller presents a request for approval of mode switching to the user, the controller switching from the manual operation mode to the semi-automatic operation mode in response to a user semi-automatic mode approval input received from the GUI.

15. The portable control device in accordance with claim 9 wherein the controller provides GUI operational signals to the GUI for presenting one or more predetermined user input devices to the user of the portable control device as a portion of the reduced set of user interfaces for the semi-automatic operation graphical interface, and wherein at least a portion of the one or more predetermined user input devices comprise one or more of low dexterity input devices and limited action input devices.

16. The portable control device in accordance with claim 15 wherein the controller also provides GUI operational signals to the GUI for presenting one or more predetermined user input devices along with the video signal view from the unmanned vehicle as a portion of the reduced set of user interfaces for the semi-automatic operation graphical interface, and wherein at least a portion of the one or more predetermined user input devices comprise one or more input devices selected from a set of input devices comprising a low dexterity input device, a move now limited action input device, a stop now limited action input device, a follow user limited action input device, a follow an identified object limited action input device, a go to an identified location limited action input device, a travel in an identified direction limited action input device, a search an identified object limited action input device, and a search within an identified radius limited action input device.

17. The portable control device in accordance with claim 9 further comprising communication circuitry for receiving operational parameter signals from the unmanned vehicle, wherein the controller provides GUI operational signals to the GUI for presenting a graphical map view depicting at least a location of the unmanned vehicle as a portion of the reduced set of user interfaces for the semi-automatic operation graphical interface, wherein the graphical map view is determined in response to a portion of the operational parameter signals and the operational environment of the portable control device.

18. The portable control device in accordance with claim 9 further comprising communication circuitry for receiving visual signals from the unmanned vehicle, wherein the controller provides the portion of the visual signals to the GUI as a portion of the GUI operational signals for presenting a video feed view from the unmanned vehicle as a portion of the reduced set of user interfaces for the semi-automatic operation graphical interface, wherein the video feed view is determined in response to the portion of the visual signals and the operational environment of the portable control device.

19. A surveillance system comprising:

an unmanned vehicle comprising;

communication circuitry for receiving operational control signals and transmitting operational parameter signals and visual signals;

a movement control module for controlling movement of the unmanned vehicle in response to movement control signals and for generating a first portion of the operational parameter signals in response to operation of the movement control module;

an umanned vehicle controller coupled to the movement control module and the communication circuitry, wherein the unmanned vehicle controller generates the movement control signals in response to a first portion of the operational control signals; and

a visual recording module for recording visual signals representative of one or more views from the unmanned vehicle in response to visual control signals and for generating a second portion of the operational parameter signals in response to operation of the visual recording module,

wherein the unmanned vehicle controller is coupled to the visual recording module and generates the visual control signals in response to a second portion of the operational control signals; and

a portable control device coupled to the unmanned vehicle for providing the operational control signals thereto and receiving the operational parameter signals and the visual signals therefrom, the portable control device comprising;

communication circuitry for receiving the operational parameter signals and the visual signals from the unmanned vehicle and for transmitting the operational control signals to the unmanned vehicle;

a graphic user interface (GUI) for presenting visual information to a user of the portable control device and for receiving user inputs thereby; and

a portable control device controller coupled to the GUI and controlling the operational mode of the portable control device, wherein the portable control device controller monitors an operational environment of the portable control device and switches from a manual operation mode to a semi-automatic operation mode in response to measured and weighted combination of detected inputs and monitored parameters within the operational environment of the portable control device, the portable control device controller further providing GUI operational signals to the GUI for presenting a semi-automatic operation graphical user interface to the user while in the semi-automatic operation mode, wherein the semi-automatic operation graphical user interface comprises a reduced set of user interfaces for the semi-automatic operation mode presented by the portable control device.

20. The surveillance system in accordance with claim 19 wherein the unmanned vehicle further comprises a Global Positioning System (GPS) receiver for generating location signals corresponding to a location of the unmanned vehicle and providing the location signals to the surveillance vehicle controller, the surveillance vehicle controller providing the location signals to the communication circuitry of the unmanned vehicle as a third portion of the operational parameter signals, and wherein the wherein the portable control device controller provides GUI operational signals to the GUI for presenting a graphical map view depicting at least the location of the unmanned vehicle as a portion of the reduced set of user interfaces for the semi-automatic operation graphical user interface, wherein the graphical map view is determined in response to the location signals received by the communication circuitry of the portable control device as a portion of the operational parameter signals and the operational environment of the portable control device.