Adaptive user interface for semi-automatic operation

• US 8,977,407 B2
• Filed: 05/27/2009
• Issued: 03/10/2015
• Est. Priority Date: 05/27/2009
• Status: Expired due to Fees

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Dorneich, Michael Christian, Whitlow, Stephen, Rogers, William, Feigh, Karen, De Mers, Robert E.
• Primary Examiner(s): Keith, Jack W
• Assistant Examiner(s): Nguyen, Chuong P
• Application Number:
  US12/473,114
• Publication Number:
  US 20100305778A1
• Time in Patent Office: 2,113 Days
• Field of Search: 701/2, 701/3, 701/23, 701/24, 701/28, 701/200, 701/207, 701/213, 701/400, 701/408, 701/412, 701/418, 701/439, 348113-120, 348/143, 348/144, 348/211.99
• US Class Current: 701/2
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

A method is provided for semi-automatic operation of a portable control device for a remote-controlled, unmanned vehicle. The method includes the steps of monitoring parameters of an operational environment of the portable control device, switching from a manual operation mode to a semi-automatic operation mode in response to occurrence of predetermined criteria within the operational environment, and presenting a semi-automatic operation graphical user interface to a user of the portable control device. The semi-automatic operation graphical user interface includes a reduced set of user interfaces for the semi-automatic operation mode presented by the portable control device.

Claims

1. A method for controlling a portable control device for an unmanned vehicle, the portable control device configured to selectively provide manual control of the unmanned vehicle in a manual operation mode or semi-automatic control of the unmanned vehicle in a semi-automatic operation mode, the method comprising the steps of:

monitoring parameters of an operational environment of the portable control device;

presenting, when in the manual operation mode, a manual operation graphical user interface to a user of the portable control device, wherein the manual operation graphical user interface comprises a first number of user interfaces for the manual operation mode presented by the portable control device;

automatically switching the portable control device, by the portable control device, from the manual operation mode to the semi-automatic operation mode in response to a measured and weighted combination of detected inputs and the monitored parameters within the operational environment; and

presenting, when in the semi-automatic operation mode, a semi-automatic operation graphical user interface to a user of the portable control device, wherein the semi-automatic operation graphical user interface comprises a second number of user interfaces for the semi-automatic operation mode presented by the portable control device, wherein the first number of user interfaces is greater than the second number of user interfaces.

2. The method in accordance with claim 1 wherein the step of monitoring parameters of the operational environment includes the step of determining one of movement of the portable control device, noise within the operational environment of the portable control device, user body position, user operation of the portable control device within a predetermined period of time, or entry of a user semi-automatic mode input.

3. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting one or more user input devices to the user of the portable control device, wherein at least one of the one or more user input devices comprise one or more of low dexterity input devices.

4. The method in accordance with claim 3 wherein the step of presenting the one or more user input devices comprises the step of presenting the one or more user input devices to the user of the portable control device, wherein the one or more of low dexterity input devices comprise one or more quick button input devices performing an input command selected from a set of limited action input commands comprising move now, stop now, follow user, follow an identified object, go to an identified location, travel in an identified direction, search an identified object, and search within an identified radius.

5. The method in accordance with claim 1 further comprising the step of the user of the portable control unit adjusting the weighted combination to provide means for calculating an adjusted weighted combination, and wherein the step of switching to the semi-automatic operation mode comprises the step of switching from the manual operation mode to the semi-automatic operation mode in response to the adjusted weighted combination of the detected inputs and the monitored parameters detected and monitored within the operational environment.

6. The method in accordance with claim 1 further comprising the step of adjusting the weighted combination in response to a learning function learning from the monitored parameters over time to provide means for calculating an adjusted weighted combination, and wherein the step of switching to the semi-automatic operation mode comprises the step of switching from the manual operation mode to the semi-automatic operation mode in response to the adjusted weighted combination of the detected inputs and the monitored parameters detected and monitored within the operational environment.

7. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting a graphical map view depicting at least a location of the unmanned vehicle, wherein the graphical map view is determined in response to the operational environment of the portable control device.

8. The method in accordance with claim 1 wherein the step of presenting the semi-automatic operation graphical user interface comprises the step of presenting a display of video feed view generated in response to visual signals received from the unmanned vehicle, the video feed view displayed determined in response to the operational environment of the portable control device.

9. A portable control device for an unmanned vehicle, the portable control device comprising:

a graphic user interface (GUI) for presenting information to a user of the portable control device and for receiving user inputs thereby; and

a controller coupled to the GUI and controlling the operational mode of the portable control device, wherein the controller is configured to;

monitor an operational environment of the portable control device;

automatically switch the portable control device from a manual operation mode to a semi-automatic operation mode in response to a measured and weighted combination of detected inputs and the monitored parameters within the operational environment of the portable control device;

provide GUI operational signals to the GUI for presenting a manual operation graphical user interface to the user while in the manual operation mode, wherein the manual operation graphical user interface comprises a first number of user interfaces for the manual operation mode presented by the portable control device; and

provide GUI operational signals to the GUI for presenting a semi-automatic operation graphical user interface to the user while in the semi-automatic operation mode, wherein the semi-automatic operation graphical user interface comprises a second number of user interfaces for the semi-automatic operation mode presented by the portable control device,

wherein the first number of user interfaces is greater than the second number of user interfaces.

10. The portable control device in accordance with claim 9 further comprising an environmental sensor for generating a signal in response to detecting environmental changes within the operational environment of the portable control device, wherein the controller is coupled to the environmental sensor to monitor the environmental changes as one of the predetermined criteria within the operational environment of the portable control device in response to the signal from the environmental sensor.

11. The portable control device in accordance with claim 10 wherein the environmental sensor generates the signal in response to detecting environmental changes within the operational environment, the environmental changes selected from the group of environmental changes comprising a change in noise level within the operational environment, movement of the portable control unit, and a change in a body position of an operator of the portable control unit.

12. The portable control device in accordance with claim 9 further comprising a workload measuring module which generates a user workload signal in response to detection of user operation of the portable control device within a predetermined period of time, wherein the controller is coupled to the workload measuring module to monitor user workload as one of the predetermined criteria within the operational environment of the portable control device.

13. The portable control device in accordance with claim 9 wherein the controller switches from the manual operation mode to the semi-automatic operation mode in response to a user semi-automatic mode input received from the GUI.

14. The portable control device in accordance with claim 13 wherein the controller presents a request for approval of mode switching to the user, the controller switching from the manual operation mode to the semi-automatic operation mode in response to a user semi-automatic mode approval input received from the GUI.

15. The portable control device in accordance with claim 9 wherein at least one of the reduced set of user interfaces for the semi-automatic operation graphical interface is a low dexterity input device.

16. The portable control device in accordance with claim 9, wherein at least one of the reduced set of user interfaces for the semi-automatic operation graphical interfaces is selected from a set of input devices comprising a low dexterity input device, a move now limited action input device, a stop now limited action input device, a follow user limited action input device, a follow an identified object limited action input device, a go to an identified location limited action input device, a travel in an identified direction limited action input device, a search an identified object limited action input device, and a search within an identified radius limited action input device.

17. The portable control device in accordance with claim 9 further comprising communication circuitry for receiving operational parameter signals from the unmanned vehicle, wherein the controller provides GUI operational signals to the GUI for presenting a graphical map view depicting at least a location of the unmanned vehicle as one of the reduced set of user interfaces for the semi-automatic operation graphical interface, wherein the graphical map view is determined in response to one of the operational parameter signals and the operational environment of the portable control device.

18. The portable control device in accordance with claim 9, further comprising communication circuitry for receiving visual signals from the unmanned vehicle, wherein at least one of the reduced set of user interfaces for the semi-automatic operation graphical interfaces is a video feed view from the unmanned vehicle, wherein the video feed view is determined in response to the visual signals and the operational environment of the portable control device.

19. A surveillance system comprising:

an unmanned vehicle comprising;

communication circuitry configured to receive operational control signals and transmitting operational parameter signals and visual signals;

a movement control module configured to control movement of the unmanned vehicle in response to movement control signals and to generate a first operational parameter signal in response to operation of the movement control module;

an unmanned vehicle controller coupled to the movement control module and the communication circuitry, wherein the unmanned vehicle controller is configured to generate the movement control signals in response to a first operational control signal; and

a visual recording module configured to record visual signals representative of one or more views from the unmanned vehicle in response to visual control signals and to generate a second operational control signal in response to operation of the visual recording module,

wherein the unmanned vehicle controller is coupled to the visual recording module and is configured to generate the visual control signals in response to the second operational control signal; and

a portable control device coupled to the unmanned vehicle configured to provide the operational control signals thereto and to receive the operational parameter signals and the visual signals therefrom, the portable control device comprising;

communication circuitry configured to receive the operational parameter signals and the visual signals from the unmanned vehicle and to transmit the operational control signals to the unmanned vehicle;

a graphic user interface (GUI) configured to present visual information to a user of the portable control device and to receive user inputs thereby; and

a portable control device controller coupled to the GUI and configured to control the operational mode of the portable control device, wherein the portable control device controller is configured to;

monitor an operational environment of the portable control device;

automatically switch the portable control device from a manual operation mode to a semi-automatic operation mode in response to measured and weighted combination of detected inputs and monitored parameters within the operational environment of the portable control device

provide GUI operational signals to the GUI for presenting a manual operation graphical user interface to the user while in the manual operation mode, wherein the manual operation graphical user interface comprises a first number of user interfaces for the manual operation mode presented by the portable control device; and

provide GUI operational signals to the GUI for presenting a semi-automatic operation graphical user interface to the user while in the semi-automatic operation mode, wherein the semi-automatic operation graphical user interface comprises a second number of user interfaces for the semi-automatic operation mode presented by the portable control device, wherein the first number of user interfaces is greater than the second number of user interfaces.

20. The surveillance system in accordance with claim 19 wherein the unmanned vehicle further comprises a Global Positioning System (GPS) receiver configured to generate location signals corresponding to a location of the unmanned vehicle and to provide the location signals to a surveillance vehicle controller, the surveillance vehicle controller configured to provide the location signals to the communication circuitry of the unmanned vehicle as a third operational parameter signal, and wherein the portable control device controller provides GUI operational signals to the GUI for presenting a graphical map view depicting at least the location of the unmanned vehicle as one of the reduced set of user interfaces for the semi-automatic operation graphical user interface, wherein the graphical map view is determined in response to the location signals received by the communication circuitry of the portable control device as one of the operational parameter signals and the operational environment of the portable control device.