User interface to facilitate control of unmanned aerial vehicles (UAVs)

• US 9,927,809 B1
• Filed: 10/30/2015
• Issued: 03/27/2018
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Tofte, Nathan L., Harvey, Brian N.
• Primary Examiner(s): Lin, Abby Y
• Application Number:
  US14/928,310
• Time in Patent Office: 879 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/41422 located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4227 Providing Remote input by a user located remotely from the client device, e.g. at work
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Various techniques are described to facilitate the control of an unmanned aerial vehicle (UAV). A graphical user interface (GUI) is provided that allows a user to control a UAV using familiar gestures. The GUI may include live video that is captured and transmitted by a UAV, which may be displayed in a display window on a portable computing device. Upon a user interacting with the display window, the video may be updated to display a new UAV perspective by changing a zoom level, panning, rotating, etc., based upon the type of user interaction. In response to the user interaction, a command may be generated and transmitted to the UAV, causing the UAV to navigate to a location to reflect the change in perspective. Upon the UAV moving to the new position, the live video data that is displayed may match the perspective indicated by the user gesture.

Claims

1. A computer-implemented method for controlling an unmanned aerial vehicle (UAV), comprising:

displaying, by one or more processors, live video received from a UAV overlaid onto a map image displayed within a display window;

receiving, by one or more processors, a first user input to update the overlaid position of the live video within the map image in response to a user interacting with the map image displayed in the display window;

adjusting, by one or more processors, the live video in response to the first user input to update the overlaid position of the live video within the map image displayed within the display window;

translating, by one or more processors, a change in the overlaid position of the live video with respect to the map image as an equivalent change in the UAV'"'"'s mapped position;

generating, by one or more processors, one or more UAV commands based upon the equivalent change in the UAV'"'"'s mapped position using the translated change in the-overlaid position of the live video with respect to the map image; and

transmitting, by one or more processors, the one or more UAV commands to the UAV.

2. The computer-implemented method of claim 1, wherein the first user input includes

a gesture to pan the overlaid position of the live video displayed within the display window.

3. The computer-implemented method of claim 1, wherein:

the first user input includes a user panning the overlaid position of the live video by a panning distance,

the act of translating the change in the position of the overlaid position of the live video with respect to the map image comprises;

translating the panning distance as a change in UAV geographical position, and the act of generating the one or more UAV commands comprises;

generating the one or more UAV commands to cause the UAV to change its geographical position as a result of the live video being panned by the panning distance.

4. The computer-implemented method of claim 1, further comprising:

receiving, by one or more processors, a second user input corresponding to the user changing a zoom level of the live video displayed in the display window.

5. The computer-implemented method of claim 4, further comprising:

translating the change in zoom level as a change in UAV altitude; and

generating one or more UAV commands to cause the UAV to change its altitude as a result of the change in zoom level.

6. The computer-implemented method of claim 1, further comprising:

receiving, by one or more processors, a third user input corresponding to the user selecting an object within the display window; and

generating, by one or more processors, one or more UAV commands to cause the UAV to perform an image analysis of the object based upon the third user input.

12. The computer-implemented method of claim 1, wherein the user interface is further configured to receive a third user input in response to the user selecting an object within the display window, and wherein the processor is further configured to generate the one or more UAV commands to cause the UAV to perform an image analysis of the object based upon the third user input.

7. A portable computing device, comprising:

an interactive display configured to display live video received from a UAV overlaid onto a map image displayed within a display window;

a user interface configured to receive first user input to update the overlaid position of the live video within the map image in response to a user interacting with the map image displayed in the display window;

a graphical processing unit (GPU) configured to adjust the live video in response to the user input to update the overlaid position of the live video within the map image displayed within the display window;

a processor configured to translate a change in the overlaid position of the live video with respect to the map image as an equivalent change in the UAV'"'"'s mapped position and to generate one or more UAV commands based upon the equivalent change in the UAV'"'"'s position using the translated change in the overlaid position of the live video with respect to the map image; and

a communication unit configured to transmit the one or more UAV commands to the UAV.

8. The portable computing device of claim 7, wherein the first user input includes a gesture to pan the overlaid position of the live video displayed in the display window.

9. The portable computing device of claim 7, wherein:

the first user input includes a user panning the overlaid position of the live video by a panning distance, and

the processor is further configured to translate the change in the overlaid position of the live video with respect to the map image by translating the panning distance as a change in UAV geographical position, and to generate the one or more UAV commands to cause the UAV to change its geographical position as a result of the live video being panned by the panning distance.

10. The portable computing device of claim 7, wherein the user interface is further configured to receive a second user input corresponding to the user changing a zoom level of the live video feed displayed in the display window.

11. The portable computing device of claim 10, wherein the processor is further configured to translate the change in zoom level as a change in UAV altitude, and to generate the one or more UAV commands to cause the UAV to change its altitude as a result of the change in zoom level.

13. A computer-implemented method for controlling an unmanned aerial vehicle (UAV), comprising:

displaying, by one or more processors, live video received from a UAV in a first display window;

displaying, by one or more processors, a map image in a second display window that includes the first display window, the first display window being superimposed over the second display window at a location on the map image that corresponds to the geographic location of the UAV while the live video is being captured;

receiving, by one or more processors a first user input to update the live video in the first display window in response to a user interacting with the first display window;

receiving, by one or more processors, a second user input corresponding to a user panning the map image displayed in the second display window to move the superimposed position of the first display window with respect to the map image;

adjusting, by one or more processors, the live video in response to the first user input to update the live video in the first display window;

translating, by one or more processors, (i) a change in the UAV'"'"'s perspective resulting from the live video being adjusted as an equivalent change in the UAV'"'"'s altitude, and (ii) a change in the superimposed position of the first display window with respect to the map image as an equivalent change in the UAV'"'"'s mapped position;

generating, by one or more processors, one or more UAV commands based upon the change in the UAV'"'"'s altitude and mapped position; and

transmitting, by one or more processors, the one or more UAV commands to the UAV.

14. The computer-implemented method of claim 13, wherein:

the first display window maintains a superimposed position relative to the edges of the second display window while the second display window is panned such that the superimposed position of the first display window is moved to an updated map position within the map image included in the second display window, and

the one or more UAV commands cause the UAV to move to a geographical position that corresponds to the updated mapped position within the map image.

15. The computer-implemented method of claim 13, further comprising:

receiving, by one or more processors, a third user input corresponding to the user interacting with the second display window to perform a zoom operation by selecting a portion of the map image within the second display window outside of the first display window; and

updating the second display window in response to the zoom operation while maintaining the live video in the first display window.

16. The computer-implemented method of claim 13, wherein the first user input includes the user interacting with the first display window to perform a zoom gesture, and further comprising:

updating the first display window to indicate a change in zoom level as a result of the change in the UAV'"'"'s perspective in response to the first user input.

17. The computer-implemented method of claim 16, wherein the act of translating the change in the UAV'"'"'s perspective comprises:

translating the change in zoom level as a change in the UAV'"'"'s altitude, and

wherein the act of generating the one or more UAV commands comprises;

generating one or more UAV commands to cause the UAV to change altitude as a result of the change in zoom level.

18. A portable computing device, comprising:

an interactive display configured to display (i) live video received from a UAV in a first display window, and (ii) a map image in a second display window that includes the first display window, the first display window being superimposed over the second display window at a location on the map image that corresponds to the geographic location of the UAV while the live video is being captured;

a user interface configured to receive (i) a first user input to update the live video in the first display window in response to a user interacting with the first display window, and (ii) a second user input corresponding to a user panning the map image displayed in the second display window to move the superimposed position of the first display window with respect to the map image;

a graphical display unit (GDU) configured to adjust the live video in response to the first user input to update the live video in the first display window;

a processor configured to translate (i) a change in UAV'"'"'s perspective resulting from the live video being adjusted as an equivalent change in the UAV'"'"'s altitude, and (ii) a change in the superimposed position of the first display window with respect to the map image as an equivalent change in the UAV'"'"'s mapped position to generate one or more UAV commands based upon the change in the UAV'"'"'s altitude and mapped position; and

a communication unit configured to transmit the one or more UAV commands to the UAV.

19. The portable computing device of claim 18, wherein:

the first display window maintains a superimposed position relative to the edges of the second display window while the second display window is panned such that the superimposed position of the first display window is moved to an updated map position within the map image included in the second display window, and

the one or more UAV commands cause the UAV to move to a geographical position corresponding to the updated mapped position within the map image.

20. The portable computing device of claim 18, wherein the user interface is further configured to receive a third user input corresponding to the user interacting with the second display window to perform a zoom operation by selecting a portion of the map image within the second display window outside of the first display window, and

wherein the GDU is further configured to update the second display window in response to the zoom operation while maintaining the live video in the first display window.

21. The portable computing device of claim 18, wherein the first user input includes the user interacting with the first display window to perform a zoom gesture, and

wherein the GDU is further configured to update the first display window to indicate a change in zoom level as a result of the change in the UAV'"'"'s perspective in response to the first user input.

22. The portable computing device of claim 21, wherein the processor is further configured to translate the change in zoom level as a change in the UAV'"'"'s altitude, and

wherein the processor is further configured to generate the one or more UAV commands to cause the UAV to change its altitude as a result of the change in zoom level.