Feedback to facilitate control of unmanned aerial vehicles (UAVs)

• US 10,031,518 B1
• Filed: 11/29/2016
• Issued: 07/24/2018
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Tofte, Nathan L., Harvey, Brian N.
• Primary Examiner(s): Beaulieu, Yonel
• Application Number:
  US15/363,541
• Time in Patent Office: 602 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/41422 located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4227 Providing Remote input by a user located remotely from the client device, e.g. at work
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Various techniques are described to facilitate controlling an unmanned aerial vehicle (UAV) and viewing feedback received from a UAV. A graphical user interface (GUI) is provided that allows a user to view a display window. The display window may indicate structures or portions of structures in which additional image data is desired by highlighting these portions within the display window. Static imagery may be leveraged to provide smooth and consistent feedback transitions. When a delay exists between the time the UAV sends live video data and the time it may be displayed in the GUI, the static images may be shown in the display window initially until the live video data may be displayed. The opacity of structures included in an initial display window may also transition to a greater opacity over time, with the live video eventually being displayed.

Claims

1. A computer-implemented method for controlling an unmanned aerial vehicle (UAV), comprising:

displaying, by one or more processors, video of a structure received from a UAV;

adjusting, by the one or more processors, the displayed video to indicate one or more portions of the structure where additional video needs to be acquired;

generating, by the one or more processors, one or more UAV commands in response to a user input indicating a selection of the one or more portions of the structure; and

transmitting, by the one or more processors, the one or more UAV commands to the UAV to cause the UAV to navigate to the one or more portions of the structure and to capture the additional video.

2. The computer-implemented method of claim 1, wherein the act of adjusting the displayed video includes highlighting the one or more portions of the structure where the additional video needs to be acquired.

3. The computer-implemented method of claim 1, further comprising:

generating, by the one or more processors, a partial three-dimensional wireframe of the structure from the video of the structure; and

determining the one or more portions of the structure where the additional video needs to be acquired based upon missing portions of the partial three-dimensional wireframe of the structure.

4. The computer-implemented method of claim 3, further comprising:

generating, by the one or more processors, a complete three-dimensional wireframe of the structure from the partial three-dimensional wireframe of the structure utilizing the additional video captured by the UAV corresponding to the missing portions of the structure.

5. The computer-implemented method of claim 4, further comprising:

displaying, by the one or more processors, video while the UAV remains stationary to facilitate virtual navigation of the structure utilizing the complete three-dimensional wireframe.

6. The computer-implemented method of claim 1, further comprising:

determining the one or more portions of the structure where the additional video needs to be acquired based upon a received command.

7. The computer-implemented method of claim 6, wherein the received command includes (i) a command received from an external computing device, or (ii) a command generated in response to another received user input.

8. A portable computing device, comprising:

a display configured to present video of a structure received from an unmanned aerial vehicle (UAV);

a graphics processing unit (GPU) configured to adjust the displayed video to cause the display to indicate one or more portions of the structure where additional video needs to be acquired;

a controller configured to generate one or more UAV commands in response to a user input indicating a selection of the one or more portions of the structure; and

a communication unit configured to transmit the one or more UAV commands to the UAV to cause the UAV to navigate to the one or more portions of the structure and to capture the additional video.

9. The portable computing device of claim 8, wherein the GPU is configured to adjust the displayed video by highlighting the one or more portions of the structure where the additional video needs to be acquired.

10. The portable computing device of claim 8, wherein the controller is further configured to generate a partial three-dimensional wireframe of the structure from the video of the structure, and to determine the one or more portions of the structure where the additional video needs to be acquired based upon missing portions of the partial three-dimensional wireframe of the structure.

11. The portable computing device of claim 10, wherein the controller is further configured to generate a complete three-dimensional wireframe of the structure from the partial three-dimensional wireframe of the structure utilizing the additional video captured by the UAV corresponding to the missing portions of the structure.

12. The portable computing device of claim 11, wherein the GPU is further configured to cause the display to present video while the UAV remains stationary to facilitate virtual navigation of the structure utilizing the complete three-dimensional wireframe.

13. The portable computing device of claim 8, wherein the controller is further configured to determine the one or more portions of the structure where the additional video needs to be acquired based upon a received command.

14. The portable computing device of claim 13, wherein the received command includes (i) a command received from an external computing device, or (ii) a command generated in response to another received user input.

15. A system, comprising:

an unmanned aerial vehicle (UAV) configured to capture video of a structure and to transmit the video as part of a video data transmission; and

a portable computing device configured to;

present video of a structure based upon the video data transmission received from the UAV,

adjust the presented video to indicate one or more portions of the structure where additional video needs to be acquired,

generate one or more UAV commands in response to a user input indicating a selection of the one or more portions of the structure, and

transmit the one or more UAV commands to the UAV,

wherein the UAV is further configured, in response to receiving the one or more UAV commands, to navigate to the one or more portions of the structure, to capture the additional video of the structure, and to transmit the additional video as part of an additional video data transmission.

16. The system of claim 15, wherein the portable computing device is configured to adjust the displayed video by highlighting the one or more portions of the structure where the additional video needs to be acquired.

17. The system of claim 15, wherein the portable computing device is further configured to generate a partial three-dimensional wireframe of the structure from the video of the structure, and to determine the one or more portions of the structure where the additional video needs to be acquired based upon missing portions of the partial three-dimensional wireframe of the structure.

18. The system of claim 17, wherein the portable computing device is further configured to generate a complete three-dimensional wireframe of the structure from the partial three-dimensional wireframe of the structure utilizing the additional video data received from the UAV corresponding to the missing portions of the structure.

19. The system of claim 18, wherein the portable computing device is further configured to present video while the UAV remains stationary to facilitate virtual navigation of the structure utilizing the complete three-dimensional wireframe.

20. The system of claim 15, wherein the portable computing device is further configured to determine one or more portions of the structure where additional video needs to be acquired based upon a received command, the received command including (i) a command received from an external computing device, or (ii) a command generated in response to another received user input.