External microphone for an unmanned aerial vehicle

• US 9,598,182 B2
• Filed: 04/08/2016
• Issued: 03/21/2017
• Est. Priority Date: 05/11/2015
• Status: Expired due to Fees

• Current Assignee: LR Acquisition LLC
• Original Assignee: Lily Robotics, Inc.
• Inventors: Bradlow, Henry W., Balaresque, Antoine
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Nolan, Peter D
• Application Number:
  US15/094,796
• Publication Number:
  US 20160332747A1
• Time in Patent Office: 347 Days
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor
• CPC Class Codes: H04R 1/086 Protective screens, e.g. all weather or wind screens
• CPC Class Codes: H04R 2420/01 Input selection or mixing for amplifiers or loudspeakers for hearing aids H04R25/43
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A videography drone can communicate with a microphone device. The videography drone can receive spatial information and audio data from a remote microphone device (e.g., a remote tracker, a mobile device running a drone control application, and/or a standalone audio recording device separate from the videography drone without drone control functionalities). The videography drone can utilize the spatial information to navigate the videography drone to follow the remote microphone device. The videography drone can stitch a video segment captured by its camera with an audio segment from the received audio data to generate an audio/video (A/V) segment. The stitching can be performed by matching spatial or temporal information (e.g., from the received spatial information) associated with the audio segment against spatial or temporal information associated with the video segment.

Claims

1. A videography drone comprising:

a spatial information sensor configured to continuously determine and update spatial locations of the videography drone;

a camera configured to capture video data, wherein the video data includes at least a video segment decorated by a video segment timestamp of when the video segment is captured and a video segment spatial coordinate from the spatial information sensor of where the video segment is captured;

a network interface configured to communicate, wirelessly, with a remote control device, wherein the network interface is configured to receive audio data and spatial location data from the remote control device in an open-ended stream, wherein the audio data includes an audio segment associated with an audio segment spatial coordinate from the spatial location data and an audio segment timestamp;

a flight system configured to navigate the videography drone based at least on the spatial locations from the spatial information sensor and the received spatial location data from the remote control device;

a microphone to record background noise data; and

a processor configured to

filter the background noise data from the received audio data, and

generate an audio/video (A/V) segment at least from aligning the video segment and the audio segment,

wherein said aligning is based at least on matching the video segment spatial coordinate against the audio segment spatial coordinate or matching the video segment timestamp against the audio segment timestamp.

2. The videography drone of claim 1, wherein the processor is configured to generate the A/V segment while the videography drone is in flight.

3. The videography drone of claim 1, wherein the spatial information sensor is an accelerometer, a global positioning system (GPS) module, a motion detector, a gyroscope, a cellular triangulation module, an inertial sensor, or any combination thereof.

4. A method of operating a videography drone comprising:

capturing video data with a camera of the videography drone, wherein the video data comprises an open-ended sequence of video segments;

recording background noise data with a microphone of the videography drone;

receiving spatial location data and audio data from a microphone device separate from the videography drone, wherein said receiving includes receiving an open-ended sequence of spatial coordinates and an open-ended sequence of audio segments from the microphone device;

filtering the background noise data from the received audio data;

navigating the videography drone based at least on the spatial location data; and

synchronizing the received audio data with the captured video data by stitching at least an audio segment of the audio data with a video segment of the video data, and wherein said stitching is based on at least matching a first spatial coordinate associated with the audio segment from the microphone device with a second spatial coordinate associated with the video segment.

5. The method of claim 4, further comprising:

tracking a spatial location of the videography drone; and

said navigating is based at least on comparing the spatial location data from the microphone device to the tracked spatial location of the videography drone.

6. The method of claim 4, further comprising:

identifying the second spatial coordinate as a spatial location of the videography drone when the video segment is taken; and

associating the second spatial coordinate with the video segment.

7. The method of claim 4, wherein said synchronizing includes combining the captured video data and the received audio data in an audio/video (A/V) file stored in a persistent data memory of the videography drone.

8. The method of claim 4, wherein said synchronizing is performed in real-time as the video segment is captured and the audio segment is received.

9. The method of claim 4, wherein said synchronizing is performed continuously as an additional video segment is captured and an additional audio segment is received.

10. The method of claim 4, wherein said synchronizing is performed asynchronously from when the video segment is captured and from when the audio segment is received.

11. The method of claim 4, wherein said synchronizing is based at least on matching a first timestamp of the video segment to a second timestamp of the audio segment within a preset tolerance range.

12. The method of claim 11, wherein the first timestamp and the second timestamp are global positioning system (GPS) timestamps.

13. The method of claim 4, further comprising:

analyzing the audio data from the microphone device to select a voice command by matching the audio data against one or more preset voice patterns associated with one or more preset voice commands; and

executing the selected voice command on the videography drone.

14. The method of claim 4, further comprising analyzing the audio data to identify an audio pattern event and executing a preset action in response to identifying the audio pattern event.

15. The method of claim 14, wherein the preset action includes stitching the video segment and the audio segment differently than previously before the preset action is executed.

16. The method of claim 14, wherein the preset action includes navigating the videography drone differently than previously before the preset action is executed.

17. The method of claim 14, wherein the audio pattern event is a high noise volume event.

18. A method of operating a remote control device, comprising:

establishing a wireless connection between the remote control device and a videography drone;

capturing audio data via a microphone on the remote control device;

determining location data associated with the remote control device utilizing a spatial information sensor of the remote control device;

sending, continuously, an open-ended stream of the location data and the audio data from the remote control device to the videography drone via the wireless connection,

wherein the audio data is decorated with location-based metadata based on the location data synchronized to when the audio data is captured;

wherein the audio data is decorated with one or more timestamps synchronized to when the audio data is captured;

capturing background noise data by a microphone on the videography drone; and

filtering the background noise data from the audio data sent by the remote control device via the open-ended stream by a processor of the videography drone.

19. The method of claim 18, wherein the microphone device is a general-purpose mobile device configured by a drone control application with a user interface implemented on a touch screen, an application-specific wearable tracker, or a standalone microphone device without drone control capability.