Electronic device for generating video data

• US 10,021,339 B2
• Filed: 12/01/2015
• Issued: 07/10/2018
• Est. Priority Date: 12/01/2015
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Taesu, Yun, Sungrack, Park, Min-Kyu, Kim, Heeman
• Primary Examiner(s): Anderson, II, James M
• Application Number:
  US14/956,252
• Publication Number:
  US 20170201714A1
• Time in Patent Office: 952 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 21/034 Automatic adjustment
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/911 for the suppression of noise H04N5/932 takes precedence
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04N 9/802 involving processing of the sound signal H04N9/806, H04N9/835 take precedence
• CPC Class Codes: H04R 2410/01 Noise reduction using microphones having different directional characteristics
• CPC Class Codes: H04R 2420/01 Input selection or mixing for amplifiers or loudspeakers for hearing aids H04R25/43
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 2430/01 Aspects of volume control, not necessarily automatic, in sound systems
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2430/23 Direction finding using a sum-delay beam-former
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An electronic device for generating video data is disclosed. The electronic device may include a communication unit configured to wirelessly receive a video stream captured by a camera, wherein the camera is located in an unmanned aerial vehicle. The electronic device may also include at least one sound sensor configured to receive an input sound stream. In addition, the electronic device may include an audio control unit configured to generate an audio stream associated with the video stream based on the input sound stream. Further, the electronic device may include a synthesizer unit configured to generate the video data based on the video stream and the audio stream.

Claims

1. An electronic device for generating video data, comprising:

communication circuitry configured to wirelessly receive a video stream captured by a camera, wherein the camera is located in an unmanned aerial vehicle;

at least one sound sensor configured to receive an input sound stream;

audio control circuitry configured to;

generate an audio stream associated with the video stream based on the input sound stream; and

adjust a gain applied to the audio stream to generate an adjusted audio stream, the gain adjusted based on a distance between the unmanned aerial vehicle and a location external to the unmanned aerial vehicle; and

synthesizer circuitry configured to generate the video data based on the video stream and the adjusted audio stream.

2. The electronic device of claim 1, wherein the electronic device comprises a controller configured to manually or automatically pilot the unmanned aerial vehicle, and wherein the location external to the unmanned aerial vehicle comprises a location in a target scene associated with the video stream.

3. The electronic device of claim 1, wherein the synthesizer circuitry comprises:

synchronization circuitry configured to synchronize the video stream and the adjusted audio stream to generate a synchronized video stream and a synchronized audio stream; and

combining circuitry configured to generate the video data based on the synchronized video stream and the synchronized audio stream.

4. The electronic device of claim 1, wherein the audio control circuitry is configured to selectively extract the audio stream from the input sound stream based on a direction of the camera in the unmanned aerial vehicle.

5. The electronic device of claim 4, wherein the audio control circuitry is configured to selectively extract the audio stream further based on at least one of an orientation or a spatial arrangement of the at least one sound sensor.

6. The electronic device of claim 1, wherein the communication circuitry is further configured to wirelessly receive one or more camera parameters and location information from the unmanned aerial vehicle, and wherein the audio control circuitry is further configured to selectively extract the audio stream from the input sound stream based on a location of the unmanned aerial vehicle indicated by the location information, a location or an orientation of the electronic device, and a focal distance of the camera, the focal distance of the camera included in the one or more camera parameters.

7. The electronic device of claim 1, wherein the video stream includes audio, and wherein the synthesizer circuitry is configured to generate the video data by combining the adjusted audio stream with the audio of the video stream.

8. The electronic device of claim 1, wherein the audio control circuitry is further configured to adjust, prior to generation of the video data, a level of noise suppression applied to audio of the video stream based on the distance.

9. The electronic device of claim 8, wherein the audio control circuitry is further configured to reduce the level of noise suppression applied to the audio of the video stream and to decrease the gain applied to the audio stream responsive to an increase in the distance.

10. The electronic device of claim 8, wherein the audio control circuitry is further configured to increase the level of noise suppression applied to the audio of the video stream and to increase the gain applied to the audio stream responsive to a decrease in the distance.

29. The electronic device of claim 1, wherein the communication circuitry is further configured to wirelessly receive a camera parameter from the unmanned aerial vehicle, and wherein the audio control circuitry is further configured to determine the distance between the unmanned aerial vehicle and the external location based on the camera parameter of the unmanned aerial vehicle.

30. The electronic device of claim 29, wherein the camera parameter comprises a focal distance of the camera.

11. A method for generating video data in an electronic device, comprising:

wirelessly receiving, by communication circuitry, a video stream captured by a camera, wherein the camera is located in an unmanned aerial vehicle;

receiving, by at least one sound sensor, an input sound stream;

generating, by a processor, an audio stream associated with the video stream based on the input sound stream;

adjusting, by the processor, a gain applied to the audio stream to generate an adjusted audio stream, the gain adjusted based on a distance between the unmanned aerial vehicle and a location external to the unmanned aerial vehicle; and

generating, by the processor, the video data based on the video stream and the adjusted audio stream.

12. The method of claim 11, further comprising sending one or more instructions from the communication circuitry to the unmanned aerial vehicle to manually or automatically control the unmanned aerial vehicle.

13. The method of claim 11, wherein generating the video data comprises:

synchronizing the video stream and the adjusted audio stream to generate a synchronized video stream and a synchronized audio stream; and

combining the synchronized video stream and the synchronized audio stream to generate the video data.

14. The method of claim 11, wherein generating the audio stream comprises selectively extracting the audio stream from the input sound stream based on a direction of the camera in the unmanned aerial vehicle, an orientation of the at least one sound sensor in the electronic device, and a spatial arrangement of the at least one sound sensor in the electronic device.

15. The method of claim 11, further comprising wirelessly receiving, by the communications circuitry, one or more camera parameters and location information from the unmanned aerial vehicle, wherein generating the audio stream comprises selectively extracting the audio stream from the input sound stream based on a location of the unmanned aerial vehicle indicated by the location information, a location or an orientation of the electronic device, and a focal distance of the camera, the focal distance of the camera indicated by the one or more camera parameters.

16. The method of claim 11, wherein the video stream includes audio, and wherein generating the video data comprises combining the adjusted audio stream with the audio of the video stream.

17. The method of claim 11, further comprising, prior to generating the video data:

adjusting a level of noise suppression applied to audio of the video stream based on the distance.

18. The method of claim 17, wherein the level of noise suppression applied to the audio of the video stream is reduced and the gain applied to the audio stream is decreased responsive to an increase in the distance.

19. The method of claim 17, wherein the level of noise suppression applied to the audio of the video stream is increased and the gain applied to the audio stream is increased responsive to a decrease in the distance.

28. The method of claim 11, further comprising:

wirelessly receiving, via the communication circuitry, a camera parameter from the unmanned aerial vehicle; and

determining, by the processor, the distance between the unmanned aerial vehicle and the location external to the unmanned aerial vehicle based on a camera parameter of the unmanned aerial vehicle, wherein the location external to the unmanned aerial vehicle comprises a location in a target scene associated with the video stream.

20. A non-transitory computer-readable storage medium comprising instructions causing at least one processor of an electronic device to perform operations of:

wirelessly receiving, via communication circuitry, a video stream captured by a camera and a camera parameter, wherein the camera is located in an unmanned aerial vehicle;

receiving, via at least one sound sensor, an input sound stream;

generating an audio stream associated with the video stream based on the input sound stream;

determining a distance between the unmanned aerial vehicle and a location external to the unmanned aerial vehicle based on the camera parameter;

adjusting a gain applied to the audio stream to generate an adjusted audio stream, the gain adjusted based on the distance between the unmanned aerial vehicle and the location external to the unmanned aerial vehicle; and

generating video data based on the video stream and the adjusted audio stream.

21. The non-transitory computer-readable storage medium of claim 20, wherein generating the video data comprises:

synchronizing the video stream and the adjusted audio stream to generate a synchronized video stream and a synchronized audio stream; and

combining the synchronized video stream and the synchronized audio stream to generate the video data.

22. The non-transitory computer-readable storage medium of claim 20, wherein generating the audio stream comprises selectively extracting the audio stream from the input sound stream based on a beamforming angle associated with the audio stream, the beamforming angle determined based on a direction of the camera in the unmanned aerial vehicle, location information of the unmanned aerial vehicle, and location information of the electronic device.

23. The non-transitory computer-readable storage medium of claim 22, wherein the beamforming angle is further determined based on at least one of an orientation or a spatial arrangement of the at least one sound sensor in the electronic device.

24. The non-transitory computer-readable storage medium of claim 20, wherein the operations further comprise wirelessly receiving, via the communication circuitry, one or more camera parameters, including the camera parameter, and location information from the unmanned aerial vehicle, and wherein generating the audio stream comprises selectively extracting the audio stream based on the location information, a location or an orientation of the electronic device, and the one or more camera parameters.

25. The non-transitory computer-readable storage medium of claim 20, wherein the video stream includes audio, and wherein generating the video data comprises combining the adjusted audio stream with the audio of the video stream, and wherein the operations further comprise adjusting a level of noise suppression applied to audio of the video stream based on the distance.

26. The non-transitory computer-readable storage medium of claim 20, wherein generating the video data based on the video stream and the adjusted audio stream is performed substantially in real time with receiving the video stream, and wherein the location external to the unmanned aerial vehicle corresponds to a center focus point of a target scene associated with the camera of the unmanned aerial vehicle.

27. The non-transitory computer-readable storage medium of claim 20, wherein the operations further comprise outputting the video data, wherein outputting the video data is performed substantially in real time with receiving the video stream.