Connection with remote internet of things (IoT) device based on field of view of camera

• US 10,447,394 B2
• Filed: 09/15/2017
• Issued: 10/15/2019
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kasilya Sudarsan, Vishnu Vardhan, Ketti Gangadharan, Ganesh Babu, Karuppusamy, Rameshkumar, Kadavamganam, Hariprasad, Gunjan, Somya
• Primary Examiner(s): Tran, Jimmy H
• Application Number:
  US15/706,322
• Publication Number:
  US 20190089456A1
• Time in Patent Office: 760 Days
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04B 10/116 Visible light communication
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04W 4/38 for collecting sensor information
• CPC Class Codes: H04W 40/246 Connectivity information discovery
• CPC Class Codes: H05B 47/175 by remote control

Abstract

This disclosure provides systems, methods, and apparatus, including computer programs encoded on computer storage media, for network communication when recording audio and video (A/V) of a subject. In one aspect, a device may determine one or more internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the device, and that are capable of remote audio recording and audio streaming. The device may select a first IoT device having a first microphone. The device may receive an audio stream from the first IoT device after selecting the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the device. When the FoV of the camera changes, the device may dynamically switch from receiving the audio stream from a first IoT device to receiving the audio stream from a second IoT device.

Claims

1. A method for network communication performed by a computing device for recording audio and video (A/V), comprising:

determining, by the computing device, a plurality of internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the computing device and that are capable of remote audio recording and audio streaming;

determining, within the FoV of the camera of the computing device, distance measurements from each of the plurality of IoT devices displayed within the FoV to a focal point of the FoV;

selecting, by the computing device, a first IoT device from the plurality of IoT devices based, at least in part, on the distance measurements, the first IoT device having a first microphone that is capable of remotely recording audio of a subject for the computing device; and

receiving, by the computing device, an audio stream from the first IoT device after selecting the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the computing device.

2. The method of claim 1, wherein selecting the first IoT device from the plurality of IoT devices based, at least in part, on the distance measurements comprises:

determining, based on the distance measurements, that the first IoT device from the plurality of IoT devices is located closest to the focal point of the FoV; and

selecting the first IoT device from the plurality of IoT devices based, at least in part, on determining that the first IoT device is located closest to the focal point of the FoV.

3. The method of claim 1, further comprising:

determining a microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices; and

comparing the microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices,

wherein selecting the first IoT device from the plurality of IoT devices is further based, at least in part, on the microphone type and performance characteristics associated with the first microphone of the first IoT device.

4. The method of claim 1, further comprising:

determining that the first IoT device and a second IoT device of the plurality of IoT devices are displayed within the focal area of the FoV of the camera of the computing device;

receiving a first audio stream from the first IoT device and a second audio stream from the second IoT device; and

comparing audio quality characteristics of the first audio stream to audio quality characteristics of the second audio stream,

wherein selecting the first IoT device is further based, at least in part, on the audio quality characteristics of the first audio stream received from the first IoT device.

5. The method of claim 1, further comprising:

receiving, by the computing device, user input selecting the first IoT device having the first microphone,

wherein selecting the first IoT device from the plurality of IoT devices is further based, at least in part, on the user input.

6. The method of claim 1, further comprising:

receiving, by the computing device, user input selecting a second IoT device from the plurality of IoT devices, the second IoT device having a second microphone that is capable of remotely recording the audio of the subject;

overriding a selection of the first IoT device by the computing device based, at least in part, on receiving the user input selecting the second IoT device; and

selecting the second IoT device from the plurality of IoT devices based, at least in part, on the user input.

7. The method of claim 1, further comprising:

recording video of the subject using the camera of the computing device; and

generating an A/V recording of the subject using the video of the subject recorded by the camera of the computing device and the audio of the subject remotely recorded by the first microphone of the first IoT device.

8. The method of claim 1, further comprising:

detecting a change in the FoV of the camera of the computing device; and

determining to switch from receiving the audio stream from the first IoT device to receiving an audio stream from a second IoT device after detecting the change in the FoV of the camera, the second IoT device having a second microphone that is capable of remotely recording the audio of the subject.

9. The method of claim 8, wherein determining to switch from receiving the audio stream from the first IoT device to receiving the audio stream from the second IoT device after detecting the change in the FoV of the camera comprises:

determining that the second IoT device is located closer to the focal point of the FoV than the first IoT device after detecting the change in the FoV of the camera; and

determining to switch from receiving the audio stream from the first IoT device to receiving the audio stream from the second IoT device in response to determining that the second IoT device is located closer to the focal point of the FoV than the first IoT device after detecting the change in the FoV of the camera.

10. The method of claim 1, further comprising:

determining the audio stream from the first IoT device is not received at the computing device for a period of time; and

determining to switch to recording the audio of the subject using a local microphone of the camera of the computing device in response to determining the audio stream from the first IoT device is not received at the computing device for the period of time.

11. The method of claim 1, wherein determining the plurality of IoT devices that are displayed within the FoV of the camera of the computing device and that are capable of remote audio recording and audio streaming comprises:

obtaining a map of locations of the plurality of IoT devices in a local network from a master device of the local network;

determining that the plurality of IoT devices each have a microphone and are capable of remote audio recording and audio streaming; and

displaying a visual indicator within the FoV of the camera for each of the plurality of IoT devices, the visual indicator indicating a location of each of the plurality of IoT devices within the FoV of the camera and indicating each of the plurality of IoT devices is capable of remote audio recording and audio streaming.

12. The method of claim 1, wherein determining the plurality of IoT devices that are displayed within the FoV of the camera of the computing device and that are capable of remote audio recording and audio streaming comprises at least one member selected from the group consisting of:

obtaining information from a master device of a local network, the information indicating which of the plurality of IoT devices are capable of remote audio recording and audio streaming; and

obtaining information from each of the plurality of IoT devices, the information indicating which of the plurality of IoT devices are capable of remote audio recording and audio streaming.

13. The method of claim 1, wherein the computing device is a mobile phone, a tablet computer, a pair of smart glasses, a dedicated camera, or a drone.

14. The method of claim 1, wherein determining, within the FoV of the camera of the computing device, the distance measurements from each of the plurality of IoT devices displayed within the FoV to the focal point of the FoV further comprises:

determining coordinates within the FoV of the camera for the focal point of the FoV, for the first IoT device displayed within the FoV, and for a second IoT device displayed within the FoV; and

determining the distance measurements, within the FoV of the camera, from each of the plurality of IoT devices displayed within the FoV to the focal point of the FoV based, at least in part, on the coordinates within the FoV of the camera.

15. A computing device, comprising:

a processor; and

memory having instructions stored therein which, when executed by the processor cause the computing device to;

determine a plurality of internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the computing device and that are capable of remote audio recording and audio streaming;

determine, within the FoV of the camera of the computing device, distance measurements from each of the plurality of IoT devices displayed within the FoV to a focal point of the FoV;

select a first IoT device from the plurality of IoT devices based, at least in part, on the distance measurements, the first IoT device having a first microphone that is capable of remotely recording audio of a subject for the computing device; and

receive an audio stream from the first IoT device after selection of the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the computing device.

16. The computing device of claim 15, wherein the instructions, when executed by the processor to select the first IoT device from the plurality of IoT devices based, at least in part, on the distance measurements, further cause the computing device to:

determine, based on the distance measurements, that the first IoT device from the plurality of IoT devices is located closest to the focal point of the FoV; and

select the first IoT device from the plurality of IoT devices based, at least in part, on a determination that the first IoT device is located closest to the focal point of the FoV.

17. The computing device of claim 15, wherein the instructions, when executed by the processor, further cause the computing device to:

determine a microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices; and

compare the microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices,

wherein selection of the first IoT device from the plurality of IoT devices is further based, at least in part, on the microphone type and performance characteristics associated with the first microphone of the first IoT device.

18. The computing device of claim 15, wherein the instructions, when executed by the processor, further cause the computing device to:

receive user input selecting the first IoT device having the first microphone,

wherein selection of the first IoT device from the plurality of IoT devices is further based, at least in part, on the user input.

19. The computing device of claim 15, wherein the instructions, when executed by the processor, further cause the computing device to:

record video of the subject using the camera of the computing device; and

generate an A/V recording of the subject using the video of the subject recorded by the camera of the computing device and the audio of the subject remotely recorded by the first microphone of the first IoT device.

20. The computing device of claim 15, wherein the instructions, when executed by the processor, further cause the computing device to:

detect a change in the FoV of the camera of the computing device; and

determine to switch from reception of the audio stream from the first IoT device to reception of an audio stream from a second IoT device after detection of the change in the FoV of the camera, the second IoT device having a second microphone that is capable of remotely recording the audio of the subject.

21. The computing device of claim 15, wherein the instructions, when executed by the processor to determine the plurality of IoT devices that are displayed within the FoV of the camera of the computing device and that are capable of remote audio recording and audio streaming, further cause the computing device to:

obtain a map of locations of the plurality of IoT devices in a local network from a master device of the local network;

determine that the plurality of IoT devices each have a microphone and are capable of remote audio recording and audio streaming; and

display a visual indicator within the FoV of the camera for each of the plurality of IoT devices, the visual indicator indicating a location of each of the plurality of IoT devices within the FoV of the camera and indicating each of the plurality of IoT devices is capable of remote audio recording and audio streaming.

22. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a processor of a computing device, cause the computing device to:

determine a plurality of internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the computing device and that are capable of remote audio recording and audio streaming;

determine, within the FoV of the camera of the computing device, distance measurements from each of the plurality of IoT devices displayed within the FoV to a focal point of the FoV;

select a first IoT device from the plurality of IoT devices based, at least in part, on the distance measurements, the first IoT device having a first microphone that is capable of remotely recording audio of a subject for the computing device; and

receive an audio stream from the first IoT device after selection of the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the computing device.

23. The non-transitory computer-readable storage medium of claim 22, wherein the instructions, when executed by the processor, further cause the computing device to:

determine, based on the distance measurements, that the first IoT device from the plurality of IoT devices is located closest to the focal point of the FoV; and

select the first IoT device from the plurality of IoT devices based, at least in part, on a determination that the first IoT device is located closest to the focal point of the FoV.

24. The non-transitory computer-readable storage medium of claim 22, wherein the instructions, when executed by the processor, further cause the computing device to:

determine a microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices; and

compare the microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices,

wherein selection of the first IoT device from the plurality of IoT devices is further based, at least in part, on the microphone type and performance characteristics associated with the first microphone of the first IoT device.

25. The non-transitory computer-readable storage medium of claim 22, wherein the instructions, when executed by the processor, further cause the computing device to:

detect a change in the FoV of the camera of the computing device; and

determine to switch from reception of the audio stream from the first IoT device to reception of an audio stream from a second IoT device after detection of the change in the FoV of the camera, the second IoT device having a second microphone that is capable of remotely recording the audio of the subject.