Establishing audio connections between users being continuously tracked by collaborative sensors

• US 9,830,554 B2
• Filed: 11/23/2016
• Issued: 11/28/2017
• Est. Priority Date: 10/22/2013
• Status: Active Grant

• Current Assignee: Lumin, LLC
• Original Assignee: Lumin, LLC
• Inventors: Shams, Nima Lahijani, Maheshaiah, Suhas
• Primary Examiner(s): Hill, Stnaley K
• Assistant Examiner(s): Lee, Tsu-Chang
• Application Number:
  US15/360,444
• Publication Number:
  US 20170076212A1
• Time in Patent Office: 370 Days
• Field of Search: 706 46
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

Audio sensors collaborate for geo-location and tracking of multiple users. Different users can be independently geo-located and tracked within the AI environment. Location is determined from two or more AI clients of known locations that detect an event such as a human voice command to connect a call with a specific user. Responsive to classification of the event in view of the estimated location, a command for an AI action, such as connecting a call between users, is received for a response to the event at the AI clients that detected the event, or others.

Claims

1. A computer-implemented method in an AI server of an artificial intelligence (AI) system, coupled to a plurality of AI clients, for establishing an audio connection between users within audio proximity of the plurality of AI clients, the method comprising the steps of:

storing location information for each of the plurality of AI clients, each of the AI clients comprising an audio sensor, a network controller, and a feedback mechanism within a common enclosure, and the plurality of AI clients being geographically-dispersed;

detecting audio event data from a first user at a first location that is received by at least a first AI client of the plurality of AI clients;

determining that audio event data comprises a request for audio connection with a second user;

responsive to the determination, estimating a location of a second user based on audio event data received by at least a second AI client of the plurality of AI clients, wherein the second AI client concerning the second user is different from the first AI clients concerning the first user; and

establishing an audio connection between the first and second users by connecting a first microphone of the first AI client to a second speaker of the second AI client and a second microphone of the second AI client to a first speaker of the first AI client.

2. The method of claim 1, wherein detecting audio event data from the first user at the first location comprises detecting a voice command.

3. The method of claim 1, wherein establishing an audio connection comprises at least one of:

connecting two or more microphones from two or more AI clients proximate to the first user to the to the second speaker of the second AI client, and connecting the first microphone of the first AI client to two or more speakers from two of more AI clients proximate to the second user.

4. The method of claim 1, further comprising:

receiving new audio event data associated with the second user; and

updating the estimated location of the second user based on the new audio event data; and

updating the audio connection for the second user according to the updated estimated location of the second user.

5. The method of claim 1, wherein establishing the audio connection comprises:

requesting through a human voice interface from the second AI client, for the call connection to the second user, including an identification of the first user.

6. The method of claim 1, wherein establishing the audio connection comprises:

establishing the audio connection for the first user and the second user, wherein both the first user and the second user are under the AI controller.

7. The method of claim 1, wherein establishing the audio connection comprises:

establishing the audio connection for the first user and the second user, wherein at least one of the first user and the second user are under the AI controller and at least one of the first user and the second user are under a second AI controller, wherein the AI controller and the second AI controller are compatible for call connections.

8. The method of claim 1, wherein establishing the audio connection comprises:

establishing the audio connection for the first user and the second user, wherein at least one of the first user and the second user are under the AI controller and at least one of the first user and the second user are connected through an external voice network.

9. The method of claim 1, wherein the audio event data comprises a sound file capturing audio from the event.

10. The method of claim 9, wherein the sound file capturing the audio event is formed by audio sensors of the two or more of the plurality of geographically-dispersed audio sensors, wherein the audio sensors continuously store ambient audio in a temporary memory and, upon detecting the event, define a beginning point and an end point of the event corresponding to the sound file.

11. The method of claim 1, further comprising:

determining a direction of movement for a subject of the event,

wherein the audio event data comprises audio event data at a first time corresponding to a first location of the event and event data at a second time corresponding to a second location, and

wherein the response command is based on the direction of movement of the subject.

12. The method of claim 11, further comprising:

predicting a future location for response to the event subject based on a difference between the audio event at the first time and the audio event at the second time,

wherein the response command is based on the predicted location.

13. The method of claim 1, further comprising:

classifying the audio event data by comparing the audio event data against a library of known audio data and corresponding identification of a human or machine event; and

mapping the event classification to a command response that causes a change in an electronic device connected to the one of the plurality of geographically-dispersed AI clients.

14. The method of claim 1, wherein executing the response command comprises outputting a sound file corresponding to the event through a speaker at the at least one of the plurality of geographically-dispersed AI appliances.

15. The method of claim 1, further comprising:

responsive to classification of the audio event data, sending a request to an Internet resource and receiving a response from the Internet resource that affects the command response.

16. The method of claim 1, wherein the event originates from a human activity, an electronic device, an environmental activity, or an ambient condition.

17. The method of claim 1, wherein executing the response command comprises:

executing the response command to the event in order to produce an event response involving an application that is an application that controls at least one of the plurality of geographically-dispersed AI clients.

18. The method of claim 1, wherein executing the response command comprises:

executing the response command to the event in order to produce an event response involving an application that is a remote application.

19. A non-transitory computer-readable medium storing source code that, when executed by a processor, performs a method in an AI controller of an artificial intelligence (AI) system, coupled to a plurality of AI clients, for establishing an audio connection between users within audio proximity of the plurality of AI clients, the method comprising the steps of

storing location information for each of the plurality of AI clients, each of the AI clients comprising an audio sensor, a network controller, and a feedback mechanism within a common enclosure, and the plurality of AI clients being geographically-dispersed;

detecting audio event data from a first user at a first location that is received by at least a first AI client of the plurality of AI clients;

determining that audio event data comprises a request for audio connection with a second user;

responsive to the determination, estimating a location of a second user based on audio event data received by at least a second AI client of the plurality of AI clients, wherein the second AI client concerning the second user is different from the first AI clients concerning the first user; and

establishing an audio connection between the first and second users by connecting a first microphone of the first AI client to a second speaker of the second AI client and a second microphone of the second AI client to a first speaker of the first AI client.

20. An AI controller of an artificial intelligence (AI) system, coupled to a plurality of AI clients, to establish an audio connection between users within audio proximity of the plurality of AI clients, the controller comprising:

a processor;

a communication interface; and

a memory, comprising;

a first module to store location information for each of the plurality of AI clients, each of the AI clients comprising an audio sensor, a network controller, and a feedback mechanism within a common enclosure, and the plurality of AI clients being geographically-dispersed;

a second module to detect audio event data from a first user at a first location that is received by at least a first AI client of the plurality of AI clients;

a third module to determine that audio event data comprises a request for audio connection with a second user;

a fourth module to, responsive to the determination, estimate a location of a second user based on audio event data received by at least a second AI client of the plurality of AI clients, wherein the second AI client concerning the second user is different from the first AI clients concerning the first user; and

a fifth module to establish an audio connection between the first and second users by connecting a first microphone of the first AI client to a second speaker of the second AI client and a second microphone of the second AI client to a first speaker of the first AI client.