Social sensing and behavioral analysis system

• US 10,049,336 B2
• Filed: 05/13/2014
• Issued: 08/14/2018
• Est. Priority Date: 02/14/2013
• Status: Active Grant

• Current Assignee: Sociometric Solutions, Inc.
• Original Assignee: Sociometric Solutions, Inc.
• Inventors: Olguin Olguin, Daniel, Jaanu, Tuomas, Heyman, Derek, Waber, Benjamin
• Primary Examiner(s): Kassim, Hafiz A
• Application Number:
  US14/276,871
• Publication Number:
  US 20140249891A1
• Time in Patent Office: 1,554 Days
• Field of Search: 705 738
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

A method and system for capturing and analyzing human behavior data is disclosed. The present disclosure describes a method and system for a plurality of people, wherein each person wears a badge. The badge transmits data collected from a plurality of sensors from the badge to a base station. The data is sent from the base station to a server, which aggregates the data from a plurality of base stations, and then analyzes and processes the data to create raw human behavior data. From the raw human behavior data and plurality of metrics is calculated, which can be displayed on a computer screen according to whichever metrics a user wishes to view.

Claims

1. A method of tracking human interactions using electronic sensors, the method comprising:

providing a first wearable electronic badge to be worn by a first person and a second wearable electronic badge to be worn by a second person, each wearable electronic badge comprising a plurality of sensors capturing sensor data, the plurality of sensors including a wireless communication module and a compass;

measuring an orientation of the first person using the compass of the first wearable electronic badge;

measuring an orientation of the second person using the compass of the second wearable electronic badge;

measuring a wireless signal strength between the first wearable electronic badge and the second wearable electronic badge, the measured wireless signal strength being representative of a distance between the first person and the second person;

transmitting from the wearable electronic badges being worn captured sensor data to a server via a base station or directly via the wireless communication module, the captured sensor data including the measured orientations and the measured wireless signal strength;

in response to the server receiving the transmitted sensor data, analyzing the captured sensor data and calculating a plurality of human interaction metrics, said analyzing the captured sensor data including determining existence of an interaction between the first person and the second person by comparing the measured orientations in combination with the measured wireless signal strength; and

creating and displaying on a computer screen a visualization of the calculated plurality of human interaction metrics according to at least one user specification.

2. The method of claim 1, wherein the plurality of sensors capturing sensor data of each wearable electronic badge further includes an infrared sensor, a camera, an accelerometer and a plurality of microphones, and the captured sensor data further includes data about the person wearing the respective wearable electronic badge, including at least one of:

proximity to another person, direct face-to-face interaction, location, individual speech patterns, conversational turn-taking patterns, orientation, body movement patterns, and posture patterns.

3. The method of claim 2, wherein the captured sensor data further includes information regarding interactions with a third person.

4. The method of claim 3, wherein the third person is not wearing a wearable electronic badge and the camera of the first wearable electronic badge is used to detect the third person'"'"'s face, and said analyzing the captured sensor data further includes utilizing said detection of the third person'"'"'s face in combination with other captured sensor data from the plurality of sensors to determine when the first and third persons are having a conversation.

5. The method of claim 3 further including determining an interaction between the first person and the third person by utilizing a combination of at least two of the following:

the infrared sensor, the wireless communication module, the compass, and the camera.

6. The method of claim 1, wherein the plurality of sensors capturing sensor data of each wearable electronic badge further includes an infrared sensor, a camera, an accelerometer and a plurality of microphones, and the captured sensor data further includes data about the person wearing the respective wearable electronic badge, including conversational turn-taking patterns of the person, said conversational turn-taking patterns being measured by a combination of at least two of the following:

the infrared sensor, the wireless communication module, the compass, the camera, and the plurality of microphones.

7. The method of claim 6, wherein the plurality of human interaction metrics include a plurality of conversational turn-taking metrics, said plurality of conversational turn-taking metrics including at least one of:

number of turns taken, number of pauses, number of successful interruptions, number of unsuccessful interruptions, average turn length, average pause length, turn-taking matrix, speaking time, silence time, listening time, overlapping speech time, dominance, interactivity, participation percentage, participation balance, and turn-taking balance.

8. The method of claim 1, wherein the wearable electronic badges are implemented into an existing personnel identification network, and

for each wearable electronic badge, a unique identification number is assigned by the existing personnel identification network and communicated to the wearable electronic badge through the wireless communication module, or manually assigned.

9. The method of claim 1, wherein the plurality of human interaction metrics are at least one of:

a plurality of social network metrics, a plurality of conversational turn-taking metrics, or a plurality of social signaling metrics.

10. The method of claim 1, wherein analyzing the captured sensor data includes mathematically representing a plurality of persons as a graph of nodes, each node representing a person in relation to the other persons of the plurality of persons;

and

wherein the plurality of human interaction metrics include a plurality of social network metrics, said plurality of social network metrics including at least one of;

betweenness centrality, degree centrality, cohesion, integration, and exploration, betweenness centrality being defined as a number of shortest paths from all end nodes to all other nodes that pass through a node in the graph of nodes, degree centrality being defined as a number of ties that a node has in the graph of nodes, cohesion being defined as a minimum number of nodes in the graph of nodes that need to be removed in order to disconnect a group of nodes, integration being defined as a number of triads that a node is part of divided by a maximum number of possible triads, and exploration being defined as a percentage of time that a person of a group of persons has spent with people outside of the group divided by total interaction time of the person.

11. The method of claim 1, wherein the plurality of human interaction metrics include a plurality of social signaling metrics, said plurality of social signaling metrics including include at least one of:

activity, consistency, mirroring, and influence; and

the plurality of social signaling metrics are calculated by any of;

body movement patterns, individual speech patterns, and conversational turn-taking patterns.

12. The method of claim 1, wherein the visualization of the plurality of human interaction metrics illustrate the relationship between any one of the plurality of human interaction metrics and a set of performance metrics, the set of performance metrics selected from:

productivity, job satisfaction, employee engagement, and drive.

13. The method of claim 12, wherein the visualization further illustrates correspondence between a change in the performance metrics and a change in sensed human interaction.

14. The method of claim 1, wherein the visualization of the plurality of metrics further includes a visualization of the plurality of human interaction metrics according to location.

15. The method of claim 1, wherein the wireless communication module is an electromagnetic data transfer module, the electromagnetic data transfer module being at least one of:

a near field communication (NFC) module and a radio frequency identification (RFID) module.

16. The method of claim 1, wherein the plurality of human interaction metrics are calculated for respective locations of different interactions between persons wearing the wearable electronic badges, wherein the respective locations of the different interactions are determined from the captured sensor data, or from the wearable badges proximity to one or more base stations, each base station having a known location.

17. A system for tracking human interactions using electronic sensors, the system comprising:

a first wearable electronic badge to be worn by a first person and a second wearable electronic badge to be worn by a second person, each wearable electronic badge comprising a plurality of sensors capturing sensor data, the plurality of sensors including a wireless communication module and a compass;

one or more base stations configured to receive the captured sensor data from the wearable electronic badges;

a server configured to;

(i) receive the captured sensor data from the one or more base stations or the wearable electronic badges, the captured sensor data including an orientation of the first person measured by the compass of the first wearable electronic badge, an orientation of the second person measured by the compass of the second wearable electronic badge, and a measured wireless signal strength between the first wearable electronic badge and the second wearable electronic badge, the measured wireless signal strength being representative of a distance between the first person and the second person, and (ii) analyze the captured sensor data and calculate a plurality of human interaction metrics, said analyzing the captured sensor data including determining existence of an interaction between the first person and the second person by comparing the measured orientations in combination with the measured wireless signal strength; and

a user terminal having a processor and a display monitor, the user terminal providing a visualization of the calculated plurality of human interaction metrics by rendering the visualization on the display monitor according to at least one user specification.

18. The system of claim 17, wherein the plurality of sensors capturing sensor data of each wearable electronic badge further includes an infrared sensor, a camera, an accelerometer and a plurality of microphones, and the captured sensor data further includes data about the person wearing the respective wearable electronic badge, including at least one of:

proximity to another person, direct face-to-face interaction, location, individual speech patterns, conversational turn-taking patterns, orientation, body movement patterns, and posture patterns.

19. The system of claim 18, wherein the captured sensor data further includes information regarding interactions with a third person.

20. The system of claim 18, wherein the wireless communication module is an electromagnetic data transfer module, the electromagnetic data transfer module being at least one of:

a near field communication (NFC) module and a radio frequency identification (RFID) module.

21. The system of claim 19, wherein the third person is not wearing a wearable electronic badge and the camera of the first wearable electronic badge is used to detect the third person'"'"'s face, and said analyzing the captured sensor data further includes utilizing said detection of the third person'"'"'s face in combination with other captured sensor data from the plurality of sensors to determine when the first and third persons are having a conversation.

22. The system of claim 19, wherein the server is further configured to determine an interaction between the first person and the third person by utilizing a combination of at least two of the following:

the infrared sensor, the wireless communication module, the compass, and the camera.

23. The system of claim 17, wherein the plurality of sensors to capture sensor data of each wearable electronic badge further includes an infrared sensor, a camera, an accelerometer and a plurality of microphones, and the captured sensor data further includes data about the person wearing the respective wearable electronic badge, including conversational turn-taking patterns of the person, said conversational turn-taking patterns being measured by a combination of at least two of the following:

the infrared sensor, the wireless communication module, the compass, the camera, and the plurality of microphones.

24. The system of claim 23, wherein the plurality of human interaction metrics include a plurality of conversational turn-taking metrics, said plurality of conversational turn-taking metrics including further include at least one of:

number of turns taken, number of pauses, number of successful interruptions, number of unsuccessful interruptions, average turn length, average pause length, turn-taking matrix, speaking time, silence time, listening time, overlapping speech time, dominance, interactivity, participation percentage, participation balance, and turn-taking balance.

25. The system of claim 17, wherein the wearable electronic badges are implemented into an existing personnel identification network, and

for each wearable electronic badge, a unique identification number is assigned by the existing personnel identification network and communicated to the wearable electronic badge through the wireless communication module, or manually assigned.

26. The system of claim 17, wherein the plurality of human interaction metrics are at least one of:

a plurality of social network metrics, a plurality of conversational turn-taking metrics, or a plurality of social signaling metrics.

27. The system of claim 17, wherein said analyzing the captured sensor data includes mathematically representing a plurality of persons as a graph of nodes, each node representing a person in relation to the other persons of the plurality of persons;

and

wherein the plurality of human interaction metrics include a plurality of social network metrics, said plurality of social network metrics including include at least one of;

betweenness centrality, degree centrality, cohesion, integration, and exploration, betweenness centrality being defined as a number of shortest paths from all end nodes to all other nodes that pass through a node in the graph of nodes, degree centrality being defined as a number of ties that a node has in the graph of nodes, cohesion being defined as a minimum number of nodes in the graph of nodes that need to be removed in order to disconnect a group of nodes, integration being defined as a number of triads that a node is part of divided by a maximum number of possible triads, and exploration being defined as a percentage of time that a person of a group of persons has spent with people outside of the group of persons divided by total interaction time of the person.

28. The system of claim 17, wherein the plurality of human interaction metrics include a plurality of social signaling metrics, said plurality of social signaling metrics including include at least one of:

activity, consistency, mirroring, and influence; and

the plurality of social signaling metrics are calculated from any of;

body movement patterns, individual speech patterns, and conversational turn-taking patterns.

29. The system of claim 17, wherein the visualization of the plurality of human interaction metrics illustrate the relationship between any one of the plurality of human interaction metrics and a set of performance metrics, the set of performance metrics selected from:

productivity, job satisfaction, employee engagement, and drive.

30. The system of claim 29, wherein the visualization further illustrates correspondence between a change in the performance metrics and a change in sensed human interaction.

31. The system of claim 17, wherein the visualization of the plurality of metrics further includes a visualization of the plurality of human interaction metrics according to location.

32. The system of claim 17, wherein the plurality of human interaction metrics are calculated for respective locations of different interactions between persons wearing the wearable electronic badges, wherein the respective locations of the different interactions are determined from the captured sensor data, or from the wearable badges proximity to one or more base stations, each base station having a known location.