Large venue surveillance and reaction systems and methods using dynamically analyzed emotional input

• US 9,996,736 B2
• Filed: 09/06/2016
• Issued: 06/12/2018
• Est. Priority Date: 10/16/2014
• Status: Active Grant

• Current Assignee: Software AG USA Incorporated (Software AG)
• Original Assignee: Software AG USA Incorporated (Software AG)
• Inventors: Smith, Leighton, Smith, Gareth
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US15/257,082
• Publication Number:
  US 20170046566A1
• Time in Patent Office: 644 Days
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Certain example embodiments relate to large venue surveillance and reaction systems and/or methods that take into account both subjective emotional attributes of persons having relations to the large venues, and objective measures such as, for example, actual or expected wait times, current staffing levels, numbers of customers to be serviced, etc. Pre-programmed scenarios are run in real-time as events stream in over one or more electronic interfaces, with each scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state. The scenarios are run to (a) determine whether an incident might be occurring and/or might have occurred, and/or (b) dynamically determine a responsive action to be taken. A complex event processing engine may be used in this regard. The analysis may be used in certain example embodiments to help improve customer satisfaction at the large venue.

Claims

1. An information processing system, comprising:

at least one data feed relevant to an area under surveillance, the area under surveillance having operational resources deployed thereto, the area under surveillance operating at an operational efficiency attributable at least in part to the operational resources deployed thereto; and

a memory storing a plurality of trigger scenarios, at least some of the trigger scenarios being represented by a set of rules that takes into account at least (a) information regarding an inferred emotional state of a plurality of individuals located in the area under surveillance and (b) additional data that pertains to the area under surveillance but is unrelated to inferred emotional state information;

a machine learning system; and

processing resources including at least one processor;

wherein the processing resources are configured to at least;

facilitate the receipt of information corresponding to system-relevant events over the at least one data feed;

evaluate at least some of the trigger scenarios stored in the memory in view of at least some of the system-relevant events corresponding to the information received via the at least one data feed to determine whether an incident might be occurring and/or might have occurred in connection with the operational resources deployed to the area under surveillance;

in response to a determination that a given incident might be occurring and/or might have occurred, select an action to be taken, the action being selected as an appropriate response for the given incident and involving a change impacting the deployment of the operational resources to cause a related alteration in the operational efficiency of the area under surveillance in a desired manner, the action being flagged for one of immediate dispatch and delayed dispatch;

add to a priority queue maintained in the memory representations of actions flagged for delayed dispatch, the priority queue facilitating dynamic reprioritization for actions flagged for delayed dispatch;

prompt actions flagged for immediate dispatch to be undertaken immediately, and prompt actions flagged for delayed dispatch to be undertaken based on their relative positions in the priority queue;

determine effectiveness metrics for prompted actions; and

provide the determined effectiveness metrics for the prompted actions to the machine learning system to cause the machine learning system to (i) assess, in response to the prompted actions, how the operational efficiency has changed and whether there has been a dynamic reaction based on inferred emotional state information, and (ii) influence how future actions are selected and/or implemented.

2. The system of claim 1, wherein at least some of the trigger scenarios specify one or more possible actions that may be taken and criteria for determining when such possible action(s) should be taken.

3. The system of claim 1, wherein at least some of the trigger scenarios are pre-programmed default trigger scenarios.

4. The system of claim 3, wherein at least some other trigger scenarios are adaptable over time.

5. The system of claim 1, wherein at least some of the trigger scenarios are automatically adaptable over time.

6. The system of claim 1, wherein the processing resources are further configured to at least assess feedback regarding evaluated trigger scenarios and automatically generate suggested modifications thereto.

7. The system of claim 6, wherein suggestable modifications include adding a new trigger scenario, removing an existing trigger scenario, and updating an existing trigger scenario.

8. The system of claim 1, wherein information receivable via the at least one data feed is structured to embed a representation of an inferred emotional state.

9. The system of claim 8, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals, derived from video and/or social media related instructions of said individuals.

10. The system of claim 1, wherein quantified representations of inferred emotional states are scored in accordance with a rubric.

11. The system of claim 10, wherein the rubric includes assignments of numerical scores to different discrete levels of each emotional state.

12. The system of claim 11, wherein the rubric is automatically updatable over time.

13. The system of claim 1, wherein system-relevant events include one or more objective measures of activities associated with the area under surveillance.

14. The system of claim 1, wherein at least some of the trigger scenarios specify a plurality of possible actions that may be taken, wherein the plurality of possible actions in the respective trigger scenarios are defined to escalate in intensity in accordance with a corresponding increase in severity of the incident.

26. The system of claim 1, wherein the area under surveillance is surveilled using video cameras.

27. The system of claim 1, wherein the area under surveillance is surveilled by monitoring social media posts.

28. The system of claim 1, wherein the representations of actions flagged for delayed dispatch have associated therewith severity and time-to-live values, the priority queue being managed to adjust relative positions of elements therein based on changing severity values and to remove representations based on lapsed time-to-live values.

29. The system of claim 28, wherein actions to be taken are selected from programmatically-defined action scenarios, and wherein the machine learning system is configured to automatically adapt at least some trigger scenarios over time in response to the determined effective metrics.

30. The system of claim 29, wherein the machine learning system is configured to rate for accuracy at least some trigger scenarios.

15. A method of monitoring an area under surveillance, the area under surveillance having operational resources deployed thereto, the area under surveillance operating at an operational efficiency attributable at least in part to the operational resources deployed thereto, the method comprising:

receiving information corresponding to system-relevant events over at least one data feed;

evaluating, using at least one processor, one or more trigger scenarios in view of at least some of the system-relevant events corresponding to the information received via the at least one data feed to determine whether an incident might be occurring and/or might have occurred in connection with the operational resources deployed to the area under surveillance, a plurality of trigger scenarios being stored to a memory, at least some of the trigger scenarios being represented by a set of rules that takes into account at least (a) information regarding an inferred emotional state of a plurality of individuals located in the area under surveillance and (b) additional data that pertains to the area under surveillance but is unrelated to inferred emotional state information;

responsive to a determination that a given incident might be occurring and/or might have occurred, selecting an action to be taken, the action being selected as an appropriate response for the given incident and involving a change impacting the deployment of the operational resources to cause a related alteration in the operational efficiency of the area under surveillance in a desired manner, the action being flagged for one of immediate dispatch and delayed dispatch;

adding to a priority queue maintained in the memory representations of actions flagged for delayed dispatch, the priority queue facilitating dynamic reprioritization for actions flagged for delayed dispatch;

prompting actions flagged for immediate dispatch to be undertaken immediately, and prompting actions flagged for delayed dispatch to be undertaken based on their relative positions in the priority queue;

determining effectiveness metrics for prompted actions; and

providing the determined effectiveness metrics for the prompted actions to a machine learning system to cause the machine learning system to (i) assess, in response to the prompted actions, how the operational efficiency has changed and whether there has been a dynamic reaction based on inferred emotional state information, and (ii) influence how future actions are selected and/or implemented.

16. The method of claim 15, wherein at least one of the trigger scenarios is a default trigger scenario that has been automatically adapted over time.

17. The method of claim 15, wherein information receivable via the at least one data feed is structured to embed a representation of an inferred emotional state.

18. The method of claim 17, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals, derived from video and/or social media related instructions of said individuals.

19. The method of claim 17, wherein quantified representations of inferred emotional states are scored in accordance with an automatically updatable rubric that includes assignments of numerical scores to different discrete levels of each emotional state.

20. The method of claim 15, wherein system-relevant events include one or more objective measures of activities associated with the area under surveillance.

21. A non-transitory computer readable storage medium tangibly storing a program comprising instructions that, when executed, in connection with a computer system including a processor that is controlled by the instructions, at least:

monitor an area under surveillance, the area under surveillance having operational resources deployed thereto, the area under surveillance operating at an operational efficiency attributable at least in part to the operational resources deployed thereto;

receive information corresponding to system-relevant events over at least one data feed;

evaluate one or more trigger scenarios in view of at least some of the system-relevant events corresponding to the information received via the at least one data feed to determine whether an incident might be occurring and/or might have occurred in connection with the operational resources deployed to the area under surveillance, a plurality of trigger scenarios being stored to a memory of the computer system, at least some of the trigger scenarios being represented by a set of rules that takes into account at least (a) information regarding an inferred emotional state of a plurality of individuals located in the area under surveillance and (b) additional data that pertains to the area under surveillance but is unrelated to inferred emotional state information;

responsive to a determination that a given incident might be occurring and/or might have occurred, select an action to be taken, the action being selected as an appropriate response for the given incident and involving a change impacting the deployment of the operational resources to cause a related alteration in the operational efficiency of the area under surveillance in a desired manner, the action being flagged for one of immediate dispatch and delayed dispatch;

add to a priority queue maintained in the memory representations of actions flagged for delayed dispatch, the priority queue facilitating dynamic reprioritization for actions flagged for delayed dispatch;

prompt actions flagged for immediate dispatch to be undertaken immediately, and prompt actions flagged for delayed dispatch to be undertaken based on their relative positions in the priority queue;

determine effectiveness metrics for prompted actions; and

provide the determined effectiveness metrics for the prompted actions to a machine learning system to cause the machine learning system to (i) assess, in response to the prompted actions, how the operational efficiency has changed and whether there has been a dynamic reaction based on inferred emotional state information, and (ii) influence how future actions are selected and/or implemented.

22. The non-transitory computer readable storage medium of claim 21, wherein at least one of the trigger scenarios is a default trigger scenario that has been automatically adapted over time.

23. The non-transitory computer readable storage medium of claim 21, wherein information receivable via the at least one data feed is structured to embed a representation of an inferred emotional state.

24. The non-transitory computer readable storage medium of claim 23, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals, derived from video and/or social media related instructions of said individuals.

25. The non-transitory computer readable storage medium of claim 23, wherein quantified representations of inferred emotional states are scored in accordance with an automatically updatable rubric that includes assignments of numerical scores to different discrete levels of each emotional state.