Large venue surveillance and reaction systems and methods using dynamically analyzed emotional input

• US 9,449,218 B2
• Filed: 10/16/2014
• Issued: 09/20/2016
• Est. Priority Date: 10/16/2014
• Status: Active Grant

• Current Assignee: Software AG USA Incorporated (Software AG)
• Original Assignee: Software AG USA Incorporated (Software AG)
• Inventors: Smith, Leighton, Smith, Gareth
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US14/515,663
• Publication Number:
  US 20160110591A1
• Time in Patent Office: 705 Days
• Field of Search: 382/118
• US Class Current: 1/1
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Certain example embodiments relate to large venue surveillance and reaction systems and/or methods that take into account both subjective emotional attributes of persons having relations to the large venues, and objective measures such as, for example, actual or expected wait times, current staffing levels, numbers of customers to be serviced, etc. Pre-programmed scenarios are run in real-time as events stream in over one or more electronic interfaces, with each scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state. The scenarios are run to (a) determine whether an incident might be occurring and/or might have occurred, and/or (b) dynamically determine a responsive action to be taken. A complex event processing engine may be used in this regard. The analysis may be used in certain example embodiments to help improve customer satisfaction at the large venue.

Claims

1. An information processing system, comprising:

processing resources including at least one processor; and

a first interface over which quantified representations of inferred emotional states of one or more individuals are electronically receivable;

wherein the processing resources are configured to at least;

dynamically perform a set of first pre-programmed scenarios in real-time as events continue to stream into the system via the first interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state, wherein the logic sequence takes into account both an inferred emotional state and some other data that is unrelated to inferred emotional state; and

dynamically determine an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident.

2. The system of claim 1, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals derived from a video of said individuals.

3. The system of claim 1, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals derived from a video of said individuals, and/or results from computations applied thereto.

4. The system of claim 1, wherein quantified representations of inferred emotional states are scored in accordance with a predefined rubric.

5. The system of claim 4, wherein the predefined rubric includes assignments of numerical scores to different discrete levels of each emotional state.

6. The system of claim 1, wherein representations of inferred emotional states include quantifications of perceived subjective emotional feelings of individuals derived from social media related messages, and/or results from computations applied thereto.

12. The system of claim 1, further comprising a third interface over which objective measures of activities associated with a venue of interest are electronically receivable.

13. The system of claim 12, wherein the objective measures include crowd volume at the venue of interest and/or average processing time at defined location of the venue of interest.

14. The system of claim 1, wherein the dynamic determination of an action to be taken is made from a plurality of possible actions defined in the first pre-programmed scenario associated with the given incident.

16. The system of claim 1, wherein the dynamic determination of an action to be taken is made in consultation with a plurality of second pre-programmed scenarios.

17. The system of claim 1, further comprising a fourth interface configured to receive feedback on the efficacy of actions taken in response to detected incidents.

18. The system of claim 1, wherein the dynamic performance is practiced in connection with a complex event processing engine that operates as new events are received over the first interface.

7. An information processing system, comprising:

processing resources including at least one processor;

a first interface over which quantified representations of inferred emotional states of one or more individuals are electronically receivable; and

a second interface;

wherein the processing resources are configured to at least;

dynamically perform a set of first pre-programmed scenarios in real-time as events continue to stream into the system via the first interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state; and

dynamically determine an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident,

wherein representations of inferred emotional states received over the first interface include quantifications of perceived subjective emotional feelings of individuals derived from a video of said individuals, and/or results from computations applied thereto, and

wherein representations of inferred emotional states received over the second interface include quantifications of perceived subjective emotional feelings of individuals derived from social media related messages, and/or results from computations applied thereto.

8. The system of claim 7, wherein at least one of said first scenarios takes into account at least an aspect of a representation of an inferred emotional state received over the second interface.

9. The system of claim 7, wherein quantified representations of inferred emotional states are scored in accordance with a predefined rubric.

10. The system of claim 9, wherein the predefined rubric includes an extensible lexicon enabling sentiments expressed in social media related messages to be gauged and quantified.

11. The system of claim 10, wherein the predefined rubric includes assignments of numerical scores to different discrete levels of each emotional state.

15. An information processing system, comprising:

processing resources including at least one processor; and

a first interface over which quantified representations of inferred emotional states of one or more individuals are electronically receivable;

wherein the processing resources are configured to at least;

dynamically perform a set of first pre-programmed scenarios in real-time as events continue to stream into the system via the first interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state; and

dynamically determine an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident,

wherein the dynamic determination of an action to be taken is made from a plurality of possible actions defined in the first pre-programmed scenario associated with the given incident, and

wherein the plurality of possible actions are defined to escalate in intensity in accordance with a corresponding increase in severity of the incident.

19. A method of monitoring a venue of interest, the method comprising:

receiving, over a first electronic interface, quantified representations of inferred emotional states of one or more individuals;

dynamically performing a set of pre-programmed scenarios in real-time as events continue to stream in over the first electronic interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state, wherein the logic sequence takes into account both an inferred emotional state and some other data that is unrelated to inferred emotional state;

dynamically determining an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident; and

issuing a message to cause the action to be taken.

23. The method of claim 19, wherein the dynamic determining includes selecting an appropriate action to be taken from a plurality of possible actions defined within an operative scenario.

20. A method of monitoring a venue of interest, the method comprising:

receiving, over a first electronic interface, quantified representations of inferred emotional states of one or more individuals;

receiving, over a second electronic interface, quantified representations of inferred emotional states of one or more individuals;

dynamically performing a set of pre-programmed scenarios in real-time as events continue to stream in over the first electronic interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state;

dynamically determining an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident and

issuing a message to cause the action to be taken,

wherein representations of inferred emotional states received over the first interface include quantifications of perceived subjective emotional feelings of individuals derived from a video of said individuals, and/or results from computations applied thereto, and

wherein representations of inferred emotional states received over the second interface include quantifications of perceived subjective emotional feelings of individuals derived from social media related messages, and/or results from computations applied thereto.

21. The method of claim 20, further comprising receiving, over a third electronic interface, indications of objectively quantifiable actions taken at the venue.

22. The method of claim 20, wherein the dynamic performing and/or dynamic determining is performed each time an event is received over the first interface and each time an event is received over the second interface.

24. A non-transitory computer readable storage medium tangibly storing instructions that, when executed, monitor a venue of interest in connection with a computer system that is controlled by the instructions to at least:

receive, over a first electronic interface, quantified representations of inferred emotional states of one or more individuals;

dynamically perform a set of pre-programmed scenarios in real-time as events continue to stream in over the first electronic interface to determine whether an incident might be occurring and/or might have occurred, each said first scenario being implemented as a logic sequence that takes into account at least an aspect of a representation of an inferred emotional state, wherein the logic sequence takes into account both an inferred emotional state and some other data that is unrelated to inferred emotional state;

dynamically determine an action to be taken, in response to a determination that a given incident might be occurring and/or might have occurred, the action being selected as an appropriate response for the given incident; and

issue a message to cause the action to be taken.

25. The computer readable storage medium of claim 24, further comprising additional instructions that, when executed, further control the system to at least:

receive, over a second electronic interface, quantified representations of inferred emotional states of one or more individuals; and

receive, over a third electronic interface, indications of objectively quantifiable actions taken at the venue,

wherein representations of inferred emotional states received over the first interface include quantifications of perceived subjective emotional feelings of individuals derived from a video of said individuals, and/or results from computations applied thereto, and

wherein representations of inferred emotional states received over the second interface include quantifications of perceived subjective emotional feelings of individuals derived from social media related messages, and/or results from computations applied thereto.