Method and user interface for forensic video search

• US 9,269,243 B2
• Filed: 10/08/2012
• Issued: 02/23/2016
• Est. Priority Date: 10/07/2011
• Status: Active Grant

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Shet, Vinay Damodar, Zheng, Xianjun Sam, Hutter, Andreas, Gao, Xiang, Kuo, Cheng-Hao
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): Nichols, Jennifer
• Application Number:
  US13/646,867
• Publication Number:
  US 20130091432A1
• Time in Patent Office: 1,233 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over

Abstract

A forensic video search user interface is disclosed that accesses databases of stored video event metadata from multiple camera streams and facilitates the workflow of search of complex global events that are composed of a number of simpler, low complexity events.

Claims

1. A forensic video search User Interface (UI) method with query processing for a facility having video camera surveillance, the method comprising:

generating a display partitioned into “

when”

, “

where” and

“

who”

panels;

in the “

when”

panel, entering a point in time and/or a timeline;

in the “

where”

panel, selecting an area or a floor of the facility bringing into view a map or a floor plan wherein the map or the floor plan includes one or more camera icons showing a Field Of View (FOV) representing actual video camera surveillance locations on the selected facility area or floor;

based on the entered time and/or timeline and the selected facility area or floor, generating a time based query to discover events and objects involving human and non-human objects that occurred comprising;

populating the “

who”

panel with one or more object thumbnails wherein an object thumbnail is an image that represents an object from a previously recorded video segment where the image is associated with an event;

populating the “

where”

panel with a plurality of markers in at least one of the one or more camera icons'"'"' FOV wherein each camera icon FOV marker represents a different object thumbnail in the “

who”

panel;

overlaying levels of transparency of the FOV of the one or more camera icons to indicate numbers of events observed by each represented actual video camera surveillance location during the time entered in the “

when”

panel.

2. The method according to claim 1 further comprising a “

what”

panel and populating the “

what”

panel with textual descriptions of discovered events that occurred during the entered time and/or timeline and in the selected facility area or floor, wherein an event includes one or more objects, location, time of occurrence and/or type of event.

3. The method according to claim 1 wherein the object thumbnail has an associated video segment with a predefined time duration.

4. The method according to claim 3 further comprising selecting the object thumbnail and playing back its video segment in a video panel or pop up.

5. The method according to claim 1 further comprising selecting the object thumbnail, bringing into view a list of events the object in the selected object thumbnail appears in, and a beginning time for the previously recorded video segment containing the object represented by the object thumbnail.

6. The method according to claim 1, wherein the display is further partitioned into a “

what”

panel populated with textual descriptions of discovered events that occurred during the entered time and/or timeline and in the selected facility area or floor, wherein an event includes one or more objects, location, time of occurrence and/or type of event, the method further comprising;

populating the timeline with one or more timeline markers wherein each timeline marker represents the object thumbnail shown in the “

who”

panel; and

populating the timeline with a metric of a number of events displayed in the “

what”

panel at a time each event occurred.

7. The method according to claim 2 further comprising including an associated statistic of occurrence with each event in the “

what”

panel.

8. The method according to claim 2 wherein events include categories selected from a group consisting of person events, vehicular events, appear events, disappear events, enter events, exit events, action events, alarm events, sensor events, human enter area 1, human exit area 2 and door alarm triggered at time T1.

9. The method according to claim 2 further comprising populating the “

what”

panel with textual descriptions of object types.

10. The method according to claim 9 wherein the object types include persons, vehicles and packages.

11. The method according to claim 2 further comprising a “

query”

panel, further comprising;

in the “

what”

panel, selecting an event to search;

dragging the selected event into the “

query”

panel;

generating a query to find events with corresponding human and/or non-human objects with the selected event; and

updating the “

when”

, “

where”

, “

who” and

“

what”

panels.

12. The method according to claim 2 further comprising a “

query”

panel, further comprising;

in the “

who”

panel, selecting an object thumbnail to search;

dragging the selected object thumbnail into the “

query”

panel;

generating a query to find events with corresponding human and/or non-human objects with the selected object thumbnail; and

updating the “

when”

, “

where”

, “

who” and

“

what”

panels.

13. The method according to claim 1 further comprising:

selecting a camera icon;

highlighting its FOV; and

bringing into view a live view from the selected camera icon'"'"'s corresponding surveillance video camera.

14. The method according to claim 1 further comprising:

in the “

when”

panel, entering a calendar date, bringing into view a timeline of the calendar date;

generating a query to find events with corresponding human and/or non-human objects that occurred during the calendar date; and

updating the “

when”

, “

where” and

“

who”

panels.

15. The method according to claim 14 further comprising:

adjusting the timeline of the calendar date as a time window wherein the time window may be narrowed or broadened depending on a desired search;

generating a query to find events with corresponding human and/or non-human objects that occurred during the time window; and

updating the “

when”

, “

where” and

“

who”

panels.