METHOD AND USER INTERFACE FOR FORENSIC VIDEO SEARCH

• US 20130091432A1
• Filed: 10/08/2012
• Published: 04/11/2013
• Est. Priority Date: 10/07/2011
• Status: Active Grant

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG, Siemens Corp. (Siemens AG)
• Inventors: Shet, Vinay Damodar, Hutter, Andreas, Gao, Xiang, Kuo, Cheng-Hao, Zheng, Xianjun Sam
• Granted Patent: US 9,269,243 B2
• Time in Patent Office: Days
• US Class Current: 715/719
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over

Abstract

A forensic video search user interface is disclosed that accesses databases of stored video event metadata from multiple camera streams and facilitates the workflow of search of complex global events that are composed of a number of simpler, low complexity events.

Claims

1. A forensic video search User Interface (UI) for a facility having video camera surveillance, the method comprising:

generating a display partitioned into “

where”

, “

when”

, “

who” and

“

what”

panels;

in the “

when”

panel, selecting a calendar date, bringing into view a timeline;

in the “

where”

panel, if the facility has more than one floor, selecting a floor of the facility, bringing into view a floor plan of the selected facility floor wherein the facility floor plan includes one or more camera icons showing a Field Of View (FOV) representing actual video camera surveillance locations on the selected facility floor;

based on the selected date, timeline and facility floor, generating a time-based query to discover events and objects involving human and non-human objects that occurred comprising;

populating the “

what”

panel with textual descriptions of events that occurred wherein an event includes one or more objects, location, time of occurrence and type of event;

populating the “

who”

panel with one or more object thumbnails wherein an object thumbnail represents a previously recorded video segment of an object associated with an event displayed in the “

what”

panel; and

populating the “

where”

panel with a marker in one or more camera icon'"'"'s FOV wherein a camera icon FOV marker represents an object thumbnail in the “

who”

panel.

2. The method according to claim 1 wherein an object thumbnail video segment has a predefined time duration.

3. The method according to claim 1 further comprising selecting an object thumbnail, bringing into view a list of events the object in the selected object thumbnail appears in, a beginning time for the object thumbnail video segment and a tracking button.

4. The method according to claim 1 further comprising:

populating the timeline with markers representing the instance object thumbnails shown in the “

who”

panel appear; and

displaying a metric of the number of events displayed in the “

what”

panel at the time each event occurred.

5. The method according to claim 1 further comprising in the “

what”

panel, including an associated statistic of occurrence with each event.

6. The method according to claim 1 wherein events include categories such as person events, vehicular events, appear events, disappear events, enter events, exit events, action events, alarm events, sensor events, human enter area 1, human exit area 2, door alarm triggered at time T1, and others.

7. The method according to claim 1 further comprising populating the “

what”

panel with textual descriptions of object types.

8. The method according to claim 7 wherein object types include persons, vehicles, packages, and others.

9. The method according to claim 1 further comprising:

in the “

what”

panel, selecting an event to search;

dragging the selected event into a “

query”

panel;

generating a query to find events with corresponding human and/or non-human objects within the selected event; and

updating the “

where”

, “

when”

, “

who” and

“

what”

panels.

10. The method according to claim 1 further comprising:

in the “

what”

panel, selecting an object to search;

dragging the selected object into a “

query”

panel;

generating a query to find objects with corresponding human and/or non-human objects within the selected object; and

updating the “

where”

, “

when”

, “

who” and

“

what”

panels.

11. The method according to claim 1 further comprising:

selecting a camera icon;

highlighting its FOV; and

bringing into view a live view from the selected camera icon'"'"'s corresponding surveillance video camera.

12. The method according to claim 1 further comprising:

in the “

when”

panel, entering a calendar date, bringing into view a timeline;

generating a query to find events and objects with corresponding human and/or non-human objects that occurred during the date; and

updating the “

where”

, “

when”

, “

who” and

“

what”

panels.

13. The method according to claim 12 further comprising:

adjusting the timeline as a time window wherein the time window may be narrowed (<

24 hours) or broadened (>

24 hours) depending on a desired search;

generating a query to find events and objects with corresponding human and/or non-human objects that occurred during the time window; and

updating the “

where”

, “

when”

, “

who” and

“

what”

panels.

14. The method according to claim 1 further comprising object tracking comprising:

in the “

who”

panel, selecting a first object thumbnail;

magnifying the selected first object thumbnail, bringing into view a list of events and objects;

selecting a “

track”

button comprising;

opening an object tracking clipboard;

moving the first selected object thumbnail to the object tracking clipboard wherein object thumbnails posted in the object tracking clipboard are query object thumbnails; and

marking the camera icon responsible for the first selected object thumbnail; and

performing a manual search or an assisted re-identification search.

15. The method according to claim 14 wherein the manual search comprises:

in the “

who”

panel, selecting another object thumbnail;

magnifying the selected another object thumbnail, bring into view a list of events and objects;

selecting an “

add”

button comprising;

moving the another selected object thumbnail to the object tracking clipboard;

marking an another camera icon responsible for the another selected object thumbnail; and

linking the another marked camera icon with the first marked camera icon.

16. The method according to claim 14 wherein the re-identification search comprises:

selecting an “

assisted re-identification”

button comprising;

generating a query to search for all objects matching one or more query object thumbnails;

obtaining a similarity score for each of the object thumbnails in the “

who”

panel and the query object thumbnails;

filtering the similarity scores of each of the object thumbnails in the “

who”

panel against the query object thumbnails similarity score; and

identifying an object thumbnail of interest from the similarity filtered “

who”

panel object thumbnails.

17. The method according to claim 1 further comprising an avatar search comprising:

in the “

who”

panel, selecting an avatar icon button, bringing into view a blank avatar, a color palette for defining the avatar'"'"'s upper body and a color palette for defining the avatar'"'"'s lower body;

coloring the avatar'"'"'s upper and lower body; and

generating a query to search for all human and/or non-human objects having similar visual attributes with the avatar that occurred during the timeline.

18. The method according to claim 17 further comprising:

obtaining a similarity score for each of the object thumbnails in the “

who”

panel and for the avatar object;

filtering the similarity scores of each of the object thumbnails in the “

who”

panel against the avatar object similarity score; and

identifying an object thumbnail of interest from the similarity filtered “

who”

panel object thumbnails.