AUTOMATIC EVENT DETECTION, TEXT GENERATION, AND USE THEREOF

• US 20170300150A1
• Filed: 02/23/2017
• Published: 10/19/2017
• Est. Priority Date: 11/23/2011
• Status: Active Grant

• Current Assignee: Avigilon Fortress Corporation
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Choe, Tae Eun, Lee, Mun Wai, Gunda, Kiran, Haering, Niels
• Granted Patent: US 10,768,769 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/48 Starting a game, e.g. activating a game device or waiting for other players to join a multiplayer session
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: A63F 13/795 for finding other players; for building a team; for providing a buddy list
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G08B 13/19665 Details related to the storage of video surveillance data television signal recording H04N5/76
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A system and method for video surveillance and searching are disclosed. Video is analyzed and events are automatically detected. Based on the automatically detected events, textual descriptions are generated. The textual descriptions may be used to supplement video viewing and event viewing, and to provide for textual searching for events.

Claims

46. A video surveillance system, comprising:

a first video camera configured to capture at least a first video sequence including a plurality of video images;

a processing system configured to automatically detect events;

a storage system configured to receive and store records of the automatically detected events with respective geographical locations in association with automatically detected events, the associated geographical location being at least in part based on map data; and

a text generator configured to automatically generate textual descriptions of the automatically detected events, and

wherein the automatically detected events are events that relate to map data and are based on rules that relate to the map data, such that the text generator generates text that combines the map data with the automatically detected events into a report.

47. The video surveillance system of claim 46, further comprising:

a file generator configured to generate a file including the first video sequence and including embedded text that reflects a first textual description associated with a first detected event that occurs at a first time of the first video sequence and is inserted in the first video sequence based on the first time, and a second textual description associated with a second detected event that occurs at a second time of the first video sequence and is inserted in the first video sequence based on the second time,

wherein the file is arranged so that when played back, the first textual description appears in the first video sequence at the first time and the second textual description appears in the first video sequence at the second time, such that the text displayed with the first video sequence changes,

wherein the file for the first video sequence includes a first set of video frames, each video frame of the first set corresponding to one of the video images and including for simultaneous display a video image and embedded text.

48. The video surveillance system of claim 47, further comprising:

a display system configured to play back the first video sequence and the embedded text, such that during the play back, individual video images of the first video sequence that are associated with an automatically detected event are displayed at the same time as embedded text describing the event.

49. The video surveillance system of claim 47, wherein the first time and second time include at least one of a temporal location within the video sequence and a universal time.

50. The video surveillance system of claim 46, wherein the processing system is further configured to automatically detect events that involve two or more agents in the video sequence.

51. The video surveillance system of claim 46, wherein the textual information includes a natural language sentence.

52. A video surveillance system comprising:

a first video camera configured to capture a first video sequence including a plurality of video images;

a processing system configured to automatically detect in the video sequence;

a storage system configured to receive and store records of the automatically detected events and to associate each automatically detected event with a geographical location where the automatically detected event occurred;

a text generator configured to automatically generate a textual description of a first event of the automatically detected events, the textual description enhanced to include text that reflects the geographical location;

a file generator configured to generate a file including the first video sequence and including embedded text that reflects the textual description and that is inserted in the video sequence; and

a display system,

wherein for each automatically detected event, and based on the textual description for the event, the display system is configured to display information for the event, the information including the text that reflects the geographical location, and to overlay the information on a map of a geographical area that includes the geographical location, such that the information for the event including the text that reflects the geographical location is visually associated with a particular location on the map.

53. The video surveillance system of claim 52, wherein the processing system is further configured to receive a search request including one or more natural language terms relating to the first event, and to return the information to be displayed and overlaid on the map of the geographical area, based on the search request.

54. The video surveillance system of claim 53, wherein the search request further includes at least one of:

geographical information, or time information.

55. The video surveillance system of claim 53, wherein the display system includes a hand-held device configured to submit the search request and receive and display the information for the first event overlaid on the map of the geographical area.

56. The video surveillance system of claim 53, wherein the processing system is configured to determine search results based on the search request and based on a geographical location of a device that makes the request.

57. The video surveillance system of claim 56, wherein the device is a smart phone.

58. The video surveillance system of claim 56, wherein the file generator is further configured to generate clips of the first video sequence, each clip including a set of video images that comprises an event, and the processing system is further configured to associate data related to the event with the clip.

59. The video surveillance system of claim 58, wherein the processing system is further configured to receive a search request including one or more natural language terms relating to the first event, and return a clip of the first event based on the search request.

59-1. The video surveillance method of claim 52, further comprising:

for the first event, automatically displaying on the map, separately from the textual description for the first event, an indication of a route taken, the route corresponding to the first event.

60. A video surveillance method, comprising:

analyzing a first video sequence including a plurality of video images;

automatically detecting events in the first video sequence;

receiving and storing records of the automatically detected events and associating each of the automatically detected events with a geographical location at which the event occurs;

automatically generating a textual description of a first event of the automatically detected events, the textual description including text describing an actor, an action, and the geographical location; and

for the first event of the automatically detected events, causing a display system to display information based on the textual description and overlay the information on a map of a geographical area, such that the text describing the actor, the action, and the geographical location is visually associated with a particular geographical location depicted in the map.

61. The video surveillance method of claim 60, further comprising receiving a search request including one or more natural language terms relating to the first event, and returning the information to be overlaid on the map of the geographical area, based on the search request.

62. The video surveillance method of claim 61, wherein the search request further includes at least one of:

geographical information, or temporal information.

63. The video surveillance method of claim 61, further comprising, receiving the search request from a display device, and based on the search request, causing the display device to display the information for the first event overlaid on the map of the geographical area.

64. The video surveillance method of claim 61, further comprising determining search results based on the search request and based on a geographical location of a device that makes the request.

65. The video surveillance method of claim 64, wherein the device is a smart phone.

66. The video surveillance method of claim 60, further comprising generating a file including a video clip of the first video sequence and including embedded text that reflects the textual description and is inserted in the video clip based on a time associated with the first video sequence.

67. The video surveillance method of claim 66, further comprising:

causing a display system to display the video clip and the embedded text, such that individual video images of the first video sequence that are associated with an automatically detected event are displayed at the same time as embedded text describing the event.

68. The video surveillance method of claim 60, wherein automatically detecting events includes automatically detecting events between two agents in the video sequence.

69. The video surveillance method of claim 60, further comprising:

automatically displaying on the map, separately from the textual description, an indication of a route taken by the actor, the route corresponding to the first event.

70. A video searching method, comprising:

capturing a plurality of video sequences from a plurality of respective video sources, each video sequence including a plurality of video images;

for each video sequence, automatically detecting an event associated with a set of video images;

for each detected event, storing a record of the event and associating with the record geographical information about the event;

receiving a search request related to one or more events, the search request including one or more natural language terms;

searching from among the stored records for events that satisfy the search request;

based on the results of the search, providing a natural language description for each automatically detected event, the natural language description for each automatically detected event including the geographical information about that event; and

based on the results of the search, for each automatically detected event, causing a display device to display a map of a geographical area with the natural language description including the geographical information overlaid on the map.

71. The video searching method of claim 70, further comprising:

based on the results of the search, for each detected event, additionally transmitting a video clip described by the natural language description and associated with the geographical information, and/or with universal time information.

72. The video searching method of claim 71, further comprising:

based on the results of the search, for each detected event, additionally providing, along with the natural language description and with the geographical information, universal time information, and/or a single image associated with the event.

73. The video searching method of claim 70, wherein part of the natural language description for an associated event comprises a link to a video clip for the associated event.

74. The video searching method of claim 73, wherein the video clip includes a series of video images of the associated event and an embedded natural language textual description of the event.

75. The video searching method of claim 70, wherein each video source is a video camera.