Synchronous video and data annotations

• US 7,446,803 B2
• Filed: 12/15/2003
• Issued: 11/04/2008
• Est. Priority Date: 12/15/2003
• Status: Active Grant

• Current Assignee: Columbia Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Honeywell International Inc.
• Inventors: Ng, Kevin, Leow, Woei Ling, Tham, Minope, Liew, Hai Seng Michael
• Primary Examiner(s): Ometz; David
• Assistant Examiner(s): Negron; Wanda M
• Application Number:
  US10/736,113
• Publication Number:
  US 20050128318A1
• Time in Patent Office: 1,786 Days
• Field of Search: 348/153, 348/211.3, 348/231.2, 348/231.3, 348/231.5, 348/231.99
• US Class Current: 348/231.5
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/231 Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/9201 involving the multiplexing of an additional signal and the video signal
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A video surveillance system captures video of a protected area, stores the video in a computer readable memory, stores data annotations in the computer readable memory where the data annotations are searchable using a search criteria, and stores links that link the stored data annotations to corresponding video segments of the stored video so that the data annotations can be used to search for a video segment of interest. The data annotations are searched using the search criteria, and the video segment of interest linked to the data annotation found as a result of the search is displayed.

Claims

1. A surveillance system comprising:

a camera arranged to output images of a protected area;

an input device arranged to provide a data annotation; and

a server arranged to synchronously store the images and the data annotation so that the data annotation can be used to search for a segment of the images;

wherein the server is arranged to time stamp the data annotation; and

further

wherein the server is arranged to directly compare the time stamp of the data annotation to an image count when searching for the segment of the images.

2. The surveillance system of claim 1 wherein the server is arranged to cause the segment of the images matching the time stamp to be displayed.

3. The surveillance system of claim 1 wherein the camera comprises a video camera, and wherein the server comprises a video server.

4. The surveillance system of claim 1 wherein the camera comprises an IR camera.

5. The surveillance system of claim 1 wherein the camera comprises a thermal imager.

6. The surveillance system of claim 1 wherein the server is arranged to save the data annotation in SQL readable form.

7. The surveillance system of claim 6 wherein the server is arranged to time stamp the data annotation.

8. The surveillance system of claim 7 wherein the server is arranged to match the time stamp of the data annotation to an image count when searching for the segment of the images.

9. The surveillance system of claim 6 wherein the server is arranged to receive an SQL search string corresponding to the data annotation to be searched and to search for the data annotation based on the SQL search string.

10. The surveillance system of claim 9 wherein the server is arranged to match the data annotation found as a result of the search to the segment of the images.

11. A method comprising:

storing surveillance video in a memory;

storing data annotations in the memory, wherein the data annotations are useful in searching for a video segment of the surveillance video of interest; and

,

synchronizing the stored data annotations to the corresponding video segments of the stored video so that the data annotations can be used to search for the video segment of interest;

wherein the data annotations include a time stamp; and

further comprising directly comparing the time stamp of the data annotations to an image count when searching for the segment of the images.

12. The method of claim 11 further comprising searching for a particular data annotation.

13. The method of claim 12 further comprising comparing the time stamp of the particular data annotation to a timing of the video when searching for the video segment of interest.

14. The method of claim 13 further comprising displaying the video segment of interest that matches the time stamp of the particular data annotation.

15. The method of claim 11 further comprising searching the data annotations using a search criteria.

16. The method of claim 15 wherein the synchronizing of the stored data annotations to the corresponding video segments comprises time stamping the data annotations with corresponding time stamps.

17. The method of claim 16 further comprising searching for a particular data annotation.

18. The method of claim 17 wherein the searching includes matching the time stamp of the particular data annotation to a timing of the video.

19. The method of claim 18 further comprising displaying the video segment of interest that matches the time stamp of the particular data annotation.

20. The method of claim 15 wherein the searching of the data comprises using SQL to conduct the search.

21. The method of claim 20 wherein the synchronizing of the stored data annotations to the corresponding video segments of the stored video comprises time stamping the data annotations with corresponding time stamps, wherein the searching comprises matching the time stamp associated with the stored data annotation that corresponds to the search criteria to a timing of the video, and wherein the method further comprises displaying the video segment of interest whose timing matches the time stamp associated with the stored data annotation that corresponds to the search criteria.

22. A surveillance method comprising:

capturing images of a protected area;

storing the images in a computer readable memory;

storing data annotations in the computer readable memory, wherein the data annotations are searchable using a search criteria; and

,

storing a link that links the stored data annotations to corresponding image segments of the stored images so that the data annotations can be used to search for an image segment of interest;

wherein the data annotations include a time stamp; and

further comprising directly comparing the time stamp of the data annotations to an image count when searching for the segment of the images.

23. The surveillance method of claim 22 further comprising searching for a particular data annotation using the search criteria.

24. The surveillance method of claim 23 further comprising displaying the image segment of interest linked to the particular data annotation found as a result of the search.

25. The surveillance method of claim 22 further comprising searching for a particular data annotation using the search criteria.

26. The surveillance method of claim 25 further comprising comparing the time stamp corresponding to the particular data annotation found as a result of the search to a timing of the images when searching for the image segment of interest.

27. The surveillance method of claim 26 further comprising displaying the image segment of interest whose timing matches the time stamp of the particular data annotation.

28. The surveillance method of claim 22 wherein the search criteria comprises an SQL search criteria.

29. The surveillance method of claim 28 further comprising:

searching for a particular data annotation using the SQL search criteria;

finding the image segment of interest linked to the particular data annotation; and

,

displaying the image segment of interest.

30. The surveillance method of claim 22 wherein the link comprises a data attribute stamp, and wherein the data attribute serves as an index to retrieve video and data segments of the same characteristic inferred by the data attribute.

31. The surveillance method of claim 30 wherein the data attribute comprises temperature.

32. The surveillance method of claim 30 wherein the data attribute comprises luminosity.

33. The surveillance method of claim 30 wherein the data attribute comprises a biometric signature.