Specifying search criteria for searching video data

• US 8,553,084 B2
• Filed: 09/12/2006
• Issued: 10/08/2013
• Est. Priority Date: 09/12/2005
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Vallone, Robert P., Wells, Michael E., Hale, Shannon P., Fleischer, Stephen D., Haupt, Gordon T., Pitts, Colvin H.
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US11/520,116
• Publication Number:
  US 20070061696A1
• Time in Patent Office: 2,583 Days
• Field of Search: 348/143, 348/152, 348/154, 348/155, 348/699
• US Class Current: 348/143
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19676 Temporary storage, e.g. cyclic memory, buffer storage on pre-alarm
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/322 used signal is digitally coded
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/45455 applied to a region of the image
• CPC Class Codes: H04N 21/45457 applied to a time segment
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739

Abstract

A method and apparatus is described for specifying regions of interest within a two-dimensional view of visual information that comprises a series of frames. Visual changes that occur in the view are stored. A user enters search criteria that specify at least one first region of interest within the view and a visual change. A visual change may include a change in pixel values or a detection of motion of one or more objects within the view. The first search criteria are compared against the stored visual changes to identify a sequence of frames in which the specified visual change occurred within the first region of interest. The search criteria may specify multiple regions of interest, each with one or more types of visual changes. If a motion is specified, then a direction, speed, and behavior of a moving object may also be specified.

Claims

1. A machine-implemented method, comprising:

storing change information about visual changes that occur in a two-dimensional view of visual information;

causing a user interface to be displayed that allows a user to specify search criteria;

wherein one search criteria option that the user interface allows the user to specify is a scope-of-change indication;

wherein the user interface presents controls for selecting between (1) a first scope-of-change option that indicates that a visual change must occur anywhere within a specified region of interest and (2) a second scope-of-change option that indicates that a visual change must occur across the entire specified region of interest;

receiving, from the user through the user interface, first search criteria that specifies;

a first region of interest within the view,

a particular visual change, and

a scope-of-change indication that indicates one of the first scope-of-change option or the second scope-of-change option for said particular visual change; and

comparing the first search criteria against said change information to identify a sequence of frames, within the visual information, (a) in which said particular visual change occurred within said first region of interest and (b) that satisfy the scope-of-change indication;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the particular visual change is a detection of motion.

3. The method of claim 1, wherein said particular visual change indicates a change in pixel values.

4. The method of claim 2, wherein said particular visual change further indicates a direction of said motion.

5. The method of claim 4, wherein said scope-of-change indication indicates that the particular visual change must occur anywhere within the first region of interest.

6. The method of claim 2, wherein said particular visual change indicates a speed of said motion.

7. The method of claim 6, wherein said scope-of-change indication indicates that the particular visual change must occur across the entire first region of interest.

8. The method of claim 2, wherein said particular visual change indicates a shape of said motion.

10. The method of claim 1, wherein:

said visual information comprises a plurality of frames;

comparing the first search criteria against said change information to identify the sequence of frames is performed on change data corresponding to frames that are appended to the plurality of frames after the first search criteria is received.

11. The method of claim 1, wherein:

said visual information comprises a plurality of frames;

comparing the first search criteria against said change information to identify the sequence of frames is performed on change data corresponding to frames that were added to the plurality of frames before the first search criteria is received.

16. The method of claim 1, wherein:

comparing said first search criteria includes;

identifying a first set of video data change records based on a second region within the view, wherein;

a video data change record corresponds to a plurality of frames of the visual information and includes change data that indicates visual changes that were detected relative to the corresponding plurality of frames, and

the second region includes said first region of interest and is larger than said first region of interest; and

based on the first set of video data change records and said first region, identifying a second set of video data change records, wherein the change data of each video data change record in the second set satisfies the first search criteria.

17. The method of claim 1, wherein:

comparing said first search criteria includes;

identifying a first set of video data change records based on a first time interval, wherein a video data change record;

corresponds to a plurality of frames of the visual information, and

includes change data that indicates visual changes that were detected relative to the corresponding plurality of frames; and

based on said first set of video data change records, identifying a second set of video data change records, wherein the change information of each video data change record in the second set satisfies the first search criteria.

9. A machine-implemented method comprising:

storing change information about visual changes that occur in a two-dimensional view of visual information;

receiving, from a user, first search criteria that specifies;

a first region of interest within the view,

a second region of interest within the view, wherein said second region of interest occupies a portion of the view that is not occupied by said first region of interest,

one or more visual changes,

a first scope-of-change indication for a first visual change of the one or more visual changes, and

a second scope-of-change indication for a second visual change of the one or more visual changes;

the first scope-of-change indication indicates that the first visual change must occur across the entire first region of interest and the second scope-of-change indication indicates that the second visual change must occur anywhere within the second region of interest; and

comparing the first search criteria against said change information to identify a sequence of frames, within the visual information, in which one of said one or more visual changes occurred within said first region of interest and in which one of said one or more visual changes occurred within said second region of interest;

wherein the method is performed by one or more computing devices.

12. The method of claim 9, wherein at least one of the one or more visual changes indicates:

a detection of motion, and

one or more characteristics of said motion.

13. The method of claim 9, where said second region of interest and said first region of interest have at least one portion of the view in common.

14. The method of claim 9, wherein:

at least one of the one or more visual changes indicates a detection of motion of one or more objects and a direction of said motion.

15. The method of claim 9, wherein:

at least one of the one or more visual changes indicates a detection of motion of one or more objects and a speed of said motion.

18. The method of claim 9, further comprising:

causing, to be displayed to the user, a user interface that allows the user to select a Boolean OR or a Boolean AND that dictates whether one or more visual changes must occur in each of the first region of interest and the second region of interest in order to satisfy the first search criteria.

19. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause:

storing change information about visual changes that occur in a two-dimensional view of visual information;

causing a user interface to be displayed that allows a user to specify search criteria;

wherein one search criteria option that the user interface allows the user to specify is a scope-of-change indication;

wherein the user interface presents controls for selecting between (1) a first scope-of-change option that indicates that a visual change must occur anywhere within a specified region of interest and (2) a second scope-of-change option that indicates that a visual change must occur across the entire specified region of interest;

receiving, from the user through the user interface, first search criteria that specifies;

a first region of interest within the view,

a particular visual change, and

a scope-of-change indication that indicates one of the first scope-of-change option or the second scope-of-change option for said particular visual change; and

comparing the first search criteria against said change information to identify a sequence of frames, within the visual information, (a) in which said particular visual change occurred within said first region of interest and (b) that satisfy the scope-of-change indication.

20. The one or more non-transitory computer-readable media of claim 19, wherein said particular visual change indicates a change in pixel values.

21. The one or more non-transitory computer-readable media of claim 19, wherein the particular visual change is a detection of motion.

22. The one or more non-transitory computer-readable media of claim 21, wherein said particular visual change further indicates a direction of said motion.

23. The one or more non-transitory computer-readable media of claim 22, wherein said scope-of-change indication indicates that the particular visual change must occur anywhere within the first region of interest.

24. The one or more non-transitory computer-readable media of claim 21, wherein said particular visual change indicates a speed of said motion.

25. The one or more non-transitory computer-readable media of claim 24, wherein said scope-of-change indication indicates that the particular visual change must occur across the entire first region of interest.

26. The one or more non-transitory computer-readable media of claim 21, wherein said particular visual change indicates a shape of said motion.

27. The one or more non-transitory computer-readable media of claim 19, wherein:

said visual information comprises a plurality of frames;

comparing the first search criteria against said change information to identify the sequence of frames is performed on change data corresponding to frames that are appended to the plurality of frames after the first search criteria is received.

28. The one or more non-transitory computer-readable media of claim 19, wherein:

said visual information comprises a plurality of frames;

comparing the first search criteria against said change information to identify the sequence of frames is performed on change data corresponding to frames that were added to the plurality of frames before the first search criteria is received.

29. The one or more non-transitory computer-readable media of claim 19, wherein:

comparing said first search criteria includes;

identifying a first set of video data change records based on a second region within the view, wherein;

a video data change record corresponds to a plurality of frames of the visual information and includes change data that indicates visual changes that were detected relative to the corresponding plurality of frames, and

the second region includes said first region of interest and is larger than said first region of interest; and

based on the first set of video data change records and said first region, identifying a second set of video data change records, wherein the change data of each video data change record in the second set satisfies the first search criteria.

30. The one or more non-transitory computer-readable media of claim 19, wherein:

comparing said first search criteria includes;

identifying a first set of video data change records based on a first time interval, wherein a video data change record;

corresponds to a plurality of frames of the visual information, and

includes change data that indicates visual changes that were detected relative to the corresponding plurality of frames; and

based on said first set of video data change records, identifying a second set of video data change records, wherein the change information of each video data change record in the second set satisfies the first search criteria.

31. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause:

storing change information about visual changes that occur in a two-dimensional view of visual information;

receiving, from a user, first search criteria that specifies;

a first region of interest within the view,

a second region of interest within the view, wherein said second region of interest occupies a portion of the view that is not occupied by said first region of interest,

one or more visual changes,

a first scope-of-change indication for a first visual change of the one or more visual changes, and

a second scope-of-change indication for a second visual change of the one or more visual changes;

the first scope-of-change indication indicates that the first visual change must occur across the entire first region of interest and the second scope-of-change indication indicates that the second visual change must occur anywhere within the second region of interest; and

comparing the first search criteria against said change information to identify a sequence of frames, within the visual information, in which one of said one or more visual changes occurred within said first region of interest and in which one of said one or more visual changes occurred within said second region of interest.

32. The one or more non-transitory computer-readable media of claim 31, where said second region of interest and said first region of interest have at least one portion of the view in common.

33. The one or more non-transitory computer-readable media of claim 31, wherein:

at least one of the one or more visual changes indicates a detection of motion of one or more objects and a direction of said motion.

34. The one or more non-transitory computer-readable media of claim 31, wherein:

at least one of the one or more visual changes indicates a detection of motion of one or more objects and a speed of said motion.

35. The one or more non-transitory computer-readable media of claim 31, wherein at least one of the one or more visual changes indicates:

a detection of motion, and

one or more characteristics of said motion.

36. The one or more non-transitory computer-readable media of claim 31, wherein the instructions, when executed by the one or more processors, further cause:

causing, to be displayed to the user, a user interface that allows the user to select a Boolean OR or a Boolean AND that dictates whether one or more visual changes must occur in each of the first region of interest and the second region of interest in order to satisfy the first search criteria.