Method and apparatus for qualitative spatiotemporal data processing

• US 6,535,620 B2
• Filed: 03/12/2001
• Issued: 03/18/2003
• Est. Priority Date: 03/10/2000
• Status: Expired due to Fees

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Bergen, James Russell, Wildes, Richard Patrick
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US09/803,774
• Publication Number:
  US 20010043722A1
• Time in Patent Office: 736 Days
• Field of Search: 382/128, 382/130, 382/131, 382/132, 382/133, 382/254, 382/256, 382/260, 382/266, 382/276, 382/277, 382/303, 382/304, 348/25, 348/26, 348/428.1, 348/429.1, 348/430.1, 348/431.1, 348/439.1, 348/526, 348/620, 348/701, 348/715, 348/905, 736/09, 369/124.08, 386/5, 386/111, 386/121, 700/55
• US Class Current: 382/107
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is embodied in a method for representing and analyzing spatiotemporal data in order to make qualitative yet semantically meaningful distinctions among various regions of the data at an early processing stage. In one embodiment of the invention, successive frames of image data are analyzed to classify spatiotemporal regions as being stationary, exhibiting coherent motion, exhibiting incoherent motion, exhibiting scintillation and so lacking in structure as to not support further inference. The exemplary method includes filtering the image data in a spatiotemporal plane to identify regions that exhibit various spatiotemporal characteristics. The output data provided by these filters is then used to classify the data.

Claims

1. A method to generate a plurality of relevant spatiotemporal descriptors for a sequence of N-dimensional data sets, each data set including a plurality of scalar quantifies arranged in the N-dimensional coordinates and a temporal coordinate, where N is an integer, the method, comprising the steps of:

(a) filtering a volume of die scalar quantities according to a plurality of oriented filter characteristics to produce a respective plurality of spatiotemporal filter values corresponding to the scalar quantities in the selected volume; and

(b) determining the plurality of relevant spatiotemporal descriptors directly from the plurality of spatiotemporal filter values;

wherein;

at least one of the plurality of relevant spatiotemporal descriptors is determined from at least two of the plurality of spatiotemporal filter values; and

the plurality of relevant spatiotemporal descriptors includes at least one descriptor of image information other than coherent motion.

2. A method according to claim 1, wherein the volume is a plane of scalar quantities and step (a) further includes the step of filtering each image in the sequence of data sets in a spatial direction orthogonal to the plane of scalar quantities.

3. A method according to claim 1, wherein the plurality of relevant spatiotemporal descriptors are selected from a set of spatiotemporal descriptors including unstructured, static, flicker, coherent motion, incoherent motion, and scintillation.

4. A method according to claim 1, wherein step (a) includes the step of filtering the volume of scalar quantities to determine the plurality of filter values corresponding to the scalar quantities of the volume using at least one of an oriented energy filter, an angularly tuned Gabor filter, a lognormal filter, and a derivative of Gaussian filter.

5. A method according to claim 1, wherein the step (b) includes the step of filtering the volume of scalar quantities to determine the plurality of filter values corresponding to the scalar quantities using at least one derivative of Gaussian filter and its corresponding Hilbert transform filter.

6. The method of claim 1, further comprising the step of filtering a subset of the plurality of filter values to determine a boundary between respective first and second regions of scalar quantities of the volume, the first region of scalar quantities having a common relevant spatiotemporal descriptor.

7. The method of claim 6, wherein the step of filtering a subset of the plurality of filter values to determine the boundary between the first and second regions of scalar quantities in the volume, includes the step of filtering the subset of the plurality of filter values applying a difference of Gaussians operator to generate a plurality of boundary filter values.

8. A method according to claim 1, wherein step (a) includes the steps of:

(a1) defining a plurality of spatiotemporal sub-band data sets for the volume of scalar quantities; and

(a2) filtering at least one spatiotemporal sub-band image of the plurality of spatiotemporal sub-band data sets to determine the plurality of filter values corresponding to the plurality of scalar quantities of the volume.

9. A method according to claim 8, wherein:

the at least one spatiotemporal sub-band data set includes at least two spatiotemporal sub-band data sets; and

step (b) includes the steps of;

(b1) determining a first plurality of spatiotemporal sub-descriptors responsive to a subset of the filter values corresponding to a first spatiotemporal sub-band data set of the at least two spatiotemporal sub-band data sets;

(b2) determining a second plurality of spatiotemporal sub-descriptors responsive to a subset of the filter values corresponding to a second spatiotemporal sub-band data set of the at least two spatiotemporal sub-band data sets; and

(b3) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of first spatiotemporal sub-descriptors and the plurality of second-spatiotemporal sub-descriptors.

10. A method according to claim 1, wherein the sequence of data sets includes only every N-th data set in a group of consecutive data sets, where N is an integer greater than 1.

11. A method according to claim 10, further including the steps of:

generating a first set of descriptors by performing steps (a) and (b) with N equal to a first integer I;

generating a second set of descriptors by performing steps (a) and (b) with N equal to a second integer K; and

combining the first set of descriptors and the second set of descriptors to generate the plurality of relevant spatiotemporal descriptors.

12. A method to generate a plurality of relevant spatiotemporal descriptors for a sequence of N-dimensional data sets, each data set including a plurality of scalar quantities arranged in the N-dimensional coordinates and a temporal coordinate, where N is an integer, the method, comprising the steps of:

(a) filtering a volume of the scalar quantifies according to a plurality of oriented filter characteristics to produce a respective plurality of spatiotemporal filter values corresponding to the scalar quantities in the selected volume; and

(b) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of spatiotemporal filter values;

wherein the plurality of oriented filter characteristics in step (a) include;

a first direction motion filter characteristic, a second direction motion filter characteristic, a static filter characteristic, and a flicker filter characteristic.

13. A method to generate a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of vertical columns, comprising the steps of:

(a) storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

(b) filtering each image in the sequence of video images using a one-dimensional spatial filter oriented in the y dimension;

(c) selecting a plane of pixels of the sequence of video images corresponding to the t dimension and the x dimension;

(d) filtering the selected plane of pixels to produce a plurality of horizontal spatiotemporal filter values corresponding to the pixels in the selected plane of pixels; and

(e) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of horizontal spatiotemporal filter values.

14. A method according to claim 13, further including the steps of:

(f) filtering each image in the sequence of video images using a one-dimensional spatial filter oriented in the x dimension;

(g) selecting a plane of pixels of the sequence of video images corresponding to the t dimension and the y dimension;

(h) filtering the selected plane of pixels to produce a plurality of vertical spatiotemporal filter values corresponding to the pixels in the selected plane of pixels;

wherein step (e) includes the step of determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of horizontal spatiotemporal filter values and the plurality of vertical spatiotemporal values.

15. A method of indexing a sequence of N-dimensional data sets of scalar quantities for storage in a database of N-dimensional data sets, where N is an integer, the method comprising the steps of:

(a) filtering a volume of the scalar quantities using a plurality of oriented filter characteristics to produce a respective plurality of spatiotemporal filter values corresponding to the scalar quantities in the data sets;

(b) combining the plurality of spatiotemporal filter values to define a corresponding plurality of spatiotemporal descriptors, wherein;

at least one of the plurality of spatiotemporal descriptors is defined by at least two of the plurality of spatiotemporal filter values; and

the plurality of relevant spatiotemporal descriptors includes at least one descriptor of image information other than coherent motion;

(c) associating the plurality of spatiotemporal descriptors with the sequence of N-dimensional data sets as an index value; and

(d) storing the sequence of N-dimensional data sets in the database.

16. A method of automatically selecting a region of interest of a scene using a sequence of N-dimensional data sets, each N-dimensional data set including a plurality of scalar quantities arranged in N-dimensions, where N is an integer, the method comprising the steps of:

(a) filtering a volume of the scalar quantities to produce a plurality of spatiotemporal filter values corresponding to the scalar quantities data sets;

(b) determining a plurality of relevant spatiotemporal descriptors directly from the plurality of spatiotemporal filter values, wherein the plurality of relevant spatiotemporal descriptors includes at least one descriptor of image information other than coherent motion; and

(c) analyzing the plurality of spatiotemporal descriptors to identify the region of interest.

17. Apparatus for generating a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of vertical columns, comprising:

a memory for storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

a plurality of filters which filter a volume of pixels to produce a respective plurality of spatiotemporal filter values corresponding to the pixels in the sequence of video images; and

a classifier which determines the plurality of relevant spatiotemporal descriptors directly from the plurality of spatiotemporal filter values;

wherein;

at least one of the plurality of relevant spatiotemporal descriptors is determined from at least two of the plurality of spatiotemporal filter values; and

the plurality of relevant spatiotemporal descriptors includes at least one descriptor of image information other than coherent motion.

18. Apparatus according to claim 17, wherein the plurality of filters include a plurality of oriented low-pass filters including:

a first filter oriented in the temporal dimension;

a second filter oriented in the at least one of the x-dimension and the y-dimension; and

a third filter oriented in a direction including both the temporal dimension and the at least one of the x-dimension and the y-dimension.

19. Apparatus according to claim 18, further including a fourth filter oriented in a direction opposite to the orientation direction of the third filter.

20. Apparatus according to claim 19, wherein each of the first, second, third and fourth filters includes a second derivative Gaussian filter and its corresponding Hilbert transform filter.

21. A computer-readable carrier including computer program instructions that instruct a computer to generate a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of vertical columns, comprising the steps of:

(a) storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

(b) filtering a volume of pixels to produce a plurality of spatiotemporal filter values corresponding to the pixels in sequence of video images; and

(c) determining the plurality of relevant spatiotemporal descriptors directly from the plurality of spatiotemporal filter values;

wherein;

at least one of the plurality of relevant spatiotemporal descriptors is determined from at least two of the plurality of spatiotemporal filter values; and

the plurality of relevant spatiotemporal descriptors includes at least one descriptor of image information other than coherent motion.