Method and apparatus for qualitative spatiotemporal data processing

• US 20010043722A1
• Filed: 03/12/2001
• Published: 11/22/2001
• Est. Priority Date: 03/10/2000
• Status: Active Grant

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Bergen, James Russell, Wildes, Richard Patrick
• Granted Patent: US 6,535,620 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is embodied in a method for representing and analyzing spatiotemporal data in order to make qualitative yet semantically meaningful distinctions among various regions of the data at an early processing stage. In one embodiment of the invention, successive frames of image data are analyzed to classify spatiotemporal regions as being stationary, exhibiting coherent motion, exhibiting incoherent motion, exhibiting scintillation and so lacking in structure as to not support further inference. The exemplary method includes filtering the image data in a spatiotemporal plane to identify regions that exhibit various spatiotemporal characteristics. The output data provided by these filters is then used to classify the data.

Claims

1. A method to generate a plurality of relevant spatiotemporal descriptors for a sequence of N-dimensional data sets, each data set including a plurality of scalar quantities arranged in the N-dimensional coordinates and a temporal coordinate, where N is an integer, the method, comprising the steps of:

(a) filtering a volume of the scalar quantities according to a plurality of oriented filter characteristics to produce a respective plurality of spatiotemporal filter values corresponding to the scalar quantities in the selected volume; and

(b) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of spatiotemporal filter values.

2. A method according to

claim 1

, wherein the volume is a plane of scalar quantities and step (a) further includes the step of filtering each image in the sequence of data sets in a spatial direction orthogonal to the plane of scalar quantities.

3. A method according to

claim 1

, wherein the plurality of oriented filter characteristics in step (a) include:

a first direction motion filter characteristic, a second direction motion filter characteristic, a static filter characteristic, and a flicker filter characteristic.

4. A method according to

claim 1

, wherein the plurality of relevant spatiotemporal descriptors are selected from a set of spatiotemporal descriptors including unstructured, static, flicker, coherent motion, incoherent motion, and scintillation.

5. A method according to

claim 1

, wherein step (a) includes the step of filtering the volume of scalar quantities to determine the plurality of filter values corresponding to the scalar quantities of the volume using at least one of an oriented energy filter, an angularly tuned Gabor filter, a lognormal filter, and a derivative of Gaussian filter.

6. A method according to

claim 1

, wherein the step (b) includes the step of filtering the volume of scalar quantities to determine the plurality of filter values corresponding to the scalar quantities using at least one derivative of Gaussian filter and its corresponding Hilbert transform filter.

7. The method of

claim 1

, further comprising the step of filtering a subset of the plurality of filter values to determine a boundary between respective first and second regions of scalar quantities of the volume, the first region of scalar quantities having a common relevant spatiotemporal descriptor.

8. The method of

claim 7

, wherein the step of filtering a subset of the plurality of filter values to determine the boundary between the first and second regions of scalar quantities in the volume, includes the step of filtering the subset of the plurality of filter values applying a difference of Gaussians operator to generate a plurality of boundary filter values.

9. A method according to

claim 1

, wherein step (a) includes the steps of:

(a1) defining a plurality of spatiotemporal sub-band data sets for the volume of scalar quantities; and

(a2) filtering at least one spatiotemporal sub-band image of the plurality of spatiotemporal sub-band data sets to determine the plurality of filter values corresponding to the plurality of scalar quantities of the volume.

10. A method according to

claim 9

, wherein:

the at least one spatiotemporal sub-band data set includes at least two spatiotemporal sub-band data sets; and

step (b) includes the steps of;

(b1) determining a first plurality of spatiotemporal sub-descriptors responsive to a subset of the filter values corresponding to a first spatiotemporal sub-band data set of the at least two spatiotemporal sub-band data sets;

(b2) determining a second plurality of spatiotemporal sub-descriptors responsive to a subset of the filter values corresponding to a second spatiotemporal sub-band data set of the at least two spatiotemporal sub-band data sets; and

(b3) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of first spatiotemporal sub-descriptors and the plurality of second-spatiotemporal sub-descriptors.

11. A method according to

claim 1

, wherein the sequence of data sets includes only every N-th data set in a group of consecutive data sets, where N is an integer greater than 1.

12. A method according to

claim 11

, further including the steps of:

generating a first set of descriptors by performing steps (a) and (b) with N equal to a first integer I;

generating a second set of descriptors by performing steps (a) and (b) with N equal to a second integer K; and

combining the first set of descriptors and the second set of descriptors to generate the plurality of relevant spatiotemporal descriptors.

13. A method to generate a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of vertical columns, comprising the steps of:

(a) storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

(b) filtering each image in the sequence of video images using a one-dimensional spatial filter oriented in the y dimension;

(c) selecting a plane of pixels of the sequence of video images corresponding to the t dimension and the x dimension;

(d) filtering the selected plane of pixels to produce a plurality of horizontal spatiotemporal filter values corresponding to the pixels in the selected plane of pixels; and

(e) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of horizontal spatiotemporal filter values.

14. A method according to

claim 13

, further including the steps of:

(f) filtering each image in the sequence of video images using a one-dimensional spatial filter oriented in the x dimension;

(g) selecting a plane of pixels of the sequence of video images corresponding to the t dimension and the y dimension;

(h) filtering the selected plane of pixels to produce a plurality of vertical spatiotemporal filter values corresponding to the pixels in the selected plane of pixels;

wherein step (e) includes the step of determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of horizontal spatiotemporal filter values and the plurality of vertical spatiotemporal values.

15. A method of indexing a sequence of N-dimensional data sets of scalar quantities for storage in a database of N-dimensional data sets, where N is an integer, the method comprising the steps of:

(a) filtering a volume of the scalar quantities using a plurality of oriented filter characteristics to produce a respective plurality of spatiotemporal filter values corresponding to the scalar quantities in the data sets;

(b) combining the plurality of spatiotemporal filter values to define a corresponding plurality of spatiotemporal descriptors;

(c) associating the plurality of spatiotemporal descriptors with the sequence of N-dimensional data sets as an index value; and

(d) storing the sequence of N-dimensional data sets in the database.

16. A method of automatically selecting a region of interest of a scene using a sequence of N-dimensional data sets, each N-dimensional data set including a plurality of scalar quantities arranged in N-dimensions, where N is an integer, the method comprising the steps of:

(a) filtering a volume of the scalar quantities to produce a plurality of spatiotemporal filter values corresponding to the scalar quantities data sets;

(b) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of spatiotemporal filter values; and

(c) analyzing the plurality of spatiotemporal descriptors to identify the region if interest.

17. Apparatus for generating a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of vertical columns, comprising:

a memory for storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

a plurality of filters which filter a volume of pixels to produce a respective plurality of spatiotemporal filter values corresponding to the pixels in the sequence of video images; and

a classifier which determines the plurality of relevant spatiotemporal descriptors from the plurality of spatiotemporal filter values.

18. Apparatus according to

claim 17

, wherein the plurality of filters include a plurality of oriented low-pass filters including:

a first filter oriented in the temporal dimension;

a second filter oriented in the at least one of the x-dimension and the y-dimension; and

a third filter oriented in a direction including both the temporal dimension and the at least one of the x-dimension and the y-dimension.

19. Apparatus according to

claim 18

, further including a fourth filter oriented in a direction opposite to the orientation direction of the third filter.

20. Apparatus according to

claim 19

, wherein each of the first, second, third and fourth filters includes a second derivative Gaussian filter and its corresponding Hilbert transform filter.

21. A computer-readable carrier including computer program instructions that instruct a computer to generate a plurality of relevant spatiotemporal descriptors for a sequence of video images, each video image including a plurality of pixels arranged in a matrix having a plurality of horizontal rows and a plurality of s vertical columns, comprising the steps of:

(a) storing the sequence of video images as a three dimensional array having an x dimension corresponding to the horizontal rows of pixels in the video images, a y dimension corresponding to the vertical columns of pixels in the video images, and a t dimension corresponding to other images in the sequence;

(b) filtering a volume of pixels to produce a plurality of spatiotemporal filter values corresponding to the pixels in sequence of video images; and

(c) determining the plurality of relevant spatiotemporal descriptors responsive to the plurality of spatiotemporal filter values.