Adaptive artificial vision method and system

• US 20040071347A1
• Filed: 10/07/2003
• Published: 04/15/2004
• Est. Priority Date: 10/08/2002
• Status: Active Grant

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Kaplan, Frederick
• Granted Patent: US 7,477,759 B2
• Time in Patent Office: Days
• US Class Current: 382/217
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

The adaptive artificial vision method comprises the following steps: (a) defining successive couples of timesteps (t₋₁, t; t, t₊₁; . . . ) synchronized by a clock (101), (b) comparing two successive images (I_t−, I_t; I_t, I_t+1, . . . ) from an input device (102, 103) at each couple of synchronized timesteps (t₋₁, t; t, t₊₁; . . . ) spaced by a predetermined time delay τ₀ for obtaining a delta image Δ_t which is the result of the computation of the distance between each pixel of the two successive images (I_t−1, I_t; I_t, I_t+1, . . . ) in view of characterizing movements of objects, (c) extracting features from the delta image Δ_t for obtaining a potential dynamic patch P_t which is compared with dynamic patches previously recorded in a repertory which is progressively constructed in real time from an initial void repertory, (d) selecting the closest dynamic patch D_i in the repertory or if no sufficiently close dynamic patch still exists, adding the potential dynamic patch P_t to the repertory and therefore obtaining and storing a dynamic patch D_i from the comparison of two successive images (I_t−1, I_t; I_t, I_t+1, . . . ) at each couple of synchronized timesteps (t₋₁, t; t, t₊₁; . . . ), and (e) temporally integrating stored dynamic patches Di of the repertory in order to detect and store stable sets of active dynamic patches representing a characterization of a reoccuring movement or event which is observed. A process of static pattern recognition may then be efficiently used.

Claims

1. An adaptive artificial vision method comprising the following steps:

(a) defining successive couples of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) such that the time difference τ

between two synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) of a couple of synchronized timesteps is equal to a predetermined time delay τ

₀,

(b) comparing two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) at each couple of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) spaced by said predetermined time delay τ

₀ for obtaining a delta image Δ

_t which is the result of the computation of the distance between each pixel of said two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) in view of characterizing movements of objects between said two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ),

(c) extracting features from said delta image Δ

_t for obtaining a potential dynamic patch P_t which is compared with dynamic patches previously recorded in a first repertory R_d which is progressively constructed in real time from an initial void repertory,

(d) selecting the closest dynamic patch D_i in the first repertory R_d or if no sufficiently close dynamic patch still exists, adding the potential dynamic patch P_t to the first repertory R_d and therefore obtaining and storing a dynamic patch D_i from the comparison of two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) at each couple of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ), and

(e) temporally integrating stored dynamic patches Di of the first repertory R_d in order to detect and store stable sets of active dynamic patches representing a characterization of a reoccuring movement or event which is observed.

2. A method according to claim 1, wherein when stable sets of active dynamic patches representing a characterization of a reoccuring movement have been detected, the center of the movement is identified and static patches which are at a predetermined distance d from the movement center and are obtained by a process of static pattern recognition are analyzed to constitute at a given timestep a set of active static patches S_i which are stored in a second repertory R_s.

3. A method according to claim 2, wherein stored static patches S_i of the second repertory R_s are spatially integrated in order to detect and store stable sets of active static patches representing a characterization of an object which is recurrently involved in observed known reoccuring movements.

4. A method according to claim 2 or claim 3, wherein the process of static pattern recognition and production of static patches is initiated after stable sets of active dynamic patches representing a characterization of a reoccuring movement have been detected.

5. A method according to claim 2 or claim 3, wherein the process of static pattern recognition and production of static patches is initiated at the same time as the process of dynamic movement recognition and production of dynamic patches and when stable sets of active dynamic patches representing a characterization of a reocccuring movement have been detected, the process of static pattern recognition is continued exclusively with static patches which are located in a restricted area of the image which is centered on said identified movement center.

6. A method according to anyone of claims 1 to 5, wherein during the computation of the distance between each pixel of two successive images (I_t−

1, I_t), a filter function f_th is used to keep only the most significant differences and therefore obtain a delta image Δ

_t such that

Δ

_t=f_th(∥

(I<sub>t−

1</sub>, I_t)∥

)

7. A method according to claim 6, wherein the filter function f_th is a threshold function.

8. A method according to anyone of claims 1 to 7, wherein the step of extracting features from the delta image Δ

_t comprises computing a gaussian color model of the distribution for each color component.

9. A method according to anyone of claims 1 to 7, wherein the step of extracting features from the delta image Δ

_t comprises using histograms to model the distribution for color components, shape or texture.

10. A method according to anyone of claims 2 to 5, wherein static patches are obtained on the basis of salient points (x,y) in an image I_t provided at a synchronized timestep t when a salient point (x,y) is detected, a region R_x,y corresponding to the surrounding pixels is defined and features are extracted from this region R_x,y to define a potential static patch S_x,y.

11. A method according to claim 10, wherein the extraction of features from the region R_x,y comprises measuring the color change of a pixel compared to its neighbors and computing a color model of the color distribution in the region R_x,y.

12. A method according to anyone of claims 1 to 11, wherein successive steps of synchronized timesteps (t₋

1, t;

T+t<sub>−

1</sub>;

T+t;

. . . ) are separated by a period of time T which is equal to n times the predetermined time delay τ

₀, where n is an integer which is positive or equal to zero.

13. A method according to claim 12, wherein successive couples of synchronized timesteps (t₋

1, t;

t, t₊₁;

. . . ) are contiguous without any time interruption between two successive couples of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁).

14. A method according to anyone of claims 1 to 13, wherein it further comprises the step of detecting transitions between stable sets of active dynamic patches representing a characterization of reoccuring movements and of constructing transition graphs for predicting complex events comprising a sequence of identified movements.

15. An adaptive artificial vision system comprising:

a clock (101) for defining successive couples of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) such that the time difference τ

between two synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) of a couple of synchronized timesteps is equal to a predetermined time delay τ

₀,

inputting means (102) for inputting images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) provided by a camera (103) at said synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ),

first comparator means (104) for comparing two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) inputted at each couple of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ) spaced by said predetermined time delay τ

₀ for obtaining a delta image Δ

_t which is the result of the computation of the distance between each pixel of said two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ),

first memory means (M_d) for storing dynamic patches Di representing elementary visual parts for describing characterized movements of objects,

feature extraction means (105) for extracting features from said delta image Δ

_t and producing a potential dynamic patch P_t,

second comparator means (106) for comparing a potential dynamic patch P_t which is compared with dynamic patches previously recorded in said first memory means (M_d),

selection means (107) for selecting the closest dynamic patch D_i in the first memory means (M_d) or if no sufficiently close dynamic patch still exists, for recording the potential dynamic patch P_t into the first memory means so that a dynamic patch D_i is stored in the first memory means for each comparison of two successive images (I<sub>t−

1</sub>, I_t;

I_t, I_t+1, . . . ) at each couple of synchronized timesteps (t<sub>−

1</sub>, t;

t, t₊₁;

. . . ),

first temporal integration means (108) comprising computing means (108A) for computing during a time T_F1 corresponding to a predetermined number N1 of couples of synchronized timesteps the frequency of each dynamic patch D_i stored in the first memory means and threshold means (108B) for defining a set of active dynamic patches comprising dynamic patches D_i whose frequency is higher than a predetermined threshold, and,

second temporal integration means (107) comprising computing means (109A) for computing during a time T_F2 corresponding to a predetermined number N2 of couples of synchronized timesteps the frequency of each set of defined active dynamic patches and threshold means (109B) for defining a stable set of dynamic patches corresponding to a reoccuring movement for each set of active dynamic patches whose frequency is higher than a predetermined threshold.

16. A system according to claim 15, wherein it further comprises means (110) for identifying the center of a reoccuring movement represented by a stable set of active dynamic patches and means (111) for triggering static pattern recognition (112) for analyzing static patches which are at a predetermined distance d from said center of a reoccuring movement.