Passive and interactive real-time image recognition software method

• US 20070292033A1
• Filed: 06/19/2006
• Published: 12/20/2007
• Est. Priority Date: 06/19/2006
• Status: Abandoned Application

• Current Assignee: Chao-Wang Hsiung, Chih Hung Chuang, Hsien-Wen Chang
• Original Assignee: Chao-Wang Hsiung, Chih Hung Chuang, Hsien-Wen Chang
• Inventors: Chang, Hsien-Wen, Hsiung, Chao-Wang, Chuang, Chih Hung
• Application Number:
  US11/455,187
• Publication Number:
  US 20070292033A1
• Time in Patent Office: Days
• US Class Current: 382/218
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching

Abstract

This invention relates to a passive and interactive real-time image recognition software method, particularly to a real-time image recognition software method without the effects of the ambient light sources and noises, which includes passive and interactive recognition methods.

Claims

1. Whereas, the passive real-time image recognition method is described as follows:

Step 1;

Capture an image projected by an image projection apparatus to image areas as reference images (5×

5 grey-level value) by using a video camera;

Step 2;

Continuously capture real-time images (5×

5 grey-level value) projected by an image projection apparatus to image areas by using a video camera, and check if any foreign object touches the reactive area.

The difference value between the reference image from step 1 and the real-time image from step 2 can be denoted as follows (1);

DIFF(x,y)=|REF(x,y)−

NEW(x,y)| 

 

(1)

Step 3;

Difference of the grey-level value of real-time image in step 2 and the grey-level value of reference image in step 1 to have the grey-level distribution of remaining images, which means foreign objects exist.

Step 4;

The image which is subject to differencing through step 3 usually has noises, which can be present as in formula (2)

$\begin{array}{cc}\mathrm{BIN}<br><br>\left(x,y\right)=\{\begin{array}{cc}255& \mathrm{DIFF}<br><br>\left(x,y\right)\ge <br><br>T^{*}\\ 0& \mathrm{DIFF}<br><br>\left(x,y\right)<<br><br>T^{*}\end{array}& \left(2\right)\end{array}$

The binarization method eliminates the noises;

in which, T* represents a threshold, in 8 bit grey-scale image and the threshold ranges from 0 to 255. The optimal threshold can be decided by a statistical method. The optimal threshold is on the wave trough of the grey-level value;

when T* is decided, the image can be segmented into two sections. The requirement for the optimal threshold T* is when the sum of variances in C₁ and the variances in C₂ has the minimum value. It is assumed that the size of the image is N=5×

5, and the grey-level value number of 8 bit grey-level image is I=256. Then the probability of grey-level value is I can be denoted as;

$\begin{array}{cc}P<br><br>\left(i\right)=\frac{n_i}{N}& \left(3\right)\end{array}$

Wherein n_i indicates the appearance number of grey-level value I, and the range of I is 0≦

i≦

I−

1. According to the probability principle, the following can be obtained;

$\begin{array}{cc}\underset{i=0}{\overset{I-1}{\sum <br><br>}}<br><br>P<br><br>\left(i\right)=1& \left(4\right)\end{array}$

Suppose the ratio of the pixel number in C₁ is;

$\begin{array}{cc}{W}_1=\mathrm{Pr}<br><br>\left(C_1\right)=\underset{i=0}{\overset{T^{*}}{\sum <br><br>}}<br><br>P<br><br>\left(i\right)& \left(5\right)\end{array}$

While the ratio of the pixel number in C₂ is;

$\begin{array}{cc}{W}_2=\mathrm{Pr}<br><br>\left(C_2\right)=\underset{i=T^{*}+1}{\overset{I-1}{\sum <br><br>}}<br><br>P<br><br>\left(i\right)& \left(6\right)\end{array}$

Here W₁+W₂=1 can be satisfied.

The expect value of C₁ can be calculated as;

$\begin{array}{cc}{U}_1=\underset{i=0}{\overset{T^{*}}{\sum <br><br>}}<br><br>\frac{P<br><br>\left(i\right)}{W_1}\times <br><br>i& \left(7\right)\end{array}$

The expect value of C₂ is;

$\begin{array}{cc}{U}_2=\underset{i=T^{*}+1}{\overset{I-1}{\sum <br><br>}}<br><br>\frac{P<br><br>\left(i\right)}{W_2}\times <br><br>i& \left(8\right)\end{array}$

The variance of C₁ and C₂ can be obtained by using the formula (7) and (8).

$\begin{array}{cc}{\mathrm{\sigma <br><br>}}_1^2=\underset{i=0}{\overset{T^{*}}{\sum <br><br>}}<br><br>{\left(i-U_1\right)}^2<br><br>\frac{P<br><br>\left(i\right)}{W_1}& \left(9\right)\\ {\mathrm{\sigma <br><br>}}_2^2=\underset{i=T^{*}+1}{\overset{I-1}{\sum <br><br>}}<br><br>{\left(i-U_2\right)}^2<br><br>\frac{P<br><br>\left(i\right)}{W_2}& \left(10\right)\end{array}$

The sum of variance in C₁ and C₂ are;

σ

_w²=W₁σ

₁²+W₂σ

₂²  

 

(11)

Substitute the value 0-255 for formula (11). When the formula (11) has the minimum value, then the optimal threshold T* can be obtained.

Step 5;

Although the residual noises have been removed through binarization in step 4, however, the moving object becomes dilapidated. This can be removed by using four connected masks and the inflation and erosion algorithm.

The inflation algorithm is described as follows;

when M_b(i,j)=255, set the mask of the 4-neighbor points as

M_b(i,j−

1)=M_b(i,j+1)=M_b(i−

1,i)=M_b(i+1,j)=255  

 

(12)

The erosion algorithm is described as follows;

when M_b(i,j)=0, set the mask of the 4 neighbor points as

M_b(i,j−

1)=M_b(i,j+1)=M_b(i−

1,j)=M_b(i+1,j)=0  

 

(13)

Convoluting the above-mentioned mask and binarized image can eliminate the dilapidation.

Step 6;

Next, the lateral mask can be used to obtain the contours of the moving object. Where, the Sobel (the image contour operation mask) is used to obtain the object contours.

Convolute the Sobel (the image contour operation mask) mask and the real-time image, which can be denoted by formula (14) and (15);

G_x(x,y)=(NEW(x−

1,y+1)+2×

NEW(x,y+1)+NEW(x+1,y+1))−

(NEW(x−

1,y−

1)+2×

NEW(x,y−

1)+NEW(x+1,y−

1))  

 

(14)

G_y(i,j)=(NEW(x+1,y−

1)+2×

NEW(x+1,y)+NEW(x+1,y+1))−

(NEW(x−

1,y−

1)+2×

NEW(x−

1,y)+NEW(x−

1,y+1))

The rim of the acquired image can be obtained by using formula (16).

G(x,y)=√

{square root over (G_x(x,y)²+G_y(x,y)²)}{square root over (G_x(x,y)²+G_y(x,y)²)} 

 

(16)

Then the above rim image is binarized.

$\begin{array}{cc}E<br><br>\left(x,y\right)=\{\begin{array}{cc}255& G<br><br>\left(x,y\right)\ge <br><br>T_e^{*}\\ 0& G<br><br>\left(x,y\right)<<br><br>T_e^{*}\end{array}& \left(17\right)\end{array}$

Wherein T_e* represents the optimal threshold, the optimal threshold can be obtained using the prior method;

then, after mixing the binarization contour pattern of the real-time image and the differentiated binary image BIN(x,y), the periphery contour of the moving object can be obtained.

Step 7;

Check if the contour point coordinates of the moving object is touched by the reactive area and run the corresponding movement.

Step 8;

Repeat all the steps above;

2. The said interactive real-time image recognition software method is described as follows:

Step 1;

Capture the image projected to the image region by an image projection apparatus as reference images by using video camera;

Step 2;

Capture the real-time image continuously projected by an image projection apparatus to the image region by using a video camera, wherein images have active images. Then, check if the reactive area is touched by any foreign object.

The difference value between reference images in step 1 and real-time images in step 2 can be defined by the following formula (1);

DIFF(x,y)=|REF(x,y)−

NEW(x,y)| 

 

(1)

Step 3;

Difference of the grey-level values of said reference image from step 1 with grey-level values of real-time images from step 2 and get the remaining image, which is denoted by formula (2)

$\begin{array}{cc}\mathrm{BIN}<br><br>\left(x,y\right)=\{\begin{array}{cc}255& \mathrm{DIFF}<br><br>\left(x,y\right)\ge <br><br>T^{*}\\ 0& \mathrm{DIFF}<br><br>\left(x,y\right)<<br><br>T^{*}\end{array}& \left(2\right)\end{array}$

The binarization method removes the effect of noises.

Step 4;

After binarization, the white segments refer to the active images and within the images. The active images and can be segmented by using the Line Segment Coding Method, said line segment coding method is a line segment restore method to store every bit of data in an object. Once the segmented images are detected in line 1, it can be regarded as the first line of the first object denoted as 1-1. Then, two lines are detected in the second line. Since the first line is under 1-1 that is denoted as 1-2, the second line is a new object denoted as 2-1. Accordingly, there is only 1 line under object 1 and object 2 in the forth line. Therefore, the image originally regarded as two objects actually is an object, which is denoted as 1-4. After all the images are scanned, then the merge procedure is performed.

Wherein, the information of every object includes;

square area, circumference, object characteristic, segmented image size, width and the total number of the object.

Step 5;

when the active images and activity reactive area are segmented, every object characteristic value is calculated. Seven unchanged matrixes are used to represent the object characteristics. The solution is described as follows;

The (k+1) matrix definition of a binary image b(m, n) is

$\begin{array}{cc}{M}_{k,l}=\underset{m=0}{\overset{M-1}{\sum <br><br>}}<br><br>\underset{n=0}{\overset{N-1}{\sum <br><br>}}<br><br>m^k<br><br>n^l<br><br>b<br><br>\left(m,n\right)& \left(18\right)\end{array}$

Wherein, the center matrix is defined as;

$\begin{array}{cc}{\mathrm{\mu <br><br>}}_{k,l}=\underset{m=0}{\overset{M-1}{\sum <br><br>}}<br><br>\underset{n=0}{\overset{N-1}{\sum <br><br>}}<br><br>{\left(m-\stackrel{\_}{x}\right)}^k<br><br>{\left(n-\stackrel{\_}{y}\right)}^l<br><br>b<br><br>\left(m,n\right)& \left(19\right)\end{array}$

Wherein,

$\stackrel{\_}{x}=\frac{M_{1,0}}{M_{0,0}},\stackrel{\_}{y}=\frac{M_{0,1}}{M_{0,0}}$

represents the mass center of the object respectively.

Then the normalized center matrix of the formula (19) is defined as follows;

$\begin{array}{cc}{\mathrm{\eta <br><br>}}_{k,l}=\frac{{\mathrm{\mu <br><br>}}_{k,l}}{{\left(\sqrt{{\mathrm{\mu <br><br>}}_{0,0}}\right)}^{k+l+2}}& \left(20\right)\end{array}$

The seven unchanged matrixes can be obtained by the normalized second and third order matrix;

${\mathrm{\phi <br><br>}}_1={\mathrm{\eta <br><br>}}_{2,0}+{\mathrm{\eta <br><br>}}_{0,2}$ ${\mathrm{\phi <br><br>}}_2={\left({\mathrm{\eta <br><br>}}_{2,0}-{\mathrm{\eta <br><br>}}_{0,2}\right)}^2+4<br><br>{\mathrm{\eta <br><br>}}_{1,1}^2$ ${\mathrm{\phi <br><br>}}_3={\left({\mathrm{\eta <br><br>}}_{3,0}-3<br><br>{\mathrm{\eta <br><br>}}_{1,2}\right)}^2+{\left(3<br><br>{\mathrm{\eta <br><br>}}_{2,1}-{\mathrm{\eta <br><br>}}_{0,3}\right)}^2$ ${\mathrm{\phi <br><br>}}_4={\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2+{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)}^2$ $\begin{array}{c}{\mathrm{\phi <br><br>}}_5=<br><br>\left({\mathrm{\eta <br><br>}}_{3,0}-3<br><br>{\mathrm{\eta <br><br>}}_{1,2}\right)<br><br>\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)<br><br>\left[{\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2-3<br><br>{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)}^2\right]+\\ <br><br>\left(3<br><br>{\mathrm{\eta <br><br>}}_{2,1}-{\mathrm{\eta <br><br>}}_{0,3}\right)<br><br>\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)<br><br>\left[3<br><br>{\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2-{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)}^2\right]\end{array}$ $\begin{array}{c}{\mathrm{\phi <br><br>}}_6=<br><br>\left({\mathrm{\eta <br><br>}}_{2,0}-{\mathrm{\eta <br><br>}}_{0,2}\right)<br><br>\left[{\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2-{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)}^2\right]+\\ <br><br>4<br><br>{\mathrm{\eta <br><br>}}_{1,1}<br><br>\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)<br><br>\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)\end{array}$ $\begin{array}{c}{\mathrm{\phi <br><br>}}_7=<br><br>\left(3<br><br>{\mathrm{\eta <br><br>}}_{2,1}-{\mathrm{\eta <br><br>}}_{0,3}\right)<br><br>\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)<br><br>\left[{\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2-3<br><br>{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)}^2\right]+\\ <br><br>\left(3<br><br>{\mathrm{\eta <br><br>}}_{1,2}-{\mathrm{\eta <br><br>}}_{0,3}\right)<br><br>\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\eta <br><br>}}_{0,3}\right)<br><br>\left[3<br><br>{\left({\mathrm{\eta <br><br>}}_{3,0}+{\mathrm{\eta <br><br>}}_{1,2}\right)}^2-{\left({\mathrm{\eta <br><br>}}_{2,1}+{\mathrm{\mu <br><br>}}_{0,3}\right)}^2\right]\end{array}$

Step 6;

In the realistic pattern recognition process, the pattern of each category has different characteristic vectors within a range, while the falling point within the range cannot be predicted precisely even the range is known. Such kind of random problem can be described using the probability concept. Here, the Bayesian classifier of Gaussian pattern category is adopted to recognize patterns to be identified in real time, which can be described as;

$\begin{array}{cc}{D}_j<br><br>\left(x\right)=-\frac{1}{2}<br><br>\ln <br><br><br><br><br><br>C_j<br><br>-\frac{1}{2}<br><br>\left[{\left(x-m_j\right)}^T<br><br>C_j^{-1}<br><br>\left(x-m_j\right)\right],<br><br>j=1,2<br><br><br><br>\mathrm{\Lambda <br><br>}<br><br><br><br>M-& \left(21\right)\end{array}$

Wherein, D_j is the j^th pattern decision function;

x=[φ

₁Aφ

₇] is the j^th eigenvector;

m_j and C_j is the j^th average eigenvector and covariance matrix. When D is the maximum, it is classified as the j^th pattern. After the pattern recognition is completed, the position of the reactive area is decided. The recognition process can be summarized as;

(1). Practice the pattern template in advance, calculate each category φ

₁Aφ

₇, and calculate m_j and C_j of each category, then the decision rules of each categorizer are completed.

(2). Segment the images acquired by video camera 12 into several sub images through step 4, and then calculate each D_j (x) of sub images.

(3) Compare the size of D_j (x), identify the maximum, and set the pattern as the k^th category.

After the recognition, the activity reactive area can be located precisely.

Step 7;

Check if the activity reactive area is touched by foreign objects and perform the corresponding actions.

Step 8;

Repeat all the steps above;

3. This is the same above-mentioned claim 2 of the said interactive real-time image recognition software method, wherein said step 6:

if there are several reactive areas in the images, there are several sub reference images. The passive recognition step 1 through 8 is utilized to determine whether the foreign object touches the sub reference images.