Bionic automatic vision and line of sight control system and method

• US 20050185945A1
• Filed: 04/18/2005
• Published: 08/25/2005
• Est. Priority Date: 10/22/2002
• Status: Active Grant

• Current Assignee: Shanghai Eyevolution Technology Co., Ltd.
• Original Assignee: Guangrang Zhang, Xiaolin Zhang
• Inventors: Zhang, Xiaolin, Zhang, Guangrang
• Granted Patent: US 7,272,306 B2
• Time in Patent Office: Days
• US Class Current: 396/73
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/40613 Camera, laser scanner on end effector, hand eye manipulator, local
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention presents a bionic automatic vision and line of sight control system and method, wherein the system comprises multiple camera sets, wherein each camera set comprises one or more parallel camera with different shooting range, wherein each camera set is driven by one or three actuators, capable of rotating in two or three degree of freedom. The present system is capable of fast object positioning, high accuracy tracking and wide range compensation for line of sight deviation caused by the movement of the system. More specifically, the system can ensure two camera sets to position and track the same object at the same time. The present system not only can be used in various fixed places to monitor, protect and care taking, it can also be installed on various moving objects, such as cars, airplanes, boats, military equipments, and it can be used as eyes for various types of robots.

Claims

1. A bionic automatic vision and line of sight control system comprises:

multiple camera sets, wherein each camera set comprises multiple parallel cameras;

a base where the camera sets are mounted;

multiple actuators which enable each camera set to rotate in three degrees of freedom relative to the base;

multiple transducers for detecting the rotation angle and speed of the actuators; and

a central controller which processes the signals from the camera sets and the transducers, and controls the rotation of the actuators and sends results from processing signals and other information to peripheral equipments.

2. The bionic automatic vision and line of sight control system as claimed in claim 1, wherein each of the camera set comprises wide angle lens cameras and telephoto lens cameras.

3. The bionic automatic vision and line of sight control system as claimed in claim 1, wherein the base is moveable, wherein the acceleration transducers are installed on the base to measure six rotation and translation degrees of freedom, wherein the acceleration transducers are in even numbers and symmetric horizontally with respect to center axis of the base.

4. The bionic automatic vision and line of sight control system as claimed in claim 2, wherein the base are driven by three actuators to enable three degrees of freedom rotation.

5. The bionic automatic vision and line of sight control system as claimed in claim 1, wherein the system comprises microphones installed on the base for receiving audio signals and sending the signals to the central controller.

6. The bionic automatic vision and line of sight control system as claimed in claim 1, wherein each camera set is driven by three actuators for three degrees of freedom rotation, wherein the third degree of freedom is for rotating around one center axis of the camera, wherein the distance between each rotation center axis can be adjusted according to the distance with the object to obtain more accurately measured object distance.

7. The bionic automatic vision and line of sight control system as claimed in claim 1, wherein the system further comprises wide angle lens cameras fixed in an open space or omnidirectional cameras controlled by central controller.

8. A bionic automatic vision and line of sight control method of the bionic automatic vision and line of sight control system as claimed in claim 1, wherein

(1) When an object enters the visual field of the telephoto lens camera, the system searches the feature point of the object and put the feature point at the visual field center;

(2) When the feature point of the object in the telephoto camera'"'"'s visual field moves, the image process program corresponding to the telephoto camera will calculate the horizontal and vertical deviation and speed of the feature point in relevant to the center of the visual field, wherein the deviation and speed signals will be used to control the rotation of the camera set to enable the line of sight of the camera automatically track the feature point.

(3) When multiple feature points appear in the visual field of the telephoto lens camera, the system controls the camera'"'"'s line of sight to sequentially align each feature point with the visual field center at given intervals, according to the relationship between each feature point and the visual field center.

9. A bionic automatic vision and line of sight control method of the bionic automatic vision and line of sight control system as claimed in claim 1, wherein the rotatable camera set includes wide angle lens camera, wherein

(1) When the wide angle lens camera of the camera set finds an interested object, the camera set rotates at high speed, corresponding to the saccade eye movement, focus the line of sight on the object to make the feature point as the visual field center;

(2) Because the wide angle lens camera and the telephoto lens camera of the camera set are parallel and adjacent to each other, when an object is at the visual field center of the wide angle lens camera, the object is automatically at the visual field center of the telephoto lens camera.

10. The bionic automatic vision and line of sight control method as claimed in claim 8, wherein when using fixed wide angle lens camera or omnidirectional camera,

When the fixed wide angle lens camera or the omnidirectional camera finds an interested object, through position transformation of the object, the system calculates the direction of the object related to the rotatable camera sets, and quickly turns the line of light of the camera set towards the object.

11. The bionic automatic vision and line of sight control method as claimed in claim 9, wherein when multiple interested objects appear in the visual field of the wide angle lens camera or omnidirectional camera, according to the sequence of appearance or position, the camera sets focus and track repsectively each object at set time intervals.

12. The bionic automatic vision and line of sight control method as claimed in claim 10, wherein when multiple interested objects appear in the visual field of the wide angle lens camera or omnidirectional camera, according to the sequence of appearance or position, the camera sets focus and track repsectively each object at set time intervals.

13. The bionic automatic vision and line of sight control method as claimed in claim 8, wherein when the camera sets comprises two camera sets for corresponding control, the response speed of two camera sets'"'"' conjugate movement is faster than the response speeds of their vergence movement, wherein the control system is called bionic binocular automatic vision and line of sight control system.

14. The bionic automatic vision and line of sight control method as claimed in claim 13, wherein the transforming functions of the respective actuator control systems of the two camera sets are:

$\begin{array}{c}{\mathrm{\theta <br><br>}}_1<br><br>\left(s\right)+{\mathrm{\theta <br><br>}}_2<br><br>\left(s\right)=\frac{T_{\mathrm{vm}}<br><br>\left(\mathrm{\rho <br><br>}-{\mathrm{\rho <br><br>}}_r\right)}{\begin{array}{c}1+T_{\mathrm{vm}}<br><br>\left(\mathrm{\rho <br><br>}-{\mathrm{\rho <br><br>}}_{r<br><br>\text{\hspace{1em}<br><br>}}\right)<br><br>\left(\mathrm{\sigma <br><br>}-{\mathrm{\sigma <br><br>}}_r\right)+T_{\mathrm{vm}}<br><br>\left(1+\left(\mathrm{\rho <br><br>}-{\mathrm{\rho <br><br>}}_r\right)<br><br>\left(\mathrm{\eta <br><br>}-{\mathrm{\eta <br><br>}}_r\right)\right)<br><br>s\times <br><br>\\ [-\left({\mathrm{\kappa <br><br>}}_x-{\mathrm{\kappa <br><br>}}_{\mathrm{xr}}\right)<br><br>\frac{T_v}{T_v<br><br>s+1}<br><br>\left({\stackrel{\mathrm{¨<br><br>}}{x}}_{A-l}<br><br>\left(s\right)+{\stackrel{\mathrm{¨<br><br>}}{x}}_{A-r}<br><br>\left(s\right)\right)-\left({\mathrm{\kappa <br><br>}}_y-{\mathrm{\kappa <br><br>}}_{y<br><br>\text{\hspace{1em}<br><br>}<br><br>r}\right)<br><br>\frac{T_v}{T_v<br><br>s+1}<br><br>({\stackrel{\mathrm{¨<br><br>}}{y}}_{A-l}<br><br>\left(s\right)+\\ {\stackrel{\mathrm{¨<br><br>}}{y}}_{A-r}<br><br>\left(s\right))-\left({\mathrm{\kappa <br><br>}}_{\mathrm{\phi <br><br>}}-{\mathrm{\kappa <br><br>}}_{\mathrm{\phi <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>r}\right)<br><br>\frac{T_s}{T_s<br><br>s+1}<br><br>\left({\stackrel{\mathrm{¨<br><br>}}{\mathrm{\phi <br><br>}}}_{z-l}<br><br>\left(s\right)+{\stackrel{\mathrm{¨<br><br>}}{\mathrm{\phi <br><br>}}}_{z-r}<br><br>\left(s\right)\right)+\\ \left(\mathrm{\sigma <br><br>}-{\mathrm{\sigma <br><br>}}_r\right)<br><br>\left({\mathrm{\phi <br><br>}}_{t-l}<br><br>\left(s\right)+{\mathrm{\phi <br><br>}}_{t-r}<br><br>\left(s\right)\right)+\left(\mathrm{\eta <br><br>}-{\mathrm{\eta <br><br>}}_r\right)<br><br>\left({\stackrel{.}{\mathrm{\phi <br><br>}}}_{t-l}<br><br>\left(s\right)+{\stackrel{.}{\mathrm{\phi <br><br>}}}_{t-r}<br><br>\left(s\right)\right)]\end{array}}\\ {\mathrm{\theta <br><br>}}_1<br><br>\left(s\right)-{\mathrm{\theta <br><br>}}_2<br><br>\left(s\right)=\frac{T_{\mathrm{vm}}<br><br>\left(\mathrm{\rho <br><br>}+{\mathrm{\rho <br><br>}}_r\right)}{\begin{array}{c}1+T_{\mathrm{vm}}<br><br>\left(\mathrm{\rho <br><br>}+{\mathrm{\rho <br><br>}}_{r<br><br>\text{\hspace{1em}<br><br>}}\right)<br><br>\left(\mathrm{\sigma <br><br>}+{\mathrm{\sigma <br><br>}}_r\right)+T_{\mathrm{vm}}<br><br>\left(1+\left(\mathrm{\rho <br><br>}+{\mathrm{\rho <br><br>}}_r\right)<br><br>\left(\mathrm{\eta <br><br>}+{\mathrm{\eta <br><br>}}_r\right)\right)<br><br>s\times <br><br>\\ [-\left({\mathrm{\kappa <br><br>}}_x+{\mathrm{\kappa <br><br>}}_{\mathrm{xr}}\right)<br><br>\frac{T_v}{T_v<br><br>s+1}<br><br>\left({\stackrel{\mathrm{¨<br><br>}}{x}}_{A-l}<br><br>\left(s\right)-{\stackrel{\mathrm{¨<br><br>}}{x}}_{A-r}<br><br>\left(s\right)\right)-\left({\mathrm{\kappa <br><br>}}_y+{\mathrm{\kappa <br><br>}}_{\mathrm{yr}}\right)<br><br>\frac{T_v}{T_v<br><br>s+1}<br><br>({\stackrel{\mathrm{¨<br><br>}}{y}}_{A-l}<br><br>\left(s\right)-\\ {\stackrel{\mathrm{¨<br><br>}}{y}}_{A-r}<br><br>\left(s\right))-\left({\mathrm{\kappa <br><br>}}_{\mathrm{\phi <br><br>}}+{\mathrm{\kappa <br><br>}}_{\mathrm{\phi <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>r}\right)<br><br>\frac{T_s}{T_s<br><br>s+1}<br><br>\left({\stackrel{\mathrm{¨<br><br>}}{\mathrm{\phi <br><br>}}}_{z-l}<br><br>\left(s\right)-{\stackrel{\mathrm{¨<br><br>}}{\mathrm{\phi <br><br>}}}_{z-r}<br><br>\left(s\right)\right)+\\ \left(\mathrm{\sigma <br><br>}+{\mathrm{\sigma <br><br>}}_r\right)<br><br>\left({\mathrm{\phi <br><br>}}_{t-l}<br><br>\left(s\right)-{\mathrm{\phi <br><br>}}_{t-r}<br><br>\left(s\right)\right)+\left(\mathrm{\eta <br><br>}+{\mathrm{\eta <br><br>}}_r\right)<br><br>\left({\stackrel{.}{\mathrm{\phi <br><br>}}}_{t-l}<br><br>\left(s\right)-{\stackrel{.}{\mathrm{\phi <br><br>}}}_{t-r}<br><br>\left(s\right)\right)]\end{array}}\end{array}$

wherein, Φ

_t-l.Φ

_t-r are respectively the horizontal deviation angle between the object and the line of sight measured by the left and right telephoto camera sets, {umlaut over (x)}_A-l.ÿ

_A-l.{umlaut over (Φ

)}_z-l and {umlaut over (x)}_A-r.ÿ

_A-r.{umlaut over (Φ

)}_z-r are the rotation acceleration and translation acceleration measured by the left and right acceleration transducers, T_vm, T_v, T_s are time constant, ρ

, ρ

_r, σ

, σ

_r, η

, η

_r, κ

_k, κ

_xr, κ

_y, κ

_yr, κ

₁₀₁ , κ

<sub>Φ

r</sub> are positive parameters, {dot over (Φ

)}_t-l.{dot over (Φ

)}_t-r are respectively the speed of the object related to the line of sight measured by the left and right telephoto camera sets, wherein the first equation is the equation of the relative movement between the two camera set, and the second equation is the equation of the common movement of the two camera set, wherein by adding and subtracting of the two equations, the rotation angle of each actuator is obtained.

15. The bionic automatic vision and line of sight control method as claimed in claim 9, wherein the saccade control curve ƒ

(t) is from linear Fourier transformation, as;

$f<br><br>\left(t\right)=\frac{a_0}{2}+\underset{n=1}{\overset{m}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(a_n<br><br>\cos <br><br>\frac{2<br><br>\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t+b_n<br><br>\sin <br><br>\frac{2<br><br>\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t\right)$

wherein, a₀, a_n, b_n are obtained by reverse transformation of the control deviation by repeating learning Fourier reverse transformation;

$\begin{array}{cc}{a}_{n<br><br>\text{\hspace{1em}<br><br>}}<br><br>\left[i+1\right]=a_n<br><br>\left[i\right]+k_a<br><br>\frac{4}{T}<br><br>{\int <br><br>}_0^T<br><br>\left(R<br><br>\left(t\right)-{\hspace{0.17em}<br><br>}^i<br><br>\mathrm{\theta <br><br>}<br><br>\left(t\right)\right)<br><br>\cos <br><br>\frac{2<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t<br><br>\text{\hspace{1em}<br><br>}<br><br>d<br><br>t<br><br>\text{\hspace{1em}<br><br>}& \text{\hspace{1em}<br><br>}<br><br>\left(n=0,1,2,3<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>m\right)\\ b_{n<br><br>\text{\hspace{1em}<br><br>}}<br><br>\left[i+1\right]=b_n<br><br>\left[i\right]+k_b<br><br>\frac{4}{T}<br><br>{\int <br><br>}_0^T<br><br>\left(R<br><br>\left(t\right)-{\hspace{0.17em}<br><br>}^i<br><br>\mathrm{\theta <br><br>}<br><br>\left(t\right)\right)<br><br>\sin <br><br>\frac{2<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t<br><br>\text{\hspace{1em}<br><br>}<br><br>d<br><br>t& \text{\hspace{1em}<br><br>}<br><br>\left(n=1,2,3<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>m\right)\end{array}$

wherein, T is the time required for moving the line of sight from the current position to the target position, i is the times of learning, R(t) is optimum actuator movement lotus of the by transforming deviation measured by wide angle lens camera, ⁱθ

(t) is the corresponding actuators'"'"' rotation angle of the i learning times.

16. The bionic automatic vision and line of sight control method as claimed in claim 10 wherein the saccade control curve ƒ

(t) is from linear Fourier transformation, as;

$f<br><br>\left(t\right)=\frac{a_0}{2}+\underset{n=1}{\overset{m}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(a_n<br><br>\cos <br><br>\frac{2<br><br>\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t+b_n<br><br>\sin <br><br>\frac{2<br><br>\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t\right)$

wherein, a₀, a_n, b_n are obtained by reverse transformation of the control deviation by repeating learning Fourier reverse transformation;

$\begin{array}{cc}{a}_{n<br><br>\text{\hspace{1em}<br><br>}}<br><br>\left[i+1\right]=a_n<br><br>\left[i\right]+k_a<br><br>\frac{4}{T}<br><br>{\int <br><br>}_0^T<br><br>\left(R<br><br>\left(t\right)-{\hspace{0.17em}<br><br>}^i<br><br>\mathrm{\theta <br><br>}<br><br>\left(t\right)\right)<br><br>\cos <br><br>\frac{2<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t<br><br>\text{\hspace{1em}<br><br>}<br><br>d<br><br>t<br><br>\text{\hspace{1em}<br><br>}& \text{\hspace{1em}<br><br>}<br><br>\left(n=0,1,2,3<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>m\right)\\ b_{n<br><br>\text{\hspace{1em}<br><br>}}<br><br>\left[i+1\right]=b_n<br><br>\left[i\right]+k_b<br><br>\frac{4}{T}<br><br>{\int <br><br>}_0^T<br><br>\left(R<br><br>\left(t\right)-{\hspace{0.17em}<br><br>}^i<br><br>\mathrm{\theta <br><br>}<br><br>\left(t\right)\right)<br><br>\sin <br><br>\frac{2<br><br>n<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\pi <br><br>}}{T}<br><br>t<br><br>\text{\hspace{1em}<br><br>}<br><br>d<br><br>t& \text{\hspace{1em}<br><br>}<br><br>\left(n=1,2,3<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>m\right)\end{array}$

wherein, T is the time required for moving the line of sight from the current position to the target position, i is the times of learning, R(t) is optimum actuator movement lotus of the by transforming deviation measured by wide angle lens camera, ⁱθ

(t) is the corresponding actuators'"'"' rotation angle of the i learning times.

17. The bionic automatic vision and line of sight control method as claimed in claim 8, wherein the camera set sets on the base, wherein when the base moves, the vibrations or movements of the base are compensated in two ways:

1) When the base moves, the rotation acceleration transducer and translation acceleration transducer installed on the base send motion signals measured from the base to the central controller, wherein the signals are processed through leakage integral to adjust the line of sight position of each camera set to compensate the vibration or movement of the base, wherein the leakage integral processes the signals through transmission function T/Ts+1), wherein T is time constant, s is the complex parameter of Laplace Transformation;

2) When the actuators for driving the base movement rotate, the rotation transducers and rotation speed transducers of the actuators send the measured actuator movement signals to central controller to adjust the position of the line of sight of the camera set to compensate the vibration or movement of the base.