Hardware-in-the-loop simulation system and method for computer vision

• US 7,768,527 B2
• Filed: 11/20/2006
• Issued: 08/03/2010
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: Beihang University
• Original Assignee: Beihang University
• Inventors: Liu, Zhen, Li, Suqi, Wei, Zhenzhong, Zhang, Guangjun
• Primary Examiner(s): GOOD JOHNSON, MONTILEWA
• Application Number:
  US11/561,696
• Publication Number:
  US 20080050042A1
• Time in Patent Office: 1,352 Days
• Field of Search: 345/632, 352/67, 352/90
• US Class Current: 345/619
• CPC Class Codes: G01B 11/2504 Calibration devices
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

The disclosure relates to a hardware-in-the-loop simulation system and method for computer vision. An embodiment of the disclosed system comprises a software simulation and a hardware simulation. The software simulation includes a virtual scene and an observed object that are generated by virtual reality software. The virtual scene images are obtained at different viewpoints. The hardware simulation includes the virtual scene images being projected onto a screen by a projector, wherein the projected scene images are shot by a camera, and where in the direction of the camera is controlled by a pan-tilt.

Claims

1. A hardware-in-the-loop simulation system to verify computer vision, the system comprising:

a virtual reality imaging unit that generates a virtual scene and an observed object using a virtual reality software, wherein virtual scene images of the scene at different viewpoints are obtained;

a projector that projects the virtual scene images generated by the virtual reality software onto a screen; and

a camera that shoots the virtual scene images projected by said projector on the screen, wherein the direction of the camera is controlled by a pan-tilt, wherein the camera is fixed on the pan-tilt, and wherein a position and orientation of the projector and the pan-tilt are fixed to the screen;

wherein the virtual reality imaging unit comprises a first computer and wherein said projector connects to the virtual reality imaging unit; and

wherein the camera sends image data gathered while shooting the projected scene images to a second computer by a frame grabber board and the second computer is configured to control the pan-tilt to select a proper direction for the camera to shoot the virtual scene images projected onto the screen.

2. A hardware-in-the-loop simulation method for computer vision, the method comprising:

generating virtual objects or scenes with a first computer using virtual reality software installed on the first computer;

setting parameters for the virtual reality software to acquire an image of the virtual objects or scenes;

projecting the image onto a screen;

taking the image of the virtual objects or scenes on the screen using an actual camera to acquire a result of a simulation;

calculating parameters of a virtual camera with a second computer, and calibrating models for the actual camera and a projector for a high-accuracy parameter estimation with the second computer; and

achieving computer vision experiments based on the result of the simulation with the second computer.

3. The hardware-in-the-loop simulation method according to claim 2, wherein the step of setting the parameters for the virtual reality software further comprise:

setting a vertical field of view angle in degrees, an aspect ratio of a width to height of a viewport, a distance from the viewpoint to near and far clipping planes, coordinates in a world coordinate system of the viewpoint, and yaw, pitch and roll angles.

4. The hardware-in-the-loop simulation method according to claim 2, wherein the step of calculating the parameters of the virtual camera is relative to the virtual reality software parameters;

the intrinsic parameters of the virtual camera are calculated according to parameters of a projection transformation of the virtual reality software, and extrinsic parameters of the virtual camera are calculated according to parameters of a model view transformation of the virtual reality software.

5. The hardware-in-the-loop simulation method according to claim 2, wherein in the step of calibrating the actual camera and projector models, the projector model and the actual camera model are regarded as a whole module for calibration.

6. The hardware-in-the-loop simulation method according to claim 5, wherein a nonlinear camera calibration technique is used for high-accuracy parameter estimation of said whole module for calibration, comprising:

determining the center of a camera image using a method of varying focal length;

achieving lens distortion calibration using a calibration approach based on cross ratio invariability;

calculating a coordinate of a corner of a calibration pattern in a camera coordinate system based on the center of the camera image and the lens distortion, according to an image coordinate in pixel;

setting up linear equation groups according to a linear transformation model from the camera coordinate system to a projector computer image coordinate system;

wherein a linear transformation matrix is estimated using a least-squares method to achieve the parameter estimation.