Attentive panoramic visual sensor

• US 7,130,490 B2
• Filed: 05/14/2002
• Issued: 10/31/2006
• Est. Priority Date: 05/14/2001
• Status: Active Grant

• Current Assignee: Fadi Dornaika, James H. Elder, Ronen Goldstein, Yuqian Hou
• Original Assignee: Fadi Dornaika, James H. Elder, Ronen Goldstein, Yuqian Hou
• Inventors: Dornaika, Fadi, Elder, James H., Goldstein, Ronen, Hou, Yuqian
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): Kassa, Yosef
• Application Number:
  US10/143,848
• Publication Number:
  US 20030026588A1
• Time in Patent Office: 1,631 Days
• Field of Search: 382/284, 382/289, 382/293, 382/294, 382/299, 348/36, 348/575, 348/584
• US Class Current: 382/294
• CPC Class Codes: G08B 13/19643 wherein the cameras play different roles, e.g. different resolution, different camera type, master-slave camera
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The present invention provides a process for accurately combining low-resolution and high-resolution video streams into a single seamless display to a human observer. The present invention also provides a method for the smooth blending of high- and low-resolution video boundary. In order to create an effective visual display, it is advantageous that the boundary between high- and low-resolution video not be overly distracting and irregular. The present method to reduce distractions from the quadrilateral boundary between low-and high-resolution video by smoothly blending the two in an annular region between the two. This results in a much more visually pleasing, effective display. The present invention also provides a method for progressively mosaicking high resolution imagery over a low resolution image captured by our panoramic sensor. The observer is continuously presented with a low-resolution panoramic image of a remote scene, and this image is updated with high-resolution imagery from the high-resolution sensor, at selected locations of interest, as they are available.

Claims

1. A method for simultaneously recording foveal and panoramic video streams and fusing the foveal and panoramic video streams into a single seamless display, the foveal video stream being recorded using a high-resolution video sensor and the panoramic video stream being recorded using a low-resolution video sensor, the method comprising the steps of:

a) unwarping a panoramic video stream from said low-resolution video sensor and transforming said foveal video stream from said high-resolution video sensor and said unwrapped panoramic video stream to common display coordinates;

b) fusing said transformed foveal video stream and said transformed unwrapped panoramic video stream; and

c) displaying the fused video stream on a visual display means to produce a fused visual display.

2. The method according to claim 1 including directing the high-resolution video sensor by clicking on points of interest in the low-resolution portion of the display.

3. The method according to claim 1 including detecting motion in a moving object in the panoramic video stream and directing the high-resolution video sensor toward the moving object.

4. The method according to claim 1 wherein said step of fusing said transformed foveal video stream and said transformed unwrapped panoramic video stream includes blending said transformed foveal video stream and said transformed unwrapped panoramic video stream using a set of concentric alpha masks to give a high-resolution circular fovea smoothly inset within a low-resolution panoramic image.

5. The method according to claim 1 including a step of having saccades to points in the panorama where motion is detected, wherein successive panoramic RGB image pairs are differenced, rectified, and summed to form a primitive motion map, convolving said primitive motion map with a separable square kernel that approximates the field of view of the foveal sensor in panoramic coordinates, wherein said primitive motion map is a thresholded to prevent the generation of saccades due to sensor noise and vibration.

6. A method for reducing distractions from a quadrilateral boundary between low-and high-resolution video stream by smoothly blending the two in an annular region between the two, the high-resolution video stream being recorded using a high-resolution video sensor and the low-resolution video stream being recorded using a low-resolution video sensor, the method comprising the steps of:

c) computing an annular region inscribing a quadrilateral boundary of the high-resolution image, as projected in the coordinate frame of the panoramic sensor; and

d) blending the high- and low-resolution video streams within this annular region to yield a smooth seamless display.

7. A method for progressively mosaicking high-resolution imagery captured by a high-resolution video sensor over a low resolution image captured by a low-resolution panoramic video sensor, the method comprising the steps of:

f) timestamping and archiving high-resolution images from previous fixations of the high-resolution video sensor;

g) providing a formula defining a time decay memory parameter formula of high-resolution images from previous fixations of the high-resolution sensor;

h) adjusting the contrast of pixels from previous fixations of the high-resolution sensor according to said time decay memory parameter formula;

i) combining the contrast-adjusted high-resolution pixels with cren data from the low-resolution panoramic sensor; and

j) adjusting the time decay memory parameter to trade off temporal and spatial resolution over a continuous range.

8. A method for estimating static homographies mapping coordinates of a high-resolution video stream to panoramic coordinates, allowing accurate fusion of data for static or distanct scenes, comprising the steps of:

c) Construction of a planar calibration rig with well-defined, high-contrast corner features

d) recording simultaneous low- and high-resolution video frames using a low-resolution panoramic video sensor and a high-resolution video sensor of a calibration rig at regular intervals over a field-of-view of the low-resolution panoramic sensor;

e) localizing corresponding points on the calibration rig in the high- and low-resolution images;

f) estimating a homography for each pan/tilt position of the high-resolution video sensor; and

g) interpolating over a resulting lattice of homographies to yield a homography relating high- and low-resolution coordinate frames at arbitrary pan/tilt angles.

9. The method according to claim 8 wherein said step of estimating a homography for each pan/tilt position of the high-resolution video sensor includes using a least-squares estimation.

10. The method according to claim 8 wherein said step of interpolating over a resulting lattice includes using a bilinear estimation procedure.

11. A method for estimating dynamic homographies mapping high-resolution to panoramic coordinates, allowing accurate fusion of data for close-range, dynamic scenes, comprising the steps of:

a. providing a coarse registration using parametric templates over a discrete scale space on the high-resolution image and using Pearson correlation to estimate translation and scaling; and

b. providing a staged refinement for estimating photometric, affine parameters and then a full homography, using Pearson correlation.

12. An apparatus for simultaneously recording high- and low-resolution video streams and combining these video streams into a single seamless display, comprising:

g) a low-resolution panoramic video sensor for recording low-resolution video streams;

h) a high-resolution video sensor for recording high-resolution video streams mounted on a motorized gimballed pan/tilt platform that permits the high-resolution sensor to be rotated about its optical centre toward any visual direction within the panoramic field of view of the low-resolution video sensor;

i) processing means for fusing the two video streams in real time;

j) processing means for displaying the fused video stream on a visual display means;

k) user-interface means that allows a human user to direct the high-resolution sensor by clicking on points of interest in the low-resolution portion of the display; and

l) processing means for detecting motion in the low-resolution video stream and directs the high-resolution sensor toward the moving object.

13. A device for panoramic sensing for visual telepresence, comprising:

a video sensor having a panoramic field of view, a motion sensor and a display means connected to said video sensor; and

control means connected to said video sensor, said motion sensor and said display means, said control means being operable in either a slaved mode in which an operator controls positioning of said video sensor, an autonomous mode in which saccades are determined by motion detected by said motion sensor, or a semi-autonomous mode in which saccades are determined by a combination of motion detected by said motion sensor and operator interest.

14. The device according to claim 13 wherein said video sensor includes a foveal component comprising a video camera, and wherein display duration of foveal images from past fixations is determined by a memory parameter.

15. The device according to claim 14 wherein said control means includes a calibration providing a table of projective parameter parameters, indexed by foveal pan/tilt coordinates, that allows rapid transfer of pixels between foveal and panoramic coordinate frames, including a transform means for mapping between the motion sensor and display frames produced on said display means.

16. The device according to claim 15 including alpha masking means for displaying a high resolution smoothly blended fovea embedded in lower resolution panoramic image.