Attentive panoramic visual sensor
First Claim
1. A method for simultaneously recording foveal and panoramic video streams and fusing the foveal and panoramic video streams into a single seamless display, the foveal video stream being recorded using a high-resolution video sensor and the panoramic video stream being recorded using a low-resolution video sensor, the method comprising the steps of:
- a) unwarping a panoramic video stream from said low-resolution video sensor and transforming said foveal video stream from said high-resolution video sensor and said unwrapped panoramic video stream to common display coordinates;
b) fusing said transformed foveal video stream and said transformed unwrapped panoramic video stream; and
c) displaying the fused video stream on a visual display means to produce a fused visual display.
0 Assignments
0 Petitions
Accused Products
Abstract
The present invention provides a process for accurately combining low-resolution and high-resolution video streams into a single seamless display to a human observer. The present invention also provides a method for the smooth blending of high- and low-resolution video boundary. In order to create an effective visual display, it is advantageous that the boundary between high- and low-resolution video not be overly distracting and irregular. The present method to reduce distractions from the quadrilateral boundary between low-and high-resolution video by smoothly blending the two in an annular region between the two. This results in a much more visually pleasing, effective display. The present invention also provides a method for progressively mosaicking high resolution imagery over a low resolution image captured by our panoramic sensor. The observer is continuously presented with a low-resolution panoramic image of a remote scene, and this image is updated with high-resolution imagery from the high-resolution sensor, at selected locations of interest, as they are available.
-
Citations
16 Claims
-
1. A method for simultaneously recording foveal and panoramic video streams and fusing the foveal and panoramic video streams into a single seamless display, the foveal video stream being recorded using a high-resolution video sensor and the panoramic video stream being recorded using a low-resolution video sensor, the method comprising the steps of:
-
a) unwarping a panoramic video stream from said low-resolution video sensor and transforming said foveal video stream from said high-resolution video sensor and said unwrapped panoramic video stream to common display coordinates; b) fusing said transformed foveal video stream and said transformed unwrapped panoramic video stream; and c) displaying the fused video stream on a visual display means to produce a fused visual display. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A method for reducing distractions from a quadrilateral boundary between low-and high-resolution video stream by smoothly blending the two in an annular region between the two, the high-resolution video stream being recorded using a high-resolution video sensor and the low-resolution video stream being recorded using a low-resolution video sensor, the method comprising the steps of:
-
c) computing an annular region inscribing a quadrilateral boundary of the high-resolution image, as projected in the coordinate frame of the panoramic sensor; and d) blending the high- and low-resolution video streams within this annular region to yield a smooth seamless display.
-
-
7. A method for progressively mosaicking high-resolution imagery captured by a high-resolution video sensor over a low resolution image captured by a low-resolution panoramic video sensor, the method comprising the steps of:
-
f) timestamping and archiving high-resolution images from previous fixations of the high-resolution video sensor; g) providing a formula defining a time decay memory parameter formula of high-resolution images from previous fixations of the high-resolution sensor; h) adjusting the contrast of pixels from previous fixations of the high-resolution sensor according to said time decay memory parameter formula; i) combining the contrast-adjusted high-resolution pixels with cren data from the low-resolution panoramic sensor; and j) adjusting the time decay memory parameter to trade off temporal and spatial resolution over a continuous range.
-
-
8. A method for estimating static homographies mapping coordinates of a high-resolution video stream to panoramic coordinates, allowing accurate fusion of data for static or distanct scenes, comprising the steps of:
-
c) Construction of a planar calibration rig with well-defined, high-contrast corner features d) recording simultaneous low- and high-resolution video frames using a low-resolution panoramic video sensor and a high-resolution video sensor of a calibration rig at regular intervals over a field-of-view of the low-resolution panoramic sensor; e) localizing corresponding points on the calibration rig in the high- and low-resolution images; f) estimating a homography for each pan/tilt position of the high-resolution video sensor; and g) interpolating over a resulting lattice of homographies to yield a homography relating high- and low-resolution coordinate frames at arbitrary pan/tilt angles. - View Dependent Claims (9, 10)
-
-
11. A method for estimating dynamic homographies mapping high-resolution to panoramic coordinates, allowing accurate fusion of data for close-range, dynamic scenes, comprising the steps of:
-
a. providing a coarse registration using parametric templates over a discrete scale space on the high-resolution image and using Pearson correlation to estimate translation and scaling; and b. providing a staged refinement for estimating photometric, affine parameters and then a full homography, using Pearson correlation.
-
-
12. An apparatus for simultaneously recording high- and low-resolution video streams and combining these video streams into a single seamless display, comprising:
-
g) a low-resolution panoramic video sensor for recording low-resolution video streams; h) a high-resolution video sensor for recording high-resolution video streams mounted on a motorized gimballed pan/tilt platform that permits the high-resolution sensor to be rotated about its optical centre toward any visual direction within the panoramic field of view of the low-resolution video sensor; i) processing means for fusing the two video streams in real time; j) processing means for displaying the fused video stream on a visual display means; k) user-interface means that allows a human user to direct the high-resolution sensor by clicking on points of interest in the low-resolution portion of the display; and l) processing means for detecting motion in the low-resolution video stream and directs the high-resolution sensor toward the moving object.
-
-
13. A device for panoramic sensing for visual telepresence, comprising:
-
a video sensor having a panoramic field of view, a motion sensor and a display means connected to said video sensor; and control means connected to said video sensor, said motion sensor and said display means, said control means being operable in either a slaved mode in which an operator controls positioning of said video sensor, an autonomous mode in which saccades are determined by motion detected by said motion sensor, or a semi-autonomous mode in which saccades are determined by a combination of motion detected by said motion sensor and operator interest. - View Dependent Claims (14, 15, 16)
-
Specification