Teleconferencing imaging system with automatic camera steering

• US 5,686,957 A
• Filed: 06/30/1995
• Issued: 11/11/1997
• Est. Priority Date: 07/27/1994
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baker, Robert G.
• Primary Examiner(s): Britton, Howard W.
• Application Number:
  US08/496,742
• Time in Patent Office: 865 Days
• Field of Search: 348/15, 348/214, 348/36, 348/53, 348/580
• US Class Current: 348/36
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G02B 3/0087 with index gradient
• CPC Class Codes: G02B 6/04 formed by bundles of fibres G02B6/24 takes precedence
• CPC Class Codes: G02B 6/06 the relative position of the fibres being the same at both ends, e.g. for transporting images
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

An automatic, voice-directional video camera image steering system specifically for use for teleconferencing that electronically selects segmented images from a selected panoramic video scene typically around a conference table so that the participant in the conference currently speaking will be the selected segmented image in the proper viewing aspect ratio, eliminating the need for manual camera movement or automated mechanical camera movement. The system includes an audio detection circuit from an array of microphones that can instantaneously determine the direction of a particular speaker and provide directional signals to a video camera and lens system that provides a panoramic display that can electronically select portions of that image and, through warping techniques, remove any distortion from the most significant portions of the image which lie from the horizon up to approximately 30 degrees in a hemispheric viewing area.

Claims

1. An automatic audio controlled video camera steering system for electronic imaging and manipulation of a hemispheric field of view, comprising:

at least two microphones;

audio detection circuitry connected to said microphones for determining which of said microphones is receiving audio energy;

means for generating a signal representing the direction of the received audio energy based on signals from the microphones;

a video camera for receiving optical images of the field of view and for producing video image output signals;

an optical system associated with said video camera for producing the optical images from a hemispheric field of view for optical conveyance to said video camera, said optical system having a configuration that emphasizes the peripheral content of the panoramic field of view (when the central lens axis is oriented vertically) as compared to the central content of a hemispheric field of view, this being accomplished through differential magnification;

an imager device associated with said camera for receiving the optical images from said lens and for providing digitized output signals;

input image memory for receiving the digitized output signals from said imager device and for storing the digitized output signals;

an image transform processor or set of circuits for selectively accessing and processing the digitized output signals from said input image memory according to user defined criteria;

output image memory for receiving the processed signals from the image transform processor, and

means connected to said output image memory from said hemispheric field of view for selecting a particular segmentized image from the hemispheric field of view representative of the direction from the camera of the microphone array determined to be receiving sound wave energy;

an output display connected to said output image memory for displaying the signals in said output image memory according to the user defined criteria.

2. The system as in claim 1, wherein said optical system has a configuration that images the peripheral portion of the hemispheric scene onto at least 50% -90% of the usable surface area of the imaging device, when used in an electronic image capture apparatus.

3. The system as in claim 2, wherein said optical system includes a color aberrated wide field lens and a gradient index hemispheric lens.

4. The system as in claim 2, wherein said optical system has a configuration which emphasizes the field of view from 1 to 45 degrees up from the horizon in a hemispherical captured image, the lens representing one of many configurations that may selectively emphasize some subset of the aforesaid 1 to 45 degrees covered.

5. The system as in claim 4, wherein said optical system includes a wide field multi-element lens and a coordinated fiber array geometrically fabricated with an annular input end and a rectangular output end.

6. The system as in claim 4, wherein said optical system includes a wide field multi-element lens and a coordinated fiber array geometrically fabricated with an annular input end and a circular output end.

7. A teleconferencing system for electronic manipulation of a hemispheric scene, comprising:

a plurality of microphones;

means connected to the output of said microphones for determining the amplitude levels of audio signals from the microphones;

means for detecting and comparing the audio signals from said microphones to determine the direction of a particular speaker;

means for providing an output signal representing the direction of a speaker at a given time;

a camera imaging system for receiving optical images of the field of view and for producing output video signals corresponding to the optical images;

an optical system associated with said camera imaging system for producing the optical images from a field of view for optical conveyance to said camera imaging system;

an imager device associated with said camera for receiving the optical images from said lens and for providing digitized output signals;

input image memory for receiving the digitized output signals from said imaging device and for storing the digitized output signals;

an image transform processor for selectively accessing and processing the digitized output signals from said input image memory according to user-defined criteria;

means connected to said image transform processor from said microphone direction means to provide the image segment of a hemispheric scene to be selected representing the direction of the speaker based on the audio signals from that array of microphones;

output image memory for receiving the processed signals from the image transform processor means;

an output display device connected to said output image memory for displaying the signals in said output image memory according to user-defined criteria;

wherein the improvement comprises said optical systems having a configuration that emphasizes the peripheral content of field of view of a hemispheric scene as compared to the central content, such that said imager device receives magnified optical images of the peripheral portion of the hemispheric field of view.

8. A teleconferencing method for electronically capturing, storing, and manipulating a hemispheric field of view, having a plurality of individual human speakers, comprising the steps of:

providing a plurality of microphones connected to a microphone audio detection circuit that can provide an output signal that determines which microphone is in use at a given point in time, indicative of the direction of the human speaker;

providing an optical system having a configuration that enhances the peripheral portion of the field of view in the direction of the human speaker;

capturing the hemispheric field of view with the periphery-enhancing optical system and imaging the field of view onto an imager device by enhancing the peripheral field of view;

storing the captured image as a single image;

selectively accessing a portion of the stored image according to user-defined criteria;

transforming the stored image so that the stored image can be displayed as a perspective-correct image;

selecting from a portion of the field of view a specific image segment representative of the direction of the speaker;

displaying the perspective-correct image in a user-defined format.

9. The method as in claim 8, wherein said transforming step comprises manipulating the peripheral-enhanced image into a perspective-correct image.

10. The method as in claim 8, wherein said periphery-enhancing optical system selectively magnifies visual content within an arc of between 10 and 45 degrees up from the horizon in a hemispheric field of view onto the imager device.

11. The method as in claim 8, wherein the storage step comprises storing the captured image in electronic storage.

12. A method for electronically manipulating a hemispheric scene having an enhanced peripheral field of view stored as an image on a video camera, comprising the steps of:

providing a plurality of microphones connected to an audio detection circuit that determines the direction of a given speaker;

converting the image on a video camera into electronic output signals;

selectively accessing a portion of the output signals according to user-defined criteria;

transforming the accessed portion of the output signals by manipulating the peripheral-enhanced field of view so that the stored image can be displayed as a perspective-correct image in a direction based on audio signals from said audio detection circuit;

selecting a particular image segment as a function of speaker direction for display;

displaying the perspective-correct image in the user-defined format.

13. A videoconferencing imaging system having automatic camera steering comprising:

plurality of microphones disposed in a common plane strategically positioned relative to each other and relative to conference participants to be captured on video for remote video and audio transmission of each participant'"'"'s image and spoken word;

microphone output circuit connected to each of said microphones;

audio detection processor having an input connected to said microphone output circuit;

conference platform for arranging participants in individual locations about said conference platform;

video camera including a hemispheric lens mounted strategically in a predetermined location on said conference platform capable of presenting a hemispheric view to said video camera from said platform;

video image view warping logic circuit connected to the output of said video camera and said output of said audio direction processor, said video image view warping logic circuit having a video output of specific regions of interest based on said hemispheric lens image and having an audio output;

a computer processor or other controller circuits having input connected to said view warping logic circuit for controlling the desired specific region of interest as a function of audio directed processor input related to the participant speaking to create a normal aspect ratio view based on the participants using the system; and

video and audio transmission medium connected to the output of said view warping logic circuit for transmitting said audio and video signals to said remote video conference.

14. A videoconferencing image system having automatic camera steering as in claim 13, including:

said video image view warping logic circuit including a source image buffer connected to the input from the video camera lens;

a row (X) warp engine having an output connected to said source image buffer;

column (Y) warp engine, said row warp engine and said column warp engine connected to panoramic image transformation parameter look up table and said computer control output;

interpolation coefficient buffer connected to the output of said row (X) warp engine and said column (Y) warp engine;

multiply accumulation unit connected to the output of said source image buffer and said interpolation coefficient buffer for receiving coefficient data;

memory control device connected to the output of said panoramic image transformation parameter look up table and said computer controller;

warped image buffer connected to the output of said column (Y) warp engine and said memory control and having a warp data output connected to said transmission medium whereby the selection of a normal aspect ratio view containing specific regions of interest selected by the determination of the participant speaking is based on the participant'"'"'s direction, as determined by the audio direction processor.