Panoramic image navigation system using neural network for correction of image distortion

• US 6,671,400 B1
• Filed: 10/26/2000
• Issued: 12/30/2003
• Est. Priority Date: 09/28/2000
• Status: Expired due to Fees

• Current Assignee: Tateyama R&D Co., Ltd.
• Original Assignee: Tateyama R&D Co., Ltd.
• Inventors: Ekpar, Frank Edughom, Yamauchi, Shinya
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): Bali, Vikkram
• Application Number:
  US09/695,881
• Time in Patent Office: 1,160 Days
• Field of Search: 382/156, 382/157, 706/2, 706/6, 706/15, 706/16, 700/47, 700/48
• US Class Current: 382/157
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A system permitting one or more users to navigate a wide-angle image, and in particular a panoramic image, using a neural network to correct distortion in that image. To train the neural network, a calibration pattern which defines an array of calibration points is disposed to occupy a field of view of a wide-angle imaging apparatus, and a calibration image is thereby captured by the apparatus. Respective view directions of the calibration points and the positions of these points in the calibration image are used as data for training the neural network to correctly match view directions to corresponding intra-image positions. Subsequently, to generate an undistorted sub-image of an arbitrary distorted wide-angle image, an array of display pixel positions are expressed as a corresponding set of view directions, and the neural network used to convert these to a corresponding set of positions within the wide-angle image, with the video attributes at these positions being then assigned to the corresponding pixels of the sub-image.

Claims

1. A method of navigating within a distorted panoramic image by selectively extracting undistorted sub-images from said distorted panoramic image, comprising steps of:

training a neural network to relate respective view directions of a panoramic imaging system to corresponding positions within a distorted panoramic image that is captured by said panoramic imaging system;

specifying a central view direction for a desired sub-image of an arbitrary distorted panoramic image that has been captured by said panoramic imaging system, with said sub-image generated by a rectangular array of pixels of a display device;

based on a known field of view of said panoramic imaging system and on a configuration of said arbitrary distorted panoramic image, expressing a lateral distance in said array equal to a pixel pitch of said array as an equivalent amount of view direction displacement in azimuth, expressing a vertical distance in said array equal to said pixel pitch as an equivalent amount of view direction displacement in elevation, and calculating respective view directions corresponding to said pixels, based on said equivalent amounts of view direction displacement and said central view direction;

successively inputting said calculated view directions to said neural network to thereby obtain information specifying respectively corresponding locations within said arbitrary distorted panoramic image; and

obtaining respective video attributes for each of said locations and assigning said video attributes to corresponding ones of said pixels, to thereby generate said sub-image.

2. The method according to claim 1, wherein said step of training said neural network comprises steps of:

preparing a calibration pattern which defines an array of visible calibration points, with dimensions and a shape of said calibration pattern being predetermined to enable said calibration pattern to be oriented substantially completely occupying said field of view of said panoramic imaging system;

disposing said calibration pattern to substantially completely occupy said field of view and capturing a distorted panoramic image of said calibration pattern, to constitute a calibration image;

generating a first set of data which relates each of said calibration points to a corresponding view direction with respect to said panoramic imaging system;

generating a second set of data which relates each of said calibration points to a corresponding location at which said calibration point appears within said calibration image; and

performing a repetitive training process of successively inputting data expressing said view directions from said first data set to said neural network, comparing each output response produced from said neural network with data expressing a corresponding location within said calibration image, obtained from said second data set, to thereby obtain an error amount, and applying said error amount in accordance with a predetermined algorithm to modify internal parameters of said neural network, with said training process being continued until said error amounts reach a predetermined permissible level.

3. The method according to claim 2, wherein said calibration pattern is formed on an internal surface of a cylinder and said panoramic imaging system includes an optical lens unit having a central optical axis, and wherein said central optical axis is oriented to substantially coincide with an axis of said cylinder while capturing said calibration image.

4. An apparatus for navigating within a distorted panoramic image by selectively extracting undistorted sub-images from said distorted panoramic image, comprising:

a panoramic imaging system adapted to capture a real-world scene as a distorted panoramic image;

a calibration pattern defining an array of visible calibration points, adapted to be oriented such as to substantially occupy a field of view of said panoramic imaging system with said calibration points arranged at respectively different view directions with respect to said panoramic imaging system;

a neural network, and means for training said neural network to relate respective view directions of said panoramic imaging system to corresponding locations within an arbitrary distorted panoramic image captured by said panoramic imaging system, said training utilizing a calibration image which has been captured as an image of said calibration pattern, and being based upon said view directions of said calibration points in conjunction with respective locations at which said calibration points appear within said calibration image;

display means having a plurality of pixels arrayed with a fixed pitch, for generating a sub-image extracted from an arbitrary distorted panoramic image which is captured by said panoramic imaging system subsequent to training of said neural network; and

data processing means responsive to externally supplied information specifying a central view direction for a desired sub-image of said arbitrary distorted panoramic image, for expressing a lateral distance in said array equal to said pixel pitch as an equivalent amount of view direction displacement in azimuth and expressing a vertical distance in said array equal to said pixel pitch as an equivalent amount of view direction displacement in elevation, based on a known field of view of said panoramic imaging system and a configuration of said arbitrary distorted panoramic image, and calculating respective view directions corresponding to the pixels of said array, based on said equivalent amounts and said central view direction;

successively inputting said calculated view directions to said neural network to thereby obtain information specifying respectively corresponding locations within said arbitrary distorted panoramic image; and

obtaining respective video attributes for each of said locations, and assigning said video attributes to corresponding ones of said array of pixels, to thereby generate said sub-image.

5. The apparatus according to claim 4, wherein said means for training said neural network comprises:

means for generating a first set of data which relates each of said calibration points to a corresponding one of said view directions;

means for generating a second set of data which relates each of said calibration points to a corresponding location at which said calibration point appears within said calibration image; and

means for performing a repetitive training process of successively inputting data expressing said view directions from said first data set to said neural network, comparing each output response produced from said neural network with data expressing a corresponding location within said calibration image, obtained from said second data set, to thereby obtain an error amount, and applying said error amount in accordance with a predetermined algorithm to modify internal parameters of said neural network, with said training process being continued until said error amounts reach a predetermined permissible level.

6. The apparatus according to claim 4, wherein said calibration pattern is formed on an internal surface of a cylinder and said panoramic imaging system includes an optical lens unit having a central optical axis which is oriented to coincide with an axis of said cylinder while capturing said calibration image.

7. A computer-usable medium storing computer-executable instructions, said instructions when executed implementing a method of navigating within a distorted panoramic image by selectively extracting undistorted sub-images from said distorted panoramic image, comprising steps of:

training a neural network to relate respective view directions of a panoramic imaging system to corresponding positions within a distorted panoramic image that is captured by said panoramic imaging system;

specifying a central view direction for a desired sub-image of an arbitrary distorted panoramic image that has been captured by said panoramic imaging system, with said sub-image generated by a rectangular array of pixels of a display device;

based on a known field of view of said panoramic imaging system and on a configuration of said arbitrary distorted panoramic image, expressing a lateral distance in said array equal to a pixel pitch of said array as an equivalent amount of view direction displacement in azimuth, expressing a vertical distance in said array equal to said pixel pitch as an equivalent amount of view direction displacement in elevation, and calculating respective view directions corresponding to said pixels, based on said equivalent amounts of view direction displacement and said central view direction;

successively inputting said calculated view directions to said neural network to thereby obtain information specifying respectively corresponding locations within said arbitrary distorted panoramic image; and

obtaining respective video attributes for each of said locations and assigning said video attributes to corresponding ones of said pixels, to thereby generate said sub-image.

8. The computer-usable medium according to claim 7, wherein said step of training said neural network comprises steps of:

preparing a calibration pattern which defines an array of visible calibration points, with dimensions and a shape of said calibration pattern being predetermined to enable said calibration pattern to be oriented substantially completely occupying said field of view of said panoramic imaging system;

disposing said calibration pattern to substantially completely occupy said field of view and capturing a distorted panoramic image of said calibration pattern, to constitute a calibration image;

generating a first set of data which relates each of said calibration points to a corresponding view direction with respect to said panoramic imaging system;

generating a second set of data which relates each of said calibration points to a corresponding location at which said calibration point appears within said calibration image; and

performing a repetitive training process of successively inputting data expressing said view directions from said first data set to said neural network, comparing each output response produced from said neural network with data expressing a corresponding location within said calibration image, obtained from said second data set, to thereby obtain an error amount, and applying said error amount in accordance with a predetermined algorithm to modify internal parameters of said neural network, with said training process being continued until said error amounts reach a predetermined permissible level.

9. The computer-usable medium according to claim 7, wherein said calibration pattern is formed on an internal surface of a cylinder and said panoramic imaging system includes an optical lens unit having a central optical axis, and wherein said central optical axis is oriented to substantially coincide with an axis of said cylinder while capturing said calibration image.