Pyramid processor for building large-area, high-resolution image by parts

• US 4,797,942 A
• Filed: 03/02/1987
• Issued: 01/10/1989
• Est. Priority Date: 03/02/1987
• Status: Expired due to Term

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Burt, Peter J.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Couso, Jose L.
• Application Number:
  US07/020,818
• Time in Patent Office: 680 Days
• Field of Search: 250/208, 356/114, 358/133, 358/160, 358/166, 358/167, 358/109, 358/204, 358/293, 358/294, 358/87, 358/903, 382/1, 382/17, 382/41, 382/43, 382/49, 382/54, 382/57, 382/62
• US Class Current: 382/284
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting

Abstract

A technique, employing image-processing pyramids, which is capable of combining a plurality of small-area high-resolution sub-images, such as may be derived from an array of television cameras, into a single large-area high-resolution image, with substantially no introduction by such processing of image-artifacts in the combined single image.

Claims

1. A pyramid-processing method for deriving, substantially without the introduction of spurious spatial-frequency image artifacts, the spatial-frequency spectrum analysis of a single high-resolution image extending over the entirety of a relatively-large-area two dimensional field of view and comprised of pixels having a given pixel density;

said deriving being done from an array of relatively-small-area, high resolution sub-images each of which also has said given pixel density and extends over a mutually exclusive two dimensional portion of said two dimensional field of view;

said method comprising the steps of;

(a) pyramid-processing said array to derive a separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given positive integer, for each individual one of said sub-images, wherein Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level;

(b) pyramid-processing said array to derive a single Gaussian-output pyramid level G_K for the entirety of said relatively large area, wherein said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of said Laplacian-output pyramid level L'"'"'_K-1 ; and

(c) separately storing respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 and respective pixels of said single Gaussian-output pyramid level G_K.

2. The method defined in claim 1, wherein said sub-images are similar in size and are contiguous, and wherein each of said steps (a) and (b) of pyramid processing said array comprises the step of pyramid-processing in parallel all of said respective ones of said sub-images of said array.

12. The method defined in claim 1, further including the preliminary step of deriving from an array of television cameras having overlapping fields of view said array of small-areas, high-resolution sub-images, whereby the respective small-areas of adjacent ones of said television-derived high-resolution sub-images of said array overlap one another.

13. The method defined in claim 1, further including the preliminary step of separately storing the pixels of said respective high-resolution sub-images in a first memory, and performing each of steps a) and b) on pixels of each individual one of said high-resolution sub-images separately read out of said first memory;

and wherein

step (c) comprises the step of storing the respective pixels of said single Gaussian output level G_K in a second memory as a single G_K image; and

said method comprises the further steps of (1) dividing the single G_K image stored in said second memory into a second array of separate sub-images of G_K pixels;

(2) substituting the G_K pixel sub-images of said second array for the high-resolution images of said first-mentioned array and performing the pyramid processing of each of steps (a) and (b) on said G_K pixel sub-images thereby to derive a separate set of Laplacian-output pyramid levels L'"'"'_K . . . L'"'"'_K'"'"'-1 lower than a certain pyramid level K'"'"', where K'"'"' is a given positive integer greater than K, for each individual one of said sub-images of said second array and to derive a single Gaussian output pyramid level G_K, for said entire single G_K image stored in said second memory; and

(3) separately storing in a third memory the respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'_K . . . L'"'"'_K'"'"'-1 and the respective pixels of said single Gaussian output pyramid level G.sub. K'"'"'.

3. A pyramid-processing method for deriving, substantially without the introduction of spurious spatial-frequency image artifacts, the spatial-frequency spectrum analysis of a single high-resolution image extending over the entirety of a relatively-large-area field of view and comprised of pixels having a given pixel density;

said deriving being done from an array of relatively-small-area, high resolution sub-images disposed in abutting relationship with one another, each of which also has said given pixel density, said method comprising the steps of;

(a) pyramid-processing said array to derive a separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given positive integer, for each individual one of said sub-images, wherein Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level, including separately pyramid-processing a given set of pixels associated with each individual one of said abutting sub-images of said array with multi-tap spatial filters, all of which filters exhibit a characteristic in each spatial dimension defined by a given kernel weighting function comprised of a certain odd plural number of coefficients symmetrically disposed about the centrally-located coefficient of said kernel weighting function, said given set of pixels associated with any individual one of said abutting sub-images including all the pixles situated within the boundary of that abutting sub-image plus a certain number of additional bordering pixels situated outside the boundary of that abutting sub-image which extend the integer portion of one-half said odd plural number of pixels beyond the boundaries of that abutting sub-image in each dimension thereof, said separate pyramid-processing resulting in the derivation of a separate set of Laplacian-output pyramid levels L₀ . . . L_K-1 for each individual one of said abutting sub-images, each of said separate sets having the same pixel density as its corresponding one of said L'"'"'₀ . . . L'"'"'_K-1 pyramid levels and including additional bordering Laplacian-output pyramid level pixels situated outside the boundary of that abutting sub-image which corresponds thereto; and

separating trimming said additional bordering pixels of each of said L₀ . . . L_K-1 Laplacian-output pyramid levels of the separate set for each individual one of said abutting sub-images to derive thereby said corresponding L'"'"'₀ . . . L'"'"'_K-1 Laplacian output pyramid levels for that abutting sub-image which corresponds thereto;

(b) pyramid-processing said array to derive a single Gaussian-output pyramid level G_K for the entirety of said relatively large area, wherein said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of said Laplacian-output pyramid level L'"'"'_K-1 ; and

(c) separately storing respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 and respective pixels of said single Gaussian-output pyramid level G_K.

4. The method defined in claim 3, wherein K is equal to two.

5. The method defined in claim 4, wherein said certain odd plural number is five.

6. A pyramid-processing method for deriving, substantially without the introduction of spurious spatial-frequency image artifacts, the spatial-frequency spectrum analysis of a single high-resolution image extending over the entirety of a relatively-large-area field of view and comprised of pixels having a given pixel density;

said deriving being done from an array of relatively-small-area, high-resolution sub-images having respective boundaries and disposed in overlapping relationship each of which also has said given pixel density said method comprising the steps of;

(a) pyramid-processing said array to derive a separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given positive integer, for each individual one of said sub-images, wherein Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level, including separately pyramid-processing a given set of pixels associated with each individual one of said overlapping sub-images of said array with multi-tap spatial filters, all of which filters exhibit a characteristic in each spatial dimension defined by a given kernel weighting function comprised of a certain odd plural number of coefficients symmetrically disposed about the centrally-located coefficient of said kernel weighting function, said given set of pixels associated with any individual one of said overlapping sub-images including all the pixels situated within the boundary of that overlapping sub-image plus a certain number of additional bordering pixels situated outside the boundary of that overlapping sub-image which extend the integer portion of one-half said odd plural number of pixels beyond the boundaries of that overlapping sub-image in each dimension thereof, said separate pyramid-processing resulting in the derivation of a separate set of Laplacian-output pyramid levels L₀ . . . L_K-1 for each individual one of said overlapping sub-images, each of said separate sets having the same pixel density as its corresponding one of said L'"'"'₀ . . . L'"'"'_K-1 pyramid levels and including additional bordering Laplacian-output pyramid level pixels situated outside the boundary of that overlapping sub-image which corresonds thereto; and

separately trimming said additional bordering pixels of each of said L₀ . . . L_K-1 Laplacian-output pyramid levels of the separate set for each individual one of said overlaping sub-images to derive thereby said corresponding L'"'"'₀ . . . L'"'"'_K-1 Laplacian output pyramid levels for that overlapping sub-image which corresponds thereto;

(b) pyramid-processing said array to derive a single Gaussian-output pyramid level G_K for the entirety of said relatively large area, wherein said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of said Laplacian-output pyramid level L'"'"'_K-1 ; and

(c) separately storing respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 and respective pixels of said single Gaussian-output pyramid level G_K.

7. The method defined in claim 6, wherein K is equal to two.

8. The method defined in claim 7, wherein said certain odd plural number is five.

9. A pyramid-processing method for deriving, substantially without the introduction of spurious spatial-frequency image artifacts, the spatial-frequency spectrum analysis of a single high-resolution image extending over the entirety of a relatively-large-area field of view and comprised of pixels having a given pixel density;

said deriving being done from an array of relatively-small-area, high-resolution sub-images disposed in abutting relationship with one another each of which also has said given pixel density;

said method comprising the steps of;

(a) storing the pixels of each of said high-resolution sub-images of said array in a first memory, whereby the pixels stored in said first memory include a limited first set of significant level-value pixels associated with each individual one of said high-resolution sub-images that are wholly within and define a sub-image domain of that individual one of said sub-images;

(b) pyramid-processing said array to derive a separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given positive integer, for each individual one of said sub-images, wherein Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level;

(c) pyramid-processing said array to derive a single Gaussian-output pyramid level G_K for the entirety of said relatively large area, wherein said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of said Laplacian-output pyramid level L'"'"'_K-1, including reading out from said first memory and separately pyramid processing a given second set of stored significant level-value pixels associated with each individual one of said high-resolution sub-images with multi-tap spatial filters, all of which filters exhibit a characteristic in each spatial dimension defined by a given kernel weighting function comprised of a certain odd plural number of coefficients symmetrically disposed about the centrally-located coefficient of said kernel weighting function, said given second set of stored pixels associated with any individual one of said sub-images including all the stored pixels of said limited first set plus a certain number of bordering pixels which extend the integer portion of one-half said odd plural number of pixels beyond the boundaries of the sub-image domain of that individual one of said sub-images in each spatial dimension thereof, said separate pyramid processing resulting in the derivation of G_K pixels for each individual one of said sub-images that includes certain significant level-value G_K pixels situated wholly within the sub-image domain defined by that individual one of said sub-images; and

(d) separately storing respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 and for each individual one of said sub-images, storing in a second memory only its derived certain significant level-value G_K pixels.

10. The method defined in claim 9, wherein K is equal to two.

11. A pyramid-processing method for deriving, substantially without the introduction of spurious spatial-frequency image artifacts, the spatial-frequency spectrum analysis of a single high-resolution image extending over the entirety of a relatively-large-area field of view and comprised of pixels having a given pixel density;

said deriving being done from an array of relatively-small-area, high-resolution sub-images each of which also has said given pixel density and extends over only a portion of said field of view;

the portions of said field of view over which said sub-images respectively extend together comprising the entirety of said relatively large-area field of view;

said method comprising the steps of;

(a) storing the pixels of each of said high-resolution sub-images of said array in a first memory, whereby the pixels stored in said first memory include a limited first set of significant level-value pixels associated with each individual one of said high-resolution sub-images that are wholly within and define a sub-image domain of that individual one of said sub-images, the sub-image domain of each individual one of said sub-images abutting the respective sub-image domains of those sub-images of said array that are situated next to that individual one of said sub-image;

(b) pyramid-processing said array to derive a separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given positive integer, for each individual one of said sub-images, wherein Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level;

(c) pyramid-processing said array to derive a single Gaussian-output pyramid level G_K for the entirety of said relatively large area, wherein said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of said Laplacian-output pyramid level L'"'"'_K-1, including reading out from said first memory and separately pyramid processing all of the limited set of stored pixels defining the sub-image domain associated with each individual one of said high-resolution sub-images with multi-tap spatial filters, all of which filters exhibit a characteristic in each spatial dimension defined by a given kernel weighting function comprised of a certain odd plural number of coefficients symmetrically disposed about the centrally-located coefficient of said kernel weighting function, said separate pyramid processing resulting in the derivation of G_K pixels for each individual one of said sub-images that includes both certain significant level-value G_K pixels situated wholly within the particular sub-image domain of that individual one of said sub-images and additional significant level-value G_K pixels extending into a sub-image domain that abuts that particular sub-image domain; and

(d) separately storing respective pixels of each separate set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 and for each individual one of said sub-images, the steps of (1) adding the level value of each additional G_K pixel that extends into the particular sub-image domain thereof from a sub-image domain that abuts that particular sub-image domain to the level value of the corresponding certain G_K pixel of that particular sub-image domain, thereby altering the respective level values of at least some of said certain G_K pixels of that particular sub-image domain, and (2), following said alteration, separately storing in a second memory the respective level-values of all of the certain G_K pixels of that particular sub-image domain.

14. In a machine vision system comprised of utilization means responsive to the image content of a high-resolution television image of a given item of interest, said given item of interest having a two dimensional area that is too large to be viewed with high resolution by a single television camera;

a combination for deriving the image content of a high-resolution television image of said given item of interest, substantially without the introduction of spurious spatial-frequency image artifacts, for use by said utilization means;

said combination comprising;

a given plural number of separate television cameras arranged in an array, such that the field of view of each individual television camera in said television camera array thus formed partially overlaps the fields of view of those television cameras situated next to that individual television camera and such that the field of view of said television camera array as a whole views said two dimensional area of said given item of interest in its entirety and with high resolution, whereby said television camera array derives a corresponding array of overlapping high-resolution sub-images, each of said high-resolution sub-images including a mutually exclusive two dimensional portion of the image content of said two dimensional area of said given item, but all of said high-resolution sub-images together including all of the image content of said given item; and

pyramid-processing means responsive to said corresponding array of overlapping high-resolution sub-images for converting said array of sub-images to a spatial-frequency spectrum analysis of a single image defining the content of said given item with high resolution and substantially without the introduction of spurious spatial-frequency image artifacts, said spatial-frequency spectrum analysis of said single image being for use by said utilization means.

15. In a machine vision system comprised of utilization means responsive to the image content of a high-resolution television image of a given item of interest, said given item of interest having an area that is too large to be viewed with high resolution by a single television camera;

a combination for deriving the image content of a high-resolution television image of said given item of interest, substantially without the introduction of spurious spatial-frequency image artifacts, for use by said utilization means;

said combination comprising;

a given plural number of separate television cameras arranged in an array, such that the field of view of each individual television camera in said television camera array thus formed partially overlaps the fields of view of those television cameras situated next to that individual television camera and such that the field of view of said television camera array as a whole views said area of said given item of interest in its entirety and with high resolution, whereby said television camera array derives a corresponding array of overlapping high-resolution sub-images, each of said high-resolution sub-images including only a portion of the image content of said given item, but all of said high-resolution sub-images together including all of the image content of said given item; and

pyramid-processing means responsive to said corresponding array of overlapping high-resolution sub-images for converting said array of sub-images to a spatial-frequency spectrum analysis of a single image defining the content of said given item with high resolution and substantially without the introduction of spurious spatial-frequency image artifacts, said spatial-frequency spectrum analysis of said single image being for use by said utilization means;

said pyramid-processing means including;

separate pyramid-analyzer modules equal in number to said given plural number, each of said modules being capable of pyramid processing pixels corresponding to a different one of said high-resolution sub-images into (K+1) pyramid levels comprised of Laplacian pyramid levels L'"'"'₀ . . . L'"'"'_K-1 pixels and Gaussian pyramid-level G_K pixels for each of said different sub-images;

a camera-image pixel memory and camera memory output control coupled to said pyramid analyzer modules for separately storing the pixels of each individual one of said high-resolution sub-images in said camera-image pixel memory, under the control of said camera output control, and reading our certain stored high-resolution pixels belonging to an individual sub-image and applying said read out stored pixels as an input to that separate pyramid analyzer module that corresponds to that individual sub-image; and

a large-area (K+1) level pyramid-output pixel memory buffer coupled to said pyramid-analyzer modules separately storing separate sets of Laplacian pyramid-level L'"'"'₀ . . . L'"'"'_K-1 pixels derived, respectively, from each different one of said pyramid analyzer modules and separately storing a single set of G_K pixels derived from all of said different ones of said modules.

16. The combination defined in claim 15, wherein said pixel memory and camera memory output control reads out in parallel the stored pixels of the different sub-images respectively applied as inputs to respective ones of said pyramid-analyzer modules.

17. The combination defined in claim 15, wherein K is equal to two.

18. The combination defined in claim 15, wherein each of said separate pyramid-analyzer modules comprises:

a K-stage pyramid having K stages and having certain stored readout pixels of the high-resolution sub-image corresponding to that separate pyramid-analyzer module applied as an input thereto, said K-stage pyramid deriving Laplacian pyramid-level L₀ . . . L_K-1 pixels and G_K pixels;

an individual sub-image domain trimmer associated with each of said stages of said K-stage pyramid for trimming away some of the pixels from that one of the Laplacian pyramid level L₀ . . . L_K-1 pixels derived by that stage and thereby obtaining said L'"'"'₀ . . . L'"'"'_K-1 Laplacian pyramid level pixels as an output from each of said respective domain trimmers; and

a module-output distributor for selectively steering different groups of G_K pixels derived from said K-stage pyramid to separate sub-image domain storage locations of said pyramid-output pixel memory buffer, one of said different groups of G_K pixels corresponding to a sub-image domain associated with the high-resolution sub-image with which that pyramid-analyzer module corresponds, and each other of said different groups of G_K pixels corresponding to a sub-image domain associated with a high-resolution sub-image that overlaps the high-resolution sub-image with which that pyramid-analyzer module corresponds.

19. A method for pyramid processing a single two dimensional large-area, high-resolution image having a given pixel density;

said method comprising the steps of;

(a) dividing said single two dimensional large-area, high-resolution image into a plurality of contiguous two dimensional sub-images each of which has said given pixel density, and each sub image including a mutually exclusive portion of said single two dimensional large area high resolution image;

(b) pyramid-processing in parallel each of said contiguous sub-images to derive a respective set of Laplacian-output pyramid levels L'"'"'₀ . . . L'"'"'_K-1 lower than a certain pyramid level K, where K is a given integer, and a respective Gaussian-output pyramid level G_K wherein each Laplacian-output pyramid level L'"'"'₀ has said given pixel density and each Laplacian output pyramid level above L'"'"'₀ has a pixel density smaller than its immediately preceding Laplacian-output pyramid level and each said Gaussian-output pyramid level G_K has a pixel density which is smaller than the pixel density of each said Laplacian-output pyramid level L'"'"'_K-1 ; and

;

(c) merging at least the Gaussian-output pyramid level G_K of each of said respective sub-images into a single large area image.