Method and system to calibrate a camera system using phase demodulation sensing

• US 7,433,029 B1
• Filed: 09/20/2006
• Issued: 10/07/2008
• Est. Priority Date: 09/22/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Hsu, Stephen C.
• Primary Examiner(s): Tarcza; Thomas H.
• Assistant Examiner(s): Ratcliffe; Luke D
• Application Number:
  US11/523,871
• Time in Patent Office: 748 Days
• Field of Search: 356 301- 315, 356 401- 41, 356 501- 515, 356 6- 22, 356/28, 356/28.5, 35613901-1391, 3561411-1415
• US Class Current: 356/141.5
• CPC Class Codes: G01B 11/26 for measuring angles or tapers; for testing the alignment of axes
• CPC Class Codes: G03B 43/00 Testing correct operation of photographic apparatus or parts thereof

Abstract

A preferably parametrically defined calibration target pattern definable in real-world coordinates (X_i,Y_i) is used to spatially calibrate a camera system acquiring images in pixel coordinates (x_i,y_i). The calibration target pattern is used to produce a unique 1:1 mapping enabling the camera system to accurately identify each point in the calibration target pattern point mapped to each pixel in the acquired image. Preferably the calibration target pattern is pre-distorted and includes at least two sinusoids that create a pattern of distorted wavefronts that when imaged by the camera system will appear substantially linearized. Since wavefront locations on the calibration target pattern were known, a mapping between those wavefronts and the substantially linearized wavefront pattern on the camera system captured image can be carried out. Demodulation of the captured image enables recovery of the continuous wave functions therein, using spectral analysis and iteration.

Claims

1. A calibration target definable in real-world (X_i,Y_i) coordinates usable to spatially calibrate a camera system that acquires a camera-captured image definable in pixel (x_i,y_i) coordinates, the calibration target including:

a target pattern including intensity levels that change continuously in any pattern direction such that when captured by said camera system at a known distance Z said target pattern produces an image that enables said camera system to discern real-world coordinates of any point in said target pattern with respect to pixel locations in the camera-captured image of said target pattern.

2. The calibration target of claim 1, wherein said calibration target has at least one characteristic selected from a group consisting of (a) said target pattern is pre-distorted such that an image of said calibration target captured by said camera system is substantially free of distortion, (b) said target pattern is printed on media with a material selected to be visible to said camera system at wavelengths used by said camera system to capture images, said material selected from a group consisting of ink and toner, (c) said target pattern is created so as to be projectable upon a substrate with optical wavelengths selected to be visible to said camera system, and (d) said calibration target is printed on a substantially planar substrate.

3. The calibration target of claim 1, wherein said target pattern has at least one characteristic selected from a group consisting of (a) said target pattern is parametrically definable, (b) said target pattern is pre-distorted and is parametrically definable, (c) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said target pattern, and (d) said target pattern is pre-distorted and within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern.

4. The calibration target of claim 1, wherein said target pattern is pre-distorted, and said image captured by said camera system is definable in pixels, and said target pattern is defined in terms of at least a first frequency and a second frequency, said first frequency and second frequency selected to produce a pattern in said camera-captured image, which pattern exhibits wavefronts having a wavelength greater than two and less than twenty pixels.

5. The calibration target of claim 1, wherein said target pattern is pre-distorted, and has at least one characteristic selected from a group consisting of (a) said pre-distorted calibration target is parametrically defined, (b) said pre-distorted calibration target includes at least one continuous phase function, (c) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term, (d) said target pattern is created to utilize substantially a full dynamic range of said camera system;

(e) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components;

(f) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, wherein said frequency components are selected to be sufficiently high to yield densely captured wavefronts in an image captured by said camera system without being more dense than said optical lens can resolve, and (g) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, and includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

6. A calibration target definable in real-world (X_i,Y_i) coordinates usable to spatially calibrate a camera system that acquires a camera-captured image definable in pixels having pixel (x_i,y_i) coordinates, the calibration target including:

a pre-distorted target pattern defined in terms of at least a first frequency and a second frequency, said first frequency and said second frequency selected to produce a pattern in said camera-captured image exhibiting wavefronts having a wavelength greater than two and less than twenty pixels, said target pattern when captured by said camera system at a known distance Z producing an image that enables said camera system to discern real-world coordinates of any point in said target pattern with respect to pixel locations in said camera-captured image of said target pattern.

7. The calibration target of claim 6,

wherein said calibration target has at least one characteristic selected from a group consisting of (a) said target pattern is pre-distorted such that an image of said calibration target captured by said camera system is substantially free of distortion, (b) said target pattern is printed on media with a material selected to be visible to said camera system at wavelengths used by said camera system to capture images, said material selected from a group consisting of ink and toner, (c) said target pattern is created so as to be projectable upon a substrate with optical wavelengths selected to be visible to said camera system, (d) said calibration target is printed on a substantially planar substrate, and (f) said camera system includes an optical lens and said calibration target includes frequencies selected to sufficiently high to yield densely captured wavefronts in a camera-captured image without being more dense than can be resolved by said optical lens.

8. The calibration target of claim 6,

wherein said target pattern has at least one characteristic selected from a group consisting of (a) said target pattern is Parametrically definable, (b) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said target pattern, (c) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern, and (d) said calibration target includes at least one continuous phase function that is recoverable using demodulation techniques from said camera-captured image.

14. The calibration target of claim 6, wherein said target pattern has at least one characteristic selected from a group consisting of (a) said pre-distorted calibration target is parametrically defined, (b) said pre-distorted calibration target includes at least one continuous phase function, (c) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term, (d) said target pattern is created to utilize substantially a full dynamic range of said camera system;

(e) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components;

(f) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, wherein said frequency components are selected to be sufficiently high to yield densely captured wavefronts in an image captured by said camera system without being more dense than said optical lens can resolve, and (g) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, and includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

9. A method of spatially calibrating a camera system that captures an image definable in pixel coordinates (x_i,y_i) on an image plane, the method comprising:

(a) providing a calibration target containing a pattern including intensity levels that change continuously in any Pattern direction, said pattern recognizable by said camera system and definable in real-world (X_i,Y_i) coordinates, such that when imaged by said camera system, said pattern enables said camera system to discern by identifying and locating real-world coordinates of said pattern to pixel coordinates in said image captured by said camera system; and

(b) using said camera system to capture an image of said pattern on said calibration target;

wherein a unique 1;

1 mapping between pixel coordinates and real-world coordinates is obtainable.

10. The method of claim 9, wherein step (a) includes at least one of (i) providing a calibration target whose said pattern enables said camera system to image process said image captured by said camera system, using adjacent pixels, to identify and locate said real-world coordinates of said pattern to said pixel coordinates in said image captured by said camera system, (ii) providing a pattern that is pre-distorted such that said image captured by said camera system will provide a substantially undistorted image on said image plane, said pattern comprising at least a first set of wavefronts, (iii) providing a parametrically defined calibration target pattern, (iv) providing a parametrically defined calibration target pattern that is pre-distorted such that a camera system image thereof is less distorted than said calibration target pattern containing a parametrically defined pattern, and (v) providing a pattern utilizing substantially full dynamic range of said camera system, (vi) providing a pattern comprising at least a first set of wavefronts, and (vii) providing said calibration target at a distance Z from said camera system.

11. The method of claim 9, wherein:

step (b) further includes examining said image captured by said camera system to build a unique 1;

1 mapping relationship between each pixel coordinate (x_i,y_i) in said image, with real-world coordinates (X_i,Y_i) of points on said calibration target pattern;

wherein said mapping relationship is storable and useable to calculate a relationship between real-world coordinates of an object to be imaged with said camera system, and pixel coordinates of an image of said object captured with said camera system.

12. The method of claim 9, wherein:

step (a) includes providing a calibration target pattern that includes at least a first set of wavefronts; and

step (b) includes identifying in said image a pattern of a first set of captured wavefronts that correspond to said first set of wavefronts in said calibration target and determining said mapping relationship from a comparison of said first set of wavefronts with said captured wavefronts.

13. The method of claim 9, wherein:

step (a) includes providing a calibration target pattern further including a second set of wavefronts; and

step (b) further includes identifying a second set of captured wavefronts corresponding to said second set of wavefronts in said calibration target, and determining said mapping relationship from a comparison of said first set and said second set of wavefronts with said first set and said second set of capture wavefronts.

15. The method of claim 9, wherein step (a) includes at least one of (i) providing a calibration target that is pre-distorted, (ii) providing a calibration target that is pre-distorted and is parametrically definable, (iii) providing a calibration target that is pre-distorted such that within a plane of said calibration target, said pre-distorted target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern, (iv) providing a calibration target that is pre-distorted and defines a pattern in terms of at least a first frequency and a second frequency where said first and second frequency are selected to produce a distortion function in said camera system captured mage, which distortion function exhibits wavefronts having a wavelength greater than two and less than twenty pixels, and (v) providing a calibration target on a substantially planar substrate.

16. The method of claim 9, wherein step (a) includes providing a calibration target that includes a pre-distorted calibration pattern having at least one characteristic selected from a group consisting of (i) said pre-distorted calibration target is parametrically defined, (ii) said pre-distorted calibration target includes at least one continuous phase function, (iii) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term.

17. The method of claim 9, wherein:

step (a) includes providing a calibration target pattern definable with at least a first phase; and

step (b) further includes at least one of (i) demodulating a signal in said image captured by said camera system to recover at least one of a linear phase carrier term, and a residual phase term, (ii) iteratively identifying each phase function in said image captured by said camera system, and (iii) determining a calibrated three-dimensional ray direction for pixel in said camera system captured image.

18. The method of claim 9, wherein step (b) further includes imposing global correction such that alignment is imposed between real-world coordinates defined in calibrated target object, and pixel coordinates in said image captured by said camera system.

19. The method of claim 11, wherein at least a portion of step (b) is carried out in a manner selected from a group consisting of (i) said portion is executed within said camera system, and (ii) said portion is executed external to said camera system.

26. The method of claim 9, wherein said target pattern has at least one characteristic selected from a group consisting of (a) said pre-distorted calibration target is parametrically defined, (b) said pre-distorted calibration target includes at least one continuous phase function, (c) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term, (d) said target pattern is created to utilize substantially a full dynamic range of said camera system;

(e) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components;

(f) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, wherein said frequency components are selected to be sufficiently high to yield densely captured wavefronts in an image captured by said camera system without being more dense than said optical lens can resolve, and (g) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, and includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

20. A camera system including an image plane definable in pixels, upon which an image of a target may be captured, said camera system including:

storage, associated with said camera system, to store data used during calibration of said camera system;

said camera system having been calibrated by using one of (a) a calibration target definable in real-world (X_i,Y_i) coordinates usable to spatially calibrate said camera system during acquisition of a camera-captured image definable in pixel (x_i,y_i) coordinates, the calibration target including a target pattern including intensity levels that change continuously in any pattern direction such that when captured by said camera system at a known distance Z said target pattern produces an image that enables said camera system to discern real-world coordinates of any point in said target pattern with respect to pixel locations in the camera-captured image of said target pattern; and

(b) a calibration target definable in real-world (X_i,Y_i) coordinates usable to spatially calibrate said camera system during acquisition of a camera-captured image definable in pixels having pixel (x_i,y_i) coordinates, the calibration target including a pre-distorted target pattern defined in terms of at least a first frequency and a second frequency, said first frequency and said second frequency selected to produce a pattern in said camera-captured image exhibiting wavefronts having a wavelength greater than two and less than twenty pixels, said target pattern when captured by said camera system at a known distance Z producing an image that enables said camera system to discern real-world coordinates of any point in said target pattern with respect to pixel locations in said camera-captured image of said target pattern.

21. The camera system of claim 20, wherein said memory stores at least one of (i) an executable software routine used during calibration of said camera system, (ii) calibration data for said camera system, which data is used during calibration of said camera system, and (iii) a look-up table storing mapping relationship data obtaining during calibration of said camera system.

22. The camera system of claim 20(a), wherein said calibration target has at least one characteristic selected from a group consisting of (a1) said target pattern is pre-distorted such that an image of said calibration target captured by said camera system is substantially free of distortion, (a2) said target pattern is printed on media with a material selected to be visible to said camera system at wavelengths used by said camera system to capture images, said material selected from a group consisting of ink and toner, (a3) said target pattern is created so as to be projectable upon a substrate with optical wavelengths selected to be visible to said camera system, and (d) said calibration target is printed on a substantially planar substrate, (a4) said target pattern is parametrically definable, (a5) said target pattern is pre-distorted and is parametrically definable, (a6) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said target pattern, and (a7) said target pattern is pre-distorted and within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern.

23. The camera system of claim 20(b), wherein said calibration target has at least one characteristic selected from a group consisting of (b1) said target pattern is pre-distorted such that an image of said calibration target captured by said camera system is substantially free of distortion, (b2) said target pattern is printed on media with a material selected to be visible to said camera system at wavelengths used by said camera system to capture images, said material selected from a group consisting of ink and toner, (b3) said target pattern is created so as to be projectable upon a substrate with optical wavelengths selected to be visible to said camera system, (b4) said calibration target is printed on a substantially planar substrate, (b5) said camera system includes an optical lens and said calibration target includes frequencies selected to sufficiently high to yield densely captured wavefronts in a camera-captured image without being more dense than can be resolved by said optical lens, and (b6) said calibration target includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

24. The camera system of claim 20(b), wherein said calibration target has at least one characteristic selected from a group consisting of (b1) said pre-distorted calibration target is parametrically defined, (b2) said pre-distorted calibration target includes at least one continuous phase function, (b3) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term, (b4) said target pattern is created to utilize substantially a full dynamic range of said camera system;

(b5) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components;

(b6) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, wherein said frequency components are selected to be sufficiently high to yield densely captured wavefronts in an image captured by said camera system without being more dense than said optical lens can resolve, and (b7) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components and includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

25. The camera system of claim 20, wherein said camera system includes an optical lens, and is calibrated to compensate for distortion other than distortion in said optical lens.

27. A method of spatially calibrating a camera system that captures an image definable in pixel coordinates (x_i,y_i) on an image plane, the method comprising:

(a) providing a calibration target definable in real-world (X_i,Y_i) coordinates usable to spatially calibrate a camera system that acquires a camera-captured image definable in pixels having pixel (x_i,y_i) coordinates, the calibration target including;

a pre-distorted target pattern defined in terms of at least a first frequency and a second frequency, said first frequency and said second frequency selected to produce a pattern in said camera-captured image exhibiting wavefronts having a wavelength greater than two and less than twenty pixels, said target pattern when captured by said camera system at a known distance Z producing an image that enables said camera system to discern real-world coordinates of any point in said target pattern with respect to pixel locations in said camera-captured image of said target pattern; and

(b) using said camera system to capture an image of said pattern on said calibration target.

28. The method of claim 27, wherein at step (a), said target pattern has at least one characteristic selected from a group consisting of (i) said target pattern is parametrically definable, (ii) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said target pattern, (iii) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern, and (iv) said calibration target includes at least one continuous phase function that is recoverable using demodulation techniques from said camera-captured image.

29. The method of claim 27, wherein at step (a), said target pattern has at least one characteristic selected from a group consisting of (i) said pre-distorted calibration target is parametrically defined, (ii) said pre-distorted calibration target includes at least one continuous phase function, (iii) said pre-distorted calibration target includes continuous phase functions that are recoverable in an image captured by said camera system as a linear phase carrier term and a residual phase term, (iv) said target pattern is created to utilize substantially a full dynamic range of said camera system;

(v) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components;

(vi) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, wherein said frequency components are selected to be sufficiently high to yield densely captured wavefronts in an image captured by said camera system without being more dense than said optical lens can resolve, and (vii) said calibration target is definable as product of a geometric transformation function that may equal identity and a sum of sinusoidal frequency components, and includes at least one continuous phase function that is recoverable from an image captured by said camera system using demodulation techniques.

30. The method of claim 27, wherein at step (a), said target pattern has at least one characteristic selected from a group consisting of (i) said target pattern is parametrically definable, (ii) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said target pattern, (iii) within a plane definable for said calibration target, said target pattern is substantially free of two-dimensional ambiguity except for singular points defined within said pre-distorted target pattern, and (iv) said calibration target includes at least one continuous phase function that is recoverable using demodulation techniques from said camera-captured image.