Methods for enhancing performance and data acquired from three-dimensional image systems

• US 20030063775A1
• Filed: 12/10/2001
• Published: 04/03/2003
• Est. Priority Date: 09/22/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Bamji, Cyrus, Torunoglu, Iihami, Sze, Cheng-Feng, Rafii, Abbas
• Granted Patent: US 6,674,895 B2
• Time in Patent Office: Days
• US Class Current: 382/106
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/14 wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/486 Receivers
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 3/0221 Arrangements for reducing keyboard size for transport or storage, e.g. foldable keyboards, keyboards with collapsible keys G06F3/0216 takes precedence; for keyboards integrated in a laptop computer G06F1/1666
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 3/155 Control of the image-sensor operation, e.g. image processing within the image-sensor

Abstract

A three-dimension distance time-of-flight system is disclosed in which distance values are acquired by a plurality of sensors independently from each other. For use with this and similar systems, Z-distance accuracy and resolution are enhanced using various techniques including over-sampling acquired sensor data and forming running averages, or forming moving averages. Acquired data may be rejected if it fails to meet criteria associated with distance, luminosity, velocity, or estimated shape information reported by neighboring sensors. A sub-target having at least one pre-calibrated reflectance zone is used to improve system measurement accuracy. Elliptical error is corrected for using a disclosed method, and reversible mapping of Z-values into RGB is provided.

Claims

1. For use with a system that acquires at least one of (x,y,z) distance and brightness measurements from a target source using independent sensors, a method of improving distance measurements comprising the following steps:

(a) causing said sensors to acquire measurement data at a repetition rate higher than required for operation of said system;

(b) combining output signals from said sensors to obtain statistical average output signals;

wherein random noise in said system is reduced proportionally to a square root of number of said output signals averaged, and accuracy of said distance measurements is enhanced.

2. The method of claim 1, wherein said sensors are formed in an array, and step (b) includes forming statistical running averages of outputs from chosen ones of said sensors based upon at least one of (i) neighboring proximity of said sensors, and (ii) successive proximity in time of sampled said outputs.

3. The method of claim 2, wherein step (b) excludes ones of said outputs if said ones do not meet at least one threshold associated with (i) distance measurement, (ii) brightness measurement, (iii) velocity measurement, and (iv) estimated shape contour of said target source.

4. The method of claim 3, wherein said sensors are formed in an array, and step (b) includes forming statistical running averages of outputs from chosen ones of said sensors based upon at least one of (i) neighboring proximity of said sensors, and (ii) successive proximity in time of sampled said outputs.

5. The method of claim 4, wherein step (b) excludes ones of said outputs if said ones do not meet at least one threshold associated with (i) distance measurement, (ii) brightness measurement, (iii) velocity measurement, and (iv) estimated shape contour of said target source.

6. The measurement of claim 2, wherein said system acquires data from said sensors at a nominal frame rate, and step (b) includes at least one of (i) averaging said outputs acquired from a same frame, and (ii) averaging said outputs acquired from at least two adjacent frames.

7. The method of claim 1, wherein said system includes a reference target having at least one region of pre-calibrated reflectivity, and further including (c) correcting said statistical average output signals made at step (b) according to detecting signals from said reference target.

8. For use with a system that acquires at least one of (x,y,z) distance and brightness measurements using energy transmitter from an emitter at a first location on a plane, said energy reflecting at least in part from a target source and being detected by independent sensors defining a sensor array on said plane but spaced-apart from said first location, a method of improving distance measurements comprising the following steps:

(a) defining a spherical coordinate for each sensor in said array, and constructing a look-up table containing spherical coordinates for each said sensor;

(b) defining a spatial coordinate of said emitter;

(c) For each sensor <

i,j>

, calculating constants k_ij and h_ij as follows;

k_ij=C_x²+C_y²C_z², andC=2(p_ij+C_x cos(a_ij)sin(b_ij)+C_y sin(a_ij)sin(b_ij)+C_z cos(b_ij))  

 

(3)

wherein sensor p has spherical coordinate (p_ij, −

a_ij, −

b_ij) and has Cartesian coordinate (p_x, p_y, p_z)=(p_ij cos(−

a_ij)sin(−

b_ij), p_ij sin(−

a_ij)sin(−

b_ij), p_ij cos(−

b_ij))

(d) constructing a look-up table containing calculated said values of k_ij and k_ij;

(e) identifying sensors <

i,j>

that actually detect energy reflected from said target object;

(f) for each sensor <

i,j>

identified at step (e), calculating calculate r_A according to r_A=((2 d−

p)²−

k_ij)/(4 d−

h_ij) where in a spherical coordinate system, point A is representable as (r_A, a_ij, b_ij) using values of k_ij and k_ij from step (d);

(g) calculating roundtrip distance 2d from said target object to sensor <

i,j>

; and

(h) calculating actual coordinate of said target object detected at sensor <

ij>

according to A_x=r_A cos(a_ij)sin(b_ij), A_y=r_A sin(a_ij)sin(b_ij), and A_z=r_A cos(β

_ij).

9. For use with a video imaging system that can encode Z values, a method of encoding said Z values as part of YIQ encoding, the method comprising the following steps:

(a) converting a RGB value for each sensor to a RGB matrix and converting said RGB matrix to a YIQ matrix;

(b) partitioning said YIQ matrix in Y, I, and Q planes;

(c) Fourier transforming I and Q dimensions of said YIQ matrix;

(d) allocating segments of said I and Q dimensions in a frequency domain to store Z-values, wherein said segments correspond to the frequencies that if eliminated from a reverse transformation would not substantially alter color perception by a human viewer;

(e) locating segments having at least one characteristic selected from a group consisting of (i) said segments are not used, and (ii) said segments fall below a predetermined threshold of visibility;

said segments being sufficiently large to store Z-values for all sensors;

(f) encoding Z(X,Y) coordinates of each sensor using said segments;

(g) adjusting amplitude coefficients of said segments;

(h) transform I″

Q″

from frequency domain to time domain, and appending Y thereto to create a YI″

Q″

matrix; and

(i) transforming from said YI″

Q″

matrix to a R″

G″

B″

matrix.

10. The method of claim 9, wherein at step (b) if said YIQ matrix comprised N-bit values per sensor, step (b) includes logically dissecting said matrix into three matrices, each having (N/3)-bit values per sensor.

11. The method of claim 9, wherein step (f) includes using Huffman encoding.

12. The method of claim 9, wherein step (g) includes just a just-noticable-differences technique.

13. The method of claim 9, wherein recovering of said Z values is carried out according to the following steps:

(a) converting R′

G′

B′

back to YI′

Q′

;

(b) converting I′

Q′

to a frequency domain matrix; and

(c) extracting Z values from said matrix resulting from step (b).

14. A computer-readable storage medium wherein is located a computer program that causes a computer sub-system having at least a processor unit to control a system that acquires at least one of (x,y,z) distance and brightness measurements from a target source using independent sensors to enhance performance of said system by:

(a) causing said sensors to acquire measurement data at a repetition rate higher than required for operation of said system;

(b) combining output signals from said sensors to obtain statistical average output signals;

wherein random noise in said system is reduced proportionally to a square root of number of said output signals averaged, and accuracy of said distance measurements is enhanced.

15. The storage medium of claim 14, wherein sensors in said system are formed in an array, and wherein execution of said computer program forms running averages of outputs from chosen ones of said sensors based upon neighboring proximity of said sensors.

16. The storage medium of claim 14, wherein sensors in said system are formed in an array, and wherein execution of said computer program excludes ones of said outputs if said ones do not meet at least one threshold associated with (i) distance measurement, (ii) brightness measurement, (iii) velocity measurement, and (iv) estimated shape contour of said target source.

17. The storage medium of claim 14, wherein said system acquires data from said sensors at a nominal frame rate, and execution of said computer program causes at least one of (i) averaging said outputs acquired from a same frame, and (ii) averaging said outputs acquired from at least two adjacent frames.

18. A computer-readable storage medium wherein is located a computer program that causes a computer sub-system having at least a processor unit to improve distance measurements in a system that acquires at least one of (x,y,z) distance and brightness measurements using energy transmitter from an emitter at a first location on a plane, said energy reflecting at least in part from a target source and being detected by independent sensors defining a sensor array on said plane but spaced-apart from said first location by carrying out the following steps:

(a) defining a spherical coordinate for each sensor in said array, and constructing a look-up table containing spherical coordinates for each said sensor;

(b) defining a spatial coordinate of said emitter;

(c) For each sensor <

i,j>

, calculating constants k_ij and h_ij as follows;

k_ij=C_x²+C_y²+C_z², and

C=2(p_ij+C_x cos(a_ij)sin(b_ij)+C_y sin(a_ij)sin(b_ij)+C_z cos(b_ij))

wherein sensor p has spherical coordinate (p_ij, −

a_ij, −

b_ij) and has Cartesian coordinate (p_x, p_y, p_z)=(p_ij cos(−

a_ij)sin(−

b_ij), p_ij sin(−

a_ij)sin(−

b_ij), p_ij cos(−

b_ij))

(d) constructing a look-up table containing calculated said values of k_ij and k_ij;

(e) identifying sensors <

i,j>

that actually detect energy reflected from said target object;

(f) for each sensor <

i,j>

identified at step (e), calculating calculate r_A according to r_A=((2 d−

p_ij)²−

k_ij)/(4 d−

h_ij) where in a spherical coordinate system, point A is representable as (r_A, a_ij, b_ij) using values of k_ij and k_ij from step (d);

(g) calculating roundtrip distance 2d from said target object to sensor <

i,j>

; and

(h) calculating actual coordinate of said target object detected at sensor <

i,j>

according to A_x=r_A cos(a_ij)sin(b_ij), A_y=r_A sin(a_ij)sin(b_ij), and A_z=r_A cos(β

_ij).

19. A computer-readable storage medium wherein is located a computer program that causes a computer sub-system having a processor unit for use with a video imaging system that can encode Z values to encode Z values as part of YIQ encoding by:

(a) converting a RGB value for each sensor to a RGB matrix and converting said RGB matrix to a YIQ matrix;

(b) partitioning said YIQ matrix in Y, I, and Q planes;

(c) Fourier transforming I and Q dimensions of said YIQ matrix;

(d) allocating segments of said I and Q dimensions in a frequency domain to store Z-values, wherein said segments correspond to the frequencies that if eliminated from a reverse transformation would not substantially alter color perception by a human viewer;

(e) locating segments having at least one characteristic selected from a group consisting of (i) said segments are not used, and (ii) said segments fall below a predetermined threshold of visibility;

said segments being sufficiently large to store Z-values for all sensors;

(f) encoding Z(X,Y) coordinates of each sensor using said segments;

(g) adjusting amplitude coefficients of said segments;

(h) transform I″

Q″

from frequency domain to time domain, and appending Y thereto to create a YI″

Q″

matrix; and

(i) transforming from said YI″

Q″

matrix to a R″

G″

B″

matrix.

20. The storage medium of claim 19, wherein if said YIQ matrix comprised N-bit values per sensor, execution of said computer program logically dissects said matrix into three matrices, each having (N/3)-bit values per sensor.