TRACKING SYSTEM CALIBRATION WITH MINIMAL USER INPUT

• US 20100150404A1
• Filed: 05/04/2009
• Published: 06/17/2010
• Est. Priority Date: 12/17/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Larsen, Eric, Marks, Richard Lee
• Granted Patent: US 8,761,434 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

To calibrate an positional sensor, a plurality of image locations and image sizes of a tracked object are received as the tracked object is moved through a rich motion path. Inertial data is received from the tracked object as the tracked object is moved through the rich motion path. Each of the plurality of image locations is converted to a three-dimensional coordinate system of the positional sensor based on the corresponding image sizes and a field of view of the positional sensor. An acceleration of the tracked object is computed in the three-dimensional coordinate system of the positional sensor. The inertial data is reconciled with the computed acceleration, calibrating the positional sensor.

Claims

1. A computer implemented method of calibrating a tracking system, comprising:

receiving positional data of a tracked object from a positional sensor by a video game console as the object is moved through a rich motion path;

receiving inertial data corresponding to the tracked object by the video game console as the tracked object is moved through the rich motion path;

converting the positional data to a three-dimensional coordinate system of the positional sensor;

computing an acceleration of the tracked object in the three-dimensional coordinate system of the positional sensor; and

reconciling the inertial data with the computed acceleration.

2. The method of claim 1, wherein the positional sensor is a video camera, wherein the positional data includes a plurality of image locations and image sizes, and wherein converting the positional data includes converting each of the plurality of image locations to the three-dimensional coordinate system of the positional sensor based on the corresponding image sizes and a field of view of the video camera.

3. The method of claim 1, wherein the rich motion path includes at least one of a motion towards the positional sensor or a motion away from the positional sensor.

4. The method of claim 1, wherein the rich motion causes accelerations that exceed a minimum acceleration threshold.

5. The method of claim 1, wherein the tracked object maintains an approximately fixed pitch and roll as it is moved through the rich motion path, the approximately fixed pitch and roll enabling the inertial data to be reconciled with the computed acceleration without converting the inertial data to a three-dimensional world coordinate system.

6. The method of claim 1, further comprising:

converting the received inertial data into a three-dimensional world coordinate system, the three-dimensional world coordinate system having an axis that is aligned with gravity.

7. The method of claim 1, wherein the inertial data includes a first set of vectors and the computed acceleration includes a second set of vectors, and wherein reconciling the inertial data with the computed acceleration comprises:

determining a first angle between the first set of vectors and the second set of vectors that is representative of a pitch of the positional sensor; and

determining a second angle between the first set of vectors and the second set of vectors that is representative of a relative yaw between the positional sensor and the tracked object.

8. The method of claim 7, wherein reconciling the inertial data with the computed acceleration further comprises:

selecting a plurality of potential first angles and a plurality of potential second angles;

rotating the first set of vectors by each combination of one of the plurality of potential first angles and one of the plurality of potential second angles;

finding a difference between the rotated first set of vectors and the second set of vectors for each combination of one of the plurality of potential first angles and one of the plurality of potential second angles; and

identifying a combination of a first potential angle and a second potential angle that causes the smallest difference between the rotated first set of vectors and the second set of vectors as the first angle and the second angle.

9. The method of claim 7, wherein reconciling the inertial data with the computed acceleration further comprises:

determining a three-dimensional angle between the first set of vectors and the second set of vectors; and

decomposing the three-dimensional angle into the first angle and the second angle.

10. The method of claim 7 further comprising:

for each vector in the first set of vectors and in the second set of vectors, filtering out the vector if it has a magnitude that is smaller than a minimum threshold.

11. The method of claim 7, further comprising:

weighting each vector in the first set of vectors and in the second set of vectors based on a magnitude of the vector, wherein vectors having a higher magnitude are weighted more heavily than vectors having a lower magnitude.

12. The method of claim 7, further comprising:

comparing the magnitude of each vector of the first set of vectors to close in time vectors from the second set of vectors; and

if a first vector of the first set of vectors is approximately equal in magnitude to a close in time second vector of the second set of vectors, determining that the first vector represents a motion in a three dimensional world coordinate system and the second vector represents the motion in the three dimensional coordinate system of the positional sensor.

13. The method of claim 1, wherein the positional sensor is one of a camera, a z-camera, an ultrasonic sensor or a photonic sensor.

14. The method of claim 1, wherein the inertial data is gathered by an inertial sensor having a fixed displacement from a centroid of the tracked object, the method further comprising:

applying an offset to one of the inertial data or the computed acceleration based on measurements provided by a gyroscope and a known distance between the accelerometer and the centroid, wherein the offset compensates for the fixed displacement.

15. The method of claim 1, wherein the field of view of the positional sensor is unknown, the method further comprising:

solving for the field of view of the positional sensor.

16. The method of claim 14, wherein solving for the field of view comprises:

assuming one or more potential fields of view;

for each potential field of view, attempting to reconcile the inertial data with the computed acceleration; and

identifying the potential field of view for which the inertial data is best able to reconcile with the computed acceleration as the field of view of the positional sensor.

17. A method of calculating a camera'"'"'s field of view comprising:

receiving image data of a sphere, the image data including an elliptical projection of the sphere onto an image plane of the camera;

determining a position of the sphere in the image plane of the camera from the image data;

computing a major axis and a minor axis of the elliptical projection;

computing the camera'"'"'s field of view based on the position of the sphere and a difference between the major axis and the minor axis; and

using the field of view to track the sphere in three-dimensional space.

18. The method of claim 17, wherein the camera has a plurality of known field of view settings.

19. The method of claim 18, further comprising:

comparing the computed field of view to each of the plurality of known fixed field of view settings; and

determining that a current field of view of the camera is equal to a fixed field of view setting closest to the computed field of view.

20. A computer readable storage medium including instructions that, when executed by a processing system, cause the processing system to perform a method comprising:

receiving a positional data of a tracked object from device positional sensor as the object is moved through a rich motion path;

receiving inertial data corresponding to the tracked object by as the tracked object is moved through the rich motion path;

converting the positional data to a three-dimensional coordinate system of the positional sensor;

computing an acceleration of the tracked object in the three-dimensional coordinate system of the positional sensor; and

reconciling the inertial data with the computed acceleration.

21. The computer readable storage medium of claim 20, wherein the positional sensor is a video camera, wherein the positional data includes a plurality of image locations and image sizes, and wherein converting the positional data includes converting each of the plurality of image locations to the three-dimensional coordinate system of the positional sensor based on the corresponding image sizes and a field of view of the video camera.

22. The computer readable storage medium of claim 20, wherein the inertial data includes a first set of vectors and the computed acceleration includes a second set of vectors, and wherein reconciling the inertial data with the computed acceleration comprises:

determining a first angle between the first set of vectors and the second set of vectors that is representative of a pitch of the positional sensor; and

determining a second angle between the first set of vectors and the second set of vectors that is representative of a relative yaw between the positional sensor and the tracked object.

23. The computer readable storage medium of claim 20, wherein the field of view of the positional sensor is unknown, and wherein the tracked object is a sphere the method further comprising:

solving for the field of view of the positional sensor, wherein solving for the field of view comprises;

receiving image data of the sphere, the image data including an elliptical projection of the sphere onto an image plane of the positional sensor;

determining a position of the sphere in the image plane of the positional sensor from the image data;

computing a major axis and a minor axis of the elliptical projection;

computing the positional sensor'"'"'s field of view based on the position of the sphere and a difference between the major axis and the minor axis.

24. A tracking system, comprising:

a tracked object that includes one or more inertial sensors, the tracked object to gather inertial data as the tracked object is moved through a rich motion path, and to transmit the inertial data to a computing device;

device positional sensor to gather positional data of the tracked object as the object is moved through the rich motion path, and to transmit the positional data to the computing device; and

the computing device, to receive the inertial data and the positional data, to convert the positional data to a three-dimensional coordinate system of the positional sensor, to compute an acceleration of the tracked object in the three-dimensional coordinate system of the positional sensor, and to reconcile the inertial data with the computed acceleration.

25. The tracking system of claim 24, wherein the positional sensor is a video camera, wherein the positional data includes a plurality of image locations and image sizes, and wherein converting the positional data includes converting each of the plurality of image locations to the three-dimensional coordinate system of the positional sensor based on the corresponding image sizes and a field of view of the video camera.

26. The tracking system of claim 25, wherein the rich motion path includes at least one of a motion towards the positional sensor or a motion away from the positional sensor.

27. The tracking system of claim 25, wherein the rich motion causes accelerations that exceed a minimum acceleration threshold.

28. The tracking system of claim 25, further comprising:

the computing device to convert the received inertial data into a three-dimensional world coordinate system, the three-dimensional world coordinate system having an axis that is aligned with gravity.

29. The tracking system of claim 25, wherein the inertial data includes a first set of vectors and the computed acceleration includes a second set of vectors, and wherein reconciling the inertial data with the computed acceleration comprises:

determining a first angle between the first set of vectors and the second set of vectors that is representative of a pitch of the positional sensor; and

determining a second angle between the first set of vectors and the second set of vectors that is representative of a relative yaw between the positional sensor and the tracked object.

30. The tracking system of claim 29, wherein reconciling the inertial data with the computed acceleration further comprises:

selecting a plurality of potential first angles and a plurality of potential second angles;

rotating the first set of vectors by each combination of one of the plurality of potential first angles and one of the plurality of potential second angles;

finding a difference between the rotated first set of vectors and the second set of vectors for each combination of one of the plurality of potential first angles and one of the plurality of potential second angles; and

identifying a combination of a first potential angle and a second potential angle that causes the smallest difference between the rotated first set of vectors and the second set of vectors as the first angle and the second angle.

31. The tracking system of claim 29, wherein reconciling the inertial data with the computed acceleration further comprises:

determining a three-dimensional angle between the first set of vectors and the second set of vectors; and

decomposing the three-dimensional angle into the first angle and the second angle.

32. The tracking system of claim 29, further comprising:

the computing device to filter out each vector in the first set of vectors and in the second set of vectors that has a magnitude that is smaller than a minimum threshold.

33. The tracking system of claim 29, further comprising:

the computing device to weight each vector in the first set of vectors and in the second set of vectors based on a magnitude of the vector, wherein vectors having a higher magnitude are weighted more heavily than vectors having a lower magnitude.

34. The tracking system of claim 29, further comprising:

the computing device to compare the magnitude of each vector of the first set of vectors to close in time vectors from the second set of vectors, and if a first vector of the first set of vectors is approximately equal in magnitude to a close in time second vector of the second set of vectors, to determine that the first vector represents a motion in a three dimensional world coordinate system and the second vector represents the motion in the three dimensional coordinate system of the positional sensor.

35. The tracking system of claim 25, wherein the positional sensor is one of a camera, a z-camera, an ultrasonic sensor or a photonic sensor.

36. The tracking system of claim 25, wherein the one or more inertial sensors have a fixed displacement from a centroid of the tracked object, the tracking system further comprising:

the computing device to apply an offset to one of the inertial data or the computed acceleration based on measurements provided by a gyroscope and a known distance between the accelerometer and the centroid, wherein the offset compensates for the fixed displacement.

37. The tracking system of claim 25, wherein the field of view of the positional sensor is unknown, further comprising:

the computing device to solve for the field of view of the positional sensor.

38. The tracking system of claim 37, wherein solving for the field of view comprises:

assuming one or more potential fields of view;

for each potential field of view, attempting to reconcile the inertial data with the computed acceleration; and

identifying the potential field of view for which the inertial data is best able to reconcile with the computed acceleration as the field of view of the positional sensor.

39. A tracking system, comprising:

a sphere that includes one or more inertial sensors, the sphere having a known radius;

a camera to gather image data of the sphere, the image data including an elliptical projection of the sphere onto an image plane of the camera, and to transmit the image data to a computing device; and

the computing device, to receive the image data, to determine a position of the sphere in the image plane of the camera from the image data, to compute a major axis and a minor axis of the elliptical projection, to compute the camera'"'"'s field of view based on the position of the sphere and a difference between the major axis and the minor axis, and to use the field of view to track the sphere in three-dimensional space.

40. The tracking system of claim 39, wherein the camera has a plurality of known field of view settings, further comprising:

the computing device to compare the computed field of view to each of the plurality of known fixed field of view settings, and to determine that a current field of view of the camera is equal to a fixed field of view setting closest to the computed field of view.