Method and system for optical-inertial tracking of a moving object

• US 10,481,679 B2
• Filed: 12/18/2017
• Issued: 11/19/2019
• Est. Priority Date: 12/18/2017
• Status: Active Grant

• Current Assignee: ALT LLC
• Original Assignee: ALT LLC
• Inventors: Sevostianov, Petr Vyacheslavovich
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US15/844,967
• Publication Number:
  US 20190187783A1
• Time in Patent Office: 701 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

Method for inside-out optical-inertial tracking of a movable object includes providing IR light by a plurality of markers; reading a picture from an optical sensor and detecting pixels satisfying predefined conditions by a first data processing device; detecting marker blobs, based on the pixels, and determining parameters of the blobs by the first data processing device; reading data from an inertial sensor; generating a consolidated data flow comprising the parameters of the marker blobs and the data of the inertial sensor by the first data processing device, and transmitting the data flow to a second communication device; determining a tracker positioning data in a coordinate system of a motion area of the object, based on the consolidated data flow, by a second data processing device; validating a model of a tracker motion path, based on tracker positioning data; extrapolating tracker positioning data onto an extrapolation horizon, based on the model.

Claims

1. A method of optical-inertial tracking a movable object, comprising the steps of:

(a) providing emission of IR light by a plurality of markers fixedly mounted so as to be visible from the movable object in order to perform inside-out tracking;

(b) reading a picture, by a first data processing device, from an optical sensor that is mounted on the movable object and detecting pixels satisfying predefined conditions in the read picture;

(c) detecting marker blobs, based on the detected pixels, and determining parameters of the marker blobs by the first data processing device;

(d) reading data, by the first data processing device, from at least one inertial sensor that is mounted on the movable object;

(e) generating, by the first data processing device, a consolidated data flow comprising the parameters of the marker blobs and the data from the at least one inertial sensor, and transmitting the consolidated data flow by a first communication device to a second communication device;

(f) receiving the consolidated data flow by the second communication device and determining, by a second data processing device, a tracker positioning data in a coordinate system of a motion area of the movable object, based on the consolidated data flow;

(g) validating, by the second data processing device, a mathematical model of a tracker motion path, based on the tracker positioning data; and

(h) extrapolating, by the second data processing device, the tracker positioning data onto an extrapolation horizon, based on the validated mathematical model of the tracker motion path.

2. The method of claim 1, wherein the picture from the optical sensor in step (b) is read form a portion of the optical sensor, wherein a size of the portion is smaller than an entire frame size of the optical sensor.

3. The method of claim 2, wherein the portion to be read is a tracing window.

4. The method of claim 1, wherein the predefined conditions in step (b) include threshold values of brightness of the pixels.

5. The method of claim 4, wherein the predefined conditions include a requirement related to aggregation of the pixels.

6. The method of claim 1, wherein detecting the pixels satisfying the predefined conditions in step (b) is started before an end of reading of the picture from the optical sensor.

7. The method of claim 1, wherein step (b) is performed by an FPGA.

8. The method of claim 1, wherein the parameters of each of the marker blobs in step (c) include coordinates of its center in a coordinate system of a frame of the optical sensor, an effective radius and a weighted brightness.

9. The method of claim 8, wherein the coordinates of the center are determined with a subpixel accuracy.

10. The method of claim 8, wherein the parameters of the marker blob include a timestamp indicating time when a frame was captured.

11. The method of claim 1, wherein step (c) is performed by a microcontroller.

12. The method of claim 1, wherein the inertial sensor in step (d) is a gyroscope.

13. The method of claim 1, wherein the inertial sensor in step (d) is an accelerometer.

14. The method of claim 1, wherein the data read from the inertial sensor in step (d) includes a timestamp indicating a time when the data was read.

15. The method of claim 1, wherein the consolidated data flow generated in step (e) includes data of multiple trackers.

16. The method of claim 15, wherein the consolidated data flow in step (f) is divided into multiple consolidated data flows, whose number corresponds to a number of the trackers.

17. The method of claim 1, wherein the first data processing device and the first communication device are located in the tracker, and the second data processing device and the second communication device are located in a host.

18. The method of claim 17, wherein the tracker is combined with the host.

19. The method of claim 1, wherein the markers are identified based on the parameters of the marker blobs, so as to determine the tracker positioning data in step (f).

20. The method of claim 19, wherein the identification of the markers is performed using a predefined optical pattern.

21. The method of claim 20, wherein the optical pattern is a linear optical pattern.

22. The method of claim 21, wherein the linear optical pattern includes blobs corresponding to a group of three markers aligned in a line, wherein a distance between a first marker and a second marker is twice as much as a distance between the second marker and a third marker.

23. The method of claim 1, wherein a set of historic tracking data is generated, based on the tracker positioning data determined in step (f) for different time points.

24. The method of claim 23, wherein the validation of the mathematical model of the tracker motion path in step (g) is performed using the set of historic tracking data.

25. The method of claim 24, wherein the set of historic tracking data includes a set of data S comprising a multi-dimensional vector that characterizes position and orientation of the tracker, and a set of data L comprising data of one or more inertial sensors.

26. The method of claim 25, wherein the set of historic tracking data includes a set of data OF comprising information on rays in a local coordinate system of the tracker, the rays directed to the markers, whose blobs were detected in step (c).

27. The method of claim 26, wherein the validation of the mathematical model of the tracker motion path in step (g) includes calculating the data S in a backward direction through an entire depth of the historic tracking data, optimizing the data S in an earliest time point of the history, and then calculating an optimized data S in a forward direction through the entire depth of the historic tracking data.

28. The method of claim 27, wherein a vector E of the tracker positioning error is calculated by comparison of the marker coordinates and based on the data OF and predetermined actual marker coordinates, in a backward direction through entire depth of the historic tracking data, and the optimization of the data S in the earliest time point of the history is performed, using the vector E of the tracker positioning error.

29. The method of claim 28, wherein the marker coordinates that are calculated based on the data OF are polar coordinates.

30. The method of claim 28, wherein the optimization of the data S in the earliest point of the history is performed using a least squares method.

31. The method of claim 30, wherein the optimization of the data S in the earliest point of the history is performed, using a Levenberg-Marquardt algorithm.

32. The method of claim 28, wherein the vector E of the tracker positioning error is calculated in a forward direction through entire depth of the historic tracking data, and the validation of the mathematical model of the tracker motion path is performed by comparing the vectors E of the tracker positioning error calculated in the forward direction and the vectors E of the tracker positioning error calculated in the backward direction, and by further calculating a target function used for judging relevance of the mathematical model of the tracker motion path.

33. The method of claim 32, wherein the target function is a weighting function.

34. The method of claim 32, wherein the validation of the mathematical model of the tracker motion path results in the optimized data S, which ensures minimal error of the tracker positioning.

35. The method of claim 1, wherein the determination of the tracker positioning data in step (f) is performed each time new data of the at least one inertial sensor is received in the consolidated data flow.

36. The method of claim 1, wherein the validation of the mathematical model of the tracker motion path in step (g) is performed each time new parameters of the marker blobs are received in the consolidated data flow.

37. The method of claim 1, wherein the extrapolation of the tracker positioning data in step (h) is performed by defining an extrapolation horizon and an extrapolation base, and by further extrapolating the tracker motion path onto the extrapolation horizon.

38. The method of claim 37, wherein the extrapolation horizon is defined based on data received from a 3D-engine that renders a 3D-scene.

39. The method of claim 1, further comprising rendering a 3D-scene based on the extrapolated tracker positioning data, and outputting the 3D-scene into a head-mounted display of a user in a VR/AR system.

40. A system for optical-inertial tracking a movable object, the system comprising:

at least one plurality of markers fixedly mounted so as to be visible from the movable object in order to perform inside-out tracking, including IR light emitting markers;

at least one tracker mounted on the movable object, and including an optical sensor, at least one inertial sensor, a first data processing device and a first communication device,

wherein the tracker is configured to

read, by the first data processing device, a picture from the optical sensor,

detect, by the first data processing device, pixels satisfying predefined conditions in the read picture, detect, by the first data processing device, marker blobs based on the detected pixels, to determine, by the first data processing device, parameters of the marker blobs,

read, by the first data processing device, data from the at least one inertial sensor,

generate, by the first data processing device, a consolidated data flow including the parameters of the marker blobs and the data of the at least one inertial sensor, and

transmit, by the first communication device, the consolidated data flow to a host;

at least one host comprising a second communication device and a second data processing device,

wherein the host is configured to

receive, by the second communication device, the consolidated data flow,

determine, by the second data processing device, a tracker positioning data in a coordinate system of a motion area of the movable object, based on the consolidated data flow,

validate, by the second data processing device, a mathematical model of a tracker motion path, based on the tracker positioning data, and

extrapolate, by the second data processing device, the tracker positioning data onto an extrapolation horizon, based on the validated mathematical model of the tracker motion path.

41. The system of claim 40, wherein the first data processing device is configured to read the picture from a portion of the optical sensor, wherein the portion is smaller than an entire frame size of the optical sensor.

42. The system of claim 41, wherein the portion is a tracing window.

43. The system of claim 40, wherein the predefined conditions include a threshold value of brightness of the pixel.

44. The system of claim 43, wherein the predefined conditions include a requirement related to aggregation of the pixels.

45. The system of claim 40, wherein the first data processing device is configured to start detecting pixels satisfying the predefined conditions before reading the picture from the optical sensor is finished.

46. The system of claim 40, wherein the first data processing device includes an FPGA configured to read the picture from the optical sensor and detect pixels satisfying the predefined conditions in the read picture.

47. The system of claim 40, wherein the parameters of each of the marker blobs include coordinates of its center in a coordinate system of a frame of the optical sensor, an effective radius and a weighted brightness.

48. The system of claim 47, wherein the first data processing device is configured to determine the coordinates of the center with a subpixel accuracy.

49. The system of claim 47, wherein the parameters of the marker blob include a timestamp indicating a time when an appropriate frame was captured.

50. The system of claim 40, wherein the first data processing device includes a microcontroller configured to detect the marker blobs, based on the detected pixels, and to determine parameters of the marker blobs.

51. The system of claim 40, wherein the inertial sensor is a gyroscope.

52. The system of claim 40, wherein the inertial sensor is an accelerometer.

53. The system of claim 40, wherein the data read from the inertial sensor includes a timestamp indicating a time when the data was read.

54. The system of claim 40, wherein the consolidated data flow includes data of multiple trackers.

55. The system of claim 54, wherein the second data processing device is configured to divide the consolidated data flow into multiple consolidated data flows whose number corresponds to a number of the trackers.

56. The system of claim 40, wherein the tracker is combined with the host.

57. The system of claim 40, wherein the second data processing device is configured to identify the markers, based on the parameters of the marker blobs, so as to determine the tracker positioning data.

58. The system of claim 57, wherein the second data processing device is configured to identify the markers using a predefined optical pattern.

59. The system of claim 58, wherein the predefined optical pattern is a linear optical pattern.

60. The system of claim 59, wherein the linear optical pattern includes blobs corresponding to a group of three markers aligned in a line, wherein a distance between a first marker and a second marker is twice a distance between the second marker and a third marker.

61. The system of claim 40, wherein the second data processing device is configured to generate a set of historic tracking data based on the tracker positioning data determined for different time points.

62. The system of claim 61, wherein the second data processing device is configured to validate the mathematical model of the tracker motion path using the set of historic tracking data.

63. The system of claim 62, wherein the set of historic tracking data includes a set of data S comprising a multi-dimensional vector characterizing position and orientation of the tracker, and a set of data L including data of one or more inertial sensors, wherein the data S and the data L correspond to parameters of the mathematical model of the tracker motion path.

64. The system of claim 63, wherein the set of historical tracking data includes a set of data OF comprising information on rays in a local coordinate system of the tracker, the rays directed to the markers, which blobs were detected.

65. The system of claim 64, wherein the second data processing device is configured to calculate the data S in a backward direction through entire depth of the historic tracking data, optimize the data S in an earliest time point of the history, and then calculate an optimized data S in a forward direction through the entire depth of the historic tracking data during the validation of the mathematical model of the tracker motion path.

66. The system of claim 64, wherein the second data processing device is configured to calculate a vector E of the tracker positioning error, which is determined by comparison of the marker coordinates calculated, based on the data OF, and predetermined actual marker coordinates, in a backward direction through an entire depth of the historic tracking data, and to optimize the data S in an earliest time point of the history, using the vector E of the tracker positioning error.

67. The system of claim 66, wherein the marker coordinates calculated based on the data OF are polar coordinates.

68. The system of claim 66, wherein the optimization of the data S in the earliest time point of the history is performed, using a least squares method.

69. The system of claim 68, wherein the optimization of the data S in the earliest time point of the history is performed, using a Levenberg-Marquardt algorithm.

70. The system of claim 66, wherein the second data processing device is configured to calculate the vector E of the tracker positioning error in a forward direction through entire depth of the historical tracking data, and validate the mathematical model of the tracker motion path by comparing the vectors E of the tracker positioning error calculated in the forward direction and the vectors E of the tracker positioning error calculated in the backward direction, and by further calculating a target function for judging relevance of the mathematical model of the tracker motion path.

71. The system of claim 70, wherein the target function is a weighting function.

72. The system of claim 70, wherein the validation of the mathematical model of the tracker motion path results in the optimized data S, which ensures minimal error of the tracker positioning.

73. The system of claim 40, wherein the second data processing device is configured to determine the tracker positioning data each time, when new data of the at least one inertial sensor is received in the consolidated data flow.

74. The system of claim 40, wherein the second data processing device is configured to validate the mathematical model of the tracker motion path each time new parameters of the marker blobs are received in the consolidated data flow.

75. The system of claim 40, wherein the second data processing device is configured to define an extrapolation horizon and an extrapolation base, and to extrapolate the tracker motion path onto the extrapolation horizon.

76. The system of claim 75, wherein the extrapolation horizon may be defined, based on data received from a 3D-engine configured to render a 3D-scene.

77. The system of claim 40, further comprising at least one head-mounted display configured to present a 3D-scene to a VR/AR system user, wherein the extrapolation of the tracker positioning data is performed so as to further render the 3D-scene, based on the extrapolated tracker positioning data, and to output the 3D-scene into the head-mounted display of the VR/AR system user.