Efficient orientation estimation system using magnetic, angular rate, and gravity sensors

• US 9,588,598 B2
• Filed: 09/24/2015
• Issued: 03/07/2017
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Li, Adam
• Primary Examiner(s): OSORIO, RICARDO
• Application Number:
  US14/864,327
• Publication Number:
  US 20170003764A1
• Time in Patent Office: 530 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system that efficiently estimates an object'"'"'s orientation using magnetic, angular rate, and gravity sensors. The object may be for example a virtual reality headset or a user in a virtual reality environment. Magnetic and gravity data are used to correct errors that accumulate from integrating angular velocity. Unlike systems that use Kalman filter approaches, embodiments of the system apply a simple, highly efficient technique to generate magnetic and gravity error vectors; these error vectors are added directly to the angular velocity prior to integration. Error calculations are performed in the sensor reference frame rather than in the Earth reference frame. Magnetic error correction uses only the horizontal component of the magnetic field, which is efficiently calculated by subtracting off the projection of the magnetic field onto the measured gravity vector. Sensors and processors for calculating orientation may be integrated into a low-latency virtual reality display system.

Claims

1. An efficient orientation estimation system using magnetic, angular rate, and gravity sensors, comprising

a three axis magnetometer configured to be attached to an object, wherein said three axis magnetometer generates magnetic field data represented in a sensor reference frame;

a three axis rate gyroscope configured to be attached to said object, wherein said three axis rate gyroscope generates angular velocity data represented in said sensor reference frame;

a three axis accelerometer configured to be attached to said object, wherein said three axis accelerometer generates acceleration data represented in said sensor reference frame, wherein said acceleration data includes a gravity measurement;

at least one processor comprising a tangible memory medium, coupled to said three axis accelerometer, to said three axis rate gyroscope, and to said three axis magnetometer, and configured to

define an Earth reference frame for an area in which said object is located, comprising

a unit vector ĵ

pointing vertically upwards and opposite to a direction of gravity;

a unit vector {circumflex over (k)} in a horizontal plane perpendicular to said unit vector ĵ

, and pointing in a direction opposite to a horizontal component of Earth'"'"'s magnetic field in said area;

a unit vector î

perpendicular to said unit vector ĵ and

perpendicular to said unit vector {circumflex over (k)};

wherein said unit vector î

, said unit vector ĵ

, and said unit vector {circumflex over (k)} form a right-handed reference frame;

define a sensor orientation as a rotation that rotates a vector in said sensor reference frame into a corresponding vector in said Earth reference frame;

set said sensor orientation to an initial sensor orientation;

iteratively calculate an updated value of said sensor orientation as

obtain said angular velocity data from said three axis rate gyroscope;

obtain said acceleration data from said three axis accelerometer;

normalize said acceleration data to form a normalized acceleration having its vector length equal to one;

obtain said magnetic field data from said three axis magnetometer;

generate an Earth to sensor rotation as an inverse of said sensor orientation;

rotate said unit vector ĵ

by said Earth to sensor rotation, to form a gravity prediction in said sensor reference frame;

apply said Earth to sensor rotation to a negative of said unit vector {circumflex over (k)}, to form a horizontal magnetic field prediction in said sensor reference frame;

project said magnetic field data onto said normalized acceleration, to form a vertical magnetic component;

subtract said vertical magnetic component from said magnetic field data, to form a horizontal magnetic component;

normalize said horizontal magnetic component to form a normalized horizontal magnetic component;

calculate a vector cross product of said normalized acceleration with said gravity prediction, to form a gravity error

calculate a vector cross product of said normalized horizontal magnetic component with said horizontal magnetic field prediction, to form a magnetic field error;

calculate an orientation derivative based on

said angular velocity,

said gravity error, and

said magnetic field error;

integrate said orientation derivative to form said updated value of said sensor orientation.

2. The system of claim 1, wherein

said sensor orientation is a sensor orientation quaternion having unit length;

said Earth to sensor rotation is a multiplicative inverse of said sensor orientation quaternion.

3. The system of claim 2, wherein

said calculate an orientation derivative comprises

calculate said orientation derivative as one half of said sensor orientation quaternion multiplied by a sum of

said angular velocity,

said gravity error multiplied by a gravity error proportional gain, and,

said magnetic field error multiplied by a magnetic field proportional gain.

4. The system of claim 2, wherein said calculate an orientation derivative comprises calculate said orientation derivative as one half of said sensor orientation quaternion multiplied

by a sum of

said angular velocity,

said gravity error multiplied by a gravity error proportional gain,

said magnetic field error multiplied by a magnetic field proportional gain,

a cumulative sum of said gravity error multiplied by a gravity error integral gain, and,

a cumulative sum of said magnetic field error multiplied by a magnetic field error integral gain.

5. The system of claim 2, wherein said calculate an orientation derivative comprises calculate said orientation derivative as one half of said sensor orientation quaternion multiplied

by a sum of

said angular velocity,

said gravity error multiplied by a gravity error proportional gain,

said magnetic field error multiplied by a magnetic field proportional gain,

a cumulative sum of said gravity error multiplied by a gravity error integral gain,

a cumulative sum of said magnetic field error multiplied by a magnetic field error integral gain,

a derivative of said gravity error multiplied by a gravity error derivative gain, and,

a derivative of said magnetic field error multiplied by a magnetic field error derivative gain.

6. The system of claim 1, further comprising

at least one display viewable by a user, wherein said object is one or more body parts of said user;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said at least one processor, and said 3D model, wherein said scene renderer

receives said sensor orientation from said at least one processor;

calculates one or more rendering virtual camera poses, based on said sensor orientation;

calculates one or more 2D projections of said 3D model, based on said one or more rendering virtual camera poses;

transmits said one or more 2D projections to said at least one display.

7. The system of claim 6, further comprising

an image warper coupled to said at least one display, said scene renderer, and said at least one processor, wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said sensor orientation from said at least one processor;

calculates a change in pose between said one or more virtual camera poses and said sensor orientation;

generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose; and

modifies one or more pixels of said at least one display based on said rerendering approximation.

8. The system of claim 7, wherein said rerendering approximation comprises

calculating a pixel translation vector;

and,

translating one or more pixels of said one or more 2D projections by said pixel translation vector.

9. The system of claim 7, wherein said image warper modifies one or more pixels of said at least one display using hardware acceleration.

10. The system of claim 7, wherein said image warper

compares said change in pose to a threshold value;

and,

generates said rerendering approximation if said change in pose is below said threshold value.

11. The system of claim 8, wherein calculating said pixel translation vector comprises approximating said change in pose as a rotation around a unit vector {circumflex over (ω

)} comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_x by an angle Δ

θ

;

calculating a spatial translation vector ({circumflex over (ω

)}_yΔ

θ

, −

{circumflex over (ω

)}_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections;

calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.

12. An efficient orientation estimation system using magnetic, angular rate, and gravity sensors, comprising

a three axis magnetometer configured to be attached to one or more body parts of a user, wherein said three axis magnetometer generates magnetic field data represented in a sensor reference frame;

a three axis rate gyroscope configured to be attached to said one or more body parts of said user, wherein said three axis rate gyroscope generates angular velocity data represented in said sensor reference frame;

a three axis accelerometer configured to be attached to said one or more body parts of said user, wherein said three axis accelerometer generates acceleration data represented in said sensor reference frame, wherein said acceleration data includes a gravity measurement;

at least one processor comprising a tangible memory medium, coupled to said three axis accelerometer, to said three axis rate gyroscope, and to said three axis magnetometer, and configured to

define an Earth reference frame for an area wherein said user is located, comprising

a unit vector ĵ

pointing vertically upwards and opposite to a direction of gravity;

a unit vector {circumflex over (k)} in a horizontal plane perpendicular to said unit vector ĵ

, and pointing in a direction opposite to a horizontal component of Earth'"'"'s magnetic field in said area;

a unit vector î

perpendicular to said unit vector ĵ and

perpendicular to said unit vector {circumflex over (k)};

wherein said unit vector î

, said unit vector ĵ

, and said unit vector {circumflex over (k)} form a right-handed reference frame;

define a sensor orientation quaternion having unit length, which rotates a vector in said sensor reference frame into a corresponding vector in said Earth reference frame;

set said sensor orientation quaternion to an initial sensor orientation quaternion;

iteratively calculate an updated value of said sensor orientation quaternion as

obtain said angular velocity data from said three axis rate gyroscope;

obtain said acceleration data from said three axis accelerometer;

normalize said acceleration data to form a normalized acceleration having its vector length equal to one;

obtain said magnetic field data from said three axis magnetometer;

generate an Earth to sensor rotation as a multiplicative inverse of said sensor orientation quaternion;

rotate said unit vector ĵ

by said Earth to sensor rotation, to form a gravity prediction in said sensor reference frame;

apply said Earth to sensor rotation to a negative of said unit vector {circumflex over (k)}, to form a horizontal magnetic field prediction in said sensor frame;

project said magnetic field data onto said normalized acceleration, to form a vertical magnetic component;

subtract said vertical magnetic component from said magnetic field data, to form a horizontal magnetic component;

normalize said horizontal magnetic component to form a normalized horizontal magnetic component;

calculate a vector cross product of said normalized acceleration with said gravity prediction, to form a gravity error

calculate a vector cross product of said normalized horizontal magnetic component with said horizontal magnetic field prediction, to form a magnetic field error;

calculate an orientation derivative as one half of said sensor orientation quaternion multiplied by a sum of

said angular velocity,

said gravity error multiplied by a gravity error proportional gain,

said magnetic field error multiplied by a magnetic field proportional gain,

a cumulative sum of said gravity error multiplied by a gravity error integral gain,

a cumulative sum of said magnetic field error multiplied by a magnetic field error integral gain,

a derivative of said gravity error multiplied by a gravity error derivative gain, and,

a derivative of said magnetic field error multiplied by a magnetic field error derivative gain;

integrate said orientation derivative to form said updated value of said sensor orientation quaternion;

at least one display viewable by said user;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said at least one processor, and said 3D model, wherein said scene renderer

receives said sensor orientation quaternion from said at least one processor;

calculates one or more rendering virtual camera poses, based on said sensor orientation quaternion;

calculates one or more 2D projections of said 3D model, based on said one or more rendering virtual camera poses;

transmits said one or more 2D projections to said at least one display;

an image warper coupled to said at least one display, said scene renderer, and said at least one processor, wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said sensor orientation quaternion from said at least one processor;

calculates a change in pose between said one or more virtual camera poses and said sensor orientation;

generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose;

modifies one or more pixels of said at least one display based on said rerendering approximation;

wherein said rerendering approximation comprises

approximating said change in pose as a rotation around a unit vector {circumflex over (ω

)} comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_x, by an angle Δ

θ

;

calculating a spatial translation vector ({circumflex over (ω

)}_yΔ

θ

, −

{circumflex over (ω

)}_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections;

calculating a pixel translation vector by scaling said spatial translation vector by said scaling factor;

translating one or more pixels of said one or more 2D projections by said pixel translation vector.