System and method for determining orientation using tracking cameras and inertial measurements

• US 10,071,306 B2
• Filed: 03/25/2016
• Issued: 09/11/2018
• Est. Priority Date: 03/25/2016
• Status: Active Grant

• Current Assignee: Zero Latency Pty., Ltd.
• Original Assignee: Zero Latency Pty., Ltd.
• Inventors: Vandonkelaar, Scott
• Primary Examiner(s): Elisca, Pierre E
• Application Number:
  US15/081,669
• Publication Number:
  US 20170274277A1
• Time in Patent Office: 900 Days
• Field of Search: 463 36
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/32 using local area network [LAN] connections
• CPC Class Codes: A63F 13/33 using wide area network [WAN] connections
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A virtual reality tracking system accurately determines one or more controller orientations using data from tracking cameras and/or an inertial measurement unit (IMU) embedded in each controller. Each controller has two or more distinctive light-emitting tracking markers. The tracking system determines the locations of the tracking markers based on the location of tracking markers in tracking camera'"'"'s images. The tracking system determines the controller orientation using the locations of the tracking markers and orientation data from the IMU. When the camera views of the markers are obstructed the tracking system relies solely on the less-accurate orientation data from the IMU.

Claims

1. A system for determining and sending controller orientation and position in a virtual reality session, the system comprising:

at least two cameras;

at least one controller comprising an inertial measurement unit configured to be used by at least one user;

at least two tracking markers attached to each of the at least one controller;

at least one server configured to track movements of the at least one controller during a virtual reality session;

a memory containing non-transitory machine readable medium comprising machine executable code having stored thereon instructions for tracking the movements of the at least one controller;

a control system comprising at least one processor coupled to the memory, the control system configured to execute the machine executable code to cause the control system to;

receive location information of a first tracking marker and a second tracking marker from among the at least two tracking markers using the at least two cameras;

determine marker locations based on the received location information;

receive orientation information from the inertial measurement unit comprised in the at least one controller; and

determine the controller orientation and position based on the determined marker information and the received orientation information.

2. The system of claim 1, wherein the control system is further configured to execute the machine executable code to cause the control system to send, using the at least one server, the determined controller orientation and position to a display device being used by the at least one user.

3. The system of claim 1, wherein for each of the at least two cameras, the determining the marker locations comprises:

searching the first tracking marker in a camera image taken by each camera;

determining a first three-dimensional (3D) vector between each camera and the first tracking marker;

adding the first 3D vectors to a vector list;

searching the second tracking marker in the camera image taken by each camera;

determining a second three-dimensional (3D) vector between each camera and the second tracking marker;

adding the second 3D vector to a second vector list;

determining, for each pair of vectors stored in each of the first and the second vector list, locations based on vector intersection and adding the determined location to a first location list and a second location list respectively; and

determining the marker locations by averaging locations in their respective location lists.

4. The system of claim 1, wherein if the location information of the first tracking marker and the second tracking marker is not received, the control system is further configured to execute the machine executable code to cause the control system to determine controller orientation and position based on the received orientation information.

5. The system of claim 1, wherein the orientation information includes at least one of pitch, roll and yaw.

6. The system of claim 1, wherein the received location information comprises direction information.

7. The system of claim 1, wherein the at least two tracking markers are programmable-color light emitting diodes (LEDs).

8. The system of claim 7, wherein an initial color choice for each of the at least two tracking markers is one of White (R,G,B), Red (R), Blue (B), Green (G), Yellow (R,G), Cyan (B,G) and Magenta (R,B).

9. The system of claim 8, wherein the color choice for each of the at least two tracking markers, after being found, further comprises half intensity secondary and tertiary colors.

10. A method for determining and sending controller orientation and position in a virtual reality session in a system that comprises;

at least two cameras, at least one controller comprising an inertial measurement unit configured to be used by at least one user, at least two tracking markers attached to each of the at least one controller, at least one processor, and at least one server, the method comprising;

receiving, using at least one of said at least one processor, location information of a first tracking marker and a second tracking marker from among the at least two tracking markers using the at least two cameras;

determining, using at least one of said at least one processor, a marker location based on the received location information;

receiving, using at least one of said at least one processor, orientation information from the inertial measurement unit comprised in the at least one controller; and

determining, using at least one of said at least one processor, the controller orientation and position based on the determined marker information and the received orientation information.

11. The method of claim 10, wherein the method further comprises sending, using the at least one server, the determined controller orientation and position to a display device being used by the at least one user.

12. The method of claim 10, wherein for each of the at least two cameras, the determining the marker locations further comprises:

searching the first tracking marker in a camera image taken by each camera;

determining, using at least one of said at least one processor, a first three-dimensional (3D) vector between each camera and the first tracking marker;

adding, using at least one of said at least one processor, the first 3D vectors to a vector list;

searching the second tracking marker in the camera image taken by each camera;

determining, using at least one of said at least one processor, a second three-dimensional (3D) vector between each camera and the second tracking marker;

adding the second 3D vector to a second vector list;

determining, for each pair of vectors stored in each of the first and the second vector list, using at least one of said at least one processor, location based on vector intersection and adding the determined location to a first location list and a second location list respectively; and

determining, using at least one of said at least one processor, the marker locations by averaging locations in their respective location lists.

13. The method of claim 10, wherein if the location information of the first tracking marker and the second tracking marker is not received, the determining the controller orientation and position further comprises determining the controller orientation and position based on the received orientation information.

14. The method of claim 10, wherein the orientation information includes at least one of pitch, roll and yaw.

15. The method of claim 10, wherein the received location information comprises direction information.

16. The method of claim 10, wherein the at least two tracking markers are programmable-color light emitting diodes (LEDs).

17. The method of claim 16, wherein an initial color choice for each of the at least two tracking markers is one of White (R,G,B), Red (R), Blue (B), Green (G), Yellow (R,G), Cyan (B,G) and Magenta (R,B).

18. The method of claim 17, wherein the color choice for each of the at least two tracking markers further, after being found, comprises half intensity secondary and tertiary colors.