Multi-user virtual reality processing

• US 9,990,689 B2
• Filed: 12/16/2015
• Issued: 06/05/2018
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Worldviz LLC
• Original Assignee: Worldviz LLC
• Inventors: Lashkari, Farshid, Beall, Andrew C.
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): TRAN, KIM THANH THI
• Application Number:
  US14/971,720
• Publication Number:
  US 20170178272A1
• Time in Patent Office: 902 Days
• Field of Search: 345633, 345504
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A multi-user virtual reality and/or augmented reality cluster system enabling the projection and tracking of users in a simulated environment is described. The disclosed cluster system enables both timely user action display in the first person and a timely view of a user'"'"'s action by other participants of the simulation. The cluster system integrates motion capture cameras, distributed computing, and tracking devices to provide a simulation which seems lifelike to all participants and observers.

Claims

1. A cluster system comprising:

a master computing device;

a first computing device configured as a first client computing device, the first client computing device associated with a first user;

a second computing device configured as a second client computing device, the second client computing device associated with a second user;

a plurality of motion tracking cameras, wherein the plurality of motion tracking cameras are configured to reside in a physical space coincident with the first user, the plurality of motion tracking cameras configured to detect infrared light and to communicate position data derived from detected infrared light to the first computing device and to the master computing device;

a first marker, attachable to a head mounted display associated with the first user, the first marker configured to emit a first repetitive pattern of infrared light pulses, the first marker associated with one or more privileged operations that are to be performed locally by the first computing device;

a second marker, attachable to the head mounted display associated with the first user, the second marker configured to emit a second repetitive pattern of infrared light pulses distinct from the first repetitive pattern of infrared light pulses, the second marker associated with one or more privileged operations that are to be performed locally by the first computing device;

a third marker, wearable or holdable by the first user, the third marker configured to emit a third repetitive pattern of infrared light pulses distinct from the first repetitive pattern of infrared light pulses and distinct from the second repetitive pattern of infrared light pulses, wherein the third marker is associated with one or more non-privileged operations that are to be performed remotely by the master computing device;

non-transitory media that stores instructions readable by the cluster system, that when executed by the cluster system, cause the cluster system to;

access configuration information comprising information indicating what types of operations are privileged and what types of operations are non-privileged;

receive, at the first computing device from one or more of the plurality of motion tracking cameras, position data corresponding to the first marker and position data corresponding to the second marker;

at least partly in response to the receipt by the first computing device of the position data corresponding to the first marker and the position data corresponding to the second marker and on the accessed configuration information, determine that a privileged operation is to be performed by the first computing device and not by the master computing device, the privileged operation enabling an orientation of a first scene of a simulation to be rendered in the head mounted display associated with the first user in synchronization with movement of the head mounted display in at least five degrees of freedom;

transmit, from the first computing device to the master computing device, a position of the first user in the physical space, and

provide a first instruction by the master computing device enabling an image corresponding to the first user to be rendered at a first virtual position in a display device associated with the second user, wherein the first virtual position is based at least in part on the position of the first user in the physical space and on a viewpoint of the second user;

receive, at the master computing device from one or more of the plurality of motion tracking cameras, position data corresponding to the third marker; and

at least partly in response to receipt of the position data corresponding to the third marker and on the accessed configuration information, determine that a non-privileged operation is to be performed by the master computing device and not by the first computing device, wherein the non-privileged operation comprises enabling;

synchronized movement of a virtual representation of the third marker from a viewpoint of the first user corresponding to received position changes associated with the third marker to be rendered in the head mounted display associated with the first user, and

synchronized movement of a virtual representation of the third marker from a viewpoint of the second user corresponding to received position changes associated with the third marker to be rendered in the head mounted display associated with the second user.

2. The cluster system of claim 1, wherein the plurality of motion tracking cameras are configured to detect infrared light and to communicate position data derived from detected infrared light to the second computing device.

3. The cluster system of claim 1, wherein the third marker is attached to a limb of the first user.

4. The cluster system of claim 1, wherein the third marker is attached to or included as a component of a handheld tracking device.

5. The cluster system of claim 1, wherein the first repetitive pattern of infrared light pulses comprise light intensity state transitions.

6. The cluster system of claim 1, wherein the plurality of motion tracking cameras are configured to derive the identity of a marker from a detected repetitive pattern of infrared light pulses.

7. The cluster system of claim 1, wherein the cluster system is configured to enable, via a user interface of the master computing device, a third user to indicate what types of operations are privileged and what types of operations are non-privileged.

8. The cluster system of claim 1, wherein the virtual marker representation of the third marker comprises a virtual representation of a glove.

9. A cluster system, comprising:

a master computing device;

a plurality of motion tracking cameras, wherein the plurality of motion tracking cameras configured to reside in a physical space coincident with the first user, the plurality of motion tracking cameras configured to detect infrared light and to communicate position data derived from detected infrared light to the master computing device;

a marker, wearable or holdable by the first user, configured to emit a repetitive pattern of infrared light pulses, wherein the marker is associated with one or more non-privileged operations that are to be performed remotely by the master computing device;

non-transitory media storing instructions readable by the cluster system, that when executed by the cluster system, cause the cluster system to;

access configuration information comprising information indicating what types of operations are privileged and what types of operations are non-privileged;

receive at the master computing device, a position of the first user in the physical space, and

provide a first instruction by the master computing device enabling an image corresponding to the first user to be rendered at a first virtual position in a display device associated with a second user, wherein the first virtual position is based at least in part on the position of the first user in the physical space and on a viewpoint of the second user;

receive, at the master computing device, from one or more of the plurality of motion tracking cameras, position data corresponding to the marker; and

at least partly in response to the receipt of the position of the marker and on the accessed configuration information, determine that a non-privileged operation is to be performed by the master computing device and not by another computing device associated with the cluster system, wherein the non-privileged operation comprises enabling;

synchronized movement of a virtual representation of the marker from a viewpoint of the first user corresponding to received position changes associated with the marker to be rendered in the head mounted display associated with the first user, and

synchronized movement of the virtual representation of the marker from a viewpoint of the second user corresponding to received position changes associated with the third marker to be rendered in the head mounted display associated with the second user.

10. The cluster system of claim 9, wherein the marker is configured to be attached to a limb of the first user.

11. The cluster system of claim 9, wherein the marker is attached to or included as a component of a handheld tracking device.

12. The cluster system of claim 9, wherein the first repetitive pattern of infrared light pulses comprise light intensity state transitions.

13. The cluster system of claim 9, wherein the plurality of motion tracking cameras are configured to derive the identity of a marker from a detected repetitive pattern of infrared light pulses.

14. The cluster system of claim 9, wherein the cluster system is configured to enable, via a third user interface of the master computing device, a user to indicate what types of operations are privileged and what types of operations are non-privileged.

15. A cluster system, comprising:

a first computing device configured as a first client computing device, the first client computing device associated with a first user;

a second computing device configured as a second client computing device, the second client computing device associated with a second user;

a plurality of motion tracking cameras, wherein the plurality of motion tracking cameras are configured to reside in a physical space coincident with the first user, the plurality of motion tracking cameras configured to detect infrared light and to communicate position data derived from detected infrared light to the first computing device;

a first marker, attachable to a head mounted display associated with the first user, the first marker configured to emit a first repetitive pattern of infrared light pulses, the first marker associated with one or more privileged operations that are to be performed locally by the first computing device;

a second marker, attachable to the head mounted display associated with the first user, the second marker configured to emit a second repetitive pattern of infrared light pulses distinct from the first repetitive pattern of infrared light pulses, the second marker associated with one or more privileged operations that are to be performed locally by the first computing device;

a third marker, wearable or holdable by the first user, the third marker configured to emit a third repetitive pattern of infrared light pulses distinct from the first repetitive pattern of infrared light pulses and distinct from the second repetitive pattern of infrared light pulses, wherein the third marker is associated with one or more non-privileged operations that are to be performed remotely by a computing device other than the first computing device;

non-transitory media that stores instructions readable by the cluster system, that when executed by the cluster system, cause the cluster system to;

access configuration information comprising information indicating what types of operations are privileged and what types of operations are non-privileged;

receive, at the first computing device from one or more of the plurality of motion tracking cameras, position data corresponding to the first marker and position data corresponding to the second marker;

at least partly in response to the receipt by the first computing device of the position data corresponding to the first marker and the position data corresponding to the second marker and on the accessed configuration information, determine that a privileged operation is to be performed locally by the first computing device and not by a computing device other than the first computing device, the privileged operation enabling an orientation of a first scene of a simulation to be rendered in the head mounted display associated with the first user in synchronization with movement of the head mounted display in at least five degrees of freedom;

transmit, from the first computing device to a master computing device, a position of the first user in the physical space;

receive, at the second computing device, a first instruction enabling;

an image corresponding to the first user to be rendered at a first virtual position in a display device associated with the second user, wherein first virtual position is based at least in part on the position of the first user in the physical space and on a viewpoint of the second user;

receive, at the first computing device, a first non-privileged instruction enabling;

synchronized movement of a virtual representation of the third marker from a viewpoint of the first user corresponding to received position changes associated with the third marker to be rendered in the head mounted display associated with the first user; and

receive, at the second computing device, a second non-privileged instruction enabling;

synchronized movement of a virtual representation of the third marker from a viewpoint of the second user corresponding to received position changes associated with the third marker to be rendered in the head mounted display associated with the second user.

16. The cluster system of claim 15, wherein the plurality of motion tracking cameras are configured to detect infrared light and to communicate position data derived from detected infrared light to the second computing device.

17. The cluster system of claim 15, wherein the plurality of motion tracking cameras are configured to detect infrared light and to communicate position data derived from detected infrared light to a computing device other than the first computing device or the second computing device.

18. The cluster system of claim 15, wherein the third marker is configured to be attached to a limb of the first user.

19. The cluster system of claim 15, wherein the third marker is attached to or included as a component of a handheld tracking device.

20. The cluster system of claim 15, wherein the at least five degrees of freedom comprise X-axis, Y-axis, Z-axis, yaw, and roll.

21. The cluster system of claim 15, wherein the first repetitive pattern of infrared light pulses comprise light intensity state transitions.

22. The cluster system of claim 15, wherein the plurality of motion tracking cameras are configured to derive the identity of a marker from a detected repetitive pattern of infrared light pulses.