System for and method of motion and force synchronization with time delay reduction in multi-user shared virtual environments

• US 7,642,989 B2
• Filed: 03/17/2006
• Issued: 01/05/2010
• Est. Priority Date: 03/17/2005
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Annaswamy, Anuradha M., Cheong, Joono, Srinivasan, Mandayam A.
• Primary Examiner(s): Nguyen; Kevin M
• Application Number:
  US11/378,151
• Publication Number:
  US 20060256080A1
• Time in Patent Office: 1,390 Days
• Field of Search: 345/156, 345/1.1, 345/2.1, 700/45, 709/204
• US Class Current: 345/2.1
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

Systems, methods and computer-executable code stored on computer readable media for synchronizing the evolving state of a dynamic object in a shared virtual environment among a plurality of geographically separated computers connected in a communications network including data communications links introducing time delays in the propagation of data between said computers. The synchronization scheme utilizes an advanced feedback controller to compensate for the state error between sites, comprised of a linear compensator and a Smith predictor based internal model, to determine correct control forces creating a smooth input while maintaining high levels of responsiveness and consistency. A recovery filter for restoring the natural motion of the virtual object distorted by synchronization control is also described.

Claims

1. Method of synchronizing the evolving state of a dynamic object in a shared virtual environment among a plurality of geographically separated computers connected in a communications network including data communications links introducing time delays in the propagation of data between said computers, each computer connected to input means permitting a user to interact with a virtual environment, comprising the steps of:

simulating, at each computer of said plurality of computers, a synchronous local model of a state of a dynamic virtual object in a shared virtual environment (SVE), each synchronous local model of said dynamic virtual object state being initially identical;

permitting users of one or more computers of said plurality of computers to interact via said input means with the corresponding one or more synchronous local models of said dynamic virtual object state so as to induce one or more local changes in said corresponding one or more synchronous local models of said dynamic virtual object state;

propagating from each computer sequences of data packets indicative of the synchronous local model of said dynamic virtual object associated with said propagating local computer to a remote computer in said network, the sequences of data packets being propagated along said communications links thereby introducing time delays into the propagation;

receiving at each computer said sequences of data packets propagated from another computer, said sequences of data packets having time delays from said propagation over said communication links;

compensating at each computer for dynamic virtual object state errors resulting from user inputs at said one or more computers and from said time delays by minimizing the differences between the corresponding local model of said dynamic virtual object state and the one or more induced local changes in dynamic virtual object state as indicated by the received time delayed data packets while maintaining consistency within a tolerable bound between said one or more computers; and

providing each of said one or more users with feedback reflecting an error-compensated, locally-simulated model of said dynamic virtual object state in said SVE.

2. The method of claim 1, wherein the compensating step further comprises generating, at each computer, corrective control forces that reduce the state differences between each of said local models of said virtual object states and the one or more induced local changes in virtual object state as indicated by the received time delayed data packets with a synchronization controller.

3. The method of claim 2, wherein:

said synchronization controller is comprised of a linear compensator and a state estimator;

said linear compensator receiving at its input the difference between the one or more induced local changes in virtual object state contained in the time delayed data packets received from said another computer and the sum of feedback including data packets output by the local computer having been circulated completely about the network and the output of said state estimator; and

said state estimator accepting as input the output of the linear compensator, said state estimator based upon a Smith predictor model P_i(s)(1−

e<sup>−

sR</sup>), where P_i(s) comprises the virtual reality object dynamics of said virtual object, and e<sup>−

sR</sup> denotes the round trip delay R in the Laplace domain representation taken by a data packet to propagate completely about the plurality of computers.

4. The method of claim 2, further comprising the step of approximating via stability analysis said tolerable bound of round trip delay, R_MAD, per

5. The method of claim 1, further comprising the steps of:

quantifying the deviation from the natural motion of said virtual object introduced by the minimization step; and

augmenting, at each of said one or more computers, said local user input with a lead compensator filter that substantially recovers the natural motion of said virtual object.

6. The method of claim 5, wherein the lead compensator filter approximates a filter transfer function

7. The method of claim 1, wherein said plurality of computers are configured in a closed, ring-type peer-to-peer topology for circulating said propagated sequences of data packets.

8. The method of claim 1, wherein:

said plurality of computers comprise client computers; and

said another computer comprises a server to which each of said plurality of computers is connected via said communication links, such that all of the data packets are propagated to and received from said server, which regulates data flow of said data packets from an (i)-th computer to an (i+1)-th computer.

9. The method of claim 1, further comprising the step of selecting a data propagation configuration among said plurality of computers that minimizes the total delay encountered by data packets making a round trip among the plurality of computers.

10. The method of claim 1, wherein said input means comprises a haptic device.

11. The method of claim 1, wherein providing feedback further comprises at least one action, taken at each of said one or more computers, selected from the group consisting of rendering local graphics, rendering local haptics, and emitting one or more audible messages.

12. The method of claim 1, wherein the data packets are comprised of temporal data, virtual object position data, and virtual object velocity data relative to each locally simulated model of the virtual object state for each of said plurality of computers, and data associated with the sum of external forces input by said users.

13. The method of claim 1, wherein said remote computer and said another computer are the same computer.

14. System for synchronizing the evolving state of a dynamic object in a SVE among a plurality of geographically separate computers connected by data communications links introducing time delays in the propagation of data between said computers, each computer connected to an input device permitting a user to interact with a virtual environment, comprising:

a plurality of computers interconnected in a data communications network;

a program component executing on each computer simulating a synchronous local model of the state of a dynamic virtual object in a SVE;

a program component executing on each computer implementing a synchronization controller adapted to perform the steps of;

simulating, at each computer of said plurality of computers, a synchronous local model of a state of the dynamic virtual object in the SVE, each synchronous local model of said dynamic virtual object state being initially identical;

permitting users of one or more computers of said plurality of computers to interact via said input means with the corresponding one or more synchronous local models of said dynamic virtual object state so as to induce one or more local changes in said corresponding one or more synchronous local model of said dynamic virtual object state;

propagating from each computer sequences of data packets indicative of the synchronous local model of said dynamic virtual object associated with said propagating local computer to a remote computer in said network, the sequences of data packets being propagated along said communications links thereby introducing time delays into the propagation;

receiving at each computer said sequences of data packets propagated from another computer, said sequences of data packets having time delays from said propagation over said communication links;

compensating at each computer for dynamic virtual object state errors resulting from user inputs at said one or more computers and from said time delays by minimizing the differences between the corresponding local model of said dynamic virtual object state and the one or more induced local changes in dynamic virtual object state as indicated by the received time delayed data packets while maintaining consistency within a tolerable bound between said one or more computers; and

providing each of said one or more users with feedback reflecting an error-compensated, locally-simulated model of said dynamic virtual object state in said SVE.

15. The system of claim 14, wherein said program component further executes instructions for generating, at each computer, corrective control forces that reduce the state differences between each of said local models of said virtual object states and the one or more induced local changes in virtual object state as indicated by the received time delayed data packets with a synchronization controller.

16. The system of claim 15, wherein:

said synchronization controller is comprised of a linear compensator and a state estimator;

said linear compensator receiving at its input the difference between the one or more induced local changes in virtual object state contained in the time delayed data packets received from said another computer and the sum of feedback including data packets output by the local computer having been circulated completely about the network and the output of said state estimator; and

said state estimator accepting as input the output of the linear compensator, said state estimator based upon a Smith predictor model P_i(s)(1−

e<sup>−

sR</sup>), where P_i(s) comprises the virtual reality object dynamics of said virtual object, and e<sup>−

sR</sup> denotes the round trip delay R in the Laplace domain representation taken by a data packet to propagate completely about the plurality of computers.

17. The system of claim 15, wherein said program component further executes instructions for approximating via stability analysis said tolerable bound of round trip delay, R_MAD, per

18. The method of claim 14, wherein said program component further executes instructions for:

quantifying the deviation from the natural motion of said virtual object introduced by the minimization step; and

augmenting, at each of said one or more computers, said local user input with a lead compensator filter that substantially recovers the natural motion of said virtual object.

19. The system of claim 18, wherein the lead compensator filter approximates a filter transfer function

20. The system of claim 14, wherein said plurality of computers are configured in a closed, ring-type peer-to-peer topology for circulating said propagated sequences of data packets.

21. The system of claim 14, wherein:

said plurality of computers comprise client computers; and

said another computer comprises a server to which each of said plurality of computers is connected via said communication links, such that all of the data packets are propagated to and received from said server, which regulates data flow of said data packets from an (i)-th computer to an (i+1)-th computer.

22. The system of claim 14, wherein said program component further executes instructions for selecting a data propagation configuration among said plurality of computers that minimizes the total delay encountered by data packets making a round trip among the plurality of computers.

23. The system of claim 14, wherein said input means comprises a haptic device.

24. The system of claim 14, wherein providing feedback further comprises at least one action, taken at each of said one or more computers, selected from the group consisting of rendering local graphics, rendering local haptics, and emitting one or more audible messages.

25. The system of claim 14, wherein the data packets are comprised of temporal data, virtual object position data, and virtual object velocity data relative to each locally simulated model of the virtual object SVE state for each of said plurality of computers, and data associated with the sum of external forces input by said users.

26. The system of claim 14, wherein said remote computer and said another computer are the same computer.

27. Machine-readable program storage medium tangibly embodying a program of instructions, the instructions executable by a plurality of computers to perform method steps for synchronizing the evolving state of a dynamic object in a shared virtual environment among a plurality of geographically separated computers connected in a communications network including data communications links introducing time delays in the propagation of data between said computers, each computer connected to input means permitting a user to interact with a virtual environment, the method steps comprising:

simulating, at each computer of said plurality of computers, a synchronous local model of a state of a dynamic virtual object in a shared virtual environment (SVE), each synchronous local model of said dynamic virtual object state being initially identical;

permitting users of one or more computers of said plurality of computers to interact via said input means with the corresponding one or more synchronous local models of said dynamic virtual object state so as to induce one or more local changes in said corresponding one or more synchronous local model of said dynamic virtual object state;

propagating from each computer sequences of data packets indicative of the synchronous local model of said dynamic virtual object associated with said propagating local computer to a remote computer in said network, the sequences of data packets being propagated along said communications links thereby introducing time delays into the propagation;

receiving at each computer said sequences of data packets propagated from another computer, said sequences of data packets having time delays from said propagation over said communication links;

compensating at each computer for dynamic virtual object state errors resulting from user inputs at said one or more computers and from said time delays by minimizing the differences between the corresponding local model of said dynamic virtual object state and the one or more induced local changes in dynamic virtual object state as indicated by the received time delayed data packets while maintaining consistency within a tolerable bound between said one or more computers; and

providing each of said one or more users with feedback reflecting an error-compensated, locally-simulated model of said dynamic virtual object state in said SVE.

28. The machine-readable program storage medium of claim 27, wherein the compensating step further comprises generating, at each computer, corrective control forces that reduce the state differences between each of said local models of said virtual object states and the one or more induced local changes in virtual object state as indicated by the received time delayed data packets with a synchronization controller.

29. The machine-readable program storage medium of claim 28, wherein:

said synchronization controller is comprised of a linear compensator and a state estimator;

said linear compensator receiving at its input the difference between the one or more induced local changes in virtual object state contained in the time delayed data packets received from said another computer and the sum of feedback including data packets output by the local computer having been circulated completely about the network and the output of said state estimator; and

said stale estimator accepting as input the output of the linear compensator, said state estimator based upon a Smith predictor model P_i(s)(1−

e<sup>−

sR</sup>), where P_i(s) comprises the virtual reality object dynamics of said virtual object, and e<sup>−

sR</sup> denotes the round trip delay R in the Laplace domain representation taken by a data packet to propagate completely about the plurality of computers.

30. The machine-readable program storage medium of claim 28, the method steps further comprising approximating via stability analysis said tolerable bound of round trip delay, R_MAD, per

31. The machine-readable program storage medium of claim 27, the method steps further comprising:

quantifying the deviation from the natural motion of said virtual object introduced by the minimization step; and

augmenting, at each of said one or more computers, said local user input with a lead compensator filter that substantially recovers the natural motion of said virtual object.

32. The machine-readable program storage medium of claim 31, wherein the lead compensator filter approximates a filter transfer function

33. The machine-readable program storage medium of claim 27, the method steps further comprising selecting a data propagation configuration among said plurality of computers that minimizes the total delay encountered by data packets making a round trip among the plurality of computers.

34. The machine-readable program storage medium of claim 27, wherein the method step of providing feedback further comprises at least one action, taken at each of said one or more computers, selected from the group consisting of rendering local graphics, rendering local haptics, and emitting one or more audible messages.

35. The machine-readable program storage medium of claim 27, wherein the data packets are comprised of temporal data, virtual object position data, and virtual object velocity data relative to each locally simulated model of the virtual object state for each of said plurality of computers, and data associated with the sum of external forces input by said users.