Real time control of hardware and software via communications network

• US 20050125150A1
• Filed: 11/21/2002
• Published: 06/09/2005
• Est. Priority Date: 11/21/2001
• Status: Abandoned Application

• Current Assignee: Handshake VR, Inc.
• Original Assignee: Handshake VR, Inc.
• Inventors: Shu, Joseph, Ni, Liya, Wang, David, Rossi, Mauro, Tuer, Kevin
• Application Number:
  US10/496,256
• Publication Number:
  US 20050125150A1
• Time in Patent Office: Days
• US Class Current: 701/213
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 19/042 using digital processors G05B19/05 takes precedence
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A Hard Real Time Control Center (HRTCC), comprised of hardware, software and firmware, with time synchronisation and time delay compensation methodologies that allows Application Hardware and/or User Input Devices to be networked together on any communications network as if there were negligible network delays in the system, is disclosed. This will allow Application Hardware and/or User Input Devices (connected to an HRTCC at one location (node) on the network) to control or operate Application Hardware and/or User Input Devices connected to another HRTCC at a remote location without the detrimental effects of network time delays. The time synchronisation of the various HRTCCs on the network can be enabled using hardware (e.g. a global positioning system (GPS)) or any other software method (e.g. Network Time Protocol). Using time stamps from the time synchronisation, the time delay of the signals (data) transferred over the network can be determined. The main embodiment of the time delay compensation methodology is an estimator/predictor algorithm. The estimator generates signal information that allows the predictor, using the time delay, to project the signal information characteristics into the future by an amount equal to the time delay. If this predicted signal is used rather than the delayed signal, there will be no readily apparent time delay in the system thereby significantly improving the stability and performance of the associated application. Any software architecture can be used such as servant-client, token ring or peer to peer.

Claims

1-58. -58. (canceled)

59. A network system for implementing hard real time control through a communications network, comprising:

a plurality of control systems, the control systems being capable of communicating with each other through the communications network,

each of the control systems including;

one or more interfaces for application hardware and associated software and/or a user input device;

a synchronizer for substantially synchronizing clocks of the control systems;

a transmitter for sending data associated with the application hardware and associated software and/or the user input device to one or more control systems via the communications network;

a receiver for receiving data from one or more control systems via the communications network; and

compensating means for compensating the received data for a time delay associated with a transmission via the communications network to remove an effect of the time delay for the hard real time control; and

means for providing the compensated data to the interface to operate the application hardware and associated software and/or the user input device.

60. A network system according to claim 59, wherein the compensating means includes predicting means for predicting current data associated with the received data.

61. A network system according to claim 60, wherein the compensating means includes:

an estimator for estimating data from the application hardware and associated software and/or the user input device, the estimated data being related to operating status of the application hardware and associated software and/or the user input device; and

a predictor for projecting the estimated data into current data to operate its own application hardware and associated software and/or the user input device in hard real time.

62. A network system according to claim 60, wherein the user input device connected to at least one of the control systems includes a haptic device, a user of the haptic device interacting an object through the communication networks and receiving haptic effect.

63. A network system according to claim 59, wherein the time delay includes a time-varying delay.

64. A network system according to claim 59, wherein the compensating means is adapted to carry out a dead reckoning prediction process, a Random Acceleration Kalman Filter prediction process, a Random Jerk Kalman Filter prediction process, a higher order Random Movement Kalman Filter prediction process or combinations thereof.

65. A network system according to claim 59, wherein at least one of the control system includes a transmission logic for determining whether or not to send data to other control systems, depending on the operating status of the application hardware/software of the control systems.

66. A network system according to claim 61, wherein one of the control systems acts as a server platform and the others act as client platforms, wherein all the client platforms send their estimated data to the server platform and the server platform distributes the estimated data to the client platform.

67. A network system according to claim 59, wherein the synchronizer includes a global positioning system (GPS) receiver, or a differential global positioning system (DGPS) receiver, a hardware time source, a software time source or a combination thereof.

68. A network system according to claim 59, wherein the data contains a time stamp.

69. A method of implementing hard real time control in a network system, the network system including a plurality of hard real time control systems, each of which has an application hardware/software and is capable of communicating with each other via a communications network, the method comprising steps of:

substantially synchronizing clocks of the hard real time control systems, one of the hard real time control systems being a transmitting side and the other being a receiving side;

estimating data from an application hardware/software on the transmitting side, the estimated data being related to operating status of the application hardware/software;

determining a time delay associated with a transmission via the communications network;

compensating the estimated data for the time delay to remove an effect of the time delay for the hard real time control; and

providing the compensated data to an application hardware/software on the receiving side for the hard real time control.

70. A method according to claim 69, wherein the compensating step includes the step of predicting current data based on the estimated data.

71. A method according to claim 70, wherein the application hardware/software of at least one of the hard real time control systems includes a haptic device, and the providing step includes the step of providing haptic effect to a user of the haptic device in response to the compensated data

72. A method according to claim 69, further comprising a step of:

determining whether or not to send data of the transmitting side to the receiving side, depending on the operating status of the application hardware/software of both sides.

73. A method according to claim 69, wherein the estimating step includes a step of generating kinematic information, the kinematic information being associated with the data and being time-varying.

74. A method according to claim 73, wherein the kinematic information includes a velocity information, and acceleration information, a jerk information and/or combinations thereof.

75. A method according to claim 69, wherein the compensating step carries out a dead reckoning prediction process, a Random Acceleration Kalman Filter prediction process, a Random Jerk Kalman Filter prediction process, a higher order Random Movement Kalman Filter prediction process or combinations thereof.

76. A server-client system for implementing a hard real time control between server and client systems via a communications network, the server-client system comprising:

a synchronizer for substantially synchronizing clocks of the server system and the client system, one of the server and client systems being a transmitting side and the other being a receiving side;

an estimator for estimating data from an application hardware/software on the transmitting side, the estimated data being related to operating status of the application hardware/software; and

a predictor for determining a time delay of the estimated data associated with a transmission via the communications network, and compensating the estimated data for the determined time delay for the hard real time control, and

means for providing the compensated data to an application hardware/software on the receiving side.

77. A server-client system according to claim 76, further comprising a transmission logic for determining whether or not to send data of the transmitting side to the receiving side, depending on the operating status of the application hardware/software of both sides.

78. A server-client system according to claim 76, wherein the application hardware/software on one of the transmitting and receiving sides includes a user input device for controlling the operation of the application hardware/software on the other side.

79. A server-client system according to claim 78, wherein the user input device includes a haptic device, a user of the haptic device operating the application hardware/software through the communications network, and receiving haptic effect.

80. A server-client system according to claim 77, wherein the time delay includes a time-varying delay.

81. A server-client system for implementing hard real time control via a communications network, the server-client system comprising:

a server system including a user input device for controlling an application hardware/software via the communications network;

a client system including the application hardware/software to be controlled by the user input device via the communications network;

a synchronization device operatively associated with each of the server system and the client system for substantially synchronizing clocks thereof; and

compensating means for compensating data transmitted between the server system and the client system for a time delay associated with the transmission via the communications network for the hard real time control between the application hardware/software and the user input device.

82. A server-client system according to claim 81, wherein the compensating means includes predicting means for predicting current data associated with data received at the server or the client through the communications network.

83. A server-client system according to claim 82, wherein the user input device includes a haptic device, a user of the haptic device operating the application hardware/software through the haptic device and receiving haptic effect.

84. A server-client system according to claim 81, wherein the time delay includes a time-varying delay.

85. A server-client system according to claim 81, wherein the application hardware/software can be interactively communicated with and force-interactively controlled by the user input device.

86. A server-client system according to claim 81, wherein the compensating means is adapted to carry out a dead reckoning prediction process, a Random Acceleration Kalman Filter prediction process, a Random Jerk Kalman Filter prediction process, a higher order Random Movement Kalman Filter prediction process, and/or combinations thereof.