Remote operation system and remote operation method thereof

• US 20020029259A1
• Filed: 07/24/2001
• Published: 03/07/2002
• Est. Priority Date: 07/26/2000
• Status: Abandoned Application

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Okada, Hidehiko
• Application Number:
  US09/910,778
• Publication Number:
  US 20020029259A1
• Time in Patent Office: Days
• US Class Current: 709/219
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/303 Terminal profiles
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display

Abstract

A remote operation system includes a screen analysis unit for analyzing the contents displayed on a screen, a screen information transmission unit for transmitting data of screen information output by the screen analysis unit to a terminal device, an operation execution unit for receiving user'"'"'s operation to execute predetermined processing corresponding to the operation and a screen change detection unit for detecting a change of display on the screen, in which the screen analysis unit detects GUI widgets which is an object displayed in the screen and operable by a user to obtain data of a kind, a name and a position in the screen of the GUI widgets, a hyper-link is set for each GUI widget, when a hyper-link corresponding to each GUI widgets is selected, a web page is generated for transmitting data for identifying the GUI widgets in question and data for identifying operation corresponding to the GUI widgets in question, and when a screen change caused by the user'"'"'s operation is completed, the screen analysis unit analyzes the current screen and transmits the analysis to a user'"'"'s terminal device to update data displayed on the user'"'"'s terminal device.

Claims

1. A remote operation system for remotely operating a device to be operated through a communication network from an operation side terminal, comprising:

screen analysis means for analyzing information displayed on a screen of said device to be operated, and

screen information transmission means for transmitting data of an analysis result obtained by said screen analysis means to said operation side terminal to display the data on said operation side terminal.

2. The remote operation system as set forth in claim 1, further comprising

operation execution means for causing said device to be operated to execute predetermined processing corresponding to user'"'"'s operation conducted using said operation side terminal.

3. The remote operation system as set forth in claim 1, further comprising

screen information conversion means for converting data of an analysis result obtained by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on a screen of said operation side terminal, wherein

said screen information transmission means transmits data of a result of conversion by said screen information conversion means to said operation side terminal to display the data on said operation side terminal.

4. The remote operation system as set forth in claim 3, further comprising

conversion rule registration means for registering and recording rules for converting data of an analysis result obtained by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal, wherein

said screen information conversion means converts data of an analysis result obtained by said screen analysis means based on the conversion rules recorded in said conversion rule registration means.

5. The remote operation system as set forth in claim 3, further comprising

device characteristic detection means for detecting device characteristics of said operation side terminal, wherein

said screen information conversion means converts data of an analysis result obtained by said screen analysis means based on device characteristics detected by said device characteristic detection means and conversion rules recorded in said conversion rule registration means.

6. The remote operation system as set forth in claim 1, further comprising

screen change detection means for detecting a change in display of the screen of said device to be operated, wherein

at a time when a change occurred in display of the screen is completed, said screen analysis means analyzes the current screen of said device to be operated.

7. The remote operation system as set forth in claim 1, wherein

said device to be operated is an information processing device having a GUI, and

said screen analysis means detects GUI widgets displayed on the screen of said device to be operated to obtain attribute data of said GUI widgets.

8. The remote operation system as set forth in claim 7, wherein

the attribute data of said GUI widgets includes at least one of a kind of said GUI widgets, a name of said GUI widgets and a location of said GUI widgets in the screen.

9. The remote operation system as set forth in claim 7, wherein

said conversion rule registration means has registration of rules for converting data of an analysis result obtained by said screen analysis means into a web page that can be displayed by a web browser,

said screen information conversion means converts data of an analysis result obtained by said screen analysis means into a web page based on said rules,

said web page includes, for GUI widgets operable by a user among GUI widgets detected by said screen analysis means, a hyper-link corresponding to the GUI widgets in question, and

when said hyper-link is selected, data for identifying GUI widgets corresponding to the hyper-link in question and data for identifying operation for the GUI widgets in question are transmitted to said operation execution means.

10. The remote operation system as set forth in claim 1, wherein

said conversion rule registration means has registration of rules for converting data of an analysis result obtained by said screen analysis means into a text of electronic mail.

11. The remote operation system as set forth in claim 10, wherein

said operation execution means receives electronic mail including information of user'"'"'s operation conducted using said operation side terminal, analyzes the received electronic mail to identify the user'"'"'s operation, and executes the operation in question.

12. The remote operation system as set forth in claim 1, wherein

data of an analysis result obtained by said screen analysis means is data of an XML format.

13. The remote operation system as set forth in claim 1, wherein

said operation side terminal is a portable communication terminal having a character data transmission and reception function.

14. The remote operation system as set forth in claim 1, wherein

said communication network is the Internet.

15. The remote operation system as set forth in claim 1, wherein

said screen analysis means, when moving picture or still picture is displayed on the screen of said device to be operated, detects the display in question to extract picture data of the display in question, and

said screen information transmission means transmits said picture data extracted by said screen analysis means to said operation side terminal to display the data on said operation side terminal.

16. The remote operation system as set forth in claim 1, wherein

said screen analysis means, when moving picture or still picture is displayed on the screen of said device to be operated, detects the display in question to extract picture data of the display in question, and

said screen information conversion means converts said picture data extracted by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal.

17. The remote operation system as set forth in claim 5, wherein

device characteristics of said operation side terminal detected by said device characteristic detection means include at least one of an inherent ID of said operation side terminal, a kind of machine, a processing rate, a communication rate, communication costs per unit volume of data, communication costs per unit time, a data format that can be displayed on a screen, a playable sound data format, executable program and script formats, a volume of data receivable at one time, a color that can be displayed, a character font that can be displayed, a screen resolution, a physical length of a screen in a vertical direction and a lateral direction and the number of characters that can be displayed within one screen in the vertical direction and the lateral direction.

19. The remote operation system as set forth in claim 18, further comprising

operation execution means for causing said device to be operated to execute predetermined processing corresponding to user'"'"'s operation conducted using said operation side terminal.

20. The remote operation system as set forth in claim 18, further comprising

screen information conversion means for converting data of an analysis result obtained by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on a screen of said operation side terminal, wherein

said screen information transmission means transmits data of a result of conversion by said screen information conversion means to said operation side terminal to display the data on said operation side terminal.

21. The remote operation system as set forth in claim 20, further comprising

conversion rule registration means for registering and recording rules for converting data of an analysis result obtained by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal, wherein

said screen information conversion means converts data of an analysis result obtained by said screen analysis means based on the conversion rules recorded in said conversion rule registration means.

22. The remote operation system as set forth in claim 20, further comprising

device characteristic detection means for detecting device characteristics of said operation side terminal, wherein

said screen information conversion means converts data of an analysis result obtained by said screen analysis means based on device characteristics detected by said device characteristic detection means and conversion rules recorded in said conversion rule registration means.

23. The remote operation system as set forth in claim 18, further comprising

screen change detection means for detecting a change in display of the screen of said device to be operated, wherein

at a time when a change occurred in display of the screen is completed, said screen analysis means analyzes the current screen of said device to be operated.

24. The remote operation system as set forth in claim 18, wherein

said device to be operated is an information processing device having a GUI, and

said screen analysis means detects GUI widgets displayed on the screen of said device to be operated to obtain attribute data of said GUI widgets.

25. The remote operation system as set forth in claim 24, wherein

the attribute data of said GUI widgets includes at least one of a kind of said GUI widgets, a name of said GUI widgets and a location of said GUI widgets in the screen.

26. The remote operation system as set forth in claim 24, wherein

said conversion rule registration means has registration of rules for converting data of an analysis result obtained by said screen analysis means into a web page that can be displayed by a web browser,

said screen information conversion means converts data of an analysis result obtained by said screen analysis means into a web page based on said rules,

said web page includes, for GUI widgets operable by a user among GUI widgets detected by said screen analysis means, a hyper-link corresponding to the GUI widgets in question, and

when said hyper-link is selected, data for identifying GUI widgets corresponding to the hyper-link in question and data for identifying operation for the GUI widgets in question are transmitted to said operation execution means.

27. The remote operation system as set forth in claim 18, wherein

said conversion rule registration means has registration of rules for converting data of an analysis result obtained by said screen analysis means into a text of electronic mail.

28. The remote operation system as set forth in claim 27, wherein

said operation execution means receives electronic mail including information of user'"'"'s operation conducted using said operation side terminal, analyzes the received electronic mail to identify the user'"'"'s operation, and executes the operation in question.

29. The remote operation system as set forth in claim 18, wherein

data of an analysis result obtained by said screen analysis means is data of an XML format.

30. The remote operation system as set forth in claim 18, wherein

said operation side terminal is a portable communication terminal having a character data transmission and reception function.

31. The remote operation system as set forth in claim 18, wherein

said communication network is the Internet.

32. The remote operation system as set forth in claim 18, wherein

said screen analysis means, when moving picture or still picture is displayed on the screen of said device to be operated, detects the display in question to extract picture data of the display in question, and

said screen information transmission means transmits said picture data extracted by said screen analysis means to said operation side terminal to display the data on said operation side terminal.

33. The remote operation system as set forth in claim 18, wherein

said screen analysis means, when moving picture or still picture is displayed on the screen of said device to be operated, detects the display in question to extract picture data of the display in question, and

said screen information conversion means converts said picture data extracted by said screen analysis means into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal.

34. The remote operation system as set forth in claim 22, wherein

device characteristics of said operation side terminal detected by said device characteristic detection means include at least one of an inherent ID of said operation side terminal, a kind of machine, a processing rate, a communication rate, communication costs per unit volume of data, communication costs per unit time, a data format that can be displayed on a screen, a playable sound data format, executable program and script formats, a volume of data receivable at one time, a color that can be displayed, a character font that can be displayed, a screen resolution, a physical length of a screen in a vertical direction and a lateral direction and the number of characters that can be displayed within one screen in the vertical direction and the lateral direction.

36. The remote operation method as set forth in claim 35, further comprising

an operation execution step of causing said device to be operated to execute predetermined processing corresponding to user'"'"'s operation conducted using said operation side terminal.

37. The remote operation method as set forth in claim 35, further comprising

a screen information conversion step of converting data of an analysis result obtained by said screen analysis step into data suitable for the transmission to said operation side terminal or for displaying on a screen of said operation side terminal, wherein

at said screen information transmission step, data of a result of conversion by said screen information conversion step is transmitted to said operation side terminal to display the data on said operation side terminal.

38. The remote operation method as set forth in claim 37, further comprising

a conversion rule registration step of registering rules for converting data of an analysis result obtained by said screen analysis step into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal, wherein

at said screen information conversion step, data of an analysis result obtained by said screen analysis step is converted based on the conversion rules registered at said conversion rule registration step.

39. The remote operation method as set forth in claim 37, further comprising

a device characteristic detection step of detecting device characteristics of said operation side terminal, wherein

at said screen information conversion step, data of an analysis result obtained by said screen analysis step is converted based on device characteristics detected by said device characteristic detection step and conversion rules registered at said conversion rule registration step.

40. The remote operation method as set forth in claim 35, further comprising

a screen change detection step of detecting a change in display of the screen of said device to be operated, wherein

at a time when a change occurred in display of the screen is completed, the contents displayed on the current screen of said device to be operated are analyzed by said screen analysis step.

41. The remote operation method as set forth in claim 35, wherein

said device to be operated is an information processing device having a GUI, and

at said screen analysis step, GUI widgets displayed on the screen of said device to be operated is detected to obtain attribute data of said GUI widgets.

42. The remote operation method as set forth in claim 41, wherein

the attribute data of said GUI widgets includes at least one of a kind of said GUI widgets, a name of said GUI widgets and a location of said GUI widgets in the screen.

43. The remote operation method as set forth in claim 41, wherein

at said conversion rule registration step, rules are registered for converting data of an analysis result obtained by said screen analysis step into a web page that can be displayed by a web browser,

at said screen information conversion step, data of an analysis result obtained by said screen analysis step is converted into a web page based on said rules,

said web page includes, for GUI widgets operable by a user among GUI widgets detected by said screen analysis step, a hyper-link corresponding to the GUI widgets in question, and

at said operation execution step, using data for identifying GUI widgets corresponding to the hyper-link in question and data for identifying operation for the GUI widgets in question which are transmitted when said hyper-link is selected, the operation side terminal is caused to execute predetermined processing.

44. The remote operation method as set forth in claim 35, wherein

at said conversion rule registration step, rules are registered for converting data of an analysis result obtained by said screen analysis step into a text of electronic mail.

45. The remote operation method as set forth in claim 35, wherein

at said screen analysis step, when moving picture or still picture is displayed on the screen of said device to be operated, the display in question is detected to extract picture data of the display in question, and

at said screen information transmission step, said picture data extracted by said screen analysis step is transmitted to said operation side terminal to display the data on said operation side terminal.

46. The remote operation method as set forth in claim 35, wherein

at said screen analysis step, when moving picture or still picture is displayed on the screen of said device to be operated, the display in question is detected to extract picture data of the display in question, and

at said screen information conversion step, said picture data extracted by said screen analysis step is converted into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal.

47. The remote operation method as set forth in claim 39, wherein

device characteristics of said operation side terminal detected by said device characteristic detection step include at least one of an inherent ID of said operation side terminal, a kind of machine, a processing rate, a communication rate, communication costs per unit volume of data, communication costs per unit time, a data format that can be displayed on a screen, a playable sound data format, executable program and script formats, a volume of data receivable at one time, a color that can be displayed, a character font that can be displayed, a screen resolution, a physical length of a screen in a vertical direction and a lateral direction and the number of characters that can be displayed within one screen in the vertical direction and the lateral direction.

49. The remote operation program as set forth in claim 48, which causes execution of operation execution processing of making said device to be operated conduct predetermined processing corresponding to user'"'"'s operation conducted using said operation side terminal.

50. The remote operation program as set forth in claim 48, which causes screen information conversion processing of converting data of an analysis result obtained by said screen analysis processing into data suitable for the transmission to said operation side terminal or for displaying on a screen of said operation side terminal, and

in said screen information transmission processing, causes execution of processing of transmitting data of a conversion result obtained by said screen information conversion processing to said operation side terminal to display the data on said operation side terminal.

51. The remote operation program as set forth in claim 50, which causes execution of conversion rule registration processing of registering rules for converting data of an analysis result obtained by said screen analysis processing into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal, and

in said screen information conversion processing, causes execution of processing of converting data of an analysis result obtained by said screen analysis processing based on the conversion rules registered by said conversion rule registration processing.

52. The remote operation program as set forth in claim 50, which causes execution of device characteristic detection processing of detecting device characteristics of said operation side terminal, and

in said screen information conversion processing, causes execution of processing of converting data of an analysis result obtained by said screen analysis processing based on device characteristics detected by said device characteristic detection processing and conversion rules registered by said conversion rule registration processing.

53. The remote operation program as set forth in claim 48, which causes execution of:

screen change detection processing of detecting a change in display of the screen of said device to be operated, and

at a time when a change occurred in display of the screen is completed, processing of analyzing the contents displayed on the current screen of said device to be operated by said screen analysis processing.

54. The remote operation program as set forth in claim 48, wherein

said device to be operated is an information processing device having a GUI, and

in said screen analysis processing, the program causes execution of processing of detecting GUI widgets displayed on the screen of said device to be operated to obtain attribute data of said GUI widgets.

55. The remote operation program as set forth in claim 54, wherein

the attribute data of said GUI widgets includes at least one of a kind of said GUI widgets, a name of said GUI widgets and a location of said GUI widgets in the screen.

56. The remote operation program as set forth in claim 54, wherein

in said conversion rule registration processing, rules are registered for converting data of an analysis result obtained by said screen analysis processing into a web page that can be displayed by a web browser,

in said screen information conversion processing, data of an analysis result obtained by said screen analysis processing is converted into a web page based on said rules,

said web page includes, for GUI widgets operable by a user among GUI widgets detected by said screen analysis processing, a hyper-link corresponding to the GUI widgets in question, and

in said operation execution processing, in order to cause the operation side terminal to execute said predetermined processing, data for identifying GUI widgets corresponding to the hyper-link in question and data for identifying operation for the GUI widgets in question transmitted when said hyper-link is selected.

57. The remote operation program as set forth in claim 48, which

in said conversion rule registration processing, causes execution of processing of registering rules for converting data of an analysis result obtained by said screen analysis processing into a text of electronic mail.

58. The remote operation program as set forth in claim 48, which causes execution of:

in said screen analysis processing, when moving picture or still picture is displayed on the screen of said device to be operated, detecting the display in question to extract picture data of the display in question, and

in said screen information transmission processing, processing of transmitting said picture data extracted by said screen analysis processing to said operation side terminal to display the data on said operation side terminal.

59. The remote operation program as set forth in claim 48, which

in said screen analysis processing, when moving picture or still picture is displayed on the screen of said device to be operated, detects the display in question to extract picture data of the display in question, and

in said screen information conversion processing, converts said picture data extracted by said screen analysis processing into data suitable for the transmission to said operation side terminal or for displaying on the screen of said operation side terminal.

60. The remote operation program as set forth in claim 52, wherein

device characteristics of said operation side terminal detected by said device characteristic detection processing include at least one of an inherent ID of said operation side terminal, a kind of machine, a processing rate, a communication rate, communication costs per unit volume of data, communication costs per unit time, a data format that can be displayed on a screen, a playable sound data format, executable program and script formats, a volume of data receivable at one time, a color that can be displayed, a character font that can be displayed, a screen resolution, a physical length of a screen in a vertical direction and a lateral direction and the number of characters that can be displayed within one screen in the vertical direction and the lateral direction.

18. A remote operation system for remotely operating a device to be operated through a communication network from an operation side terminal, wherein

said device to be operated comprises:

screen analysis means for analyzing the contents displayed on a screen of said device to be operated, and

screen information transmission means for transmitting data of an analysis result obtained by said screen analysis means to said operation side terminal to display the data on said operation side terminal.

35. A remote operation method of remotely operating a device to be operated through a communication network from an operation side terminal, comprising the steps of:

a screen analysis step of analyzing information displayed on a screen of said device to be operated, and

screen information transmission step of transmitting data of an analysis result obtained by said screen analysis step to said operation side terminal to display the data on said operation side terminal.

48. A remote operation program for remotely operating a device to be operated through a communication network from an operation side terminal by controlling an information processing system, which causes execution of:

screen analysis processing of analyzing information displayed on a screen of said device to be operated, and

screen information transmission processing of transmitting data of an analysis result obtained by said screen analysis processing to said operation side terminal to display the data on said operation side terminal.