Remotely operating target device

• US 10,331,226 B2
• Filed: 06/23/2017
• Issued: 06/25/2019
• Est. Priority Date: 08/09/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ishimoto, Kenya, Takeda, Hideo, Yoshifuji, Nobuyuki
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US15/631,207
• Publication Number:
  US 20180046262A1
• Time in Patent Office: 732 Days
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 3/1462 with means for detecting differences between the image stored in the host and the images displayed on the remote displays
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/02 Recognising information on displays, dials, clocks
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 2250/52 including functional features of a camera

Abstract

The embodiments of the present invention provide systems and methods for remotely operating a target device. A camera image is obtained, including a target screen image (which is an image of a screen of the target device). The camera image is transmitted to and displayed in a second device. Operational information is obtained on the screen of the second device, which displays the camera image. The operational information is transmitted to a first device. Human device data, which derives from the operational information, is inputted from the first device to the target device.

Claims

1. A method for remotely operating a target device, comprising:

obtaining, by one or more processors, a camera image in a first device, wherein the camera image from an image handler comprises a target screen image and a position of a pointer on the target screen, wherein the target screen image is an image of a screen of the target device;

adjusting, by one or more processors, the position displayed in an image screen mapper by analyzing the position of the pointer relative to the target screen image from the image handler;

determining whether the first device is authorized to interact with the target device;

transmitting, by one or more processors, at least a portion of the camera image from the first device to a second device responsive to determining that the first device is authorized to interact with the target device, wherein the portion of the camera image includes at least the target screen image;

displaying, by one or more processors, at least the portion of the camera image on a screen of the second device;

enabling, by one or more processors, a user to interact with the portion of the camera image to provide operational information via the second device while the first device displays at least the portion of the camera image, wherein the operational information is indicative of an operation being performed on the screen of the second device;

transmitting, by one or more processors, the operational information from the second device to the first device; and

inputting, by one or more processors, human interface device data from the first device to the target device, wherein the human interface device data is based on the operational information.

2. The method of claim 1, wherein transmitting at least the portion of the camera image, further comprises:

capturing, by one or more processors, a portion outside of the target screen image, which is included in the portion of the camera image.

3. The method of claim 1, wherein obtaining the operational information, comprises:

capturing, by one or more processors, a first set of information, wherein the first set of information is indicative of a movement direction of a pointer on the screen of the second device; and

capturing, by one or more processors, a second set of information, wherein the second set of information is indicative of an amount of movement by the pointer on the screen of the second device.

4. The method of claim 3, further comprises:

inputting, by one or more processors, the first set of information and the second set of information into human interface device data.

5. The method of claim 1, wherein obtaining the operational information, comprises:

capturing, by one or more processors, a third set of information, wherein the third set of information is indicative of a position of a pointer on the target screen image of the second device; and

a fourth set of information, wherein the fourth set of information is indicative of the position of the pointer on the target screen image corresponding to the pointer on the screen of a second device.

6. The method of claim 5, further comprises:

inputting, by one or more processors, the third set of information into human interface device data.

7. The method of claim 1, further comprises:

inputting, by one or more processors, the adjusted position in the image screen mapper the fourth set of information into human interface device data.