Mobile terminal-based virtual game controller and remote control system using the same

• US 9,612,709 B2
• Filed: 06/21/2013
• Issued: 04/04/2017
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: TS-Optics Corporation (Intellectual Discovery Co., Ltd.)
• Original Assignee: SungKyunKwan University's Research & Business Foundation (SungKyunKwan University (SKKU))
• Inventors: Shin, Dong Ryeol, Nam, Choon Sung
• Primary Examiner(s): Shah, Milap
• Application Number:
  US13/924,186
• Publication Number:
  US 20140011584A1
• Time in Patent Office: 1,383 Days
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]

Abstract

Disclosed herein are a virtual controller client mobile terminal-based virtual game controller and a remote control system using the same. The remote control system includes a virtual controller server and a virtual controller client. The virtual controller server generates button setting information including mapping relationship between key inputs and virtual input messages, transfers the button setting information, extracts a key input from a virtual input message, and provides the key input to the application. The virtual controller client specifies an arrangement and attributes of virtual buttons based on the button setting information, generates a virtual button screen on the touch screen of the mobile terminal, generates a touch input message based on touch event objects generated based on touch signals for regions corresponding to the virtual buttons, and converts the touch input message into a virtual input message and outputs the virtual input message.

Claims

1. A mobile terminal comprising a virtual controller client configured to remotely communicate with a virtual controller server running on a computer for remote key input to an application running on the computer, the virtual controller client comprising:

a button setting adjusting unit configured to receive first button setting information including a mapping relationship between key inputs to the application and associated virtual input messages, and to specify an arrangement and attributes of virtual buttons based on the received first button setting information;

a user virtual button interface configured to generate a first virtual button screen based on the first button setting information in which touch regions corresponding to the virtual buttons are visually displayed, and to display the first virtual button screen on a touch screen display device of the mobile terminal;

a touch event filter configured to generate touch input messages recognized as key input by the application, based on touch event objects that are generated from touch signals, of the touch regions corresponding to the virtual buttons, among touch signals input by the touch screen; and

a client message interfacing unit configured to convert the touch input message into a virtual input message in a form recognized by the virtual controller server, and to output, to the virtual controller server, the converted touch input message as the virtual input message,

wherein the mapping relationship dynamically redefines the virtual input message associated with a given key input of the key inputs based on an event in the application,

wherein the button setting adjusting unit is configured to receive second button setting information including the dynamically changed virtual message associated with the given key input, and

wherein, in response to an occurrence of the event in the application, the user virtual button interface is configured to generate and display, on the touch screen display device of the mobile terminal, a second virtual button screen based on the second button information.

2. The mobile terminal of claim 1, wherein:

the user virtual button interface activates an acceleration sensor of the mobile terminal to enable a detection of movements of the mobile terminal, and

the virtual controller client further comprises;

an acceleration data filter configured to generate a movement input message that is mapped to a key input of the application, based on acceleration data that is generated based on an acceleration signal generated by the acceleration sensor; and

the client message interfacing unit operable to convert the touch input message or the movement input message into a virtual input message in a form recognized by the virtual controller server.

3. The mobile terminal of claim 1, further comprising a computer-readable storage medium storing a program that is run by the virtual controller client.

4. A computer comprising:

a virtual controller server configured to remotely communicate with a virtual controller client running on a remote mobile terminal including a touch screen display device for remote key input on an application running on the computer, the virtual controller server comprising;

a button setting generating unit configured to generate and transmit, to the virtual controller client, first button setting information including a mapping relationship between key inputs to the application and associated virtual input messages;

a server message interfacing unit configured to transmit a setting message including the first button setting information to the virtual controller client, and to receive a virtual input message from the virtual controller client, the virtual input message being generated based on a touch on the touch screen display device of the mobile terminal; and

a key mapping unit configured to identify a key input value mapped to the received virtual input message based on the first button setting information,

wherein the mapping relationship dynamically redefines the virtual input message associated with a given key input based on an event in the application,

wherein the button setting generating unit is further configured to, in response to an occurrence of the event in the application, generate and transmit, to the virtual controller client, second button setting information including the dynamically changed virtual input message associated with the given key input, and

wherein the key mapping unit is further configured to identify the key input value mapped to the dynamically changed virtual input message based on the second button setting information.

5. The computer of claim 4, wherein the mobile terminal further comprises an acceleration sensor configured to detect movements, and wherein the server message interfacing unit operable to receive a virtual input message generated based on a movement of the mobile terminal.

6. The computer of claim 4, wherein the key mapping unit transfers a key input value to the application via a message transfer architecture of an operating system that runs the application on the computer.

7. The computer of claim 4, wherein the key mapping unit transfers a key input value to the application via an input and output application programming interface (API) of an operating system that runs the application on the computer.

8. The computer of claim 4, further comprising a computer-readable storage medium storing a program that is run by the virtual controller server.

9. A remote control system, comprising:

a computer including a virtual controller server configured to generate first button setting information including mapping relationship between key inputs to an application running on the computer and associated virtual input messages, and transfer the first button setting information to a virtual controller client, for extracting a key input from a virtual input message received from the virtual controller client, and for providing the key input to the application; and

a mobile terminal including a touch screen display device and the virtual controller client, the virtual controller client being configured to remotely communicate with the computer, for specifying an arrangement and attributes of virtual buttons based on the first button setting information received from the virtual controller server, for generating a first virtual button screen in which touch regions corresponding to the virtual buttons are visually displayed on the touch screen display device of the mobile terminal, for generating a touch input message recognized as a key input by the application, based on touch event objects generated based on touch signals for the touch regions corresponding to the virtual buttons, and for converting the touch input message into a virtual input message in a form recognized by the virtual controller server and output the virtual input message,

wherein the mapping relationship dynamically redefines the virtual input message associated with a given key input based on an event in the application,

wherein the virtual controller server is further configured to generate, in response to an occurrence of the event in the application, second button setting information including the dynamically changed virtual message associated with the given key input and transfer the second button setting information to the virtual controller client, and

wherein the virtual controller client is further configured to receive the second button setting information, and generate and display, on the touch screen display device of the mobile terminal, a second virtual button screen based on the second button setting information.

10. The remote control system of claim 9, wherein:

the mobile terminal further comprises an acceleration sensor configured to detect movements; and

the virtual controller client operates such that it activates an acceleration sensor of the mobile terminal to enable a detection of movements, generates a movement input message that is mapped to a key input of the application, based on acceleration data that is generated based on an acceleration signal generated by the acceleration sensor, and converts the touch input message or movement input message into a virtual input message in a form received by the virtual controller server and then outputs the virtual input message.

11. A remote controller interfacing method, the remote controller interfacing method using a virtual controller server running on a computer and a virtual controller client running based on a remote mobile terminal including a touch screen display device for remote key input on an application running on the computer, the remote controller interfacing method comprising:

generating, by the virtual controller server, first button setting information including mapping relationship between key inputs required by the application and associated virtual input messages to be transmitted by the virtual controller client and, in response to an occurrence of an event in the application, second button setting information including mapping relationship that dynamically redefines a virtual input message associated with a given key input based on the event in the application, to be transferred to the virtual controller client;

specifying, by the virtual controller client, an arrangement and attributes of virtual buttons based on the first button setting information, and displaying, by the virtual controller client, a first virtual button screen in which the virtual button regions are visually arranged on the touch screen display device of the mobile terminal, and in response to occurrence of the event, displaying, by the virtual controller client, a second virtual button screen based on the second button setting information;

generating, by the virtual controller client, touch event objects based on a touch signal generated by the touch screen, and further a touch input message based on the valid touch event objects;

transferring, by the virtual controller client, a virtual input message generated based on the touch input message to the virtual controller server;

identifying, by the virtual controller server, a key input value mapped to the received virtual input message based on the first button setting information or the second button setting information; and

transferring, by the virtual controller server, the identified key input value to the application.

12. The remote controller interfacing method of claim 11,

wherein:

the mobile terminal further comprises an acceleration sensor configured to detect movements; and

the remote controller interfacing method further comprises;

generating, by the virtual controller client, a movement input message that is mapped to a key input of the application, based on acceleration data that is generated based on an acceleration signal generated by the acceleration sensor; and

converting, by the virtual controller client, the movement input message into a virtual input message in a form received by the virtual controller server, and outputting, by the client, the virtual input message.

13. The remote controller interfacing method of claim 11, wherein the key input value identified by the virtual controller server is transferred to the application via a message transfer architecture of an operating system that runs the application on the computer.

14. The remote controller interfacing method of claim 11, wherein the key input value identified by the virtual controller server is transferred to the application via an input and output API of an operating system that runs the application on the computer.

15. A non-transitory computer-readable storage medium storing instruction that causes a computer and a remote mobile terminal to perform a remote controller interfacing method, the remote controller interfacing method using a virtual controller server running on the computer and a virtual controller client running based on the remote mobile terminal including a touch screen for remote key input on an application running on the computer, the remote controller interfacing method comprising:

generating, by the virtual controller server, first button setting information including mapping relationship between key inputs required by the application and associated virtual input messages to be transmitted by the virtual controller client and, in response to an occurrence of an event in the application, second button setting information including mapping relationship that dynamically redefines a virtual input message associated with a given key input based on the event in the application, to be transferred to the virtual controller client;

specifying, by the virtual controller client, an arrangement and attributes of virtual buttons based on the first button setting information, and displaying, by the virtual controller client, a first virtual button screen in which the virtual button regions are visually arranged on the touch screen display device of the mobile terminal, and in response to occurrence of the event, displaying, by the virtual controller client, a second virtual button screen based on the second button setting information;

generating, by the virtual controller client, touch event objects based on a touch signal generated by the touch screen, and further a touch input message based on the valid touch event objects;

transferring, by the virtual controller client, a virtual input message generated based on the touch input message to the virtual controller server;

identifying, by the virtual controller server, a key input value mapped to the received virtual input message based on the first button setting information or the second button setting information; and

transferring, by the virtual controller server, the identified key input value to the application.