Method, device, and system for recognizing gesture based on multi-terminal collaboration

• US 9,817,447 B2
• Filed: 12/11/2014
• Issued: 11/14/2017
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Song, Xingguang, Lian, Shiguo, Hu, Wei
• Primary Examiner(s): Bhargava, Anil
• Application Number:
  US14/567,253
• Publication Number:
  US 20150186006A1
• Time in Patent Office: 1,069 Days
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/1698 the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone details of antennas disposed inside a computer H01Q1/2266
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/1438 using more than one graphics controller
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2356/00 Detection of the display position w.r.t. other display screens
• CPC Class Codes: G09G 5/08 Cursor circuits

Abstract

A method, a device, and a system are provided for recognizing a gesture based on multi-terminal collaboration, which are used to recognize an air gesture operation of a user in a multi-terminal collaboration scenario. The method includes receiving, by a first terminal after receiving a collaboration request activation signal, parameter information of second terminals; recording, by the first terminal according to the parameter information of the second terminals, an arrangement manner of terminals that participate in acquiring of gesture information, where the terminals that participate in the acquiring of the gesture information include all terminals or the second terminals; receiving, by the first terminal, sensor trigger information sent by at least one second terminal; and determining, by the first terminal according to the recorded arrangement manner and the acquired sensor trigger information, a type of a gesture that triggers a sensor.

Claims

1. A method for recognizing a gesture based on multi-terminal collaboration, comprising:

receiving, by a first terminal after receiving a collaboration request activation signal triggered by a user, parameter information of second terminals sent by the second terminals, wherein the parameter information of the second terminals comprises terminal screen sizes and screen resolutions of the second terminals;

calculating multiple arrangement manners of the first terminal and the second terminals based at least in part on the terminal screen sizes and the screen resolutions of the second terminals;

receiving a user selection of one of the multiple arrangement manners to use as an arrangement manner of terminals that participate in acquiring of gesture information;

recording, by the first terminal according to the parameter information of the second terminals and the user selection, the arrangement manner of the terminals that participate in acquiring of the gesture information, wherein the terminals that participate in the acquiring of the gesture information comprise all terminals or the second terminals, and all the terminals comprise the first terminal and the second terminals;

receiving, by the first terminal, sensor trigger information sent by at least one second terminal; and

determining, by the first terminal according to the recorded arrangement manner and the acquired sensor trigger information, a type of a gesture that triggers a sensor, and

wherein determining, by the first terminal according to the recorded arrangement manner and the acquired sensor trigger information, the type of the gesture that triggers the sensor comprises;

acquiring, by the first terminal, first sensor trigger information of each triggered terminal from at least one piece of sensor trigger information detected by each triggered terminal, wherein the first sensor trigger information is provided by the sensor of a highest priority of each terminal;

determining, by the first terminal, a sliding direction of the gesture and calculating a trigger time difference between adjacent triggered terminals along the sliding direction of the gesture according to the recorded arrangement manner and trigger time that is contained in the first sensor trigger information of each triggered terminal; and

determining, by the first terminal according to the sliding direction of the gesture and the trigger time difference, the type of the gesture that triggers the sensor.

2. The method according to claim 1, wherein the parameter information comprises a device configuration parameter, and wherein recording, by the first terminal according to the received parameter information of the second terminals and the user selection, the arrangement manner of the terminals that participate in acquiring of the gesture information comprises:

displaying, in an interface by the first terminal according to a device configuration parameter of the first terminal, received device configuration parameters of the second terminals, and the number of all the terminals, at least one arrangement manner containing all the terminals, for the user to select; and

recording the arrangement manner selected by the user.

3. The method according to claim 1, wherein the parameter information comprises a device configuration parameter, and wherein recording, by the first terminal according to the received parameter information of the second terminals and the user selection, the arrangement manner of the terminals that participate in acquiring of the gesture information comprises:

displaying, in an interface by the first terminal according to received device configuration parameters of the second terminals and the number of the second terminals, at least one arrangement manner containing the second terminals, for the user to select; and

recording the arrangement manner selected by the user.

4. The method according to claim 1, wherein before determining, by the first terminal, the sliding direction of the gesture and calculating the trigger time difference between the adjacent triggered terminals along the sliding direction of the gesture according to the recorded arrangement manner and the trigger time that is contained in the first sensor trigger information of each triggered terminal, the method further comprises determining that time of the first terminal is synchronized with time of the second terminal.

5. The method according to claim 1, wherein determining, by the first terminal according to the sliding direction of the gesture and the trigger time difference, the type of the gesture that triggers the sensor comprises:

determining that the type of the gesture that triggers the sensor is a slide gesture, when the gesture slides in a unidirectional manner and the first terminal determines that the trigger time difference is within a preset time range;

determining that the type of the gesture that triggers the sensor is a zoom-in gesture, when the gesture comprises a first gesture and a second gesture, the first gesture and the second gesture slide apart from each other, and both the first gesture and the second gesture are slide gestures;

determining that the type of the gesture that triggers the sensor is the zoom-out gesture, when the first gesture and the second gesture slide toward each other, and both the first gesture and the second gesture are slide gestures; and

determining that the type of the gesture that triggers the sensor is a rotate gesture, when a sliding direction of the first gesture is different from a sliding direction of the second gesture, a start terminal of the first gesture is different from a start terminal of the second gesture, an end terminal of the first gesture is different from an end terminal of the second gesture, and the first terminal determines that both the first gesture and the second gesture are slide gestures and a difference between trigger time of the end terminal of the first gesture and that of the end terminal of the second gesture is less than a preset threshold.

6. A method for recognizing a gesture based on multi-terminal collaboration, comprising:

sending, by a second terminal after receiving a collaboration request activation signal, parameter information of the second terminal to a first terminal such that the first terminal records, according to parameter information of second terminals and a user selection, an arrangement manner of terminals that participate in acquiring of gesture information, wherein the parameter information of the second terminal comprises terminal screen sizes and screen resolutions of the second terminals, wherein multiple arrangement manners of the first terminal and the second terminals are calculated based at least in part on the terminal screen sizes and the screen resolutions of the second terminals, wherein the user selection of one of the multiple arrangement manners is received and is used as an arrangement manner of terminals that participate in acquiring gesture information, wherein the terminals that participate in the acquiring of the gesture information comprise all terminals or the second terminals, and all the terminals comprise the first terminal and the second terminals; and

sending, by the second terminal when detecting that a sensor is triggered, detected sensor trigger information to the first terminal such that the first terminal determines, according to the recorded arrangement manner and the acquired sensor trigger information, a type of a gesture that triggers the sensor, and

wherein determining, by the first terminal according to the recorded arrangement manner and the acquired sensor trigger information, the type of the gesture that triggers the sensor comprises;

acquiring, by the first terminal, first sensor trigger information of each triggered terminal from at least one piece of sensor trigger information detected by each triggered terminal, wherein the first sensor trigger information is provided by the sensor of a highest priority of each terminal;

determining, by the first terminal, a sliding direction of the gesture and calculating a trigger time difference between adjacent triggered terminals along the sliding direction of the gesture according to the recorded arrangement manner and trigger time that is contained in the first sensor trigger information of each triggered terminal; and

determining, by the first terminal according to the sliding direction of the gesture and the trigger time difference, the type of the gesture that triggers the sensor.

7. The method according to claim 6, wherein the parameter information comprises a device configuration parameter, and wherein sending, by the second terminal, the parameter information to the first terminal comprises sending, by the second terminal, the device configuration parameter to the first terminal such that the first terminal displays, in an interface according to a device configuration parameter of the first terminal, received device configuration parameters of the second terminals, and the number of all the terminals, at least one arrangement manner containing all the terminals, for a user to select, and records the arrangement manner selected by the user.

8. The method according to claim 6, wherein the parameter information comprises a device configuration parameter, and wherein sending, by the second terminal, the parameter information to the first terminal comprises sending, by the second terminal, the device configuration parameter to the first terminal such that the first terminal displays, in an interface according to received device configuration parameters of the second terminals and the number of the second terminals, at least one arrangement manner containing the second terminals, for the user to select, and records the arrangement manner selected by the user.

9. A terminal, comprising:

a computer processor;

a receiver coupled to the computer processor; and

a memory coupled to the computer processor,

wherein the receiver is configured to receive a collaboration request activation signal triggered by a user,

wherein the receiver is further configured to receive parameter information of terminals sent by the terminals,

wherein the parameter information of the terminals comprises terminal screen sizes and screen resolutions of the terminals,

wherein the computer processor is configured to calculate multiple arrangements of the terminals based at least in part on the terminal screen sizes and the screen resolutions of the terminals,

wherein the receiver is configured to receive a user selection of one of the multiple arrangement manners to use as an arrangement manner of terminals that participate in acquiring of gesture information,

wherein the memory is configured to record, according to the parameter information received by the receiver and the user selection, the arrangement manner of the terminals that participate in acquiring of the gesture information,

wherein the terminals that participate in the acquiring of the gesture information comprises all terminals or the terminals, and all the terminals comprise the terminal itself and the terminals,

wherein the receiver is further configured to receive sensor trigger information sent by at least one terminal,

wherein the computer processor is configured to determine, according to the arrangement manner recorded by the memory and the sensor trigger information received by the receiver, a type of a gesture that triggers a sensor, and

wherein the computer processor is further configured to;

acquire first sensor trigger information of each triggered terminal from at least one piece of sensor trigger information detected by each triggered terminal, wherein the first sensor trigger information is provided by a sensor of a highest priority of each terminal;

determine a sliding direction of the gesture;

calculate a trigger time difference between adjacent triggered terminals along the sliding direction of the gesture according to the arrangement manner recorded by the memory and trigger time that is contained in the first sensor trigger information of each triggered terminal; and

determine, according to the sliding direction of the gesture and the trigger time difference, the type of the gesture that triggers the sensor, wherein the type of the gesture comprises a slide gesture, a zoom gesture, and a rotate gesture.

10. The terminal according to claim 9, wherein the parameter information comprises a device configuration parameter, and wherein the computer processor is further configured to:

calculate and display at least one arrangement manner of all the terminals according to the device configuration parameter of the terminal itself, device configuration parameters of the terminals received by the receiver, and the number of all the terminals; and

record the arrangement manner selected by the user according to the display.

11. The terminal according to claim 9, wherein the parameter information comprises a device configuration parameter, and wherein the computer processor is further configured to:

calculate and display at least one arrangement manner of the terminals, according to device configuration parameters of the terminals received by the receiver and the number of terminals that send the device configuration parameters; and

record the arrangement manner selected by the user according to the display.

12. The terminal according to claim 9, further comprising a clock coupled to the computer processor and configured to determine that time of the terminal itself is synchronized with time of each terminal, and wherein the computer processor is further configured to, when the clock determines that time is synchronized, determine the sliding direction of the gesture and calculate the trigger time difference between adjacent triggered terminals along the sliding direction of the gesture according to the arrangement manner recorded by the memory and the trigger time that is contained in the first sensor trigger information of each triggered terminal.

13. The terminal according to claim 9, wherein the computer processor is further configured to:

determine that the type of the gesture that triggers the sensor is slide gesture, when the gesture slides in a unidirectional manner and determine that the trigger time difference is within a preset time range;

determine that the type of the gesture that triggers the sensor is the zoom-in gesture, when the gesture comprises a first gesture and a second gesture, the first gesture and the second gesture slide apart from each other, and both the first gesture and the second gesture are slide gestures;

determine that the type of the gesture that triggers the sensor is zoom-out gesture, when the first gesture and the second gesture slide toward each other, and both the first gesture and the second gesture are slide gestures; and

determine that the type of the gesture that triggers the sensor is the rotate gesture, when a sliding direction of the first gesture is different from a sliding direction of the second gesture, a start terminal of the first gesture is different from a start terminal of the second gesture, an end terminal of the first gesture is different from an end terminal of the second gesture, and determine that both the first gesture and the second gesture are slide gestures and a difference between trigger time of the end terminal of the first gesture and that of the end terminal of the second gesture is less than a preset threshold.

14. A terminal, comprising:

a receiver configured to receive a collaboration request activation signal; and

a transmitter configured to send parameter information to a receiving terminal when the receiver receives the collaboration request activation signal such that the receiving terminal records, according to the received parameter information, an arrangement manner of terminals that participate in acquiring of gesture information, wherein the received parameter information comprises terminal screen sizes and screen resolutions, wherein multiple arrangement manners of the terminals are calculated based at least in part on the terminal screen sizes and the screen resolutions, wherein a user selection of one of the multiple arrangement manners is used as the arrangement manner of the terminals that participate in acquiring the gesture information, wherein the terminals that participate in the acquiring of the gesture information comprise all terminals or terminals sending parameter information, and all the terminals comprise the terminals sending parameter information and the receiving terminal, and wherein the transmitter is further configured to send detected sensor trigger information to the receiving terminal when it is detected that a sensor is triggered such that the receiving terminal determines, according to the recorded arrangement manner and the acquired sensor trigger information, a type of a gesture that triggers the sensor, and

wherein determining, by the receiving terminal according to the recorded arrangement manner and the acquired sensor trigger information, the type of the gesture that triggers the sensor comprises;

acquiring, by the receiving terminal, first sensor trigger information of each triggered terminal from at least one piece of sensor trigger information detected by each triggered terminal, wherein the first sensor trigger information is provided by the sensor of a highest priority of each terminal;

determining, by the receiving terminal, a sliding direction of the gesture and calculating a trigger time difference between adjacent triggered terminals along the sliding direction of the gesture according to the recorded arrangement manner and trigger time that is contained in the first sensor trigger information of each triggered terminal; and

determining, by the receiving terminal according to the sliding direction of the gesture and the trigger time difference, the type of the gesture that triggers the sensor.

15. The terminal according to claim 14, wherein the transmitter is configured to send a device configuration parameter to the receiving terminal such that the receiving terminal displays, in an interface according to a device configuration parameter of the receiving terminal, received device configuration parameters of the terminals, and the number of all the terminals, at least one arrangement manner containing all the terminals, for a user to select, and records the arrangement manner selected by the user.

16. The terminal according to claim 14, wherein the transmitter is configured to send a device configuration parameter to the receiving terminal such that the receiving terminal displays, in an interface according to received device configuration parameters of the terminals and the number of terminals sending device configuration parameters, at least one arrangement manner containing the terminals sending device configuration parameters for a user to select, and records the arrangement manner selected by the user.