Recognizing gesture inputs using distributed processing of sensor data from multiple sensors

• US 9,557,817 B2
• Filed: 08/15/2011
• Issued: 01/31/2017
• Est. Priority Date: 08/13/2010
• Status: Active Grant

• Current Assignee: Wickr Inc. (Amazon.com, Inc.)
• Original Assignee: Wickr Inc. (Amazon.com, Inc.)
• Inventors: Lemmey, Tara, Vonog, Stanislav, Surin, Nikolay
• Primary Examiner(s): Giesy, Adam R
• Assistant Examiner(s): KIYABU, KARIN A
• Application Number:
  US13/210,370
• Publication Number:
  US 20120038550A1
• Time in Patent Office: 1,996 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

The techniques discussed herein contemplate methods and systems for providing, for example, interactive virtual experiences that are initiated or controlled using user gestures. In embodiments, the techniques provide for gestures performed by users holding devices to be recognized and processed in a cloud computing environment such that the gestures produce a predefined desired result. According to one embodiment, a server communicates with a first device in a cloud computing environment, wherein the first device can detect surrounding devices, and an application program is executable by the server, wherein the application program is controlled by the first device and the output of the application program is directed by the server to one of the devices detected by the first device.

Claims

1. A computer implemented method of communicating gestures between client devices using multiple sensors, the method comprising:

receiving, by a remote server device operated as part of an experience platform, a portfolio of attributes including sensor capabilities and data processing capabilities of a plurality of client devices in communication with the remote server device via a communication network;

generating, by the remote server device, a distributed processing graph based on the received portfolio of attributes;

receiving, by the remote server device, raw sensor data from sensors of a first and a second client device of the plurality of client devices, the first client device associated with a first user, the second client device associated with a second user, the raw sensor data indicative of a physical gesture performed by the first user of the first client device, the physical gesture indicative of a request by the first user of the first client device to execute an operation at a third client device, the third client device associated with a third user, wherein the first, second, and third client devices are proximate in physical location to each other and are connected as part of an interactive experience hosted by the experience platform;

distributing, by the remote server device, the raw sensor data to the plurality of client devices based on the distributed processing graph;

receiving, by the remote server device, processed sensor data from the plurality of client devices;

recombining, by the remote server device, the processed sensor data received from the plurality of client devices;

recognizing, by the remote server device, a type of the physical gesture performed by the first user based on the recombined processed sensor data;

interpreting, by the remote server device, the operation to be executed at the third client device based on the type of physical gesture performed by the first user; and

transmitting, by the remote server device, a command signal configured to cause an application at the third client device to execute the operation.

2. The method of claim 1, wherein the third client device is located within a visual proximity of the first client device.

3. The method of claim 1, wherein the sensors include one or more of:

a camera;

a microphone;

a gyrometer;

a motion sensor;

a temperature sensor;

a pressure sensor; and

an accelerometer.

4. The method of claim 1, wherein the physical gesture is one or more of:

a facial expression of the first user;

a movement of the first client device in a particular manner by the first user;

a vocal expression of the first user;

an accelerated movement of the first client device by the first user;

or

an application of sensory or tactile pressure by the first user.

5. The method of claim 1, wherein the communication network operates in association with a cloud computing environment, the remote server device instantiated within the cloud computing environment that is physically remote to the first, second, and third client devices.

6. The method of claim 1, wherein the first, second, and third client devices are connected as members of a social networking platform at the time the remote server device receives the raw sensor data.

7. The method of claim 1, wherein the processed sensor data indicates, in addition to the physical gesture, a magnitude of assertion of the physical gesture.

8. The method of claim 7, wherein, upon receipt of the processed sensor data, the remote server device further interprets additional parameters for execution of the operation based on the indication of the magnitude of assertion of the physical gesture.

9. The method of claim 1, wherein the raw sensor data includes indicia of a plurality of physical gestures performed by the first user.

10. The method of claim 9, wherein each of the plurality of physical gestures performed by the first user characterize different parameters associated with a particular operation to be executed on the third client device.

11. The method of claim 1, further comprising:

transmitting, by the remote server device, a command signal configured to cause an application at a fourth client device to execute an operation;

wherein the fourth client device is associated with a fourth user and is remote in physical location to the third client device;

wherein the fourth client device is within visual proximity to another user;

wherein the command signal configured to cause the application at the fourth client device to execute the operation is transmitted using a low latency network protocol; and

wherein the execution of the operation at the fourth client device is substantially synchronized with the execution of the operation at the third client device.

12. The method of claim 1, further comprising:

receiving, by the remote server device, a signal from the first client device indicating a presence of the second client device within a proximity of the first client device.

13. The method of claim 1, further comprising:

incorporating, by the remote server device, the processed sensor data from at least two types of sensors;

wherein at least one of the at least two types of sensors is an optical sensor;

wherein the processed sensor data includes a visual capture of the first user of the first client device; and

interpreting, by the remote server device, a facial or bodily expression of the first user based on the incorporated processed sensor data.

14. A server device operated as part of an experience platform, the server device comprising:

one or more processors;

a network adapter through which to communicate with a plurality of client devices via a communication network;

a memory device coupled to the network adapter and configured to store computer code, which when executed by the one or more processors, cause the server device to;

receive a portfolio of attributes including sensor capabilities and data processing capabilities of the plurality of client devices;

generate a distributed processing graph based on the received portfolio of attributes;

receive raw sensor data from sensors associated with a first and a second client device of the plurality of client devices, the first client device associated with a first user, the second client device associated with a second user, the raw sensor data indicative of a physical gesture performed by the first user of the first client device, the physical gesture indicative of a request by the first user of the first client device to execute an operation at a third client device, the third client device associated with a third user, wherein the first, second, and third client devices are proximate in physical location to each other and are connected as part of an interactive experience hosted by the experience platform;

distribute the raw sensor data to the plurality of client devices based on the distributed processing graph;

receive processed sensor data from at least one of the plurality of client devices;

recombine the processed sensor data received from the at least one of the plurality of client devices;

recognize a type of the physical gesture performed by the first user based on the recombined processed sensor data;

interpret the operation to be executed at the third client device based on the type of physical gesture performed by the first user; and

transmit a command signal configured to cause an application at the third client device to execute the operation.

15. The server device of claim 14, wherein the third client device is located within a visual proximity of the first client device.

16. The server device of claim 14, wherein the sensors include one or more of:

a camera;

a microphone;

a gyrometer;

a motion sensor;

a temperature sensor;

a pressure sensor; and

an accelerometer.

17. The server device of claim 14, wherein the physical gesture is one or more of:

a facial expression of the first user;

a movement of the first client device in a particular manner by the first user;

a vocal expression of the first user;

an accelerated movement of the first client device by the first user;

or

an application of sensory or tactile pressure by the first user.

18. The server device of claim 14, wherein the communication network operates in association with a cloud computing environment, the server device instantiated within the cloud computing environment that is physically remote to the first, second, and third client devices.

19. The server device of claim 14, wherein the first, second, and third client devices are connected as members of a social networking platform at the time the server device receives the raw sensor data.

20. The server device of claim 14, wherein the processed sensor data indicates, in addition to the physical gesture, a magnitude of assertion of the physical gesture.

21. The server device of claim 20, wherein, upon receipt of the processed sensor data, the server device further interprets additional parameters for execution of the operation based on the indication of the magnitude of assertion of the physical gesture.

22. The server device of claim 14, wherein the raw sensor data includes indicia of a plurality of physical gestures performed by the first user.

23. The server device of claim 22, wherein each of the plurality of physical gestures performed by the first user characterize different parameters associated with a particular operation to be executed on the third client device.

24. The server device of claim 14, wherein the memory device is further configured to store additional computer code, which when executed by the one or more processors, cause the server device to further:

transmit a command signal configured to cause an application at a fourth client device to execute an operation;

wherein the fourth client device is associated with a fourth user and is remote in physical location to the third client device;

wherein the fourth client device is within visual proximity to another user;

wherein, the command signal configured to cause the application at the fourth client device to execute the operation is transmitted using a low latency network protocol; and

wherein the execution of the operation at the fourth client device is substantially synchronized with the execution of the operation at the third client device.

25. The server device of claim 14, wherein the memory device is further configured to store additional computer code, which when executed by the one or more processors, cause the server device to further:

receive a signal from the first client device indicating a presence of the second client device within a proximity of the first client device.

26. The server device of claim 14, wherein the memory device is further configured to store additional computer code, which when executed by the one or more processors, cause the server device to further:

incorporate the processed sensor data from at least two types of sensors;

wherein, at least one of the at least two types of sensors is an optical sensor;

wherein, the processed sensor data includes a visual capture of the first user of the first client device; and

interpret a facial or bodily expression of the first user based on the incorporated processed sensor data.

27. A system operated as part of an experience platform for communicating gestures between client devices, the system comprising:

means for receiving a portfolio of attributes including sensor capabilities and data processing capabilities of a plurality of client devices in communication with the system via a communication network;

means for generating a distributed processing graph based on the received portfolio of attributes;

means for receiving raw sensor data from sensors associated with a first and a second client device of the plurality of client devices, the first client device associated with a first user, the second client device associated with a second user, the raw sensor data indicative of a physical gesture performed by the first user of the first client device, the physical gesture indicative of a request by the first user of the first client device to execute an operation at a third client device, the third client device associated with a third user, wherein the first, second, and third client devices are proximate in physical location to each other and are connected as part of an interactive experience hosted by the experience platform;

means for distributing the raw sensor data to the plurality of client devices based on the distributed processing graph;

means for receiving processed sensor data from the plurality of client devices;

means for recombining the processed sensor data received from the plurality of client devices;

means for recognizing the type of physical gesture performed by the first user based on the recombined processed sensor data;

means for interpreting the operation to be executed at the third client device based on the type of physical gesture performed by the first user; and

means for transmitting a command signal configured to cause an application at the third client device to execute the operation.

28. The system of claim 27, further comprising:

means for receiving a signal from the first client device indicating a presence of the second client device within a proximity of the first client device.

29. The system of claim 27, further comprising:

means for transmitting a command signal configured to cause an application at a fourth client device to execute an operation;

wherein the fourth client device is associated with a fourth user and is remote in physical location to the third client device;

wherein the fourth client device is within visual proximity to another user; and

wherein the execution of the operation at the fourth client device is substantially synchronized with the execution of the operation at the third client device.