ARCHITECTURES FOR INPUT TRACKING

• US 20150364109A1
• Filed: 06/17/2014
• Published: 12/17/2015
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sandige, Michael Lee, Heller, Geoffrey Scott, Jain, Kritarth, Kozlowski, Michal Marek, Leonard, Andrew Bartlett, Savastinuk, Paul, Roessler, Ross David
• Granted Patent: US 9,754,552 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

A tracking architecture is provided that enables data for gestures and head positions to be provided to both native and non-native clients on a computing device. A pipeline component can obtain the raw image data and sensor data and synchronize that data to be processed to determine, for example, location and/or motion data that may correspond to device input. The data can be processed by separate components, such as an event publisher and an event provider, each capable of filtering the location, motion, and/or raw sensor data to generate a set of event data. The event data then can be published to registered listeners or provided in response to polling requests. Head coordinates, gesture data, and other such information can be passed through one or more interface layers enabling the data to be processed by a non-native client on the device.

Claims

1. A computing device, comprising:

at least one processor;

memory including instructions that, when executed by the at least one processor that when executed provide;

a plurality of hardware interfaces, each hardware interface of the plurality of hardware interfaces configured to accept data from at least one of a camera or a motion sensor;

a synchronizer for storing sensor data from at least one hardware interface of the plurality of hardware interfaces until an image is received from a camera interface of the plurality of hardware interfaces;

an engine for processing the image and the sensor data to generate input data corresponding to at least one of a location of a head or a motion of the computing device;

an event publisher configured to filter the input data to produce first filtered input data and publish the first filtered input data as first event data to a listener client executing on the computing device, the listener client registered to receive event data published by the event publisher; and

an event provider configured to filter the input data to produce second filtered input data and provide the second filtered input data as second event data in response to a polling request from a pull client executing on the computing device,

wherein the listener client and the pull client are enabled to determine at least one of a change in head position or a gesture based at least in part upon at least one of the first event data or the second event data.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to provide:

a policy manager component configured to adjust the operation of the computing device according to at least one policy, wherein the plurality of hardware interfaces, the synchronizer, and the engine are part of a data pipeline, and

wherein the policy manager is configured to selectively activate the pipeline in accordance with the at least one policy.

3. The computing device of claim 1, wherein the instructions when executed further cause the computing device to provide:

at least one interface layer configured to receive the polling request from the pull client, the interface layer configured to cause the polling provider to provide the second event data, the at least one interface layer further configured to send the second event data to the pull client in response to the polling request.

4. The computing device of claim 1, further comprising:

at least one first filter in the event publisher configured to receive data from at least one hardware interface of the plurality of hardware interfaces and input data from the engine; and

at least one second filter in the event provider configured to receive data from at least one hardware interface of the plurality of hardware interfaces and input data from the engine,

wherein the at least one first filter and the at least one second filter are each configured to smooth the input data from the engine based at least in part upon the data received from the at least one hardware interface.

5. A computer-implemented method, comprising:

receiving sensor data through a first hardware interface, the sensor data being acquired using at least one motion sensor of a computing device;

receiving image data through a second hardware interface, the image data being acquired using a camera of the computing device;

processing, using at least one processor of the computing device, the image data and the sensor data to generate input data;

transmitting the input data to a publisher component and a polling component;

causing, using the publisher component, the input data to be available as first event data to at least one listener client, the at least one listener client registered to receive the first event data; and

providing, in response to a polling request from a pull client and using the polling component, second event data based at least in part on the input data.

6. The computer-implemented method of claim 5, further comprising:

storing the sensor data to a synchronizer component

storing time information indicating a time of receipt of the sensor data;

receiving the image data; and

transmitting the sensor data, with the time information, and the image data to an engine to generate the input data.

7. The computer-implemented method of claim 5, further comprising:

processing the input data using at least one first filter and the sensor data to smooth the input data to be used to generate the first event data; and

processing the input data using at least one second filter and the camera data to smooth the input data to be used to generate the first event data

8. The computer-implemented method of claim 7, wherein at least one of the first input data or the second input data is filtered using at least one of a motion adaptive fusion filter, a pass-through filter, a finite impulse response (FIR) filter, or a fixed filter.

9. The computer-implemented method of claim 5, wherein the input data corresponds to at least one of a location of a head or a motion of the computing device, and wherein the listener client and the pull client are enabled to determine at least one of a change in head position or a gesture based at least in part upon at least one of the first event data or the second event data.

10. The computer-implemented method of claim 5, further comprising:

analyzing, using the publishing component, at least one listener map to determine the at least one listener client to which to provide the first event data, the at least one listener map further including interface information for the at least one listener client.

11. The computer-implemented method of claim 5, further comprising:

determining, based at least in part upon the sensor data, that a determined motion sensor of the at least one motion sensor is in an error state; and

restarting the determined motion sensor in response to the determined device being in an error state,

wherein the processing of the image data and the sensor data continues while the determined motion sensor is restarting.

12. The computer-implemented method of claim 5, further comprising:

determining, based at least in part upon metadata received with the image data, that the camera is in an error state; and

resetting the second hardware interface in response to the camera being in an error state.

13. The computer-implemented method of claim 5, further comprising:

causing the second event data, provided in response to the polling request, to be passed through at least one interface layer and at least one inter-process communication (IPC) layer, the event data further capable of being passed through at least one language interface component for modifying the event data to be received by the pull client.

14. The computer-implemented method of claim 5, further comprising:

registering, as a listener, at least one data manager component configured to listen for a specific type of at least one of the first event data or the second event data.

15. The computer-implemented method of claim 14, further comprising:

communicating with a policy service to determine how to process a request received from at least one of a listener client, a pull client, or a data manager.

16. A computing device, comprising:

at least one processor;

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive sensor data through a first hardware interface, the sensor data being acquired using at least one motion sensor of a computing device;

receive image data through a second hardware interface, the image data being acquired using a camera of the computing device;

process, using the at least one processor of the computing device, the image data and the sensor data to generate input data;

provide, to at least one listener client, the input data as first event data, the at least one listener client registered to receive the first event data; and

provide, in response to a polling request from a pull client, the input data as second event data.

17. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

store the sensor data to a synchronizer component;

store time information indicating a time of receipt of the sensor data;

receive the image data; and

transmit the sensor data, with the time information, and the image data to an engine to generate the input data.

18. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

analyze the image data and the sensor data using at least one monitoring component to determine that at least one motion sensor or camera is in an error state, the at least one monitoring component configured to attempt to cause the at least one motion sensor or camera to move out of the error state.

19. The computing device of claim 16, wherein the at least one motion sensor includes at least one of a gyroscope, an accelerometer, an inertial sensor, or an electronic compass.

20. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

process the input data to generate at least one of the first event data or the second event data, the input data being processed to smooth the input data based at least in part upon the sensor data received through the first hardware interface, the input data being processed using at least one of a motion adaptive fusion filter, a pass-through filter, a finite impulse response (FIR) filter, or a fixed filter.