Architectures for processing of head tracking on a mobile device

• US 9,754,552 B2
• Filed: 06/17/2014
• Issued: 09/05/2017
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Jain, Kritarth, Kozlowski, Michal Marek, Sandige, Michael Lee, Leonard, Andrew Bartlett, Savastinuk, Paul, Roessler, Ross David, Heller, Geoffrey Scott
• Primary Examiner(s): Edouard, Patrick
• Assistant Examiner(s): Yeung, Matthew
• Application Number:
  US14/307,284
• Publication Number:
  US 20150364109A1
• Time in Patent Office: 1,176 Days
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

A tracking architecture is provided that enables data for gestures and head positions to be provided to both native and non-native clients on a computing device. A pipeline component can obtain the raw image data and sensor data and synchronize that data to be processed to determine, for example, location and/or motion data that may correspond to device input. The data can be processed by separate components, such as an event publisher and an event provider, each capable of filtering the location, motion, and/or raw sensor data to generate a set of event data. The event data then can be published to registered listeners or provided in response to polling requests. Head coordinates, gesture data, and other such information can be passed through one or more interface layers enabling the data to be processed by a non-native client on the device.

Claims

1. A computing device, comprising:

at least one processor;

memory including instructions that, when executed by the at least one processor that when executed provide;

a second hardware interface, of a plurality of hardware interfaces, the second hardware interface configured to accept image data from a camera, the image data indicating at least location of a head, the image data provided at a first rate;

a first hardware interface, of the plurality of hardware interfaces, the first hardware interface configured to accept motion data from a motion sensor, the motion data indicating at least motion of the computing device, the motion data provided at a second rate that is faster than the first rate;

a synchronizer configured to;

receive and store the motion data;

receive the image data; and

generate a data object that represents synchronized image and motion data, the data object based at least in part on motion data generated during a first time period and image data generated at an end of the first time period;

an engine for processing the data object to generate input data corresponding to at least the location of the head and the motion of the computing device;

an inter-process communication layer;

an event publisher configured to filter the input data to produce first filtered input data and publish, using at least the inter-process communication layer, the first filtered input data as first event data to a listener client executing on the computing device, the listener client registered to receive the first event data; and

an event provider configured to filter the input data to produce second filtered input data and provide, using at least the inter-process communication layer, the second filtered input data as second event data in response to a polling request from a pull client executing on the computing device,

wherein the listener client and the pull client are enabled to determine at least one of a change in head position or a gesture based at least in part upon at least one of the first event data or the second event data.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to provide:

a policy manager component configured to adjust operation of the computing device according to at least one policy, wherein the plurality of hardware interfaces, the synchronizer, and the engine are part of a data pipeline, and

wherein the policy manager component is configured to selectively activate the data pipeline in accordance with the at least one policy.

3. The computing device of claim 1, wherein the instructions when executed further cause the computing device to provide:

at least one interface layer configured to receive the polling request from the pull client, the interface layer configured to provide the second event data, the at least one interface layer further configured to send the second event data to the pull client in response to the polling request.

4. The computing device of claim 1, further comprising:

at least one first filter in the event publisher configured to receive first data from at least one hardware interface of the plurality of hardware interfaces and the input data from the engine; and

at least one second filter in the event provider configured to receive second data from at least one hardware interface of the plurality of hardware interfaces and the input data from the engine,

wherein the at least one first filter and the at least one second filter are each configured to smooth the input data from the engine based at least in part upon at least one of the first data or the second data.

21. The computing device of claim 1, wherein the inter-process communication layer is an ANDROID binder layer.

5. A computer-implemented method, comprising:

receiving motion sensor data indicating at least a motion of a computing device through a first hardware interface of the computing device, the motion sensor data being acquired during a first time period and at a first rate using at least one motion sensor of the computing device;

storing the motion sensor data to a storage component;

receiving image data indicating at least a location of a head through a second hardware interface of the computing device, the image data being acquired at an end of the first time period and at a second rate using a camera of the computing device, the second rate being slower than the first rate;

retrieving the motion sensor data from the storage component;

generating synchronized input data by processing, using at least one processor of the computing device, the image data and the motion sensor data, the synchronized input data including at least locations of feature points of the head and motion as represented by the motion sensor data;

transmitting the synchronized input data to a publisher component;

transmitting the synchronized input data to a polling component;

causing, using the publisher component and at least an inter-process communication layer, the synchronized input data to be available as first event data to at least one listener client, the at least one listener client registered to receive the first event data; and

providing, in response to a polling request from a pull client and using the polling component and at least the inter-process communication layer, second event data based at least in part on the synchronized input data.

6. The computer-implemented method of claim 5, further comprising:

storing the motion sensor data to a synchronizer component

storing time information indicating a time of receipt of the motion sensor data;

receiving the image data; and

transmitting the motion sensor data, with the time information, and the image data to an engine to generate the synchronized input data.

7. The computer-implemented method of claim 5, further comprising:

processing the synchronized input data using at least one first filter and the motion sensor data to smooth the synchronized input data to be used to generate the first event data; and

processing the synchronized input data using at least one second filter and the image data to smooth the synchronized input data to be used to generate the first event data.

8. The computer-implemented method of claim 7, further comprising:

filtering at least one of the synchronized input data or the motion sensor data using at least one of a motion adaptive fusion filter, a pass-through filter, a finite impulse response (FIR) filter, or a fixed filter.

9. The computer-implemented method of claim 5, further comprising:

determining at least one of a change in head position or a gesture based at least in part upon at least one of the first event data or the second event data.

10. The computer-implemented method of claim 5, further comprising:

analyzing, using the publishing component, at least one listener map to determine the at least one listener client to which to provide the first event data, the at least one listener map further including interface information for the at least one listener client.

11. The computer-implemented method of claim 5, further comprising:

determining, based at least in part upon the motion sensor data, that a determined motion sensor of the at least one motion sensor is in an error state; and

restarting the determined motion sensor in response to being in an error state,

wherein the processing of the image data and the motion sensor data continues while the determined motion sensor is restarting.

12. The computer-implemented method of claim 5, further comprising:

determining, based at least in part upon metadata received with the image data, that the camera is in an error state; and

resetting the second hardware interface in response to the camera being in an error state.

13. The computer-implemented method of claim 5, further comprising:

causing the second event data, provided in response to the polling request, to be passed through at least one interface layer and the inter-process communication layer, the second event data further capable of being passed through at least one language interface component for modifying the second event data to be received by the pull client.

14. The computer-implemented method of claim 5, further comprising:

registering, as a listener, at least one data manager component configured to listen for a specific type of at least one of the first event data or the second event data.

15. The computer-implemented method of claim 14, further comprising:

communicating with a policy service to determine how to process a request received from at least one of a listener client, a pull client, or a data manager.

16. A computing device, comprising:

at least one processor;

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive motion sensor data indicating at least a motion of the computing device through a first hardware interface of the computing device, the motion sensor data being acquired during a first time period and at a first rate using at least one motion sensor of the computing device;

store the motion sensor data to a storage component;

receive image data indicating at least a location of a head through a second hardware interface of the computing device, the image data being acquired at an end of the first time period and at a second rate using a camera of the computing device, the second rate being slower than the first rate;

retrieve the motion sensor data from the storage component;

generate synchronized input data by processing, using the at least one processor of the computing device, the image data and the motion sensor data, the synchronized input data including at least locations of feature points of the head and motion as represented by the motion sensor data;

provide, to at least one listener client and using an inter-process communication layer, the synchronized input data as first event data, the at least one listener client registered to receive the first event data; and

provide, in response to a polling request from a pull client and using the inter-process communication layer, the synchronized input data as second event data.

17. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

store the motion sensor data to a synchronizer component;

store time information indicating a time of receipt of the motion sensor data;

receive the image data; and

transmit the motion sensor data, with the time information, and the image data to an engine to generate the synchronized input data.

18. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

analyze the image data and the motion sensor data using at least one monitoring component to determine that at least one motion sensor or camera is in an error state, the at least one monitoring component configured to attempt to cause the at least one motion sensor or camera to move out of the error state.

19. The computing device of claim 16, wherein the at least one motion sensor includes at least one of a gyroscope, an accelerometer, an inertial sensor, or an electronic compass.

20. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

process the synchronized input data to generate at least one of the first event data or the second event data, the synchronized input data being processed to smooth the synchronized input data based at least in part upon the motion sensor data received through the first hardware interface, the synchronized input data being processed using at least one of a motion adaptive fusion filter, a pass-through filter, a finite impulse response (FIR) filter, or a fixed filter.