CORRELATING MOVEMENT INFORMATION RECEIVED FROM DIFFERENT SOURCES

• US 20130069931A1
• Filed: 09/15/2011
• Published: 03/21/2013
• Est. Priority Date: 09/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Benko, Hrvoje
• Granted Patent: US 9,939,888 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system is described herein which receives internal-assessed (IA) movement information from a mobile device. The system also receives external-assessed (EA) movement information from at least one monitoring system which captures a scene containing the mobile device. The system then compares the IA movement information with the EA movement information with respect to each candidate object in the scene. If the IA movement information matches the EA movement information for a particular candidate object, the system concludes that the candidate object is associated with the mobile device. For example, the object may correspond to a hand that holds the mobile device. The system can use the correlation results produced in the above-indicated manner to perform various environment-specific actions.

Claims

1. A method, implemented by computing functionality, for assessing movement of a mobile device associated with a subject in a scene, comprising:

receiving internal-assessed (IA) movement information from at least one position-determining device provided by the mobile device;

receiving external-assessed (EA) movement information from at least one monitoring system which captures the scene containing the subject and the mobile device;

comparing the IA movement information with the EA movement information with respect to a candidate object associated with the subject;

repeating, at least one time, said comparing with respect to another candidate object, each iteration of said comparing producing correlation results;

determining, based on the correlation results for each candidate object, at least one matching object which is associated with the mobile device; and

performing an environment-specific action based on an outcome of said determining,

said receiving of the IA movement information, receiving of the EA movement information, comparing, repeating, determining, and performing being implemented by the computing functionality.

2. The method of claim 1, wherein the subject comprises a human being, and where each candidate object corresponds to a part of the human being.

3. The method of claim 2, wherein at least one candidate object corresponds to a hand of the human being.

4. The method of claim 1, wherein the mobile device corresponds to a device that is held in a hand of a human being.

5. The method of claim 1, wherein the mobile device corresponds to a device that is associated with part of an anatomy of a human being.

6. The method of claim 1, wherein the monitoring system comprises an apparatus which generates the EA movement information by:

generating a depth image of the scene;

identifying each candidate object within the scene based on the depth image; and

assessing movement of each candidate object in the scene.

7. The method of claim 6, wherein the monitoring system identifies each candidate object by comparing the depth image with a model of the subject.

8. The method of claim 1, further comprising:

removing a contribution of gravity from the IA movement information, based on an orientation of the mobile device, to provide IA-corrected movement information,

wherein said comparing comprises comparing said IA-corrected movement information with the EA movement information.

9. The method of claim 8, wherein the orientation of the mobile device comprises an actual orientation of the mobile device that is assessed based on the IA movement information received from the mobile device.

10. The method of claim 8, wherein the orientation of the mobile device comprises a guessed orientation of the mobile device.

11. The method of claim 10, wherein the guessed orientation of the mobile device comprises an actual orientation of the mobile device if a matching object is identified based on use of the guessed orientation.

12. The method of claim 8, wherein said comparing comprises comparing said IA-corrected movement information with the EA movement information over a time window.

13. The method of claim 1, wherein said comparing comprises:

extracting at least one feature from the IA movement information, to provide at least one IA feature;

extracting at least one feature from the EA movement information, to provide at least one EA feature; and

comparing said at least one IA feature with said at least one EA feature.

14. The method of claim 1, wherein said performing comprising:

supplementing the EA movement information with the IA movement information to provide enhanced control information with respect to at least one matching object in the scene; and

using the enhanced control information to control operation of a target system.

15. The method of claim 14, wherein said supplementing comprises adding orientation information that is obtained from the IA movement information to the EA movement information, the orientation information pertaining to an orientation of said at least one matching object.

16. The method of claim 1, wherein said performing comprises fusing the EA movement information with the IA movement information to provide enhanced position information for at least one matching object in the scene.

17. A computer readable storage medium for storing computer readable instructions, the computer readable instructions providing a movement processing system when executed by one or more processing devices, the computer readable instructions comprising:

logic configured to receive first movement information from at least one position-determining device provided by a mobile device;

logic configured to receive second movement information from at least one other source besides the mobile device;

logic configured to compare the first movement information with the second movement information, producing correlation results; and

logic configured to identify a matching object which is associated with the mobile device, based on the correlation results.

18. The computer readable storage medium of claim 17, further comprising logic configured to remove a contribution of gravity from the first movement information prior to said comparing, based on an orientation of the mobile device, wherein the orientation of the mobile device comprises an actual orientation of the mobile device that is assessed based on the first movement information received from the mobile device.

19. The computer readable storage medium of claim 17, further comprising logic configured to remove a contribution of gravity from the first movement information prior to said comparing, based on an orientation of the mobile device, wherein the orientation of the mobile device comprises a guessed orientation of the mobile device.

20. A movement processing system, implemented by computing functionality, for assessing movement of a mobile device in a scene, comprising:

a correlation processing module, comprising;

logic configured to receive internal-assessed (IA) movement information from at least one position-determining device provided by the mobile device;

logic configured to receive external-assessed (EA) movement information from at least one monitoring system which captures the scene containing a human being and the mobile device;

logic configured to compare the IA movement information with the EA movement information with respect to a plurality of candidate parts of the human being, to produce correlation results for each candidate part; and

logic configured to determine, based on the correlation results for each candidate part, at least one matching part which is associated with the mobile device; and

at least one application module, comprising;

logic configured to supplement the EA movement information with the IA movement information to provide enhanced control information with respect to at least one matching candidate part in the scene; and

logic configured to use the enhanced control information to control operation of a target system.