ADAPTIVE EVENT RECOGNITION

• US 20140375545A1
• Filed: 06/25/2013
• Published: 12/25/2014
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ackerman, Nathan, Finocchio, Mark J., Hodge, Andrew Bert
• Granted Patent: US 8,988,345 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 1/3259 Power saving in cursor control device, e.g. mouse, joystick, trackball
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A system and related methods for adaptive event recognition are provided. In one example, a selected sensor of a head-mounted display device is operated at a first polling rate corresponding to a higher potential latency. Initial user-related information is received. Where the initial user-related information matches a pre-event, the selected sensor is operated at a second polling rate faster than the first polling rate and corresponding to a lower potential latency. Subsequent user-related information is received. Where the subsequent user-related information matches a selected target event, feedback associated with the selected target event is provided to the user via the head-mounted display device.

Claims

1. An adaptive event recognition system, comprising:

a display device operatively connected to a computing device, the display device including a display system and a plurality of input sensors; and

an adaptive event recognition program executed by a processor of the computing device, the adaptive event recognition program configured to;

operate a selected sensor of the plurality of input sensors at a first polling rate corresponding to a higher potential latency;

receive initial user-related information from the selected sensor;

determine whether the initial user-related information matches one of a plurality of pre-events, wherein each of the pre-events corresponds to one or more different patterns of pre-events, and each of the patterns leads to a different possible target event;

where the initial user-related information matches one of the plurality of pre-events, operate the selected sensor at a second polling rate that is faster than the first polling rate and that corresponds to a lower potential latency that is less than the higher potential latency;

receive subsequent user-related information from the selected sensor; and

where the subsequent user-related information matches a selected target event from among the different possible target events, provide feedback associated with the selected target event to a user via the display device.

2. The adaptive event recognition system of claim 1, wherein the adaptive event recognition program is further configured to, where the subsequent user-related information matches a predictive pre-event that is not the selected target event:

determine an estimated execution time at which the selected target event will occur; and

provide the feedback associated with the selected target event to the user either at the estimated execution time or prior to the estimated execution time.

3. The adaptive event recognition system of claim 1, wherein the adaptive event recognition program is further configured to, after receiving the initial user-related information, pre-fetch at least a portion of the feedback associated with the selected target event.

4. The adaptive event recognition system of claim 1, wherein the display device is a head-mounted display device.

5. The adaptive event recognition system of claim 1, wherein the selected target event comprises a hand gesture.

6. The adaptive event recognition system of claim 1, wherein the adaptive event recognition program is further configured to:

after receiving the initial user-related information and before receiving the subsequent user-related information, receive intermediate user-related information from the selected sensor;

determine whether the intermediate user-related information matches a pre-event from a subset of the plurality of pre-events;

where the intermediate user-related information matches a pre-event from the subset of the plurality of pre-events, operate the selected sensor at a third polling rate that is faster than the first polling rate and is lower than the second polling rate.

7. The adaptive event recognition system of claim 1, wherein the adaptive event recognition program is further configured to, where the initial user-related information is not received within a predetermined timeframe, control the selected sensor to operate at a timed out polling rate that is slower than the first polling rate.

8. The adaptive event recognition system of claim 1, wherein each of the patterns comprises a different sequence of the pre-events.

9. The adaptive event recognition system of claim 1, wherein the plurality of input sensors are selected from the group consisting of image sensors, position sensors, microphones, eye-tracking sensors, and biometric sensors.

10. A method for recognizing a selected target event, comprising:

in a display device comprising a plurality of input sensors, operating a selected sensor of the plurality of input sensors at a first polling rate corresponding to a higher potential latency;

receiving initial user-related information from the selected sensor;

determining whether the initial user-related information matches one of a plurality of pre-events, wherein each of the pre-events corresponds to one or more different patterns of pre-events, and each of the patterns leads to a different possible target event;

where the initial user-related information matches one of the plurality of pre-events, operating the selected sensor at a second polling rate that is faster than the first polling rate and that corresponds to a lower potential latency that is less than the higher potential latency;

receiving subsequent user-related information from the selected sensor; and

where the subsequent user-related information matches the selected target event from among the different possible target events, providing feedback associated with the selected target event to a user via the display device.

11. The method of claim 10, further comprising, where the subsequent user-related information matches a predictive pre-event that is not the selected target event:

determining an estimated execution time at which the selected target event will occur; and

providing the feedback associated with the selected target event to the user either at the estimated execution time or prior to the estimated execution time.

12. The method of claim 10, further comprising, after receiving the initial user-related information, pre-fetching at least a portion of the feedback associated with the selected target event.

13. The method of claim 10, wherein the display device is a head-mounted display device.

14. The method of claim 10, wherein the selected target event comprises a hand gesture.

15. The method of claim 10, further comprising:

after receiving the initial user-related information and before receiving the subsequent user-related information, receiving intermediate user-related information from the selected sensor;

determining whether the intermediate user-related information matches a pre-event from a subset of the plurality of pre-events;

where the intermediate user-related information matches a pre-event from the subset of the plurality of pre-events, operating the selected sensor at a third polling rate that is faster than the first polling rate and is lower than the second polling rate.

16. The method of claim 10, further comprising, where the initial user-related information is not received within a predetermined timeframe, controlling the selected sensor to operate at a timed out polling rate that is slower than the first polling rate.

17. The method of claim 10, wherein each of the patterns comprises a different sequence of the pre-events.

18. The method of claim 10, wherein the plurality of input sensors are selected from the group consisting of image sensors, position sensors, microphones, eye-tracking sensors, and biometric sensors.

19. A method for recognizing a selected target event, comprising:

in a display device comprising a plurality of sensors, operating a selected sensor of the plurality of sensors at a first polling rate corresponding to a higher potential latency;

receiving initial user-related information from the selected sensor;

determining whether the initial user-related information matches one of a plurality of pre-events, wherein each of the pre-events corresponds to one or more different patterns of pre-events, and each of the patterns leads to a different possible target event;

where the initial user-related information matches one of the plurality of pre-events, operating the selected sensor at a second polling rate that is faster than the first polling rate and that corresponds to a lower potential latency that is less than the higher potential latency;

receiving subsequent user-related information from the selected sensor;

where the subsequent user-related information matches a selected target event from among the different possible target events, providing feedback associated with the selected target event to the user via the display device; and

where the subsequent user-related information matches a predictive pre-event that is not the selected target event;

determining an estimated execution time at which the selected target event will occur; and

providing the feedback associated with the selected target event to the user either at the estimated execution time or prior to the estimated execution time.

20. The method of claim 19, further comprising, after receiving the initial user-related information, pre-fetching at least a portion of the feedback associated with the selected target event.