Dynamic Displays Based On User Interaction States

• US 20150070263A1
• Filed: 09/09/2013
• Published: 03/12/2015
• Est. Priority Date: 09/09/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Murillo, Oscar, Bailey, Richard
• Application Number:
  US14/021,968
• Publication Number:
  US 20150070263A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A system and method enabling dynamic interaction between users and displays. Interaction states for a user are determined by tracking user motions and position within the field of view of one or more capture devices. Interaction states are defined by any number of factors, including one or more of a user'"'"'s body position, body orientation. Once a user occupies an interaction state, an associated application layout is applied to a display. Application layout states may include which application objects are displayed for a given interaction state. Triggering an application state is driven by a transition event and a determination that a user occupies an interaction state. Monitoring user motion and position may be performed continuously, so that changes in interaction states can be determined and corresponding changes to application layout states can be applied to a display, thereby rendering the technology dynamic to user movement.

Claims

1. A computer implemented method of optimizing an experience with a processor controlled display, comprising:

tracking a human body in three dimensional space relative to a capture device;

responsive to the tracking, determining an interaction states occupied by the user human body, the interaction state defined by at least a body position and a body orientation relative to the display;

based on the interaction state, selecting an application layout state associated with the interaction state for the application, the application layout state defining application objects and a display layout for the application when the body occupies the interaction state; and

rendering application objects on the display based on the application layout state.

2. The computer implemented method of claim 1 wherein the interaction state is defined by two or more of factors selected from a body'"'"'s:

body position, dwell time in the body position, body orientation, dwell time in the body orientation, body range, dwell time at the body range, head range, dwell time at the head range, head position, dwell time at the head position, head orientation, dwell time at the head orientation, body motion, body appendage motion, body posture, body gaze and body auditory cues.

3. The computer implemented method of claim 2 wherein determining an interaction state includes retrieving a definition including a plurality of said factors and matching each of the factors to tracking data.

4. The computer implemented method of claim 2 wherein determining an interaction state includes retrieving a definition including a plurality of said factors and matching a subset of the factors to tracking data.

5. The computer implemented method of claim 1 further including determining a transition between the interaction state and a new interaction state based on a change in one or more of the position or orientation.

6. The computer implemented method of claim 2 further including determining a transition between the interaction state and a new interaction state based on a change in one or more of the plurality of said factors.

7. The computer implemented method of claim 6 wherein determining a transition includes determining a transition based on a change in said one or more of said factors by a first amount to generate a first transition event to the new interaction state, and determining a transition based on a change in said one or more of said factors by a second amount to return to the interaction state.

8. The computer implemented method of claim 1 wherein each application layout state includes one or more of which application objects are to be displayed sizes of the objects, perspective of the objects, the position of the objects, fonts, font sizes, a format of application objects displayed, a resolution of the display, a resolution of application objects, application controls presented, interactions presented/allowed, and alternative interfaces presented/allowed.

9. The computer implemented method of claim 1 wherein tracking is performed continuously and the determining, selecting and rendering are repeated based on a change in body position and orientation detected by tracking continuously.

10. A capture device, comprising:

an image capture device having a field of view;

a processing device coupled to the image capture device, the processing device including code instructing the processing device to;

retrieve image data of a human body in the field of view;

track the body position and movement in three dimensional space in the field of view;

determine an interaction state occupied by the body in the field of view, the interaction state defined by at least a body position and a body orientation relative to the capture device;

select an application layout state associated with the interaction state for an application executed by the processing device and in control of a display, the application layout state defining application objects and a display layout for the application when the body occupies the interaction state; and

render application objects based on the application layout state when the body is in the interaction state determined.

11. The capture device of claim 10 wherein the image data is continuously received that the code continuously tracks body position and movement performed continuously, the code determining, selecting and rendering being repeated based on a change in body position and orientation detected.

12. The capture device claim 11 wherein the interaction state is defined by two or more of factors selected from a body'"'"'s:

body position, dwell time in the body position, body orientation, dwell time in the body orientation, body range, dwell time at the body range, head range, dwell time at the head range, head position, dwell time at the head position, head orientation, dwell time at the head orientation, body motion, body appendage motion body posture, body gaze and body auditory cues.

13. The capture device of claim 12 further including determining a transition between the interaction state and a new interaction state based on a change in one or more of the plurality of said factors.

14. The capture device of claim 13 wherein determining a transition includes determining a transition based on a change in said one or more of the factors by a first amount to generate a first transition event to the new interaction state, and determining a transition based on a change in said one or more of said factors by a second amount to return to the interaction state.

15. The capture device of claim 14 wherein each application layout state includes one or more of which application objects are to be displayed sizes of the objects, perspective of the objects, the position of the objects, fonts, font sizes, a format of application objects displayed, a resolution of the display, a resolution of application objects, application controls presented, interactions presented/allowed, and alternative interfaces presented/allowed.

16. A depth and image capture processing device detecting movements of a body in a field of view and having an output coupled to control a display, the device including code instructing a processor to perform a method comprising:

tracking a body in three dimensional space relative to a capture device;

responsive to the tracking, determining an interaction state occupied by the body, the interaction state defined by at least a body position and a body orientation relative to the display;

selecting an application layout state associated with the interaction state for the application, the application layout state defining application objects and a display layout for the application when the body occupies the interaction state;

determining whether a transition event between the interaction state and a next interaction state has occurred by reference to one or more of a plurality of factors related to body position and orientation has changed;

responsive to the determining, determining a new interaction state occupied by the body; and

selecting a new application layout state associated with the new interaction state for the application.

17. The processing device of claim 16 wherein the interaction state is defined a plurality of factors selected from a body'"'"'s:

body position, dwell time in the body position, body orientation, dwell time in the body orientation, body range, dwell time at the body range, head range, dwell time at the head range, head position, dwell time at the head position, head orientation, dwell time at the head orientation, body motion, body appendage motion body posture, body gaze and body auditory cues.

18. The processing device of claim 17 wherein each application layout state includes one or more of which application objects are to be displayed sizes of the objects, perspective of the objects, the position of the objects, fonts, font sizes, a format of application objects displayed, a resolution of the display, a resolution of application objects, application controls presented, interactions presented/allowed, and alternative interfaces presented/allowed.

19. The processing device of claim 18 wherein determining a transition includes determining a transition based on a change in said one or more of the plurality of said factors by a first amount to generate a first transition event to the new interaction state, and determining a transition based on a change in said one or more of said plurality of factors by a second amount to return to the interaction state.

20. The processing device of claim 19 wherein tracking is performed continuously and the determining, selecting and rendering are repeated based on a change in body position and orientation detected by tracking continuously.