Interaction with virtual objects based on determined restrictions

• US 10,037,626 B2
• Filed: 06/30/2016
• Issued: 07/31/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Poulos, Adam Gabriel, Brown, Cameron Graeme, Jackson, Andrew Austin, Mathey-Owens, Cheyne Rory Quin, Thomas, Michael Robert, Tomlin, Arthur
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Lhymn, Sarah
• Application Number:
  US15/199,657
• Publication Number:
  US 20180005443A1
• Time in Patent Office: 761 Days
• Field of Search: 345619, 345632, 345633
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Motion and/or rotation of an input mechanism can be tracked and/or analyzed to determine limits on a user'"'"'s range of motion and/or a user'"'"'s range of rotation in three-dimensional space. The user'"'"'s range of motion and/or the user'"'"'s range of rotation in three-dimensional space may be limited by a personal restriction for the user (e.g., a broken arm). The user'"'"'s range of motion and/or the user'"'"'s range of rotation in three-dimensional space may additionally or alternatively be limited by an environmental restriction (e.g., a physical object in a room). Accordingly, the techniques described herein can take steps to accommodate the personal restriction and/or the environmental restriction thereby optimizing user interactions involving the input mechanism and a virtual object.

Claims

1. A device comprising:

a display configured to present virtual content;

an interface communicatively coupled to an input mechanism that is configured to collect data associated with motion of the input mechanism based on a current location of a user, wherein the motion of the input mechanism occurs within a first three-dimensional coordinate space;

one or more processors communicatively coupled to the display; and

memory having computer-executable instructions stored thereon which, when executed by the one or more processors, cause the device to perform operations comprising;

receiving, via the interface and from the input mechanism, first data associated with first motion of the input mechanism that occurs within the first three-dimensional coordinate space;

determining a location of a physical object that restricts the first motion of the input mechanism based at least in part on the current location of the user;

analyzing the first data associated with the first motion of the input mechanism and the location of the physical object to determine at least part of a range of motion of the input mechanism;

determining a restricted area of the display, wherein the restricted area comprises a portion of a display area and an interaction with a virtual object presented in the restricted area is limited by the range of motion of the input mechanism;

adjusting, based at least in part on the restricted area of the display, a parameter that correlates an amount of motion of the input mechanism to an amount of motion of a virtual element presented by the display, wherein the motion of the virtual element occurs within a second three-dimensional coordinate space that is larger than the first three-dimensional coordinate space;

receiving, via the interface and from the input mechanism, second data associated with second motion of the input mechanism that occurs within the first three-dimensional coordinate space; and

converting, using the adjusted parameter, the second motion of the input mechanism that occurs within the first three-dimensional coordinate space into correlated motion of the virtual element that occurs within the second three-dimensional coordinate space,

wherein the correlated motion enables performance of the interaction with the virtual object presented in the restricted area of the display.

2. The device of claim 1, wherein the adjusting the parameter increases the amount of motion of the virtual element within the second three-dimensional coordinate space relative to the amount of motion of the input mechanism within the first three-dimensional coordinate space.

3. The device of claim 1, wherein:

the first three-dimensional coordinate space is associated with three-dimensional space within reach of an arm and a hand of the user;

the second three-dimensional coordinate space is associated with three-dimensional space within a view of the user; and

the restricted area is further limited by the range of motion of the input mechanism due to a detected impossibility or a detected difficulty for the user to move the input mechanism to a position in the first three-dimensional coordinate space that corresponds to a position within the restricted area of the second three-dimensional coordinate space.

4. The device of claim 1, wherein the display comprises a transparent display that presents virtual content in association with real-world content within a view of the user.

5. The device of claim 1, wherein:

the analyzing reveals that the first motion of the input mechanism is indicative of a number of repeated attempts to perform a same interaction between (i) the virtual element and (ii) the virtual object or another virtual object presented by the display; and

the operations further comprise;

determining that the number of repeated attempts exceeds a threshold number of attempts; and

determining that the restricted area is limited by the range of motion of the input mechanism based at least in part on the determining that the number of repeated attempts exceeds the threshold number of attempts.

6. The device of claim 1, wherein:

the analyzing reveals that the first motion of the input mechanism is associated with a period of time during which the user is attempting to perform an individual interaction; and

the operations further comprise;

determining that the period of time exceeds a threshold period of time; and

determining that the restricted area is limited by the range of motion of the input mechanism based at least in part on the determining that the period of time exceeds the threshold period of time.

7. The device of claim 1, wherein the data associated with the motion of the input mechanism comprises a change in position of the input mechanism over time.

8. The device of claim 1, wherein the virtual element comprises at least one of a cursor element or a pointer element.

9. The device of claim 1, wherein:

the input mechanism is further configured to collect data associated with rotation of the input mechanism, the data associated with the rotation of the input mechanism comprising a change in orientation of the input mechanism; and

the operations further comprise;

receiving, via the interface and from the input mechanism, third data associated with rotation of the input mechanism;

analyzing the third data associated with the rotation of the input mechanism to determine at least part of a range of rotation of the input mechanism; and

adjusting, based at least in part on the range of rotation, another parameter that increases an amount of rotation of the input mechanism relative to an amount of rotation of the virtual element presented by the display.

10. A device comprising:

one or more sensors configured to sense data associated with motion of an input mechanism within a first three-dimensional coordinate space and to sense data indicative of a location of a physical object within the first three-dimensional coordinate space relative to a current location of a user;

a display configured to present virtual content within a second three-dimensional coordinate space that is larger than the first three-dimensional coordinate space;

one or more processors communicatively coupled to the display; and

memory having computer-executable instructions stored thereon which, when executed by the one or more processors, cause the device to perform operations comprising;

determining a range of motion of the input mechanism within the first three-dimensional coordinate space that is limited by the location of the physical object relative to the current location of the user;

determining a restricted area of the display, wherein the restricted area comprises a portion of a display area and an interaction with a virtual object presented in the restricted area is limited by the range of motion of the input mechanism;

adjusting, based at least in part on the restricted area of the display, a parameter that correlates an amount of motion of the input mechanism within the first three-dimensional coordinate space to an amount of motion of a virtual element presented by the display within the second three-dimensional coordinate space;

receiving the data associated with the motion of the input mechanism; and

converting, using the adjusted parameter, the motion of the input mechanism within the first three-dimensional coordinate space into correlated motion of the virtual element within the second three-dimensional coordinate space, wherein the correlated motion enables performance of the interaction with the virtual object presented in the restricted area of the display.

11. The device of claim 10, wherein the adjusting the parameter increases the amount of motion of the virtual element within the second three-dimensional coordinate space relative to the amount of motion of the input mechanism within the first three-dimensional coordinate space.

12. The device of claim 11, wherein:

the first three-dimensional coordinate space is associated with three-dimensional space within reach of an arm and a hand of the user;

the second three-dimensional coordinate space is associated with three-dimensional space within a view of the user; and

the display comprises a transparent display that presents virtual content in association with real-world content within the view of the user.

13. A device comprising:

an interface communicatively coupled to an input mechanism that is configured to collect data associated with motion of the input mechanism within a first three-dimensional coordinate space;

a display configured to present virtual objects within a second three-dimensional coordinate space that is larger than the first three-dimensional coordinate space;

one or more processors communicatively coupled to the display; and

memory having computer-executable instructions stored thereon which, when executed by the one or more processors, cause the device to perform operations comprising;

receiving, via the interface and from the input mechanism, the data associated with the motion of the input mechanism;

determining a location of a physical object, relative to a current location of a user, that restricts the motion of the input mechanism;

building, based at least in part on the data associated with the motion of the input mechanism and the location of the physical object, a model of a range of motion of the input mechanism within the first three-dimensional coordinate space; and

mapping the range of motion of the input mechanism within the first three-dimensional coordinate space to the second three-dimensional coordinate space to define, for the second three-dimensional coordinate space;

an interaction area within which a first interaction with a first virtual object based on the motion of the input mechanism is not limited by the range of motion of the input mechanism; and

a restricted area within which a second interaction with a second virtual object based on the motion of the input mechanism is limited by the range of motion of the input mechanism; and

adjusting a parameter to increase an amount of motion of a virtual element within the second three-dimensional coordinate space that correlates to an amount of motion of the input mechanism in the first three-dimensional space so that the second interaction with the second virtual object based on the motion of the input mechanism is no longer limited by the range of motion of the input mechanism.

14. The device of claim 13, wherein the operations further comprise:

analyzing the data associated with the motion of the input mechanism;

determining that the data indicates a number of repeated attempts to perform the second interaction with the second virtual object;

determining that the number of repeated attempts exceeds a threshold number of attempts; and

determining that a restriction limits an ability to use the input mechanism to perform the second interaction with the second virtual object based at least in part on the determining that the number of repeated attempts exceeds the threshold number of attempts.

15. The device of claim 13, wherein the operations further comprise:

analyzing the data associated with the motion of the input mechanism;

determining that the data associated with the motion of the input mechanism is associated with a period of time during which the user is attempting to perform the second interaction with the second virtual object;

determining that the period of time exceeds a threshold period of time; and

determining that a restriction limits an ability to use the input mechanism to perform the second interaction with the second virtual object based at least in part on the determining that the period of time exceeds the threshold period of time.

16. The device of claim 13, the operations comprise:

storing the adjusted parameter in an interaction profile associated with the current location of the user;

determining that the user has returned to the current location at a later time; and

activating the interaction profile to enable subsequent interactions using the adjusted parameter.

17. The device of claim 13, wherein the operations further comprise one of:

re-positioning the second virtual object from the restricted area into the interaction area;

or

causing additional virtual objects to be presented in the interaction area and not the restricted area.

18. The device of claim 13, wherein:

the first three-dimensional coordinate space is associated with three-dimensional space within reach of an arm and a hand of the user;

the second three-dimensional coordinate space is associated with three-dimensional space within a view of the user; and

the display comprises a transparent display that presents virtual content in association with real-world content within the view of the user.