Interacting with an Environment Using a Parent Device and at Least One Companion Device

• US 20180164877A1
• Filed: 12/12/2016
• Published: 06/14/2018
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Miller, Quentin Simon Charles, Kohler, Jeffrey Alan
• Granted Patent: US 10,452,133 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: H04B 1/3827 Portable transceivers
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 84/12 WLAN [Wireless Local Area Networks]

Abstract

A technique is described herein for using a parent computing device and at least one companion computing device to simultaneously interact with an environment. In one implementation, the parent computing device progressively builds parent map information through its interaction with the environment. The technique leverages the parent map information to identify a current pose of the companion computing device. In one case, for instance, the parent computing device sends the parent map information to the companion computing device, and the companion computing device determines its current pose based on the parent map information in conjunction with sensor information collected by the companion computing device.

Claims

1. A companion computing device for interacting with a parent computing device, comprising:

a companion interaction component configured to send data to and/or receive data from the parent computing device via a companion communication component; and

at least one companion environment-sensing device configured to capture companion sensor information that describes an environment,

wherein parent map information, provided by the parent computing device, is used to determine a current pose of the companion computing device, in conjunction with the companion sensor information,

the companion computing device and the parent computing device simultaneously interacting with the environment to perform respective functions, and

the parent map information representing features in the environment with reference to a world coordinate system shared by the companion computing device and the parent computing device.

2. The companion computing device of claim 1,

wherein the companion interaction component is configured to receive the parent map information from the parent computing device,

wherein said at least one companion environment-sensing device includes:

at least one companion camera for capturing companion image information that depicts the environment; and

at least one companion inertial measurement unit for capturing companion movement information that represents movement of the companion computing device within the environment, and

wherein the companion computing device further includes a companion tracking component configured to determine the current pose of the companion computing device within the environment based on the parent map information, the companion image information and the companion movement information, to provide companion pose information.

3. The companion computing device of claim 2, wherein the companion interaction component is configured to receive new instances of the parent map information from the parent computing device on a periodic and/or event-driven basis.

4. The companion computing device of claim 2,

wherein the companion interaction component is configured to transfer the companion pose information to the parent computing device, and

wherein the parent computing device is configured to control a user experience based on the companion pose information.

5. The companion computing device of claim 1,

wherein the parent computing device is further configured to produce a presentation that includes a virtual object within the world coordinate system, from a vantage point of a current pose of the parent computing device identified by the parent computing device, and

wherein the companion computing device is further configured to produce a presentation that includes the virtual object within the world coordinate system, from a vantage point of the current pose of the companion computing device.

6. The companion computing device of claim 5, wherein the companion computing device is configured to control the virtual object in response to a control action performed by a user.

7. The companion computing device of claim 1,

wherein the companion computing device further includes a companion map-building component configured to build companion map information that represents the environment based on the companion sensor information,

wherein the companion interaction component is configured to transfer the companion map information to the parent computing device, and

wherein the parent computing device is configured to integrate the companion map information with the parent map information, with reference to the world coordinate system.

8. The companion computing device of claim 7,

wherein the companion interaction component is configured to receive an instance of the parent map information from the parent computing device, and

wherein the companion map-building component is configured to integrate the instance of the parent map information with the companion map information.

9. The companion computing device of claim 1,

wherein said at least one companion environment-sensing device includes at least one companion camera for capturing companion image information that depicts the environment,

wherein the companion interaction component is configured to transfer the companion image information to the parent computing device, and

wherein the parent computing device is configured to determine the current pose of the companion computing device based on the parent map information and the companion image information.

10. A system for interacting with an environment, comprising:

a parent computing device, including;

at least one parent camera for capturing parent image information that depicts the environment;

at least one parent inertial measurement unit for capturing parent movement information, the parent movement information representing movement of the parent computing device within the environment;

a parent map-building component configured to build parent map information that represents the environment based at least on the parent image information,

the parent map information representing features in the environment with reference to a world coordinate system;

a parent tracking component configured to track a pose of the parent computing device within the environment based on the parent map information, the parent image information and the parent movement information; and

a parent interaction component configured to send data to and/or receive data from a companion computing device via a parent communication component; and

a companion computing device, including;

a companion interaction component configured to send data to and/or receive data from the parent computing device via a companion communication component,

the system leveraging the same parent map information to determine a current pose of the companion computing device, and

the companion computing device and the parent computing device simultaneously interacting with the environment to perform respective functions.

11. The system of claim 10,

wherein the companion interaction component is configured to receive the parent map information from the parent computing device, and

wherein the companion computing device further includes:

at least one companion camera for capturing companion image information that depicts the environment;

at least one companion inertial measurement unit for capturing companion movement information that represents movement of the companion computing device within the environment; and

a companion tracking component configured to determine the current pose of the companion computing device within the environment based on the parent map information, the companion image information and the companion movement information, to provide companion pose information.

12. The system of claim 10,

wherein the companion computing device further includes a companion map-building component configured to build companion map information that represents the environment based at least on companion image information collected by the companion computing device,

wherein the companion interaction component is configured to transfer the companion map information to the parent computing device, and

wherein the parent computing device is configured to integrate the companion map information with the parent map information, with reference to the world coordinate system.

13. The system of claim 10, wherein the system includes two or more companion computing devices which interact with the parent computing device, with reference to the parent map information.

14. A method for using a companion computing device to interact with a parent computing device, comprising:

capturing companion sensor information, from at least one companion environment-sensing device, that describes an environment; and

applying the companion sensor information, in conjunction with parent map information, to identify a current pose of the companion computing device,

the parent map information being provided by the parent computing device,

the companion computing device and the parent computing device simultaneously interacting with the environment to perform respective functions, and

the parent map information representing features in the environment with reference to a world coordinate system shared by the companion computing device and the parent computing device.

15. The method of claim 14,

wherein said at least one companion environment-sensing device includes:

at least one companion camera for capturing companion image information that depicts the environment; and

at least one companion inertial measurement unit for capturing companion movement information that represents movement of the companion computing device within the environment,

wherein the method further includes, at the companion computing device;

receiving parent map information from the parent computing device; and

determining the current pose of the companion computing device within the environment based on the parent map information, the companion image information and the companion movement information, to provide companion pose information.

16. The method of claim 15, wherein said receiving map information comprises receiving new instances of the parent map information from the parent computing device on a periodic and/or event-driven basis.

17. The method of claim 14, further comprising, at the companion computing device, producing a presentation that includes a virtual object within the world coordinate system, from a vantage point of the current pose of the companion computing device,

the virtual object being viewable by the parent computing device from a vantage point of a current pose of the parent computing device identified by the parent computing device.

18. The method of claim 14, further comprising, at the companion computing device:

building companion map information that represents the environment based at least on the companion sensor information; and

transferring the companion map information to the parent computing device,

wherein the parent computing device is configured to integrate the companion map information with the parent map information, with reference to the world coordinate system.

19. The method of claim 18, further comprising, at the companion computing device:

receiving an instance of the parent map information from the parent computing device; and

integrating the instance of the parent map information with the companion map information.

20. The method of claim 14,

wherein said at least one companion environment-sensing device includes at least one companion camera for capturing companion image information that depicts the environment,

wherein the method further comprises transferring, by the companion computing device, the companion image information to the parent computing device, and

wherein the parent computing device is configured to determine the current pose of the companion computing device based on the parent map information and the companion image information.