Interacting with an environment using a parent device and at least one companion device

• US 10,452,133 B2
• Filed: 12/12/2016
• Issued: 10/22/2019
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Miller, Quentin Simon Charles, Kohler, Jeffrey Alan
• Primary Examiner(s): Reed, Stephen T.
• Application Number:
  US15/375,195
• Publication Number:
  US 20180164877A1
• Time in Patent Office: 1,044 Days
• Field of Search: None
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: H04B 1/3827 Portable transceivers
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 84/12 WLAN [Wireless Local Area Networks]

Abstract

A technique is described herein for using a parent computing device and at least one companion computing device to simultaneously interact with an environment. In one implementation, the parent computing device progressively builds parent map information through its interaction with the environment. The technique leverages the parent map information to identify a current pose of the companion computing device. In one case, for instance, the parent computing device sends the parent map information to the companion computing device, and the companion computing device determines its current pose based on the parent map information in conjunction with sensor information collected by the companion computing device.

Claims

1. A companion computing device, comprising:

at least one companion environment-sensing device configured to capture companion sensor information that describes a local environment in which the companion computing device and a parent computing device are located;

a processing device; and

a storage resource storing machine-readable instructions which, when executed by the processing device, cause the processing device to;

send the companion sensor information to the parent computing device; and

receive, from the parent computing device, a current pose of the companion computing device determined by the parent computing device,

wherein the companion computing device and the parent computing device are configured to perform respective functions with respect to the local environment in which the companion computing device and the parent computing device are located, and

wherein the parent computing device determines the current pose of the companion computing device using parent map information representing features in the local environment with reference to a world coordinate system shared by the companion computing device and the parent computing device.

2. The companion computing device of claim 1,

wherein said at least one companion environment-sensing device includes:

at least one companion camera configured to capture companion image information that depicts the local environment; and

at least one companion inertial measurement unit configured to capture companion movement information that represents movement of the companion computing device within the local environment.

3. The companion computing device of claim 1, wherein the companion computing device lacks a tracking component for tracking a location of the companion computing device in the local environment.

4. The companion computing device of claim 1, wherein the companion computing device lacks a map-building component for building map information representing the local environment.

5. The companion computing device of claim 1, wherein the machine-readable instructions, when executed by the processing device, cause the processing device to:

produce a presentation that includes a virtual object within the world coordinate system, the presentation being shown from a vantage point of the current pose of the companion computing device as determined by the parent computing device.

6. The companion computing device of claim 1, wherein the machine-readable instructions, when executed by the processing device, cause the processing device to:

when a control action manipulates the current pose of the companion computing device, control a virtual object based at least on the current pose.

7. The companion computing device of claim 1, wherein the parent computing device is a head-mounted display and the companion computing device is a custom hand-held controller designed for use with the head-mounted display.

8. The companion computing device of claim 7, wherein the machine-readable instructions, when executed by the processing device, cause the processing device to:

send the companion sensor information from the custom hand-held controller to the head-mounted display over a local wireless connection; and

receive the current pose of the custom hand-held controller from the head-mounted display over the local wireless connection.

9. The companion computing device of claim 1, wherein the at least one companion environment-sensing device comprises a wide field-of-view red-green-blue camera.

10. A system for interacting with a local environment, the system comprising:

a parent computing device located in the local environment; and

a companion computing device located in the local environment,

the parent computing device including;

at least one parent camera configured to capture parent image information that depicts the local environment;

a parent processing device configured to;

build parent map information that represents the local environment based at least on the parent image information, the parent map information representing features in the local environment with reference to a coordinate system;

receive companion image information depicting the local environment from the companion computing device;

based at least on the parent map information and the companion image information, determine a pose of the companion computing device; and

send the pose of the companion computing device from the parent computing device to the companion computing device; and

the companion computing device including;

at least one companion camera configured to capture the companion image information; and

a companion processing device configured to;

send the companion image information to the parent computing device; and

receive, from the parent computing device, the pose of the companion computing device

the companion computing device and the parent computing device being configured to perform respective functions with respect to the local environment in which the companion computing device and the parent computing device are located.

11. The system of claim 10, wherein the companion processing device is further configured to:

present a mixed-reality scene of the local environment on a screen of the companion computing device, the mixed-reality scene representing a virtual object from a vantage point of the pose of the companion computing device as determined by the parent computing device.

12. The system of claim 11, wherein the parent processing device is further configured to:

based at least on the parent map information and the parent image information, determine a pose of the parent computing device; and

present another mixed-reality scene of the local environment using a display mechanism of the parent computing device, the another mixed-reality scene representing the virtual object from a vantage point of the pose of the parent computing device.

13. The system of claim 12, wherein the companion computing device is a handheld computing device and the parent computing device is a head-mounted display.

14. A method comprising:

obtaining companion sensor information describing a local environment as captured by at least one companion-environment sensing device of a companion computing device;

applying the companion sensor information, in conjunction with parent map information determined by a parent computing device in the local environment, to identify a current pose of the companion computing device; and

sending the current pose of the companion computing device to the companion computing device over a local wireless connection,

the companion computing device and the parent computing device being configured to perform respective functions with respect to the local environment, and

the parent map information representing features in the local environment with reference to a coordinate system shared by the companion computing device and the parent computing device.

15. The method of claim 14,

wherein the companion sensor information comprises image information captured by a camera of the companion computing device.

16. The method of claim 14, further comprising:

obtaining parent sensor information describing the local environment as captured by at least one parent-environment sensing device of the parent computing device; and

determining a current pose of the parent computing device based at least on the parent map information and the parent sensor information.

17. The method of claim 16, further comprising:

producing a presentation that includes a virtual object within the coordinate system from a vantage point of the current pose of the parent computing device.

18. The method of claim 17, wherein the virtual object corresponds to a graphical object displayed on a screen of the parent computing device.

19. The method of claim 17, further comprising:

receiving user input directed to the virtual object; and

manipulating the virtual object responsive to the user input.

20. The method of claim 14, performed by the parent computing device.