Projectors and depth cameras for deviceless augmented reality and interaction

• US 9,509,981 B2
• Filed: 05/19/2014
• Issued: 11/29/2016
• Est. Priority Date: 02/23/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew David, Benko, Hrvoje
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): PHAM, NAM D
• Application Number:
  US14/281,885
• Publication Number:
  US 20140253692A1
• Time in Patent Office: 925 Days
• Field of Search: 348/51
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

Architecture that combines multiple depth cameras and multiple projectors to cover a specified space (e.g., a room). The cameras and projectors are calibrated, allowing the development of a multi-dimensional (e.g., 3D) model of the objects in the space, as well as the ability to project graphics in a controlled fashion on the same objects. The architecture incorporates the depth data from all depth cameras, as well as color information, into a unified multi-dimensional model in combination with calibrated projectors. In order to provide visual continuity when transferring objects between different locations in the space, the user'"'"'s body can provide a canvas on which to project this interaction. As the user moves body parts in the space, without any other object, the body parts can serve as temporary “screens” for “in-transit” data.

Claims

1. A multi-dimensional modeling system, comprising:

an arrangement of depth-sensing cameras that sense objects in a physical space and output sensed information;

an arrangement of projectors that impose projections relative to the physical space, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,

at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view;

a mapping component that maps portions of the sensed information to the projections to create mapping information; and

a modeling component that creates a multi-dimensional model based on the mapping information, the multi-dimensional model describing a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the physical space.

2. The system of claim 1, wherein the mapping component maps portions of the sensed information to the projections to create a 3D physical coordinate system.

3. The system of claim 1, further comprising a control component that applies the sensed information, which is 3D sensed information, to the projections, which are 2D images.

4. The system of claim 1, wherein the cameras and the projectors are registered to the physical space.

5. The system of claim 1, further comprising a control component that facilitates projection of a spatial widget on a surface in the physical space, and wherein interactive manipulation of the spatial widget is accomplished using a predetermined movement in the physical space.

6. The system of claim 1, wherein the sensed information of the depth-sensing cameras is utilized to construct a 3D mesh, wherein the system comprises a control component that processes the 3D mesh to simulate positioning of a virtual object relative to a user in the physical space.

7. The system of claim 6, wherein the 3D mesh is constructed in realtime and models a sensed portion of the physical space.

8. The system of claim 1, further comprising a control component that processes the sensed information and the projections to map a virtual interactive space to the physical space to emulate an interactive display for interaction with the objects and the surfaces.

9. The system of claim 8, wherein the control component that processes the sensed information to infer depth of the objects in the physical space.

10. A multi-dimensional modeling method, comprising:

orienting an arrangement of depth-sensing cameras that sense objects in a physical space and output sensed information;

orienting an arrangement of projectors that impose projections relative to the physical space, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,

at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view;

mapping portions of the sensed information to the projections to create mapping information; and

generating a multi-dimensional model based on the mapping information, the multi-dimensional model describing a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the physical space.

11. The method of claim 10, further comprising mapping portions of the sensed information to the projections to create a 3D physical coordinate system.

12. The method of claim 10, further comprising applying the sensed information, which is 3D sensed information, to the projections, which are 2D images.

13. The method of claim 10, wherein the cameras and the projectors are registered to the physical space.

14. The method of claim 10, further comprising facilitating projection of a spatial widget on a surface in the physical space, and wherein interactive manipulation of the spatial widget is accomplished using a predetermined movement in the physical space.

15. A multi-dimensional modeling method, comprising:

orienting an arrangement of depth-sensing cameras to sense objects within a physical space;

orienting an arrangement of projectors to project images in the physical space via projections, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space, at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view; and

creating a multi-dimensional model that describes a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the space.

16. The method of claim 15, further comprising inferring depth of the objects in the physical space based on the sensed information.

17. The method of claim 15, further comprising generating a 3D mesh of the physical space in realtime based on registration of the cameras to the physical space and registration of the projectors to the physical space.

18. The method of claim 15, further comprising virtually relating two surfaces in the physical space based on connective interaction with the two surfaces.

19. The method of claim 15, further comprising virtually picking an object and virtually dropping the object based on computed proximity of virtual interactive surfaces to the object.

20. The method of claim 15, further comprising presenting a spatial widget on an interactive surface of the physical space and interactively manipulating the spatial widget using a predetermined physical movement.