Projectors and depth cameras for deviceless augmented reality and interaction

• US 8,730,309 B2
• Filed: 06/21/2010
• Issued: 05/20/2014
• Est. Priority Date: 02/23/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew David, Benko, Hrvoje
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): PHAM, NAM D
• Application Number:
  US12/819,230
• Publication Number:
  US 20110205341A1
• Time in Patent Office: 1,429 Days
• Field of Search: 348/51
• US Class Current: 348/51
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

Architecture that combines multiple depth cameras and multiple projectors to cover a specified space (e.g., a room). The cameras and projectors are calibrated, allowing the development of a multi-dimensional (e.g., 3D) model of the objects in the space, as well as the ability to project graphics in a controlled fashion on the same objects. The architecture incorporates the depth data from all depth cameras, as well as color information, into a unified multi-dimensional model in combination with calibrated projectors. In order to provide visual continuity when transferring objects between different locations in the space, the user'"'"'s body can provide a canvas on which to project this interaction. As the user moves body parts in the space, without any other object, the body parts can serve as temporary “screens” for “in-transit” data.

Claims

1. A multi-dimensional modeling system, comprising:

an arrangement of depth-sensing cameras that sense objects in a physical space and output sensed information;

an arrangement of projectors that impose projections on surfaces in the physical space; and

a control component that processes the sensed information and the projections to map a virtual interactive space to the physical space to emulate an interactive display for interaction with the objects and the surfaces,

the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,

at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view.

2. The system of claim 1, further comprising a mapping component that maps portions of the sensed information to the projections to create a 3D physical coordinate system.

3. The system of claim 1, wherein the control component applies the sensed information, which is 3D sensed information, to the projections, which are 2D images.

4. The system of claim 1, wherein the cameras and the projectors are registered to the physical space.

5. The system of claim 1, wherein the control component facilitates projection of a spatial widget on a surface in the physical space, and interactive manipulation of the spatial widget is accomplished using a predetermined movement in the physical space.

6. The system of claim 1, wherein the sensed information of the depth-sensing cameras is utilized to construct a 3D mesh that the control component processes to simulate positioning of a virtual object relative to a user in the physical space.

7. The system of claim 6, wherein the 3D mesh is constructed in realtime and models a sensed portion of the physical space.

8. The system of claim 1, wherein the control component processes the sensed information to infer depth of the objects in the physical space.

9. The system of claim 1, wherein two surfaces in the physical space are related virtually based on physical user interaction with the two surfaces.

10. A multi-dimensional modeling system, comprising:

an arrangement of depth-sensing cameras that sense a distance value per pixel of objects in a physical space and output 3D sensed information, and an arrangement of projectors that impose 2D projections on surfaces in the physical space; and

a control component that processes the 3D sensed information and the projections into virtual camera images as created from perspective views of virtual cameras, the virtual camera images employed to emulate an interactive display for user interaction with the objects and surfaces in the physical space,

the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,

at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view.

11. The system of claim 10, wherein two surfaces in the physical space are related virtually based on physical user interaction with the two surfaces.

12. The system of claim 10, wherein the control component facilitates virtual picking of an object and virtual dropping of the object based on computed proximity of virtual interactive surfaces to the object.

13. The system of claim 10, wherein the control component processes the sensed information to infer distance of the objects in the physical space relative to views of the virtual cameras.

14. The system of claim 10, wherein the control component facilitates presentation of a spatial widget on an interactive surface in the physical space and interactive manipulation of the spatial widget using a predetermined physical movement.

15. A multi-dimensional modeling method, comprising:

sensing objects within a physical space to output sensed information using an arrangement of depth-sensing cameras;

projecting images on surfaces in the physical space using an arrangement of projectors;

creating and overlaying a virtual interactive space onto the physical space based on the sensed information and the images; and

presenting interactions with an object relative to a surface in the physical space,

the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,

at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view.

16. The method of claim 15, further comprising inferring depth of the objects in the physical space based on the sensed information.

17. The method of claim 15, further comprising generating a 3D mesh of the physical space in realtime based on registration of the cameras to the physical space and registration of the projectors to the physical space.

18. The method of claim 15, further comprising virtually relating two surfaces in the physical space based on connective interaction with the two surfaces.

19. The method of claim 15, further comprising virtually picking an object and virtually dropping the object based on computed proximity of virtual interactive surfaces to the object.

20. The method of claim 15, further comprising presenting a spatial widget on an interactive surface of the physical space and interactively manipulating the spatial widget using a predetermined physical movement.