Interaction between objects and a virtual environment display

• US 7,907,128 B2
• Filed: 04/25/2008
• Issued: 03/15/2011
• Est. Priority Date: 04/29/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dehlin, Joel P., Wilson, Andrew D., Chen, Christina Summer, Kurlander, David Joshua, Bathiche, Steven
• Primary Examiner(s): TRAN, MY CHAU T
• Application Number:
  US12/110,032
• Publication Number:
  US 20080231611A1
• Time in Patent Office: 1,054 Days
• Field of Search: 345/156, 345/168, 345/173, 345/175, 345/632, 345/633, 463/9, 463/14, 463/30, 463/31, 463/36, 463/37, 463/38, 463/39, 715/700, 715/702, 715/866, 715/863
• US Class Current: 345/175
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 3/00643 Electric board games; Electric features of board games electric word or number games A63F3/0421; computer chess G06F; electric raffle games A63F3/081
• CPC Class Codes: A63F 9/183 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive table has a display surface on which a physical object is disposed. A camera within the interactive table responds to infrared (IR) light reflected from the physical object enabling a location of the physical object on the display surface to be determined, so that the physical object appear part of a virtual environment displayed thereon. The physical object can be passive or active. An active object performs an active function, e.g., it can be self-propelled to move about on the display surface, or emit light or sound, or vibrate. The active object can be controlled by a user or the processor. The interactive table can project an image through a physical object on the display surface so the image appears part of the object. A virtual entity is preferably displayed at a position (and a size) to avoid visually interference with any physical object on the display surface.

Claims

1. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed adjacent a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from an opposite side of the display surface;

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in each of a first mode as well as a second mode, wherein utilizing the first mode or second mode depends upon a type of the physical object placed adjacent the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object adjacent the display surface; and

(ii) a second mode in which the physical object that is placed adjacent the display surface comprises an active object that is electrically powered and that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, and wherein the active object, which is placed adjacent the display surface, interacts with the virtual environment in the second mode by performing said active function adjacent the display surface automatically in response to the virtual environment producing a signal that is detected by the active object and which causes the active object to perform the active function as if the physical object were included in the virtual environment.

2. The method of claim 1, wherein the method further includes:

(c) changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

3. The method of claim 1, wherein the step of producing the response in the first mode comprises the step of causing a virtual object in the virtual environment to interact with the physical object.

4. The method of claim 3, wherein the interaction of the virtual object with the physical object comprises the step of automatically preventing the virtual object from moving through a region in the virtual environment where the physical object is located adjacent the display surface.

5. The method of claim 1, further comprising the step of detecting at least one of a size and a shape of the physical object in the first mode, to enable the predefined response to the physical object within the virtual environment.

6. The method of claim 1, further comprising the step of identifying the physical object based upon the light reflected from the physical object, said light that is reflected being encoded with a predefined pattern that is optically detected through the display surface.

7. The method of claim 6, wherein the physical object to which the virtual environment responds in the first mode comprises a passive object that does not perform any active function.

8. The method of claim 1, wherein the active object is mobile and moves itself over the display surface to interact with the virtual environment.

9. The method of claim 1, further comprising the step of enabling the active function of the active object to be controlled by the user.

10. The method of claim 1, further comprising the step of enabling the active function performed by the active object to be controlled by a software program that produces the virtual environment.

11. The method of claim 1, further comprising the steps of:

(a) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and

(b) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

12. The method of claim 11, further comprising the step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

13. The method of claim 11, further comprising the step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

14. The method of claim 11, further comprising the step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

15. The method of claim 11, further comprising the step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

16. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed adjacent a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object through the display surface;

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in each of a first mode as well as a second mode, wherein utilizing the first mode or second mode depends upon a type of the physical object placed adjacent the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object adjacent the display surface; and

(ii) a second mode in which the physical object that is placed adjacent the display surface comprises an active object that is electrically powered and that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment,

wherein the active function comprises at least one of vibrating, generating sound or emitting light, and

wherein the active object, which is placed adjacent the display surface, interacts with the virtual environment in the second mode by performing said active function adjacent the display surface automatically in response to the virtual environment producing a signal that is detected by the active object and which causes the active object to perform the active function as if the physical object were included in the virtual environment.

17. A method as recited in claim 16, wherein said active function comprises vibrating.

18. A method as recited in claim 16, wherein said active function comprises generating sound.

19. A method as recited in claim 16, wherein said active function comprises emitting light.

20. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

detecting the physical object and its location when the physical object is placed in a location atop a display surface that is displaying the virtual environment, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface, wherein the physical object includes an IR encoded pattern that makes the physical object distinguishable from at least one other physical object placed on the display surface, and wherein the IR encoded pattern is substantially invisible to a human eye; and

displaying a projected image limited to the location of the physical object, the projected image being visible through the physical object at said location and such that the projected image visible through the physical object appears to be part of the physical object, even though it is part of an overall image of the virtual environment that is displayed on the display surface.

21. A method as recited in claim 20, wherein the method further includes moving the projected image, while it is being projected, to follow movement of the physical object over the display surface and so that the projected image remains visible through the physical object even while the physical object is moved over the display surface.

22. A method as recited in claim 20, wherein the projected image is a static image.

23. A method as recited in claim 20, wherein the projected image comprises a video image comprising multiple different images.