Interaction between objects and a virtual environment display

• US 7,394,459 B2
• Filed: 04/29/2004
• Issued: 07/01/2008
• Est. Priority Date: 04/29/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dehlin, Joel P., Wilson, Andrew D., Chen, Christina Summer, Kurlander, David Joshua, Bathiche, Steven
• Primary Examiner(s): TRAN, MY CHAU T
• Application Number:
  US10/834,675
• Publication Number:
  US 20050245302A1
• Time in Patent Office: 1,524 Days
• Field of Search: 345/156, 345/175, 345/632, 345/633, 345/168, 345/173, 715/700, 715/702, 715/866, 463/9, 463/14, 463/30, 463/31, 463/36, 463/37, 463/38, 463/39
• US Class Current: 345/175
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 3/00643 Electric board games; Electric features of board games electric word or number games A63F3/0421; computer chess G06F; electric raffle games A63F3/081
• CPC Class Codes: A63F 9/183 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive table has a display surface on which a physical object is disposed. A camera within the interactive table responds to infrared (IR) light reflected from the physical object enabling a location of the physical object on the display surface to be determined, so that the physical object appear part of a virtual environment displayed thereon. The physical object can be passive or active. An active object performs an active function, e.g., it can be self-propelled to move about on the display surface, or emit light or sound, or vibrate. The active object can be controlled by a user or the processor. The interactive table can project an image through a physical object on the display surface so the image appears part of the object. A virtual entity is preferably displayed at a position (and a size) to avoid visually interference with any physical object on the display surface.

Claims

1. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface;

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and

(ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment;

(c) tracking movement of the physical object relative to the display surface by sensing changes in location of the physical object on the display surface; and

(d) changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

2. The method of claim 1, wherein the step of producing the response in the first mode comprises the step of causing a virtual object in the virtual environment to interact with the physical object.

3. The method of claim 2, wherein the interaction of the virtual object with the physical object comprises the step of automatically preventing the virtual object from moving through a region in the virtual environment where the physical object is located atop the display surface.

4. The method of claim 1, further comprising the step of detecting at least one of a size and a shape of the physical object in the first mode, to enable the predefined response to the physical object within the virtual environment.

5. The method of claim 1, further comprising the step of identifying the physical object based upon the light reflected from the physical object, said light that is reflected being encoded with a predefined pattern that is optically detected below the display surface.

6. The method of claim 5, wherein the physical object to which the virtual environment responds in the first mode comprises a passive object that does not perform any active function.

7. The method of claim 5, wherein the physical object is an active object that actively performs an active function in connection with the virtual environment and the active function is perceptible to the user, said active object and the virtual environment interacting in either or both the first mode and the second mode.

8. The method of claim 7, further comprising the step of producing a signal in the virtual environment in the second mode, said signal being detected by the active object, causing the active object to perform the active function.

9. The method of claim 7, wherein the active function comprises the step of producing one of:

(a) a visually perceptible state of the active object;

(b) an audibly perceptible state of the active object;

(c) a movement of the active object over the display surface; and

(d) a vibration of at least a portion of the active object.

10. The method of claim 7, wherein the active object is mobile and moves itself over the display surface to interact with the virtual environment.

11. The method of claim 7, further comprising the step of enabling the active function of the active object to be controlled by the user.

12. The method of claim 7, further comprising the step of enabling the active function performed by the active object to be controlled by a software program that produces the virtual environment.

13. The method of claim 7, wherein electrical power is provided to the active object by one of a battery and a charged capacitor to enable the active object to perform the active function.

14. The method of claim 1, further comprising the step of displaying a projected image limited to the location of the physical object and visible through the physical object at the location of the physical object on the display surface so that the projected image visible through the physical object appears to be part of the physical object, but is part of an overall image of the virtual environment that is displayed on the display surface.

15. The method of claim 1, further comprising the steps of:

(a) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and

(b) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

16. The method of claim 15, further comprising the step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

17. The method of claim 15, further comprising the step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

18. The method of claim 15, further comprising the step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

19. The method of claim 15, further comprising the step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

20. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 1.

21. A method as recited in claim 1, wherein said active function comprises vibrating.

22. A method as recited in claim 1, wherein said active function comprises generating sound.

23. A method as recited in claim 1, wherein said active function comprises emitting light.

31. A method as recited in claim 1, wherein said active function comprises vibrating.

32. A method as recited in claim 1, wherein said active function comprises generating sound.

33. A method as recited in claim 1, wherein said active function comprises emitting light.

24. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface;

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and

(ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment;

(c) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and

(d) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

25. The method of claim 24, further comprising a step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

26. The method of claim 24, further comprising a step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

27. The method of claim 24, further comprising a step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

28. The method of claim 24, further comprising a step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

29. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 24.

30. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface;

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and

(ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, wherein the active function comprises at least one of vibrating, generating sound or emitting light, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment, the method further including;

(c) tracking movement of the physical object relative to the display surface by sensing changes in location of the physical object on the display surface; and

(d) changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

34. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 30.