Interaction between objects and a virtual environment display

• US 20050245302A1
• Filed: 04/29/2004
• Published: 11/03/2005
• Est. Priority Date: 04/29/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dehlin, Joel P., Wilson, Andrew D., Chen, Christina Summer, Kurlander, David Joshua, Bathiche, Steven
• Granted Patent: US 7,394,459 B2
• Time in Patent Office: Days
• US Class Current: 463/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 3/00643 Electric board games; Electric features of board games electric word or number games A63F3/0421; computer chess G06F; electric raffle games A63F3/081
• CPC Class Codes: A63F 9/183 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive table has a display surface on which a physical object is disposed. A camera within the interactive table responds to infrared (IR) light reflected from the physical object enabling a location of the physical object on the display surface to be determined, so that the physical object appear part of a virtual environment displayed thereon. The physical object can be passive or active. An active object performs an active function, e.g., it can be self-propelled to move about on the display surface, or emit light or sound, or vibrate. The active object can be controlled by a user or the processor. The interactive table can project an image through a physical object on the display surface so the image appears part of the object. A virtual entity is preferably displayed at a position (and a size) to avoid visually interference with any physical object on the display surface.

Claims

1. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of:

(a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface; and

(b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including;

(i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and

(ii) a second mode in which the physical object responds to the virtual environment that is displayed after the physical object is detected.

2. The method of claim 1, further comprising the step of tracking any movement of the physical object relative to the display surface by sensing changes in the location of the physical object on the display surface.

3. The method of claim 2, wherein said step of producing the response to the physical object in the second mode includes the step of changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

4. The method of claim 1, wherein the step of producing the response in the first mode comprises the step of causing a virtual object in the virtual environment to interact with the physical object.

5. The method of claim 4, wherein the interaction of the virtual object with the physical object comprises the step of automatically preventing the virtual object from moving through a region in the virtual environment where the physical object is located atop the display surface.

6. The method of claim 1, further comprising the step of detecting at least one of a size and a shape of the physical object in the first mode, to enable the predefined response to the physical object within the virtual environment.

7. The method of claim 1, further comprising the step of identifying the physical object based upon the light reflected from the physical object, said light that is reflected being encoded with a predefined pattern that is optically detected below the display surface.

8. The method of claim 7, wherein the physical object to which the virtual environment responds in the first mode comprises a passive object that does not perform any active function.

9. The method of claim 7, wherein the physical object is an active object that actively performs an active function in connection with the virtual environment and the active function is perceptible to the user, said active object and the virtual environment interacting in either or both the first mode and the second mode.

10. The method of claim 9, further comprising the step of producing a signal in the virtual environment in the second mode, said signal being detected by the active object, causing the active object to perform the active function.

11. The method of claim 9, wherein the active function comprises the step of producing one of:

(a) a visually perceptible state of the active object;

(b) an audibly perceptible state of the active object;

(c) a movement of the active object over the display surface; and

(d) a vibration of at least a portion of the active object.

12. The method of claim 9, wherein the active object is mobile and moves itself over the display surface to interact with the virtual environment.

13. The method of claim 9, further comprising the step of enabling the active function of the active object to be controlled by the user.

14. The method of claim 9, further comprising the step of enabling the active function performed by the active object to be controlled by a software program that produces the virtual environment.

15. The method of claim 9, wherein electrical power is provided to the active object by one of a battery and a charged capacitor to enable the active object to perform the active function.

16. The method of claim 1, further comprising the step of displaying a projected image limited to the location of the physical object and visible through the physical object at the location of the physical object on the display surface so that the projected image visible through the physical object appears to be part of the physical object, but is part of an overall image of the virtual environment that is displayed on the display surface.

17. The method of claim 1, further comprising the steps of:

(a) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and

(b) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

18. The method of claim 17, further comprising the step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

19. The method of claim 17, further comprising the step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

20. The method of claim 17, further comprising the step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

21. The method of claim 17, further comprising the step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

22. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 1.

23. A system for enabling interaction between a virtual environment and a physical object, comprising:

(a) an interactive display that includes;

(i) a display surface on which the virtual environment is displayed; and

(ii) a sensor for detecting a physical object placed on the display surface;

(b) a processor that is coupled to the interactive display;

(c) a memory coupled to the processor, said memory storing machine instructions that when executed by the processor, cause the processor to carry out a plurality of functions, including;

(i) using the sensor, detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed; and

(ii) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including;

(1) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and

(2) a second mode in which the physical object responds to the virtual environment that is displayed after the physical object is detected.

24. The system of claim 23, wherein the sensor comprises a camera that is disposed below the display surface and detects light reflected from the physical object disposed at the location on the display surface.

25. The system of claim 23, wherein the virtual environment is projected onto the display surface from under the display surface.

26. The system of claim 24, wherein the machine instructions further cause the processor to process a signal from the camera for tracking any movement of the physical object relative to the display surface by sensing changes in the location of the physical object on the display surface.

27. The system of claim 26, wherein the processor produces the response to the physical object in the first mode by changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

28. The system of claim 23, wherein the processor produces the response to the physical object in the first mode by causing a virtual object in the virtual environment to interact with the physical object.

29. The system of claim 28, wherein processor causes the virtual object to interact with the physical object by automatically preventing the virtual object from moving through a region in the virtual environment, where the physical object is located within the virtual environment.

30. The system of claim 23, wherein the processor uses the sensor to detect at least one of a size and a shape of the physical object to enable the predefined response to the physical object to be carried out within the virtual environment.

31. The system of claim 23, wherein the machine instructions further cause the processor to use the sensor to identify the physical object based upon light reflected from the physical object, said light that is reflected being encoded with a predefined pattern that is optically detected by the sensor.

32. The system of claim 31, wherein the physical object to which the virtual environment responds in the first mode comprises a passive object that does not perform any active function.

33. The system of claim 31, wherein the physical object is an active object that performs an active function in connection with the virtual environment, and the active function is perceptible to the user, said active object and the virtual environment interacting in either or both the first mode and the second mode.

34. The system of claim 33, wherein the machine instructions further cause the processor to produce a signal with the interactive display that is detected by the active object, causing the active object to perform the active function in the second mode.

35. The system of claim 33, where the active function comprises one of:

(a) a visually perceptible state of the active object;

(b) an audibly perceptible state of the active object;

(c) a movement of the active object over the display surface; and

(d) a vibration of at least a portion of the active object.

36. The system of claim 33, wherein the active object is mobile and moves itself over the display surface to interact with the virtual environment.

37. The system of claim 33, wherein the active function of the active object is controlled by a user.

38. The system of claim 37, wherein the processor limits control of the active object by a user, to prevent undesired results.

39. The system of claim 33, wherein the active function performed by the active object is controlled by the processor in response to the machine instructions.

40. The system of claim 33, wherein electrical power is provided to the active object by one of a battery and a charged capacitor to enable the active object to perform the active function.

41. The system of claim 23, wherein the interactive display includes a projector for displaying a projected image limited to the location of the physical object and visible through the physical object at the location of the physical object on the display surface so that the projected image visible through the physical object appears to be part of the physical object, but is part of an overall image of the virtual environment that is displayed on the display surface.

42. The system of claim 23, wherein the machine instructions further cause the processor to display a virtual entity on the interactive display, wherein the virtual entity comprises one of a virtual object and information that is visible on the display surface at a position selected to avoid a visual interference between the virtual entity and the physical object disposed on the display surface.

43. The system of claim 42, wherein the machine instructions define a plurality of prioritized positions at which the virtual entity can be selectively displayed by the processor using the interactive display to avoid the visual interference with the physical object, a position having a higher priority being selected from the plurality of prioritized positions for display of the virtual entity to avoid the virtual entity visually interfering with the physical entity.

44. The system of claim 42, wherein the machine instructions further cause the processor to determine a position of a specific user who will be visually perceiving the virtual entity on the interactive display, so that the virtual entity is oriented and displayed at a position selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

45. The system of claim 42, wherein the machine instructions further cause the processor to reduce a size of the virtual entity to avoid visually interfering with the physical object at the position where the virtual entity is displayed on the display surface.

46. A method for interacting with a physical object that is disposed on a display surface when displaying a virtual entity on the display surface, comprising the steps of:

(a) sensing a location at which the physical object is disposed on the display surface and a region of the display surface that is thereby obscured by the physical object;

(b) automatically selecting a position for displaying the virtual entity, so that the position selected does not cause a visual interference between the virtual entity and the physical entity that is disposed on the display surface; and

(c) displaying the virtual entity at the position selected.

47. The method of claim 46, further comprising the step of determining an orientation of the virtual entity to facilitate a visual perception of the virtual entity by a user who is intended to visually perceive the virtual entity that is to be displayed on the display surface

48. The method of claim 46, further comprising the step of applying a prioritized list of positions when automatically selecting the position for displaying the virtual entity.

49. The method of claim 48, further comprising the steps of:

(a) determining an orientation of the virtual entity to facilitate a visual perception of the virtual entity by a user who is intended to visually perceive the virtual entity that is to be displayed on the display surface; and

(b) creating the prioritized list as a function of the orientation of the virtual entity thus determined.

50. The method of claim 46, wherein if the virtual entity cannot be displayed at any of the possible locations without the physical object visually interfering with the display of the virtual entity on the display surface, further comprising the steps of:

(a) resizing the virtual entity to reduce its size; and

(b) displaying the virtual entity as thus reduced in size at a position selected so that the physical object does not visually interfere with the virtual entity as thus reduced in size when displayed on the display surface at the position selected.

51. The method of claim 46, wherein if the virtual entity cannot be displayed at any of the possible locations without the physical object visually interfering with the display of the virtual entity on the display surface, further comprising the steps of:

(a) reformatting the virtual entity to change one of its shape and size; and

(b) displaying the virtual entity as thus changed at a position selected so that the physical object does not visually interfere with the virtual entity as thus changed when displayed on the display surface at the position selected.

52. The method of claim 46, wherein the virtual entity comprises one of:

(a) a virtual object; and

(b) information.

53. A memory medium storing machine executable instructions for carrying out the steps of claim 46.

54. A system that avoids visually interfering with a physical object that is disposed on a display surface when displaying a virtual entity on the display surface, comprising:

(a) an interactive display that includes;

(i) the display surface on which the virtual entity is to be displayed; and

(ii) a sensor for detecting the physical object placed on the display surface;

(b) a processor that is coupled to the interactive display;

(c) a memory coupled to the processor, said memory storing machine instructions that when executed by the processor, cause the processor to carry out a plurality of functions, including;

(i) using the sensor for sensing a location at which the physical object is disposed on the display surface and a region of the display surface that is thereby obscured by the physical object;

(ii) automatically selecting a position for displaying the virtual entity, so that the position selected does not cause a visual interference between the virtual entity and the physical entity that is disposed on the display surface; and

(iii) displaying the virtual entity at the position selected.

55. The system of claim 54, wherein the machine instructions further cause the processor to determine an orientation of the virtual entity to facilitate a visual perception of the virtual entity by a user who is intended to visually perceive the virtual entity that is to be displayed on the display surface.

56. The system of claim 54, wherein the machine instructions further cause the processor to use a prioritized list of possible locations for displaying the virtual entity, when automatically selecting the position for displaying the virtual entity.

57. The system of claim 56, wherein the machine instructions further cause the processor to:

(a) determine an orientation of the virtual entity to facilitate a visual perception of the virtual entity by a user who is intended to visually perceive the virtual entity that is to be displayed on the display surface; and

(b) create the prioritized list as a function of the orientation of the virtual entity thus determined.

58. The system of claim 54, wherein if the virtual entity cannot be displayed at any of the possible locations without the physical object visually interfering with the display of the virtual entity on the display surface, the machine instructions further cause the processor to:

(a) resize the virtual entity to reduce its size; and

(b) display the virtual entity as thus reduced in size, so that the physical object does not visually interfere with the virtual entity as thus reduced in size when displayed on the display surface at the position selected.

59. The system of claim 54, wherein if the virtual entity cannot be displayed at any of the possible locations without the physical object visually interfering with the display of the virtual entity on the display surface, the machine instructions further cause the processor to:

(a) reformat the virtual entity to change at least one of its shape and size; and

(b) display the virtual entity as thus changed, so that the physical object does not visually interfere with the virtual entity as thus changed, when displayed on the display surface at the position selected.

60. The system of claim 54, wherein the vial entity comprises one of:

(a) a virtual object; and

(b) information.