Using clear-coded, see-through objects to manipulate virtual objects

• US 7,576,725 B2
• Filed: 10/19/2004
• Issued: 08/18/2009
• Est. Priority Date: 10/19/2004
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Christina S., Gillett, Donald Marcus, Bathiche, Steven N., Yee, Dawson, Dehlin, Joel P., Kurlander, David J.
• Primary Examiner(s): Patel; Nitin
• Application Number:
  US10/969,746
• Publication Number:
  US 20060092170A1
• Time in Patent Office: 1,764 Days
• Field of Search: 345156-176, 178/18.01, 178/18.02, 178/19.01, 178/19.02, 463/25, 348/169
• US Class Current: 345/156
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/214 for locating contacts on a surface, e.g. floor mats or touch pads
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/1068 being specially adapted to detect the point of contact of the player on a surface, e.g. floor mat, touch pad
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6669 using a plurality of virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character change rooms
• CPC Class Codes: G06F 2203/04805 Virtual magnifying lens, i.e. window or frame movable on top of displayed information to enlarge it for better reading or selection
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An object placed on an interactive display surface is detected and its position and orientation are determined in response to IR light that is reflected from an encoded marking on the object. Upon detecting the object on an interactive display surface, a software program produces a virtual entity or image visible through the object to perform a predefined function. For example, the object may appear to magnify text visible through the object, or to translate a word or phrase from one language to another, so that the translated word or phrase is visible through the object. When the object is moved, the virtual entity or image that is visible through the object may move with it, or can control the function being performed. A plurality of such objects can each display a portion of an image, and when correctly positioned, together will display the entire image, like a jigsaw puzzle.

Claims

1. A system that includes objects that are generally optically transparent to visible light through which portions of an image are transmitted using visible light, so that as the objects are moved about, the portions of the image move with the objects, comprising:

(a) a source of non-visible light for illuminating the objects;

(b) an image source that produces images using visible light;

(c) a sensor for producing a signal in response to non-visible light that is received from the objects;

(d) a memory in which machine instructions are stored;

(e) a processor coupled to the image source, the sensor, and the memory, said processor executing the machine instructions to carry out a plurality of functions, including;

(i) responding with the sensor to a distinctive marking applied to each of the objects in the signal produced by the sensor, the marking comprising a material that substantially transmits visible light, but which affects light in a non-visible waveband from the non-visible light source, by one of;

(A) reflecting the light in the non-visible waveband; and

(B) absorbing the light in the non-visible waveband;

(ii) identifying and detecting a location and an orientation of each object, in response to the signal produced by the sensor that receives light in the non-visible waveband from the objects; and

(iii) causing the image source to transmit the portions of the image through the objects as visible light, so that each portion of the image is associated with the object through which the portion of the image is visible, so that when the objects are assembled in a correct layout, the image is clearly apparent by collectively viewing the portions of the image visible through the objects that are thus assembled, said objects representing the image in a manner similar to pieces of a jigsaw puzzle that are correctly assembled to display a picture.

2. The system of claim 1, wherein the machine instructions further cause the processor to execute software that enables the image to be selectively changed, so that portions of a different image produced by the image source are apparent through the objects.

3. The system of claim 1, wherein the machine instructions further cause the processor to initially randomly distribute the portions of the image among the objects, so that the image then not readily apparent from viewing the portions of the image.

4. The system of claim 1, wherein the machine instructions further cause the processor to detect one of two different distinctive markings applied to each of the objects, including a first marking that is associated with the portion of the image, and a second marking associated with a mirror version of the portion of the image that is associated with the first marking, so that:

(a) all of the objects have to oriented with first marking providing the non-visible light that is received by the sensor, to enable the image produced by the image source to be clearly apparent;

or alternatively,

(b) all of the objects have to be oriented with the second marking applied providing the non-visible light that is received by the sensor, to enable a mirror version of the image produced by the image source to be clearly apparent.

5. The system of claim 1, wherein the machine instructions further cause the processor to cause the image source to produce a sequence of images that are divided into portions associated with the objects and which are visible through each of the objects, so that when the objects are correctly assembled, the sequence of images is readily apparent from the visible light from the image source passing through the objects.

6. The system of claim 5, wherein the sequence of images comprises a video.

7. The system of claim 5, wherein the machine instructions further cause the processor to enable a user to employ one of:

(a) a personal image provided by a user for the image; and

(b) a sequence of personal images provided by a user for the sequence of images.

8. An object for providing input to carry out a predefined function, comprising an encoded marking applied to at least one surface of a material that is optically transparent to visible light comprising the object, the encoded marking serving to identify the object, its position on a display surface that is coupled to a computing system, and an orientation of the object on the display surface, the encoded marking comprising a material that substantially transmits visible light without affecting the visible light, but which modifies a non-visible light so that the object is readily identified in response to the modification of the non-visible light, identification of the object providing an input to a computing system that causes a function associated with the object to be implemented by the computing system that affects a virtual entity or image formed with visible light that passes through the object.

9. The object of claim 8, wherein a different encoded marking is applied to a different surface of the object, said different encoded marking serving to provide a different input to the computing system and causing a different predefined function associated with the object to be implemented by the computing system.

10. The object of claim 8, wherein a manipulation of the object modifies a parameter related to the function associated with the object and affects the virtual entity or image that is formed with the visible light that passes through the object.

11. A method for enabling an object having at least a substantial central part that is generally optically transparent to visible light, to be optically recognized in response to non-visible light received from the object without substantially affecting visible light transmission through the object, comprising the steps of:

(a) applying a first marking on a first surface of the object, said first marking comprising a material that substantially transmits visible light, but which affects light in a non-visible waveband by one of;

(i) reflecting the light in the non-visible waveband; and

(ii) absorbing the light in the non-visible waveband;

(b) directing light in the non-visible waveband onto the first surface of the object where the first marking is applied; and

(c) detecting an image of at least a portion of the object, in response to light in the non-visible waveband that is received from the object, the image enabling the object to be recognized.

12. The method of claim 11, further comprising the step of applying a second marking to a second surface of the object, said second marking comprising substantially the same material as the first marking, said first surface being separated from the second surface by:

(a) a reflective material that substantially transmits visible light, but reflects light in the non-visible waveband, if the material comprising the first marking and the second marking absorbs the light in the non-visible waveband;

or

(b) an absorptive material that substantially transmits visible light, but absorbs light in the non-visible waveband, if the material comprising the first marking and the second marking reflects the light in the non-visible waveband.

13. The method of claim 12, further comprising the step of determining a state of the object by detecting light in the non-visible waveband from one of the first marking and the second marking, the first marking and the second marking each indicating a different state of the object.

14. The method of claim 13, further comprising the step of using the state of the object that was determined to control a software application that produces visible light, which passes through the object.

15. The method of claim 12, further comprising the steps of:

(a) determining a change in an orientation of the object over time based upon the image produced with the light in the non-visible waveband from one of the first surface and the second surface; and

(b) as a function of the change in the orientation of the object over time, controlling a software application that produces visible light that passes through the object.

16. The method of claim 15, wherein the change in the orientation of the object based upon the non-visible light from the first marking causes a different control function to be applied to the software application than the change in the orientation of the object based upon the non-visible light from the second marking.

17. The method of claim 12, wherein the second marking is applied as an encoded pattern that is automatically recognized in the image of at least the portion of the object when the light in the non-visible waveband is received from the second surface.

18. The method of claim 11, wherein the first marking is applied as an encoded pattern that is automatically recognized in the image of at least the portion of the object when the light in the non-visible waveband is received from the first surface.

19. The method of claim 11, further comprising the steps of:

(a) determining an orientation of the object based upon the light in the non-visible waveband that is received from the first surface; and

(b) detecting a change in the orientation of the object over time, for providing an input to control a software application that produces visible light passing through the object.

20. The method of claim 19, wherein the change in the orientation of the object over time appears to change a magnification factor affecting an image formed by the software application with the visible light, so that the image is either enlarged or decreased in relative size, as the orientation of the object is changed, depending upon a direction in which the orientation of the object is changed.

21. The method of claim 19, further comprising the step of changing an orientation of an image formed by the visible light that passes through the object by the software application, in a manner corresponding with a change in the orientation of the object over time, so that the image formed with the visible light appears to move with the object when the orientation of the object is changed.

22. The method of claim 21, further comprising the step of changing a position of an image formed by the visible light that passes through the object by the software application, in a manner corresponding with a change in the position of the object, so that the image formed with the visible light appears to move with the object when a position of the object is changed.

23. The method of claim 22, further comprising, for each of a plurality of objects like the object, repeating the steps of

(a) applying the first marking on the first surface of each corresponding object;

(b) directing light in the non-visible waveband onto the first surface of each corresponding object where the first marking is applied; and

(c) detecting the image of at least the portion of each corresponding object, in response to light in the non-visible waveband that is received from each corresponding object, the image enabling each corresponding object to be recognized, and wherein the image formed with the visible light that passes through each of the plurality of objects by the software application comprises a portion of a larger image that is divided into a plurality of portions, further comprising the step of causing each of a plurality of portions of the larger image to be visible through a corresponding object so that the portions of the image appear as pieces of a jigsaw puzzle, which if properly assembled adjacent to each other, correctly present the larger image.

24. The method of claim 11, wherein a software program causes visible light passing through the object to form text corresponding to a translation of at least one word into a different language than the at least one word over which the object is positioned.

25. The method of claim 11, further comprising the steps of:

(a) displaying a first image with visible light produced in accord with a software program; and

(b) enabling the object to be placed over the first image; and

(c) transmitting visible light through the object to form a second image that is related to the first image, but is different than the first image.

26. The method of claim 25, further comprising the step of stacking a plurality of objects like the object, each of plurality of objects being associated with providing a different function related to modifying the first image, so that a composite of the functions provided by each of the plurality of objects is evident in the second image.

27. The method of claim 25, wherein the first image comprises a photograph, and wherein the second image comprises the photograph in which a modification has been applied, further comprising the step of enabling a user to change an orientation of the object to adjust the modification applied to the photograph visible through the object as the second image.

28. The method of claim 25, wherein the first image represents an entity having a first appearance, and wherein the second image represents the entity having a different appearance, because some feature of the entity in the first image is modified in the second image.

29. The method of claim 28, wherein the second image represents one of a different depth, a different layer, and a subsystem of the entity.

30. The method of claim 28, further comprising the step of enabling a user to move the object to control the modification of the feature.

31. A memory medium having machine executable instructions for carrying out the steps of claim 11.

32. A method for employing objects that are generally optically transparent to visible light and through which portions of an image are transmitted using visible light, so that as the objects are moved about, the portions of the image move with the objects, comprising the steps of:

(a) applying a distinctive marking to each of the objects, the marking comprising a material that substantially transmits visible light, but which affects light in a non-visible waveband by one of;

(i) reflecting the light in the non-visible waveband; and

(ii) absorbing the light in the non-visible waveband;

(b) directing light in the non-visible waveband onto the objects where the marking is applied;

(c) identifying and detecting a location and an orientation of each object, in response to light in the non-visible waveband that is received from the objects;

(d) transmitting the portions of the image through the objects as visible light, so that each portion of the image is associated with the object through which the portion of the image is visible; and

(e) enabling the objects to be assembled in a correct layout, so that the image is clearly apparent by collectively viewing the portions of the image visible through the objects that are thus assembled, said objects representing the image in a manner similar to pieces of a jigsaw puzzle that are correctly assembled to display a picture.

33. The method of claim 32, further comprising the step of enabling the image to be selectively changed using software, so that portions of a different image are apparent through the objects.

34. The method of claim 32, further comprising the step of initially randomly distributing the portions of the image among the objects, so that the image is then not readily apparent from viewing the portions of the image.

35. The method of claim 32, further comprising the step of applying different distinctive markings to an opposite surface of each of the objects from that on which the distinctive markings are applied, each different distinctive marking applied to each object being associated with a mirror portion of the corresponding to the portion of the image associated with the distinctive marking applied to the object, so that

(a) all of the objects have to be oriented with the surface having the distinctive markings applied providing the non-visible light that is received, to enable the image to be clearly apparent;

or alternatively,

(b) all of the objects have to be oriented with the opposite surface having the different distinctive marking applied providing the non-visible light that is received, to enable a mirror version of the image to be clearly apparent.

36. The method of claim 32, further comprising the step of causing a sequence of images to be divided into portions that are visible through each of the objects, so that when the objects are correctly assembled, the sequence of images is readily apparent from the visible light passing through the objects.

37. The method of claim 36, wherein the sequence of images comprises a video.

38. The method of claim 36, further comprising the step of enabling a user to employ one of:

(a) a personal image provided by the user for the image; and

(b) a sequence of personal images provided by the user for the sequence of images.

39. A memory medium having machine executable instructions for carrying out the steps of claim 32.

40. A system on which an object, which is generally optically transparent to visible light, is optically recognized in response to non-visible light received from the object, without substantially effecting visible light transmission through the object, comprising:

(a) a source of non-visible light for illuminating an object;

(b) an image source that produces an image using visible light;

(c) a sensor for producing a signal in response to non-visible light that is received from the object;

(d) a memory in which machine instructions are stored;

(e) a processor coupled to the image source, the sensor, and the memory, said processor executing the machine instructions to carry out a plurality of functions, including;

(i) detecting light received from the object with the sensor, producing the signal, wherein the object has a first marking applied to a first surface of the object, said first marking comprising a material that substantially transmits visible light, but which affects non-visible light from the source of non-visible light, by one of;

(A) reflecting the light in the non-visible waveband; and

(B) absorbing the light in the non-visible waveband; and

(ii) in response to the signal produced by the sensor, identifying the object.

41. The system of claim 40, wherein a second marking is applied to a second surface of the object, said second marking comprising substantially the same material as the first marking, said first surface being separated from the second surface by:

(a) a reflective material that substantially transmits visible light, but reflects light in the non-visible waveband, if the material comprising the first marking and the second marking absorbs the light in the non-visible waveband;

or

(b) an absorptive material that substantially transmits visible light, but absorbs light in the non-visible waveband, if the material comprising the first marking and the second marking reflects the light in the non-visible waveband.

42. The system of claim 41, wherein the machine instructions further cause the processor to determine a state of the object by detecting light in the non-visible waveband received by the sensor from one of the first marking and the second marking, the first marking and the second marking each indicating a different state of the object.

43. The system of claim 42, wherein the machine instructions further cause the processor to control a software application that is being executed and which is producing the image with the image source using the visible light that passes through the object.

44. The system of claim 41, wherein the machine instructions further cause the processor to:

(a) determine a change in an orientation of the object over time based upon the signal produced by the sensor in response to the light in the non-visible waveband received by the sensor from one of the first surface and the second surface; and

(b) as a function of the change in the orientation of the object over time, control a software application being executed by the processor that produces the image with the image source, using the visible light that passes through the object.

45. The system of claim 44, wherein the change in the orientation of the object based upon the non-visible light received by the sensor from the first marking causes a different control function to be applied to the software application by the processor than the change in the orientation of the object based upon the non-visible light received by the sensor from the second marking.

46. The system of claim 41, wherein the second marking is applied as an encoded pattern that is automatically recognized by the processor based upon the signal produced by the sensor in response to the light in the non-visible waveband received from the second surface.

47. The system of claim 40, wherein the first marking is applied as an encoded pattern that is automatically recognized by the processor based upon the signal produced by the sensor in response to the light in the non-visible waveband received from the first surface.

48. The system of claim 40, wherein the machine instructions further cause the processor to:

(a) determine an orientation of the object based the signal produced by the sensor in response to the light in the non-visible waveband that is received by the sensor from the first surface; and

(b) detect a change in the orientation of the object over time, to provide an input to control a software application that causes the image source to produce visible light that passes through the object.

49. The system of claim 48, wherein the machine instructions further cause the processor to respond to the change in the orientation of the object over time by causing the image source to apparently change a magnification factor affecting an image formed by the software application with the visible light passing through the object, so that the image is either enlarged or decreased in relative size, as the orientation of the object is changed, depending upon a direction in which the orientation of the object is changed.

50. The system of claim 48, wherein the machine instructions further cause the processor to cause the image source to change an orientation of an image formed by the visible light that passes through the object, as determined by the software application, in a manner corresponding with a change in the orientation of the object over time, so that the image formed with the visible light appears to move with the object when the orientation of the object is changed.

51. The system of claim 50, wherein the machine instructions further cause the processor to cause the image source to change a position of an image formed by the image source using the visible light that passes through the object, as determined by the software application, in a manner corresponding with a change in the position of the object, so that the image formed with the visible light appears to move with the object when a position of the object is changed.

52. The system of claim 51, wherein the machine instructions further cause the processor to repeat functions (e)(i)-(e)(ii) of claim 40, for each of a plurality of other objects, and wherein the image formed by the image source with the visible light that passes through the object, in accord with the software application, comprises a portion of a larger image that is divided into a plurality of portions, the processor further causing the image source to produce each of a plurality of portions of the larger image with visible light that passes through a corresponding other object, so that the portions of the image appear as pieces of a jigsaw puzzle, which if properly assembled adjacent to each other, correctly present the larger image.

53. The system of claim 40, wherein the machine instructions further cause the processor to cause the image source to produce text with the visible light that passes through the object corresponding to a translation of at least one word into a different language than the at least one word over which the object is positioned.

54. The system of claim 40, wherein the machine instructions further cause the processor to cause the image source to:

(a) display a first image with visible light from the image source that is controlled by a software program executed by the processor; and

(b) enabling the object to be placed over the first image; and

(c) display a second image with visible light from the image source that passes through the object, the second image being related to the first image, but different than the first image.

55. The system of claim 54, wherein the first image comprises a photograph, and wherein the second image comprises the photograph in which a modification has been applied, the machine instructions further causing the processor to respond to a change in an orientation of the object indicated by the signal from the sensor, by adjusting the modification applied to the photograph that is visible through the object as the second image.

56. The system of claim 55, wherein the machine instructions further causing the processor to enable a user to move the object to control the modification of the feature.

57. The system of claim 54, wherein the first image represents an entity having a first appearance, and wherein the second image represents the entity having a different appearance, because some feature of the entity in the first image is modified in the second image.

58. The system of claim 57, wherein the second image represents one of a different depth, a different layer, and a subsystem of the entity.