Interactive system and method

• US 20050276444A1
• Filed: 05/28/2004
• Published: 12/15/2005
• Est. Priority Date: 05/28/2004
• Status: Active Grant

• Current Assignee: National University of Singapore
• Original Assignee: National University of Singapore
• Inventors: Cheok, Adrian David, Zhou, Zhi Ying, Pan, Jiun Horng
• Granted Patent: US 7,474,318 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive system for interacting with a device in a mixed reality environment, the system comprising an object having at least two surfaces, each surface having a marker, an image capturing device to capture images of the object in a first scene, and a microprocessor configured to track the position and orientation of the object in the first scene by identifying a marker. In addition, the microprocessor is configured to respond to manipulation of the object causes the device to perform an associated operation.

Claims

1. An interactive system for interacting with a device in a mixed reality environment, the system comprising:

an object having at least two surfaces, each surface having a marker;

an image capturing device to capture images of the object in a first scene; and

a microprocessor configured to track the position and orientation of the object in the first scene by identifying a marker;

wherein the microprocessor is configured to respond to manipulation of the object by causing the device to perform an associated operation.

2. The system according to claim 1, wherein at least two surfaces of the object are tracked to identify a marker for tracking the position and orientation of the object.

3. The system according to claim 2, wherein the marker used for tracking the position and orientation of the object is identified on a surface with the highest tracking confidence.

4. The system according to claim 3, wherein the surface with the highest tracking confidence is determined according to the extent of occlusion of its marker.

5. The system according to claim 1, wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker.

6. The system according to claim 1, wherein the device includes a television, DVD player, lighting, or an air conditioner.

7. The system according to claim 6, wherein associated operations include power on or off, volume control, dimming level control or temperature control.

8. The system according to claim 1, wherein the device is a computer.

9. The system according to claim 8, wherein the microprocessor is configured to cause other software applications on the computer to perform associated actions or tasks.

10. The system according to claim 9, wherein other software applications include an MP3 player, a Media Player for playing video clips or movies, a Photo Album to display digital photos or an Internet web browser.

11. The system according to claim 10, wherein associated actions for the Media Player application include playing, pausing, fast forwarding or rewinding a video clip.

12. The system according to claim 11, wherein translational movement of the object left or right instructs the Media Player to rewind or fast forward.

13. The system according to claim 11, wherein rotating the object clockwise or anti-clockwise instructs the Media Player to rewind or fast forward.

14. The system according to claim 5, whereby if there is more one object, a first object is used as an anchor for relative positioning of the associated multimedia content, and a second object is used to operate the device or software application.

15. The system according to claim 5, wherein the associated multimedia content includes a virtual representation of the device, or the user interface of the software application.

16. The system according to claim 1, wherein the microprocessor is configured to utilize a state transition model to determine a response to a manipulation of the object, and to determine an associated operation for the device to perform.

17. The system according to claim 16, wherein each physical action and physical property of the object is virtually coupled to a virtual method and virtual attribute.

18. The system according to claim 1, wherein the marker includes a discontinuous border that has a single gap.

19. The system according to claim 18 wherein the marker comprises an image within the border.

20. The system according to claim 19 wherein the image is a geometrical pattern.

21. The system according to claim 20, wherein the pattern is matched to an exemplar stored in a repository of exemplars.

22. The system according to claim 19, wherein the color of the border produces a high contrast to the background color of the marker, to enable the background to be separated by the computer software.

23. The system according to claim 20, wherein the microprocessor is configured to identify a marker if the border is partially occluded and if the pattern within the border is not occluded.

24. The system according to claim 1, further comprising a display device to display the second scene at the same time the second scene is generated.

25. The system according to claim 24, wherein the display device is a monitor, television screen or LCD.

26. The system according to claim 24, wherein the display device is a view finder of the image capture device or a projector to project images or video.

27. The system according to claim 24, wherein the video frame rate of the display device is in the range of twelve to thirty frames per second.

28. The system according to claim 1, wherein the image capture device is mounted above the display device.

29. The system according to claim 28, where the image capture device and display device face the user.

30. The system according to claim 29, wherein the object is manipulated between the user and the display device.

31. The system according to claim 1, wherein multimedia content includes two dimensional or three dimensional images, video or audio information.

32. The system according to claim 1, wherein the at least two surfaces of the object are substantially planar.

33. The system according to claim 32, wherein the at least two surfaces are joined together.

34. The system according to claim 32, wherein the object is a cube or polyhedron.

35. The system according to claim 1, wherein the object is foldable.

36. The system according to claim 1, wherein the microprocessor is part of a desktop or mobile computing device such as a Personal Digital Assistant (PDA), mobile telephone or other mobile communications device.

37. The system according to claim 1, wherein the image capturing device is a camera.

38. The system according to claim 37, wherein the camera is a CCD or CMOS video camera.

39. The system according to claim 37, wherein the camera, microprocessor and display device are provided in a single integrated unit.

40. The system according to claim 37, wherein the camera, microprocessor and display device are located in remote locations.

41. The system according to claim 1, wherein the associated multimedia content is superimposed over the first scene by rendering the associated multimedia content into the first scene, for every video frame to be displayed.

42. The system according to claim 1, wherein the position of the object is calculated in three dimensional space.

43. The system according to claim 42, wherein a positional relationship is estimated between the display device and the object.

44. The system according to claim 1, wherein the captured image is thresholded.

45. The system according to claim 44, wherein contiguous dark areas are identified using a connected components algorithm.

46. The system according to claim 45, wherein a contour seeking technique is used to identify the outline of these dark areas.

47. The system according to claim 45 wherein contours that do not contain four corners are discarded.

48. The system according to claim 45, wherein contours that contain an area of the wrong size are discarded.

49. The system according to claim 45, wherein straight lines are fitted to each side of a square contour.

50. The system according to claim 49, wherein the intersections of the straight lines are used as estimates of corner positions.

51. The system according to claim 50, wherein a projective transformation is used to warp the region described by the corner positions to a standard shape.

52. The system according to claim 51, wherein the standard shape is cross-correlated with stored exemplars of markers to identify the marker and determine the orientation of the object.

53. The system according to claim 51, wherein the corner positions are used to identify a unique Euclidean transformation matrix relating to the position of the display device to the position of the marker.

69. The system according to claim 1, wherein the marker is unoccluded to identify the marker.

70. The system according to claim 1, wherein the marker is a predetermined shape.

71. The system according to claim 70, wherein at least a portion of the shape is recognized by the computer software to identify the marker.

72. The platform according to claim 71, the microprocessor is configured to determine the complete predetermined shape of the marker using the recognized portion of the shape.

73. The platform according to claim 72, wherein the predetermined shape is a square.

74. The platform according to claim 73, wherein the microprocessor is configured to determine that the shape is a square if one corner of the square is occluded.

54. An interactive system for interacting with a device in a mixed reality environment, the system comprising:

at least two objects, each object having at least two surfaces, each surface having a marker;

an image capturing device to capture images of the objects in a first scene; and

a microprocessor configured to track the position and orientation of the objects in the first scene by identifying a marker on each object;

wherein the microprocessor is configured to respond to manipulation of the objects and their arrangement relative to each other causes the device to perform an associated operation.

55. The system according to claim 54, wherein at least two surfaces of the object are tracked to identify a marker for tracking the position and orientation of the object.

56. The system according to claim 55, wherein the marker used for tracking the position and orientation of the object is identified on a surface with the highest tracking confidence.

57. The system according to claim 56, wherein the surface with the highest tracking confidence is determined according to the extent of occlusion of its marker.

58. The system according to claim 54, wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

59. A method for interacting with a device in a mixed reality environment, the method comprising:

capturing images of an object having at least two surfaces, each surface having a marker; and

tracking the position and orientation of the object by identifying a marker;

wherein in response to manipulation of the object, the device is made to perform an associated operation.

60. The method according to claim 59, wherein at least two surfaces of the object are tracked to identify a marker for tracking the position and orientation of the object.

61. The method according to claim 60, wherein the marker used for tracking the position and orientation of the object is identified on a surface with the highest tracking confidence.

62. The method according to claim 61, wherein the surface with the highest tracking confidence is determined according to the extent of occlusion of its marker.

63. The method according to claim 60, wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

64. A method for interacting with a device in a mixed reality environment, the method comprising:

capturing images of at least two objects, each object having at least two surfaces, each surface having a marker; and

tracking the position and orientation of the objects by identifying a marker on each object;

wherein in response to manipulation of the objects and their arrangement relative to each other, the device is made to perform an associated operation.

65. The method according to claim 64, wherein at least two surfaces of the object are tracked to identify a marker for tracking the position and orientation of the object.

66. The method according to claim 65, wherein the marker used for tracking the position and orientation of the object is identified on a surface with the highest tracking confidence.

67. The method according to claim 66, wherein the surface with the highest tracking confidence is determined according to the extent of occlusion of its marker.

68. The method according to claim 64, wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

75. A software application for interacting with a device in a mixed reality environment, the application comprising:

an image processing module to receive captured images of an object in a first scene from an image capturing device; and

a tracking module to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein the software application in response to manipulation of the object causes the device to perform an associated operation.

76. An image capturing device for interacting with a second device in a mixed reality environment, the device comprising:

an image capture module to capture images of an object in a first scene; and

a tracking module to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein in response to manipulation of the object, the second device is made to perform an associated operation.

77. A computer program product comprised of a computer-readable medium for carrying computer-executable instructions for:

receiving captured images of an object in a first scene from an image capturing device; and

tracking the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein in response to manipulation of the object, a device is made to perform an associated operation.