Interactive system and method

• US 7,474,318 B2
• Filed: 05/28/2004
• Issued: 01/06/2009
• Est. Priority Date: 05/28/2004
• Status: Active Grant

• Current Assignee: National University of Singapore
• Original Assignee: National University of Singapore
• Inventors: Zhou, Zhi Ying, Pan, Jiun Horng, Cheok, Adrian David
• Primary Examiner(s): Nguyen; Phu K
• Application Number:
  US10/856,177
• Publication Number:
  US 20050276444A1
• Time in Patent Office: 1,684 Days
• Field of Search: 345/418, 345/419, 345/619, 345629-632, 345581-582, 345/586, 345/622, 345/651
• US Class Current: 345/632
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive system for interacting with a device in a mixed reality environment, the system comprising an object having at least two surfaces, each surface having a marker, an image capturing device to capture images of the object in a first scene, and a microprocessor configured to track the position and orientation of the object in the first scene by identifying a marker. In addition, the microprocessor is configured to respond to manipulation of the object causes the device to perform an associated operation.

Claims

1. An interactive system for interacting with a device in a mixed reality environment, the system comprising:

an object including a first surface and a second surface, wherein the first surface includes a first marker and the second surface includes a second marker;

an image capture device that captures images of the object in a first scene; and

a microprocessor that identifies an identified marker from the first and second markers to track a position and orientation of the object in the first scene, and that causes the device to perform an associated operation in response to manipulation of the object.

2. The system according to claim 1, wherein the microprocessor tracks the first and second to identify the identified marker for tracking the position and orientation of the object.

3. The system according to claim 2, wherein the first surface has a first tracking confidence, the second surface has a second tracking confidence, and the identified marker is identified by determining a highest tracking confidence of the first and second tracking confidences.

4. The system according to claim 3, wherein the first surface has a first occlusion of the first marker, the second surface has a second occlusion of the second marker, the first tracking confidence is determined by an extent of the first occlusion, and the second tracking confidence is determined by an extent of the second occlusion.

5. The system according to claim 1, wherein the microprocessor retrieves multimedia content associated with the identified marker, and generates a second scene including the multimedia content superimposed over the first scene in a relative position to the identified marker.

6. The system according to claim 5, wherein the system includes first and second objects, the object is one of the first and second objects, the first object is used as an anchor for the relative positioning of the multimedia content, and the second object is used to operate the device or a software application on the device.

7. The system according to claim 5, wherein the multimedia content includes a virtual representation of the device, or a user interface of a software application on the device.

8. The system according to claim 5, wherein the multimedia content includes two dimensional or three dimensional images, video, or audio information.

9. The system according to claim 5, wherein the multimedia content is superimposed over the first scene by rendering the multimedia content into the first scene, for every video frame of the second scene to be displayed.

10. The system according to claim 1, wherein the device includes a television, DVD player, lighting, or air conditioner.

11. The system according to claim 10, wherein the associated operation includes power on or off, volume control, dimming level control, or temperature control.

12. The system according to claim 1, wherein the device is a computer.

13. The system according to claim 12, wherein the microprocessor causes a software application on the computer to perform the associated operation.

14. The system according to claim 13, wherein the software application is a MP3 player, for playing a video clip or movie, a photo album to display digital photos, or Internet web browser.

15. The system according to claim 14, wherein the associated operation for the media player is playing, pausing, fast forwarding, or rewinding the video clip.

16. The system according to claim 15, wherein translational movement of the object left or right instructs the media player to rewind or fast forward the video clip.

17. The system according to claim 15, wherein rotating the object clockwise or anti-clockwise instructs the media player to rewind or fast forward the video clip.

18. The system according to claim 1, wherein the microprocessor uses a state transition model to determine the response to manipulation of the object, and to determine the associated operation.

19. The system according to claim 18, wherein each physical action and physical property of the object is virtually coupled to a virtual method and a virtual attribute.

20. The system according to claim 1, wherein the identified marker includes a discontinuous border that has a gap.

21. The system according to claim 20, wherein the identified marker includes a marker image within the border.

22. The system according to claim 21, wherein the marker image is a pattern.

23. The system according to claim 22, wherein the pattern is matched to an exemplar stored in a repository of exemplars.

24. The system according to claim 22, wherein the microprocessor identifies the identified marker if the border is partially occluded and if the pattern is not occluded.

25. The system according to claim 21, wherein the identified marker includes a background, and a border color of the border produces a high contrast to a background color of the background to enable the background to be separated from the identified marker by software.

26. The system according to claim 1, further comprising a display device to display the first scene at the same time the first scene is generated.

27. The system according to claim 26, wherein the display device is a monitor, television screen or LCD.

28. The system according to claim 26, wherein the display device is a view finder of the image capture device or a projector to project images or video.

29. The system according to claim 26, wherein a video frame rate of the display device is in the range of twelve to thirty frames per second.

30. The system according to claim 26, wherein the image capture device is mounted above the display device.

31. The system according to claim 26, where the image capture device and the display device face a user.

32. The system according to claim 31, wherein the manipulation of the object is by the user.

33. The system according to claim 1, wherein the first and second surfaces planar.

34. The system according to claim 33, wherein the first and second surfaces are joined together.

35. The system according to claim 33, wherein the object is a cube or a polyhedron.

36. The system according to claim 1, wherein the object is foldable.

37. The system according to claim 1, wherein the microprocessor is part of a desktop or mobile communications device.

38. The system according to claim 1, further comprising a display device to display the first scene, wherein the image capture device is a camera.

39. The system according to claim 38, wherein the camera is a CCD or CMOS video camera.

40. The system according to claim 38, wherein the camera, the microprocessor and the display device are provided in a single integrated unit.

41. The system according to claim 38, wherein the camera, the microprocessor and the display device are located in remote locations.

42. The system according to claim 1, wherein the position of the object is calculated in three dimensional space.

43. The system according to claim 42, further comprising a display device to display the first scene, wherein a positional relationship is estimated between the display device and the object.

44. The system according to claim 1, wherein the captured images are thresholded based on brightness characteristics.

45. The system according to claim 44, wherein contiguous dark areas in the captured images are identified using a connected components algorithm.

46. The system according to claim 45, wherein a contour seeking technique identifies an outline of the dark areas, and the outline provides a contour.

47. The system according to claim 46, wherein the contour is discarded if the contour does not contain four corners.

48. The system according to claim 46, wherein the contour is discarded if the contour contains an area of a wrong size.

49. The system according to claim 46, wherein straight lines are fitted to each side of the contour if the contour is a square contour.

50. The system according to claim 49, wherein intersections of the straight lines are used as estimates of corner positions.

51. The system according to claim 50, wherein a projective transformation is used to warp a region described by the corner positions to a standard shape.

52. The system according to claim 51, wherein the standard shape is cross-correlated with stored exemplars of markers to identify the identified marker and determine the orientation of the object.

53. The system according to claim 51, wherein the corner positions are used to identify a unique Euclidean transformation matrix relating to a position of the image capture device to the position of the identified marker.

54. The system according to claim 1, wherein the identified marker is unoccluded.

55. The system according to claim 1, wherein the identified marker is a predetermined shape.

56. The system according to claim 55, wherein at least a portion of the shape is recognized by the computer software to identify the identified marker.

57. The system according to claim 56, wherein the microprocessor determines a complete portion of the shape using a recognized portion of the shape.

58. The system according to claim 57, wherein the shape is a square.

59. The system according to claim 58, wherein the microprocessor determines that the shape is a square if one corner of the square is occluded.

60. An interactive system for interacting with a device in a mixed reality environment, the system comprising:

first and second objects, wherein the first object has first surfaces, the second object has second surfaces, the first surfaces each has a respective first marker, and the second surfaces each has a respective second marker;

an image capturing device to capture images of the objects in a first scene; and

a microprocessor that identifies a first identified marker from a first surface of the first object and a second identified marker from a second surface of the second object, tracks a position and orientation of the objects in the first scene by identifying the first and second identified markers, and causes the device to perform an associated operation in response to manipulation of the objects and an arrangement of the objects relative to each other.

61. The system according to claim 60, wherein the microprocessor tracks the first surfaces to identify the first identified marker for tracking the position and orientation of the first object, and tracks the second surfaces to identify the second identified marker for tracking the position and orientation of the second object.

62. The system according to claim 61, wherein the first surfaces each has a respective first tracking confidence and the first identified marker is identified by determining which of the first surfaces has a highest first tracking confidence, and the second surfaces each has a respective second tracking confidence and the second identified marker is identified by determining which of the second surfaces has a highest second tracking confidence.

63. The system according to claim 62, wherein the first surfaces each has a respective first occlusion for the respective first marker, the second surfaces each has a second occlusion of the respective second marker, the first surface with the highest first tracking confidence is determined by an extent of the first occlusions, and the second surface with the highest second tracking confidence is determined by an extent of the second occlusions.

64. The system according to claim 60, wherein the microprocessor retrieves multimedia content associated with the first and second identified markers, and generates a second scene including the multimedia content superimposed over the first scene in a relative position to the first and second identified markers, to provide the mixed reality environment to a user.

65. A method for interacting with a device in a mixed reality environment, the method comprising:

capturing images of an object having first and second surfaces, wherein the first surface includes a first marker and the second surface includes a second marker;

tracking a position and orientation of the object by identifying an identified marker from the first and second markers; and

causing the device to perform an associated operation in response to manipulation of the object.

66. The method according to claim 65, including identifying the identified marker by tracking the first and second surfaces for tracking the position and orientation of the object.

67. The method according to claim 66, including identifying the identified marker by determining a highest tracking confidence of first and second tracking confidences, wherein the first surface has the first tracking confidence, and the second surface has the second tracking confidence.

68. The method according to claim 67, including determining the highest tracking confidence by determining an extent of first and second occlusions, wherein the first surface has the first occlusion of the first marker, and the second surface has the second occlusion of the second marker.

69. The method according to claim 65, including retrieving multimedia content associated with the identified marker, and generating a second scene that includes the multimedia content superimposed over the first scene in a relative position to the identified marker, to provide the mixed reality environment to a user.

70. A method for interacting with a device in a mixed reality environment, the method comprising:

capturing images of first and second objects, wherein the first object has first surfaces, the second object has second surfaces, the first surfaces each has a respective first marker, and the second surfaces each has a respective second marker;

tracking a position and orientation of the objects by identifying a first identified marker from the first markers and a second identified marker from the second marker; and

causing the device to perform an associated operation in response to manipulation of the objects and an arrangement of the objects relative to each other.

71. The method according to claim 70, including identifying the first identified marker by tracking the first surfaces for tracking the position and orientation of the first object, and identifying the second identified marker by tracking the second surfaces for tracking the position and orientation of the second object.

72. The method according to claim 71, including identifying the first identified marker by determining a highest first tracking confidence of first tracking confidences and identifying the second identified marker by determining a highest second tracking confidence of second tracking confidences, wherein the first surfaces each has a respective first tracking confidence, and the second surfaces each has a respective second tracking confidence.

73. The method according to claim 72, including determining the highest first tracking confidence by determining an extent of first occlusions and determining the highest second tracking confidence by determining an extent of second occlusions, wherein the first surfaces each has a respective first occlusion of the respective first marker, and the second surfaces each has a respective second occlusion of the respective second marker.

74. The method according to claim 73, including retrieving multimedia content associated with the first and second identified markers, and generating a second scene that includes the multimedia content superimposed over the first scene in a relative position to the first and second identified markers, to provide the mixed reality environment to a user.

75. A computer readable medium having an encoded software application for interacting with a device in a mixed reality environment, the software application when executed on a computer causing a system to perform operations comprising:

receiving captured images of an object in a first scene from an image capturing device;

tracking a position and orientation of the object in the first scene by tracking first and second surfaces of the object, wherein the first surface includes a first marker and the second surface includes a second marker;

identifying an identified marker from the first and second markers; and

causing the device to perform an associated operation in response to manipulation of the object.

76. A computer readable medium for storing a image capturing program for interacting with a computer system in a mixed reality environment, the image capturing program comprising:

an image capture module to capture images of an object in a first scene; and

a tracking module to track a position and orientation of the object in the first scene by tracking first and second surfaces of the object, wherein the first surface includes a first marker and the second surface includes a second marker, and identifying an identified marker from the first and second marker;

wherein the computer system is made to perform an associated operation in response to manipulation of the object.

77. A computer-readable medium for storing computer-executable instructions, which when executed on a computer, cause a system to perform operations comprising:

receiving captured images of an object in a first scene from an image capturing device;

tracking a position and orientation of the object in the first scene by tracking first and second surfaces of the object, wherein the first surface includes a first marker and the second surface includes a second marker;

identifying an identified marker from the first and second markers; and

causing the device to perform an associated operation in response to manipulation of the object.