Interactive system and method

• US 20050264555A1
• Filed: 05/28/2004
• Published: 12/01/2005
• Est. Priority Date: 05/28/2004
• Status: Active Grant

• Current Assignee: National University of Singapore
• Original Assignee: National University of Singapore
• Inventors: Cheok, Adrian David, Zhou, Zhi Ying, Pan, Jiun Horng
• Granted Patent: US 7,295,220 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive system for providing a mixed reality experience to a user, the system including an object having at least two surfaces, where each surface having a marker. The system also including an image capturing device to capture images of the object in a first scene and a microprocessor configured to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object and identifying at least one marker. In addition, the microprocessor is also configured retrieve multimedia content associated with an identified marker and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker and the microprocessor is configured to provide a mixed reality experience to a user using the second scene.

Claims

1. An interactive system for providing a mixed reality experience to a user, the system comprising:

an object having at least two surfaces, each surface having a marker;

an image capturing device to capture images of the object in a first scene; and

a microprocessor configured to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object and identifying at least one marker;

wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker; and

wherein the microprocessor is configured to provide a mixed reality experience to a user using the second scene.

2. The system according to claim 1, wherein the marker is unoccluded.

3. The system according to claim 1, wherein the marker is a predetermined shape.

4. The system according to claim 3, wherein the microprocessor is configured to identify the marker by recognizing at least a portion of the shape.

5. The system according to claim 4, wherein the microprocessor is configured to determine the complete predetermined shape of the marker using the recognized portion of the shape.

6. The system according to claim 3, wherein the predetermined shape is a square.

7. The system according to claim 6, wherein the microprocessor is configured to recognize at least two corners of the square to determine the marker.

8. The system according to claim 1, wherein the marker includes a discontinuous border that has a single gap.

9. The system according to claim 8, wherein the marker comprises an image within the border.

10. The system according to claim 9, wherein the image is a geometrical pattern.

11. The system according to claim 10, wherein the pattern is matched to an exemplar stored in a repository of exemplars.

12. The system according to claim 8, wherein the color of the border produces a high contrast to the background color of the marker, to enable the background to be separated by the computer software.

13. The system according to claim 12, wherein the border color is black and the background color is white.

14. The system according to claim 10, wherein the microprocessor is configured to identify a marker if the border is partially occluded and if the pattern within the border is not occluded.

15. The system according to claim 1, further comprising a display device to display the second scene at the same time the second scene is generated.

16. The system according to claim 15, wherein the display device is a monitor, television screen, LCD or PDP.

17. The system according to claim 15, wherein the display device is a view finder of the image capture device or a projector to project images or video.

18. The system according to claim 15, wherein the video frame rate of the display device is in the range of twelve to thirty frames per second.

19. The system according to claim 1, wherein the image capture device is mounted above the display device.

20. The system according to claim 19, where the image capture device and display device face the user.

21. The system according to claim 20, wherein the object is manipulated between the user and the display device.

22. The system according to claim 1, wherein multimedia content includes two dimensional or three dimensional images, video or audio information.

23. The system according to claim 1, wherein the at least two surfaces of the object are substantially planar.

24. The system according to claim 23, wherein the at least two surfaces are joined together.

25. The system according to claim 23, wherein the object is a cube or polyhedron.

26. The system according to claim 1, wherein the object is foldable.

27. The system according to claim 26, wherein the object is a foldable cube for storytelling.

28. The system according to claim 1, wherein the microprocessor is part of a desktop or mobile computing device such as a Personal Digital Assistant (PDA), mobile telephone or other mobile communications device.

29. The system according to claim 1, wherein the image capturing device is a camera.

30. The system according to claim 29, wherein the camera is a CCD or CMOS video camera.

31. The system according to claim 29, wherein the camera, microprocessor and display device is provided in a single integrated unit.

32. The system according to claim 29, wherein the camera, microprocessor and display device is located in remote locations.

33. The system according to claim 1, wherein the associated multimedia content is superimposed over the first scene by rendering the associated multimedia content into the first scene, for every video frame to be displayed.

34. The system according to claim 1, wherein the position of the object is calculated in three dimensional space.

35. The system according to claim 34, wherein a positional relationship is estimated between the display device and the object.

36. The system according to claim 1, wherein the captured image is thresholded.

37. The system according to claim 36, wherein contiguous dark areas are identified using a connected components algorithm.

38. The system according to claim 37, wherein a contour seeking technique is used to identify the outline of these dark areas.

39. The system according to claim 38, wherein contours that do not contain four corners are discarded.

40. The system according to claim 38, wherein contours that contain an area of the wrong size are discarded.

41. The system according to claim 38, wherein straight lines are fitted to each side of a square contour.

42. The system according to claim 41, wherein the intersections of the straight lines may be used as estimates of corner positions.

43. The system according to claim 42, wherein a projective transformation is used to warp the region described by the corner positions to a standard shape.

44. The system according to claim 43, wherein the standard shape is cross-correlated with stored exemplars of markers to identify the marker and determine the orientation of the object.

45. The system according to claim 42, wherein the corner positions are used to identify a unique Euclidean transformation matrix relating to the position of the display device to the position of the marker.

46. The system according to claim 1, wherein the interactive system is an interior design application.

47. An interactive system for providing a mixed reality experience to a user, the system comprising:

an image capturing device configured to capture images of an object in a first scene; and

a microprocessor configured to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object having a marker and identifying at least one marker;

wherein the microprocessor is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker; and

wherein the microprocessor is configured to provide a mixed reality experience to a user using the superposition of the multimedia content and the first scene.

48. A software application for providing a mixed reality experience to a user, the application comprising:

an image processing module to receive captured images of an object in a first scene from an image capturing device; and

a tracking module to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein the image processing module is configured to retrieve multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker; and

wherein the image processing module is configured to provide a mixed reality experience to a user using the second scene.

49. A retrieval module for retrieving virtual objects to provide a mixed reality experience to a user, the module comprising:

a data receiver to receive marker identification data related to an identified marker; and

a searching tool to search a virtual object database for a virtual object corresponding to the marker identification data;

whereby if a match is found, the virtual object is superimposed over a real scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

50. The module according to claim 49, wherein an identified marker has more than one corresponding virtual object.

51. A repository of associated multimedia content for providing a mixed reality experience to a user, the repository comprising:

an identification data field to identify each item of multimedia content; and

a content data field to store an item of multimedia content or storage location of an item of multimedia content;

wherein marker identification data related to an identified marker is searched against the identification data field, and multimedia content associated with the identified marker is retrieved to be superimposed over a real scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

52. A signal carrying mixed reality content for providing a mixed reality experience to a user, comprising:

a signal indicative of graphical information containing a real scene and multimedia content associated with an identified marker superimposed over the real scene in a relative position to the identified marker;

wherein the identified marker is identified by tracking at least two surfaces of an object having a marker on each surface, where one marker.

53. The signal according to claim 52, further comprising a signal indicative of audio information associated with an identified marker.

54. A tangible object to be used in an interactive system for providing a mixed reality experience to a user, the object comprising:

at least two surfaces; and

a marker on each surface, the marker including a discontinuous border and an image within the border;

wherein the discontinuity in the border indicates the alignment of the image within the border and orientation of the object.

55. The object according to claim 54, wherein the position and orientation of the object is tracked by tracking at least one surface of the object.

56. A marker for a tangible object to be used in an interactive system for providing a mixed reality experience to a user, the marker comprising:

a discontinuous border and an image within the border;

whereby to identify the marker in a scene, the border is located within the scene, and the image within the border is searched to find a matching image in an image repository.

57. The marker according to claim 56, wherein computer software retrieves multimedia content associated with the marker when it is identified.

58. A method for providing a mixed reality experience to a user, the method comprising:

capturing in a first scene, images of an object having at least two surfaces, each surface having a marker;

tracking the position and orientation of the object in the first scene by tracking at least two surfaces of the object;

identifying at least one marker;

wherein multimedia content associated with an identified marker is retrieved, and a second scene is generated including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

59. A method for tracking an object to provide a mixed reality experience to a user, the method comprising:

calculating a transformation matrix for each surface of the object having a marker;

identifying a surface having the highest tracking confidence; and

calculating the transformation matrix from a marker co-ordinate system to an object co-ordinate system based on the physical relationship of the identified surface and the object;

wherein the transformation matrix from the object co-ordinate system to a camera co-ordinate system is calculated by subtracting the object co-ordinate system from the corresponding transformation matrix of the identified surface.

60. The method according to claim 59, further comprising an initial step of capturing the normal of the top surface of the object in its rest position.

61. The method according to claim 60, further comprising the step of determining the top surface of the object by calculating the surface which produces the largest dot product using the transformation matrix.

62. The method according to claim 61, wherein during manipulation of the object, the top surface of the object is determined by continuously calculating the surface which produces the largest dot product using the transformation matrix.

63. An image capturing device to provide a mixed reality experience to a user, the device comprising:

an image processing module to receive captured images of an object in a first scene; and

a tracking module to track the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein the device retrieves multimedia content associated with an identified marker, and generates a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to a user.

64. A computer program product comprised of a computer-readable medium for carrying computer-executable instructions for:

receiving captured images of an object in a first scene from an image capturing device; and

tracking the position and orientation of the object in the first scene by tracking at least two surfaces of the object where each surface has a marker, and identifying at least one marker;

wherein multimedia content associated with an identified marker is retrieved, and a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker is generated, to provide a mixed reality experience to a user.