Mobile platform

• US 20050285878A1
• Filed: 05/28/2004
• Published: 12/29/2005
• Est. Priority Date: 05/28/2004
• Status: Abandoned Application

• Current Assignee: National University of Singapore
• Original Assignee: National University of Singapore
• Inventors: Ng, Guo Loong, Singh, Siddharth, Cheok, Adrian David
• Application Number:
  US10/857,048
• Publication Number:
  US 20050285878A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations

Abstract

A mobile platform for providing a mixed reality experience to a user via a mobile communications device of the user, the platform including an image capturing module to capture images of an item in a first scene, the item having at least one marker; a communications module to transmit the captured images to a server, and to receive images in a second scene from the server providing a mixed reality experience to the user. In addition, the second scene is generated by retrieving multimedia content associated with an identified marker, and superimposing the associated multimedia content over the first scene in a relative position to the identified marker.

Claims

1. A mobile platform for providing a mixed reality experience to a user via a mobile communications device of the user, the platform comprising:

an image capturing module to capture images of an item in a first scene, the item having at least one marker;

a communications module to transmit the captured images to a server, and to receive images in a second scene from the server providing a mixed reality experience to the user;

wherein the second scene is generated by retrieving multimedia content associated with an identified marker, and superimposing the associated multimedia content over the first scene in a relative position to the identified marker.

2. The platform according to claim 1, wherein the mobile communications device is a mobile phone, Personal Digital Assistant (PDA) or a PDA phone.

3. The platform according to claim 1, wherein the images are captured as still images or images which form a video stream.

4. The platform according to claim 1, wherein the item is a three dimensional object.

5. The platform according to claim 1, wherein the communications module communicates with the server via Bluetooth, 3G, GPRS, Wi-Fi IEEE 802.11b, WiMax, ZigBee, Ultrawideband, Mobile-Fi or any other wireless protocol.

6. The platform according to claim 5, wherein the images are communicated as data packets between the mobile communications device and the server.

7. The platform according to claim 1, wherein the image capturing module comprises an image adjusting tool to enable users to change the brightness, contrast and image resolution for capturing an image.

8. The platform according to claim 1, wherein the associated multimedia content are virtual objects.

9. The platform according to claim 8, wherein a maker is associated with more than one virtual object.

10. The platform according to claim 1, wherein the associated multimedia content is locally stored on the mobile communications device.

11. The platform according to claim 1, wherein the associated multimedia content is remotely stored on the server.

12. The platform according to claim 1, wherein the marker includes a discontinuous border that has a single gap.

13. The platform according to claim 12, wherein the marker comprises an image within the border.

14. The platform according to claim 13, wherein the image is a geometrical pattern.

15. The platform according to claim 14, wherein the pattern is matched to an exemplar stored in a repository of exemplars.

16. The platform according to claim 13, wherein the color of the border produces a high contrast to the background color of the marker, to enable the background to be separated by the server.

17. The platform according to claim 1, wherein the server is able to identify a marker if the border is partially occluded and if the pattern within the border is not occluded.

18. The platform according to claim 1, further comprising a display device to display the second scene at the same time the second scene is generated.

19. The platform according to claim 18, wherein the display device is a mobile phone screen, monitor, television screen or LCD.

20. The platform according to claim 19, wherein the video frame rate of the display device is in the range of twelve to thirty frames per second.

21. The platform according to claim 1, wherein multimedia content includes two dimensional or three dimensional images, video or audio information.

22. The platform according to claim 4, wherein at least two surfaces of the object are substantially planar.

23. The platform according to claim 22, wherein the at least two surfaces are joined together.

24. The platform according to claim 23, wherein the object is a cube or polyhedron.

25. The platform according to claim 1, wherein the image capturing module captures images using a camera.

26. The platform according to claim 25, wherein the camera is a CCD or CMOS video camera.

27. The platform according to claim 1, wherein the position of the item is calculated in three dimensional space.

28. The platform according to claim 27, wherein a positional relationship is estimated between the display device and the object.

29. The platform according to claim 1, wherein the captured image is thresholded.

30. The platform according to claim 29, wherein contiguous dark areas are identified using a connected components algorithm.

31. The platform according to claim 30, wherein a contour seeking technique is used to identify the outline of these dark areas.

32. The platform according to claim 31, wherein contours that do not contain four corners are discarded.

33. The platform according to claim 31, wherein contours that contain an area of the wrong size are discarded.

34. The platform according to claim 31, wherein straight lines are fitted to each side of a square contour.

35. The platform according to claim 34, wherein the intersections of the straight lines are used as estimates of corner positions.

36. The platform according to claim 35, wherein a projective transformation is used to warp the region described by the corner positions to a standard shape.

37. The platform according to claim 36, wherein the standard shape is cross-correlated with stored exemplars of markers to identify the marker and determine the orientation of the object.

38. The platform according to claim 35, wherein the corner positions are used to identify a unique Euclidean transformation matrix relating to the position of a display device displaying the second scene to the position of the marker.

48. The platform according to claim 5, where if communication between the mobile communications device and the server is via Bluetooth, a Logical Link Control and Adaptation Protocol (L2CAP) service is initialized.

49. The platform according to claim 48, wherein the mobile communications device discovers a server for providing a mixed reality experience to a user by searching for Bluetooth devices within the vicinity of the mobile communications device.

50. The platform according to claim 1, wherein the captured image is resized to 160×

120 pixels.

51. The platform according to claim 50, wherein the resized image is compressed using the JPEG compression algorithm.

52. The platform according to claim 1, wherein the marker is unoccluded to identify the marker.

53. The platform according to claim 1, wherein the marker is a predetermined shape.

54. The platform according to claim 53, wherein at least a portion of the shape is recognized by the server to identify the marker.

55. The platform according to claim 54, the server determines the complete predetermined shape of the marker using the recognized portion of the shape.

56. The platform according to claim 55, wherein the predetermined shape is a square.

57. The platform according to claim 56, wherein the server determines that the shape is a square if one corner of the square is occluded.

39. A mobile platform for providing a mixed reality experience to a user via a mobile communications device of the user, the platform comprising:

an image capturing module to capture images of an item in a first scene, the item having at least one marker; and

a graphics module to retrieve multimedia content associated with an identified marker, and generate a second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker, to provide a mixed reality experience to the user.

40. A server for providing a mixed reality experience to a user via a mobile communications device of the user, the server comprising:

a communications module to receive captured images of an item in a first scene from the mobile communications device, and to transmit images in a second scene to the mobile communications device providing a mixed reality experience to the user, the item having at least one marker; and

an image processing module to retrieve multimedia content associated with an identified marker, and to generate the second scene including the associated multimedia content superimposed over the first scene in a relative position to the identified marker.

41. A system for providing a mixed reality experience to a user via a mobile communications device of the user, the system comprising:

an item having at least one marker;

an image capturing module to capture images of the item in a first scene;

an image display module to display images in a second scene providing a mixed reality experience to the user;

wherein the second scene is generated by retrieving multimedia content associated with an identified marker, and superimposing the associated multimedia content over the first scene in a relative position to the identified marker.

42. A method for providing a mixed reality experience to a user via a mobile communications device of the user, the method comprising:

capturing images of an item having at least one marker, in a first scene;

displaying images in a second scene to provide a mixed reality experience to the user;

wherein the second scene is generated by retrieving multimedia content associated with an identified marker, and superimposing the associated multimedia content over the first scene in a relative position to the identified marker.

43. A mixed reality application for delivering messages to a user via a mobile communications device of the user, the application comprising:

an item having at least one marker;

an image capturing module to capture images of the item in a first scene;

an image display module to display images in a second scene providing a mixed reality experience to the user;

wherein the second scene is generated by retrieving a message associated with an identified marker, and superimposing the message over the first scene in a relative position to the identified marker.

44. The application according to claim 43, wherein the message is a reminder, e-mail, calendar entry or task to perform.

45. The application according to claim 43, wherein the item is magnetic or adhesive, to enable the item to be positioned on a refrigerator door or wall, respectively.

46. A mixed reality application for reading via a mobile communications device of a user, the application comprising:

a book having at least one marker on each page;

an image capturing module to capture images of at least one page in a first scene;

an image display module to display images in a second scene providing a mixed reality experience to the user;

wherein the second scene is generated by retrieving multimedia content associated with an identified marker, and superimposing the associated multimedia content over the first scene in a relative position to the identified marker.

47. The application according to claim 46, wherein the book is a catalogue.