Teleportal face-to-face system

• US 6,774,869 B2
• Filed: 12/22/2000
• Issued: 08/10/2004
• Est. Priority Date: 12/22/2000
• Status: Expired due to Fees

• Current Assignee: The Board of Trustees of Michigan State University (Michigan State University), University of Central Florida (State University System of Florida)
• Original Assignee: The Board of Trustees of Michigan State University (Michigan State University), University of Central Florida (State University System of Florida)
• Inventors: Rolland, Jannick P., Biocca, Frank
• Primary Examiner(s): Liang, Regina
• Assistant Examiner(s): Nguyen, Jennifer T.
• Application Number:
  US09/748,761
• Publication Number:
  US 20020080094A1
• Time in Patent Office: 1,327 Days
• Field of Search: 345/7, 345/8, 345/9, 345/435, 345/425, 345/427, 345/419, 359/630, 359/633, 359/636, 359/631
• US Class Current: 345/8
• CPC Class Codes: A41D 31/32 Retroreflective
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A teleportal system which provides remote communication between at least two users. A projective display and video capture system provides video images to the users. The video system obtains and transmits 3D images which are stereoscopic to remote users. The projective display unit provides an augmented reality environment to each user and allows users to view, unobstructed, the other local users, and view a local site in which they are located. A screen transmits to the user the images generated by the projective display via a retro-reflective fabric upon which images are projected and reflected back to the user'"'"'s eyes.

Claims

1. A head mounted display unit worn by a human user for displaying images received from a remote location to the user via retro-reflective material, the head mounted display unit comprising:

an image data receiving unit that operably receives image data from the remote location;

first and second augmented reality displays connected to the image data receiving unit operably permitting generation of an image upon the retro-reflective material and which is reflected to eyes of the user, the first and second augmented reality displays permitting a substantially full range of view to the eyes of the user; and

first and second cameras worn by the user and disposed to capture plural views of user facial expressions for transmission to the remote location.

2. The head mounted display unit of claim 1 wherein the user operates within a physical environment, the user'"'"'s eyes capable of seeing images within the physical environment.

3. The head mounted display unit of claim 2 wherein the physical environment includes first and second users wearing the head mounted display unit, wherein the eyes of the first user are capable of seeing the face of the second user.

4. The head mounted display unit of claim 3 further comprising at least one glove worn by the user, the glove being made of retro-reflective fabric which operably reflects the image to the eyes of the user.

5. The head mounted display of claim 3 further comprising a body suit worn by the user, the body suit being fitted to a portion of the user and being made of retro-reflective fabric which operably reflects the image to the eyes of the user.

6. The head mounted display unit of claim 2 wherein the physical environment includes a room with a wall, and the wall includes the retro-reflective material operably reflecting the image to the eyes of the user.

7. The head mounted display unit of claim 2 wherein the physical environment includes a portable cubicle that includes the retro-reflective material operably reflecting the image to the eyes of the user.

8. The head mounted display unit of claim 2 wherein the physical environment includes a display sphere that has the retro-reflective material operably reflecting the image to the eyes of the user.

9. The head mounted display unit of claim 1 further comprising first and second optical lenses operably directing the respectively generated images by the first and second augmented-reality displays to the retro-reflective material and to the eyes of the user.

10. The head mounted display unit of claim 9 wherein the first and second optical lenses respectively include a first and second compound lens assembly that displays stereo three-dimensional images off the retro-reflective material.

11. The head mounted display unit of claim 9 wherein the first and second optical lenses include components selected from the group consisting of glass optical components, plastic optical components, diffractive optics components and combinations thereof.

12. The head mounted display unit of claim 9 wherein each of the first and second optical lenses includes a projection lens in optical communication with a beamsplitter for use in directing the image to the retro-reflective material.

13. The head mounted display unit of claim 1 wherein the first and second augmented-reality displays generate a stereoscopic image to the eyes of the user.

14. The head mounted display unit of claim 1 further comprising a computer network, wherein the image data receiving unit operably receives image data from the remote location via the computer network.

15. The head mounted display unit of claim 1 wherein the retro-reflective material is at least 98 percent reflective.

16. The head mounted display unit of claim 1, wherein said first and second cameras are mounted to the user'"'"'s head and disposed to capture plural views of user facial expressions via first and second convex mirrors mounted to the user'"'"'s head.

17. A video capture system for obtaining and transmitting 3D images of a face of at least one remote user for use in a teleportal system, the system comprising:

a first camera worn by the remote user and operably capturing a first image of at least a first portion of the face of the remote user;

a second camera worn by the remote user and operably capturing a second image of at least a second portion of the face of the remote user; and

a face warping and image stitching module operably determining image signals based on said first image and said second image, the face warping and image stitching module operably receiving image signals from the first camera and the second camera and transmitting the signals to produce stereoscopic images.

18. The system of claim 17 further comprising:

a screen;

an optic system; and

a projective augmented-reality display operably providing a stereo image without substantial occlusion of the physical objects, the projective augmented-reality display having at least two beams of light projected through the optic system directly in front of eyes of the user and reflected back to the eyes from the screen.

19. The system of claim 18 wherein the screen includes a surrounding screen of retro-reflective fabric.

20. The system of claim 18 wherein the screen is a central display sphere for displaying three-dimensional images of objects, said central display sphere having a spherical frame covered with a retro-reflective material.

21. The system of claim 17 further comprising:

a first convex mirror disposed between the first camera and the user'"'"'s face; and

a second convex mirror disposed between the second camera and the user'"'"'s face,

wherein the first and second mirrors transmit images of the user'"'"'s face to the first and second cameras in order to produce the stereoscopic images of the face.

22. The system of claim 17 further comprising an optical tracker operably determining a location of a body part of a user and providing a position signal based on the location of the body part, the optical tracker operably receiving data from the body part of the user and transmitting the data to a network.

23. The system of claim 17 wherein the first and second cameras assist in video recording and digitizing the face of users.

24. The system of claim 17 further comprising a projective augmented-reality display operably displaying the stereoscopic images of the face, the projective augmented-reality display being coupled to a network.

25. The system of claim 21 further comprising a headwear system, wherein the video capture system and the projective augmented-reality display are coupled to the headwear system.

26. A method for obtaining and transmitting images received from a remote location to at least one user by using a projective augmented display with a retro-reflective screen, such that the images are shared by users, comprising:

(a) receiving image data from the remote location;

(b) generating stereoscopic images based on the image data from a projective-augmented display onto the retro-reflective screen;

(c) reflecting the stereoscopic images from the retro-reflective screen to the eyes of the user such that stereoscopic projected images are apparently seen by the user;

(d) capturing multiple views of a present location wherein the user is located; and

(e) transmitting the multiple views to the remote location.

27. The method of claim 26 wherein the projective-augmented display includes a source display, a projection lens and a beamsplitter arranged in series such that the stereoscopic images are transmitted from the source display, the projection lens, and the beam splitter.

28. The method of claim 26 further comprising projecting the stereoscopic image from the retro-reflective screen to a beamsplitter.

29. The method of claim 26 further comprising collecting image data via at least one camera and at least mirror, the at least one camera receiving the images from the at least one mirror.

30. The method of claim 26 further comprising collecting image data based on images retrieved via a first camera from a first mirror and a second camera from a second mirror, the first camera and the second camera assisting in digitizing the image data into stereo video images.

31. The method of claim 26 further comprising collecting image data via digital processing.

32. The method of claim 26 wherein the retro-reflective screen is human body shaped.

33. The method of claim 26 further comprising processing the multiple views for transmission to the remote location.

34. A method for obtaining and transmitting images received from a remote location to at least one user by using a projective-augmented display with a retro-reflective screen positioned on a human body, such that the images are shared by users, the method comprising:

(a) receiving medical image data from the remote location;

(b) generating stereoscopic images based on the image data from a projective-augmented display onto the retro-reflective screen, the retro-reflective screen being shaped to conform to at least part of the human body; and

(c) reflecting the stereoscopic images from the retro-reflective screen to eyes of the user such that stereoscopic projected images are apparently seen by the user.

35. The method of claim 34 wherein the projective-augmented display includes a source display, a projection lens and a beamsplitter arranged in series such that the stereoscopic images are transmitted from the source display, the projection lens, and the beam splitter.

36. The method of claim 34 further comprising projecting the stereoscopic image from the retro-reflective screen to a beamsplitter.

37. The method of claim 34 further comprising collecting image data via at least one camera and at least mirror, the at least one camera receiving the images from the at least one mirror.

38. The method of claim 34 further comprising collecting image data based on images retrieved via a first camera from a first mirror and a second camera from a second mirror, the first camera and the second camera assisting in digitizing the image data into stereo video images.

39. The method of claim 34 further comprising collecting image data via digital processing.

40. The method of claim 34 wherein the retro-reflective screen is hand shaped.

41. A head mounted display unit worn by a human user for displaying images to the user, the head mounted display unit comprising:

an image data receiving unit that operably receives image data from the remote location;

retro-reflective material operably reflecting the images, the retro-reflective material reflecting a majority of the image and shaped to conform to at least part of the human body; and

first and second augmented reality displays connected to the image data receiving unit operably permitting generation of an image upon the retro-reflective material and which is reflected to the eyes of the user, the first and second augmented-reality displays permitting a substantially full range of view to the eyes of the user.

42. The head mounted display unit of claim 41 wherein the user operates within a physical environment, the user'"'"'s eyes capable of seeing images within the physical environment.

43. The head mounted display unit of claim 42 wherein the physical environment includes first and second users wearing the retro-reflective material, wherein the first user and the second user are capable of displaying the image data such that movement of the first user over the second user maintains the displaying the image data on the retro-reflective material.

44. The head mounted display unit of claim 43 further comprising at least one glove worn by the user, the glove being made of retro-reflective fabric which operably reflects the image to the eyes of the user.

45. The head mounted display of claim 43 further comprising a body suit worn by the user, the body suit being fitted to a portion of the user and being made of retro-reflective fabric which operably reflects the image to the eyes of the user.

46. The head mounted display unit of claim 43 wherein the physical environment includes a body suit and gloves that include the retro-reflective material operably reflecting the image data to the eyes of the user, the body suit associated with the first user and the gloves associated with the second user.

47. The head mounted display unit of claim 41 further comprising first and second optical lenses operably directing the respectively generated images by the first and second augmented-reality displays to the retro-reflective material and to the eyes of the user.

48. The head mounted display unit of claim 47 wherein the first and second optical lenses respectively include a first and second compound lens assembly that displays stereo three-dimensional images off the retro-reflective material.

49. The head mounted display unit of claim 47 wherein the first and second optical lenses include components selected from the group consisting of glass optical components, plastic optical components, diffractive optics components and combinations thereof.

50. The head mounted display unit of claim 47 wherein each of the first and second optical lenses include a projection lens in optical communication with a beamsplitter for use in directing the image to the retro-reflective material.

51. The head mounted display unit of claim 41 wherein the first and second augmented-reality displays generate a stereoscopic image to the eyes of the user.

52. The head mounted display unit of claim 41 further comprising a computer network, wherein the image data receiving unit operably receives image data from the remote location via the computer network.

53. The head mounted display unit of claim 41 wherein the retro-reflective material is at least 98 percent reflective.

54. The head mounted display unit of claim 41 comprising first and second head mounted video cameras disposed to capture plural views of the user location for transmission to a remote location.

55. A display screen apparatus worn by a human user for displaying image data, said apparatus comprising:

an image data transmitting device operably providing the image data;

a reflective material operably reflecting images to eyes of the user, the reflective material being of a highly reflective material such that a reflected image is transmitted, the reflected image being formed upon the reflective material as a virtual image in view, the reflected image of a brightness such that the reflective material reflects at least a majority of light to allow for viewing of the virtual image;

an image receiving device operably transferring the image data to eyes of the human user; and

at least first and second imaging devices capturing plural views at a physical location of the user for transmission to a remote location,

wherein said display screen and the virtual image create an augmented virtual reality environment in which the user observes both physical objects and virtual objects.

56. The apparatus of claim 55 further comprising first and second optical lenses operably directing the image data to the retro-reflective material and to the eyes of the human user.

57. The apparatus of claim 56 wherein the first and second optical lenses respectively include a first and second compound lens assembly that displays image data formed as stereo three-dimensional images off the retro-reflective material.

58. The apparatus of claim 56 wherein the first and second optical lenses include components selected from the group consisting of glass optical components, plastic optical components, diffractive optics components and combinations thereof.

59. The apparatus of claim 56 wherein each of the first and second optical lenses includes a projection lens in optical communication with a beamsplitter for use in directing the light to the retro-reflective material.

60. The apparatus of claim 55 wherein the retro-reflective material forms at least one glove worn by the human user.

61. The apparatus of claim 55 wherein the retro-reflective material forms a body suit worn by the user, the body suit being fitted to a portion of the user.

62. The apparatus of claim 55 wherein the physical objects include a room with a wall, and the wall includes the retro-reflective material operably reflecting the image to the eyes of the human user.

63. The apparatus of claim 55 wherein the physical objects include a portable cubicle that includes the retro-reflective material operably reflecting the image to the eyes of the user.

64. The apparatus of claim 55 wherein the physical objects include a display sphere that has the retro-reflective material operably reflecting the light to the eyes of the human user.

65. The apparatus of claim 55 wherein the light is first and second formed as a stereoscopic image to the eyes of the human user.

66. The apparatus of claim 55 further comprising a computer network, wherein the image data receiving unit operably receives image data from the remote location via the computer network.

67. The apparatus of claim 55 wherein the retro-reflective material is at least 98 percent reflective.

68. The apparatus of claim 55 wherein the retro-reflective material is at least 90 percent reflective.