Two-way interactive system, terminal equipment and image pickup apparatus having mechanism for matching lines of sight between interlocutors through transmission means

• US RE38,884 E1
• Filed: 10/24/2002
• Issued: 11/22/2005
• Est. Priority Date: 02/16/1995
• Status: Expired due to Term

• Current Assignee: Sumitomo Electric Industries Limited (Sumitomo Corporation)
• Original Assignee: Sumitomo Electric Industries Limited (Sumitomo Corporation)
• Inventors: Kakii, Toshiaki
• Primary Examiner(s): Mei, Xu
• Application Number:
  US10/278,794
• Time in Patent Office: 1,125 Days
• Field of Search: 348/14.08, 348/14.16, 348/211.12, 348/211.13, 348/373, 396/428
• US Class Current: 348/14.16
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

A two-way interactive system that includes an apparatus for matching the line of sight of interlocutors in order to enhance communication through a transmission means when the interlocutors are located in remote areas. Terminal equipment, located at the site of each interlocutor, includes a display unit, a position regulator means and an image pickup apparatus. The display unit includes a monitor that displays an image of a partner interlocutor. The position regulating means regulates the position of the interlocutor with respect to the monitor. The image pickup apparatus includes an image pickup unit which captures an image of the interlocutors and a support means which sets the image pickup unit in a predetermined position between the interlocutor and a monitor of the display unit. More particularly, the image pick up unit is positioned in a line-of-sight matching area defined by the distance from the interlocutor to the monitor and a parallactic angle formed by the lines of sight of the interlocutor to the image of the partner interlocutor and the image pickup unit.

Claims

1. A two-way interactive system comprising:

a first terminal equipment located at a first point where a first interlocutor is present;

a second terminal equipment located at a second point where a second interlocutor is present; and

transmission means, arranged between said first terminal equipment and said second terminal equipment, for enabling two-way communication of at least image data of the first interlocutor and the second interlocutor, at least said first terminal equipment comprising;

a display unit for displaying an image of the second interlocutor, which is received through said transmission means, on a monitor;

position regulation means for regulating a position of the first interlocutor with respect to a monitor screen of said display unit; and

an image pickup apparatus for enabling transmission of an image of the first interlocutor to said second terminal equipment through said transmission means, said image pickup apparatus having an image pickup unit for directly picking up the image of the first interlocutor and a support mechanism for setting said image pickup unit at a predetermined position between the first interlocutor and said monitor,

wherein while an image pickup direction of said image pickup unit is set directly toward the first interlocutor, said image pickup unit is positioned so that the first interlocutor can view said image pickup unit at a position where said image pickup unit is located within an image displaying window on said monitor screen of said display unit and above a position of an eye of a monitor image of the second interlocutor. The image displaying window being a region which the monitor image of the second interlocutor is displayed.

2. A two-way interactive system according to claim 1, wherein said support mechanism of said image pickup apparatus has a mechanism for adjusting an angle of the image pickup direction of said image pickup unit.

3. A two-way interactive system according to claim 1, wherein a parallactic angle for matching the line of sight of the first interlocutor and the line of sight of the monitor image of the second interlocutor is not more than 4.5°

in a horizontal direction of said monitor screen, not more than 12°

in an upper direction, and not more than 8°

in a lower direction with respect to the line of sight of the first interlocutor when viewed from the first interlocutor.

4. A two-way interactive system according to claim 3, wherein

the parallactic angle for matching the line of sight of the first interlocutor and the line of sight of the monitor image of the second interlocutor is not more than 3°

in horizontal and vertical directions of said monitor screen with respect to the line of sight of the first interlocutor when viewed from the first interlocutor.

5. A two-way interactive system according to claim 1, wherein said first terminal equipment comprises a person authentication unit for authenticating a user of said first terminal equipment before an interactive operation is started.

6. A two-way interactive system according to claim 5, wherein said first terminal equipment comprises a read unit for reading a predetermined item from a card-shaped external recording medium having at least an IC, in which the predetermined item is recorded in advance.

7. A two-way interactive system according to claim 6, wherein said external recording medium records, as the predetermined item, at least one of reservation information and information for performing person authentication of a user of said second terminal equipment as a partner for an-interactive operation.

8. A two-way interactive system according to claim 6, wherein said external recording medium records, as the predetermined item, interactive operation conditions including at least a reserved interactive operation start time, a reserved interactive operation time, and interactive operation contents.

9. A two-way interactive system according to claim 6, wherein said external recording medium records, as the predetermined item, interactive operation reservation data including at least a reserved interactive operation start time and information associated with a partner for an interactive operation, and

wherein said first terminal equipment starts two-way communication with said second terminal equipment after said first terminal equipment is line-connected with said second terminal equipment searched through a communication network serving as said transmission means on the basis of the interactive operation reservation data.

10. A two-way interactive system according to claim 9, wherein said external recording medium records, as the predetermined information, at least one of character information, graphic information and image information, and said first terminal equipment displays part of the information recorded in said external recording medium on said monitor of said first terminal equipment at least before an interactive operation, during an interactive operation, or after an interactive operation.

11. A two-way interactive system according to claim 10, wherein said first terminal equipment transmits control information for causing said second terminal equipment to display and output part of the information recorded in said external recording medium to said second terminal equipment through said transmission means at least before an interactive operation, during an interactive operation, or after an interactive operation.

12. A two-way interactive system according to claim 10, wherein said first terminal equipment outputs an acoustic medium including at least one of a background music and a voice recorded/stored inside or outside said first terminal equipment in advance in a manner interlocked with a display operation of said first terminal equipment of claim 8.

13. A two-way interactive system according to claim 10, wherein said first terminal equipment transmits a control signal for causing said second terminal equipment to output an acoustic medium including at least one of a background music and a voice recorded/stored inside or outside said first terminal equipment in advance to said second terminal equipment through said transmission means in a manner interlocked with a transmission operation of said first terminal equipment.

14. A two-way interactive system according to claim 1, wherein said first terminal equipment further comprises a processing unit for recognizing a position of the image of the first interlocutor on said monitor screen, the image being displayed on said monitor, and

wherein said image pickup apparatus of said first terminal equipment further comprises a driving mechanism for relatively moving a position of said image pickup unit with respect to the first interlocutor whose position is recognized by said processing unit, or changing an image pickup angle of said image pickup unit.

15. A two-way interactive system according to claim 14, wherein said second terminal equipment comprises at least a second point image pickup unit for picking up the image of the second interlocutor, and a second point driving mechanism for moving a position of said second point image pickup unit or changing an image pickup angle of said second point image pickup unit, and

wherein said processing unit of said first terminal equipment recognizes a position of the image of the second interlocutor on said monitor screen, the image being received through said transmission means and displayed on said monitor, and transmits a control signal for relatively moving the position of said second point image pickup unit with respect to the second interlocutor or changing the image pickup angle of said second point image pickup unit to said second terminal equipment through said transmission means.

16. A two-way interactive system according to claim 1, wherein said first terminal equipment further comprises a processing unit for recognizing a position of the image of the first interlocutor on said monitor screen, the image being displayed on said monitor, and

wherein said position regulation means comprises a mechanism for adjusting a position of an eye of the first interlocutor to be picked up with respect to a position of an eye of the monitor image of the first interlocutor whose position is recognized by said processing unit.

17. A two-way interactive system according to claim 16, wherein said second terminal equipment comprises at least a second point display unit for displaying the image of the first interlocutor on a monitor, and second point position regulation means for regulating a position of the second interlocutor, said second point regulation means having a mechanism for adjusting a position of an eye of the second interlocutor with respect to a position of an eye of the monitor image, and

wherein said processing unit of said first terminal equipment recognizes a position of the image of the second interlocutor on said monitor, the image being received through said transmission means and displayed on said monitor, and transmits a control signal for adjusting the position of the eye of the second interlocutor with respect to the position of the eye of the monitor image on said second point display unit to said second terminal equipment through said transmission means.

18. A two-way interactive system according to claim 1, wherein said support mechanism of said image pickup apparatus comprises a transparent member for directly supporting said image pickup unit.

19. A two-way interactive system according to claim 1, wherein said position regulation means includes a chair having a sitting height adjustment mechanism.

20. A two-way interactive system according to claim 1, wherein said position regulation means includes a table set between the first interlocutor and said display unit and having a predetermined height.

21. A two-way interactive system according to claim 1, wherein said position regulation means includes a table having a predetermined height, on which said display unit is directly set.

22. A two-way interactive system according to claim 1, wherein said first terminal equipment comprises an interactive operation processing unit for counting a total interactive operation time, and average interactive operation time, and the number of interactive operations during a two-way interactive operation between the first interlocutor and the second interlocutor, and sequentially displaying the total interactive operation time, the average interactive operation time, the number of interactive operations, and a distribution of the number of interactive operations on said display unit.

23. A two-way interactive system according to claim 1, wherein first terminal equipment comprises a sound absorbing member having external sound insulating properties and at least partially arranged around the first interlocutor.

24. A two-way interactive system according to claim 1, wherein said first terminal equipment comprises a voice/acoustic data processing unit for receiving, as an electrical signal, voice of the first interlocutor, which is gathered by a voice input unit, and a voice/acoustic output control unit for causing a voice output unit to reproduce voice data of the second interlocutor, which is transmitted through said transmission means.

25. A two-way interactive system according to claim 24, wherein said voice output unit includes a pair of speakers arranged near ears aside the first interlocutor.

26. A two-way interactive system according to claim 24, wherein said voice input unit includes a directional high-sensitive microphone.

27. A two-way interactive system according to claim 24, wherein said voice-acoustic data processing unit detects an average level of input volume levels for a predetermined time and automatically adjusts the volume when a difference between the input volume level and the average level exceeds a predetermined value.

28. A two-way interactive system according to claim 24, wherein said voice/acoustic output control unit detects that an interactive operation volume is substantially zero for a predetermined time and, on the basis of a detection result, at least starts a background music or performs adjustment to increase or decrease a volume of the background music.

29. A two-way interactive system according to claim 24, wherein said voice/acoustic data processing unit performs statistical analysis processing in which a voice spectrum of the first interlocutor or a voice spectrum according to transmitted voice information of the second interlocutor is analyzed over time, and detects an average spectrum and a voice spectrum changed beyond a predetermined value.

30. A two-way interactive system according to claim 1, wherein said display unit has a through hole causing a rear surface of said display unit to communicate with said monitor screen on a front surface of said display unit, and wherein said image pickup unit is attached to said display unit through said support mechanism while a distal end of said image pickup unit extends through the through hole of said display unit and projects forward from said monitor screen.

31. A two-way interactive system according to claim 30, wherein said image pickup unit is attached to said display unit through said support mechanism to freely rotate about a predetermined portion of said image pickup unit, whereby the image pickup direction of said image pickup unit is changed in a desired direction.

32. A two-way interactive system according to claim 31, wherein said first terminal equipment comprises a driving mechanism for changing an attachment angle between said display unit and said image pickup unit.

33. A two-way interactive system according to claim 1, wherein said first terminal equipment further comprises a processing unit for adjusting a display position of the image of the second interlocutor, which is displayed on said monitor, said processing unit

specifying a reference point at a virtual position on the monitor image where said image pickup unit should be originally present in the basis of the image of the second interlocutor, the image being received through said transmission means and displayed on said monitor,

detecting a shift of the monitor image of the second interlocutor on the basis of coordinates on said monitor corresponding to an actual set position of said image pickup unit and coordinates of the obtained reference point on said monitor,

calculating a vector starting from the coordinates of the reference point on said monitor and terminating at the coordinates on said monitor corresponding to the set position of said image pickup unit, and

translating the monitor image of the second interlocutor on the basis of the obtained vector.

34. A two-way interactive system according to claim 33, wherein said processing unit detects display positions of both eyes of the image of the second interlocutor, the image being received through and transmission means and displayed on said monitor, and specifies the reference point at a position on said monitor, which is separated upward from a center between both the eyes by a predetermined distance on said monitor.

35. A two-way interactive system according to claim 33, wherein said processing unit specifies the reference point at a position on said monitor corresponding to a parietal portion of the monitor image of the second interlocutor, the image being received through said transmission means and displayed on said monitor.

36. A method of displaying a person'"'"'s image of a first interlocutor in an image displaying window on a monitor screen of a monitor while an image pickup unit for picking up an image of a second interlocutor located in front of said monitor is set so that the second interlocutor can view that said image pickup unit overlaps the person'"'"'s image of the first interlocutor displayed within the image displaying window on said monitor screen, the image displaying window being a region which the person'"'"'s image of the first interlocutor is displayed, said method comprising the step of:

enlarging or reducing the person'"'"'s image of the first interlocutor in the image displaying window on said monitor screen, relative to a predetermined point within the image displaying window on said monitor screen is fixed, said predetermined position insuring that a line of sight of the second interlocutor matches a line of sight of the person'"'"'s image of the first interlocutor displayed on said monitor screen.

37. A method according to claim 36, wherein at least characters or graphics are displayed in a predetermined area on said monitor screen, except for the image displaying window.

38. An image pickup apparatus comprising:

a main body having a predetermined setting surface attached to a surface of a monitor;

an image pickup unit for picking up directly an image of a person positioned in front of said monitor;

a support mechanism which has one end attached to said main body and another end attached to the image pick up unit, as structured so that the one end can be adjusted through an attachment connecting the one end to the main body and set an image pickup direction of said image pickup unit toward the person directly viewing said image pickup unit covering a part of a screen of said monitor on which an image of the person is displayed when said main body is set on said monitor.

39. An image pickup apparatus according to claim 38, wherein said support mechanism has a structure for adjusting an angle of the image pickup direction of said image pickup unit.

40. An image pickup apparatus according to claim 38, further comprising a processing unit for transmitting at least an image of the person to a transmission means as image data.

41. A two-way interactive system, comprising

a first terminal equipment located at a first point where first interlocutor is present;

a second terminal equipment located at a second point where second interlocutor is present; and

transmission means, arranged between said first terminal equipment and said second terminal equipment, for enabling two-way communication of at least image data of the first interlocutor and the second interlocutor,

at least said first terminal equipment comprising;

a display unit for displaying an image of the first interlocutor or the second interlocutor, which is received through said transmission means, on a monitor screen;

an image pickup apparatus for enabling transmission of an image of the first interlocutor to said second terminal equipment through said transmission means, said image pickup apparatus having an image pickup unit for picking up the image of the first interlocutor, and a support mechanism for setting said image pickup unit at a position between the first interlocutor and said monitor screen; and

position regulation means for regulating a relative positional relationship between said image pickup unit and a person image displayed on said monitor screen such that, from a view of the first interlocutor, said image pickup unit overlaps said monitor screen and is located above an eye of the person image displayed on said monitor screen.

42. The two-way interactive system according to claim 41, wherein said position regulation means regulates a display position of the person image on said monitor screen such that, from the view of the first interlocutor, said image pickup unit overlaps said monitor screen and is located above the eye of the person image displayed on said monitor screen.

43. The two-way interactive system according to claim 41, wherein said support mechanism of said image pickup apparatus has a structure for adjusting an angle of an image pickup direction of said image pickup unit.

44. The two-way interactive system according to claim 41, wherein said position regulation means includes a processing unit electrically connected to said image pickup apparatus, for making said display unit display an image of the first interlocutor picked up by said image pickup unit on said monitor screen,

wherein said first terminal equipment further comprises a processing unit for recognizing a position of an image of the first interlocutor on said monitor screen, and

wherein said image pickup apparatus of said first terminal equipment further comprises a driving mechanism for moving a position of said image pickup unit or changing an angle of an image pickup direction of said image pickup unit, relatively with respect to a position of an image of the first interlocutor which is recognized by said processing unit.

45. The two-way interactive system according to claim 44, wherein said second terminal equipment comprises a second-point image pickup unit for picking up the image of the second interlocutor, and a second-point driving mechanism for moving a position of said second-point image pickup unit or changing an angle of an image pickup direction of said second-point image pickup unit, and

wherein said processing unit of said first terminal equipment recognizes a position of the image of the second interlocutor which is received through said transmission means and is displayed on said monitor screen, and transmits control signals used for moving the position of said second-point image pickup unit or changing the angle of the image pickup direction of said second-point image pickup unit, relatively respect to the second interlocutor to said second terminal equipment through said transmission means.

46. The two-way interactive system according to claim 41, wherein said supporting mechanism of said image pickup apparatus includes a transparent member for directly supporting said image pickup unit.

52. The two-way interactive system according to claim 41, wherein said position regulation means regulates a relative postional relationship between said image pickup unit and the person image displayed on said monitor screen such that, from the view of the first interlocutor, said image pickup unit overlaps the person image on said monitor screen.

47. A terminal equipment for enabling a two-way interactive communication through a transmission means, said terminal equipment comprising:

a display unit for displaying at least a person image received as image signals through said transmission means, or a person image picked up by said terminal equipment, on a monitor screen;

an image pickup apparatus for enabling transmission of an image of a person present in front of said monitor screen through said transmission means; and

position regulating means,

wherein said image pickup apparatus has an image pickup unit for picking up the image of the person in front of said monitor screen; and

a support mechanism for setting said image pickup unit at a position between the person in front of said monitor screen and said monitor screen, and

wherein said position regulation means regulates a relative positive relationship between said image pickup unit and the person image displayed on said monitor screen such that, from a view of the person in front of said monitor screen, said image pickup unit overlaps said monitor screen and is located above an eye of the person image displayed on said monitor screen.

48. The terminal equipment according to claim 47, wherein said position regulation means regulates a display position of the person image to be displayed on said monitor screen such that, from the view of the person in front of said monitor screen, said image pickup unit overlaps said monitor screen and is located above the eye of the person image displayed on said monitor screen.

49. The terminal equipment according to claim 47, wherein said support mechanism of said image pickup apparatus has a structure for adjusting an angle of an image pickup direction of said image pickup unit.

50. The terminal equipment according to claim 47, wherein said position regulation means includes a processing unit electrically connected to said image pickup apparatus, for making said display unit display the image of the person in front of said monitor screen picked up by said image pickup unit, on said monitor screen,

wherein said terminal equipment further comprises a processing unit for recognizing a position of the image of the person in front of said monitor screen, on said monitor screen, and

wherein said image pickup apparatus further comprises a driving mechanism for moving a position of said image pickup unit or changing an angle of an image pickup direction of said image pickup unit, relatively with respect to the position of the person in front of said monitor screen which is recognized by said processing unit.

51. The terminal equipment according to claim 47, wherein said supporting mechanism of said image pickup apparatus includes a transparent member for supporting said image pickup unit.

53. The terminal equipment according to claim 47, wherein said position regulation means regulates a relative postional relationship between said image pickup unit and the person image displayed on said monitor screen such that, from the view of the first interlocutor, said image pickup unit overlaps the person image on said monitor screen.

54. The terminal equipment according to claim 47, wherein said position regulation means is incorporated in said image pickup apparatus.

55. A method for interactive two-way communication, comprising the steps of:

acquiring a first image of a first person, located in front of a monitor screen, using an image pickup unit;

transmitting the first image from the image pickup unit to the monitor screen;

displaying the first image of the first person on the monitor screen; and

regulating a relative position between the image pickup unit and the displayed first image such that, from the view of the first person, the image pickup unit overlaps the monitor screen and is located above eye level of the first image displayed on the monitor screen.

56. The method according to claim 55, further comprising the step of:

regulating a display position of the first image on the monitor screen such that, from the view of the first person, the image pickup unit overlaps the monitor screen and is located above eye level of the first image displayed on the monitor screen.

57. The method according to claim 55, further comprising the step of:

adjusting an angle of direction for the image pickup unit.

58. The method according to claim 55, further comprising the step of:

receiving a second image of a second person, said second person located remotely from said first person;

displaying the second image of the second person on the monitor screen; and

regulating a relative position between the image pickup unit and the displayed second image such that, from the view of the first person, the image pickup unit overlaps the monitor screen and is located above eye level of the second image of the second person displayed on the monitor screen.

59. The method according to claim 55, further comprising the steps of:

detecting a position of the image of the first person on the monitor screen;

adjusting an angle of direction for the image pickup unit based on the detected position of the image of the first person.

60. The method according to claim 55, further comprising the steps of:

detecting a position of the image of the first person on the monitor screen;

moving a position of the image pickup unit based on the detected position of the image of the first person.

61. The method according to claim 55, further comprising the step of:

regulating the relative position such that, from the view of the first person, the image pickup unit overlaps the first image of the first person displayed on the monitor screen.

62. The method according to claim 58, further comprising the step of:

regulating the relative position such that from the view of the first person the image pickup unit overlaps the second image of the second person displayed on the monitor screen.