Navigable telepresence method and system utilizing an array of cameras

• US 20030229735A1
• Filed: 03/17/2003
• Published: 12/11/2003
• Est. Priority Date: 04/02/1998
• Status: Abandoned Application

• Current Assignee: Andrew H. Weber, David C. Worley, Scott Sorokin
• Original Assignee: Andrew H. Weber, David C. Worley, Scott Sorokin
• Inventors: Weber, Andrew H., Sorokin, Scott, Worley, David C.
• Application Number:
  US10/391,509
• Publication Number:
  US 20030229735A1
• Time in Patent Office: Days
• US Class Current: 710/36
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment includes an array of cameras. Each camera has an associated view of the environment and an associated camera output representing the associated view. The array also includes at least one camera path. A first user interface device has first user inputs associated with movement along a first path in the array, and a second user interface device has second user inputs associated with along a second path in the array. The user interface devices are coupled to at least one processing element that receives and interprets the user inputs. The processing element interprets the first inputs and selects outputs of cameras in the first path, and interprets second inputs and selects outputs of cameras in the second path independently of the first inputs, thereby allowing a first user and a second user to navigate simultaneously and independently through the array. In another embodiment, a telepresence system includes at least one processing element that distinguishes between permissible cameras in the array and impermissible cameras in the array.

Claims

1. A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the array including at least one camera path;

a first user interface device associated with the first user having first user inputs associated with movement along a first path in the array;

a second user interface device associated with the second user having second user inputs associated with movement along a second path in the array;

at least one processing element coupled to the user interface devices for receiving user inputs, the processing element configured to interpret received first inputs and select outputs of cameras in the first path, and interpret received second inputs and select outputs of cameras in the second path independently of the first inputs, thereby allowing the first user and second user to navigate simultaneously and independently through the array.

19. The system of claim 1 wherein the first user inputs include an indication of a speed of movement through the first path.

20. The system of claim 19 wherein the processing element is further configured to compare the indication of speed to a predetermined threshold and to select a number of outputs corresponding to fewer than all cameras in the path upon determining the indication of speed exceeds the predetermined threshold.

21. The system of claim 19 wherein the number of outputs is inversely proportional to the speed.

23. The system of claim 19 wherein the indication of relative speed of movement through the array includes an indication of moving relatively slower through the array and the processing element adjusts the switching speed by causing the view associated with the first camera to be displayed for a relatively longer period of time in accordance with the indication of moving slower.

30. The system of claim 1 wherein the user interface device includes a joystick wherein user inputs are movements of the joystick.

31. The system of claim 1 wherein the user interface device includes a mouse, wherein user inputs are movements of the mouse.

32. The system of claim 1 wherein the user interface device includes a touchscreen.

33. The system of claim 1 wherein the user interface device includes a biofeedback device.

34. The system of claim 1 wherein the user interface device includes a voice recognition device.

35. The system of claim 1 wherein the user inputs include an indication of a particular camera in the array.

36. The system of claim 1 wherein the user inputs include moving up, down, to the right, and to the left.

37. The system of claim 34 wherein the user inputs further include moving forward and backward.

38. The system of claim 1 wherein the processing element operates in accordance with a navigation control algorithm, the navigation control algorithm distinguishing between permissible movement and impermissible movement in the array.

39. The system of claim 38 further including navigation control factors, wherein the navigation control algorithm controls movement through the array in accordance with the navigation control factors.

40. The system of claim 39 wherein the navigation control factors include time of day, time duration, speed of navigation through the array, output of a device in the array, output of a motion sensor, and output of a microphone.

41. The system of claim 1 wherein the processing element is further configured to disregard certain user input.

42. The system of claim 41 wherein the processing element is further configured to disregard user input by selecting at least one predetermined camera output, thereby directing the user to a predetermined view.

43. The system of claim 42 wherein the processing element is configured to select the predetermined output based on time of day.

44. The system of claim 42 wherein the processing element is configured to select the predetermined output based on a duration of time.

45. The system of claim 41 wherein the processing element is further configured to disregard user input by waiting for a permissible user input.

46. The system of claim 1 further including an operator input device coupled to the processing element for receiving from the operator input device operator input, and wherein the processing element is controlled to override user input based on the operator input.

47. The system of claim 1 wherein the system further includes a memory storing additional source output, wherein the user inputs include an indication of viewing the additional source output, and the processing element is further configured to mix camera output and the additional source output upon receiving the indication to view the additional source output.

48. The system of claim 47 wherein the indication of viewing the additional source output is available to the user in connection with a predetermined camera.

49. The system of claim 47 wherein the processing element is configured to mix camera output and the additional source output by compositing camera output and the additional source output.

50. The system of claim 47 wherein the processing element is configured to mix camera output and the additional source output by switching from camera output to the additional source output.

51. The system of claim 47 wherein the additional source output includes output from the group of outputs including:

computer graphic imagery, virtual world imagery, applets, film clips, and animation.

52. The system of claim 1 further including a plurality of microphones, each microphone having an output, wherein the processing element is further configured to select a microphone output based on received first user input for transmission to the first user.

53. The system of claim 52 further including a memory coupled to the processing element, the memory correlating cameras and microphones, the processing element selecting a microphone by using the memory to correlate the selected camera output with a microphone output.

2. The system of claim 1 further including:

a first display device associated with the first user and coupled to the processing element; and

a second display device associated with the second user and coupled to the processing element,

wherein the processing element is further configured to select a first output based on the received first inputs and send the first output to the first the first display device, the first display device configured to display the view associated with the selected first output, and

select a second output based on the received second inputs and send the second output to the second display device, the second display device configured to display the view associated with the selected second output.

3. The system of claim 2 wherein the processing element is configured to select outputs in real time and the display devices are configure to display views in real time.

4. The system of claim 1 wherein the array includes a plurality of navigable camera paths and the first path is different than the second path.

5. The system of claim 1 wherein the processing element includes a control processing element and a plurality of mixing processing elements.

6. The system of claim 1 wherein the processing element is also coupled to the array.

7. The system of claim 1 wherein the processing element includes a plurality of processing elements.

8. The system of claim 2 wherein the first user interface device and the first display device are coupled to the processing element by different types of communication links.

8-1.

8-2.

9. The system of claim 1 wherein the user interface device is coupled to the processing element by one of the following communication links:

an internet link, a microwave link, a satellite link, a wireless link, or a fiber optics link.

10. The system of claim 1 wherein the first user interface device and second user interface device are coupled to the processing element by different communication links.

11. The system of claim 1 wherein the first user interface device and second user interface device are different.

12. The system of claim 1 wherein the array of cameras includes a plurality of modular rails having a male end and a female end, the male end of one rail connectable to the female end of another rail, and a plurality of cameras connectable to each rail.

13. The system of claim 1 wherein the array of cameras includes hollow ball joints, each of which houses a camera, and hollow rods interconnecting the ball joints, the rods housing communication paths for the outputs.

14. The system of claim 1 wherein the cameras in the array are positioned at a density of approximately one camera every inch.

15. The system of claim 1 wherein the view of the first camera overlaps the view of the second camera.

16. The system of claim 1 wherein the array includes a first camera having a first output and a second camera having a second output and wherein the processing element is further configured to mix the first and second outputs in accordance with the received first user inputs.

17. The system of claim 16 wherein the received first user inputs indicate a desire to move from the first camera to the second camera, and the processing element is configured to mix the first and second outputs by electronically switching between the first and second outputs at a switching speed.

18. The system of claim 17 wherein the processing element is further configured to mix the outputs by adjusting the switching speed.

22. The system of claim 16 wherein the first user inputs include an indication of relative speed of movement through the array and the processing element is configured to mix the first output and second out by switching from the first output to the second output of a switching speed, the processing element further configured to adjust the switching speed in accordance with the indication of relative speed.

24. The system of claim 16 wherein the processing element is further configured to mix the outputs by dissolving the view associated with the first camera with the view of the second camera.

25. The system of claim 16 further comprising a first user display device associated with the first user and coupled to the processing element and wherein the first display device has a frame refresh rate associated therewith and the processing element is configured to mix the outputs by adjusting the frame refresh rate.

26. The system of claim 16 wherein the processing element is configured to mix the outputs by creating a composite output from the first and second outputs.

27. The system of claim 16 wherein the array further includes at least one other camera interposed between the first camera and the second camera and wherein the processor is configured to mix the outputs of the first camera and the second camera by switching from the first output, to an output of the at least one camera, and then to the second output.

28. The system of claim 16 wherein the processing element is configured to mix outputs by adding motion blur.

16-3.

16-4.

16-5.

16-6.

16-7.

16-8.

16-9.

16-10.

16-11.

16-12.

16-13.

16-14.

29. The system of claim 1 wherein the user interface device includes a keyboard, wherein user inputs are keystrokes.

29-15.

29-16.

54. A method of providing users with real time views of a remote environment, the method comprising:

receiving electronic images of the environment from an array of cameras, the array including at least one camera path through the environment;

receiving a first input from a first user interface device associated with a first user, the first input indicating movement along a first path;

receiving a second input from a second user interface device associated with a second user, the second input indicating movement along a second path;

obtaining a first mixed image by mixing, with a first processing element, a first image with a second image in accordance with the first input;

obtaining a second mixed image by mixing, with a second processing element, a third image with a fourth image in accordance with the second input;

providing the first user with the first mixed image in substantially real time, thereby simulating movement along the first path; and

providing the second user with the second mixed image substantially in real time and simultaneous to providing the first user with the first mixed image, thereby independently and simulating movement along the second path.

55. The method of claim 54 wherein the first processing element is different than the second processing element.

56. The method of claim 54 wherein the array includes a plurality of navigable paths and the first path differs from the second path.

57. The method of claim 54 wherein the first image is different than the third image.

58. The method of claim 54 wherein mixing the first image with the second image includes switching from the first image to the second image at a first switching speed, the first mixed image including the second image, and mixing the third image with the fourth image includes switching from the third image to the fourth image at a second switching speed, the second mixed image including the fourth image.

59. The method of claim 58 wherein receiving the first input includes receiving an indication of a first speed of movement along the first path and wherein the first switching speed is based on the indication of the first speed of movement.

60. The method of claim 59 wherein receiving the second input includes receiving an indication of a second speed of movement along the second path, the second speed greater than the first speed, and wherein the second switching speed is based on the indication of the second speed of movement, the second switching speed greater than the first switching speed.

61. The method of claim 60 wherein mixing the third and fourth images involves providing the second user the third image for a relatively longer duration than the first image is provided to the first user.

62. The method of claim 60 wherein mixing the third and fourth images involves providing the second user the fourth image for a relatively longer duration than the second image is provided to the first user.

63. The method of claim 54 wherein mixing the first image with the second image includes compositing the first image with the second image.

64. The method of claim 63 wherein mixing the third image with the fourth image includes switching between the third image and the fourth image.

65. The method of claim 54 wherein mixing the first image with the second image includes dissolving the first image into the second image.

66. The method of claim 54 wherein the first image is from a first camera, the second image is from a second camera, the array includes another camera interposed between the first and second cameras, and mixing the first image with the second image includes providing the first user with an image from the interposed camera.

67. The method of claim 54 further including obtaining a third mixed image by mixing the second image with a fifth image in accordance with the first input and providing the first use the third mixed image.

68. The method of claim 54 further including receiving a third input from the first user interface device and overriding the third input.

69. The method of claim 68 wherein overriding the received third input includes presenting the first user with a predetermined image.

70. The method of claim 68 wherein overriding the received third input includes continuing to provide the first user with the first mixed image and awaiting a fourth input.

71. The method of claim 54 wherein receiving the first input includes receiving an indication of mixing an additional source output, the method further including obtaining a third mixed output by mixing the second image with the additional source output.

72. The method of claim 71 wherein mixing the second image with the additional source output includes switching from the second image to the additional source output.

73. A device for providing a user with a display of an environment, the device comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view; and

at least one processing element coupled to the array, the processing element configured to identify permissible cameras, the outputs of which may be provided to the user, and impermissible cameras, the outputs of which are not provided to the user.

74. The device of claim 73 wherein the permissible cameras change over time.

75. The device of claim 74 wherein the permissible cameras change based on the time of day.

76. The device of claim 73 wherein the environment being viewed is a performance and the permissible cameras change based on the duration of time from the start of the performance.

77. The device of claim 73 wherein the permissible cameras are predetermined.

78. The device of claim 77 further including memory coupled to the processing element, the memory storing an indication of the predetermined permissible cameras.

79. The device of claim 73 further including a user interface device associated with the user and coupled to the processing element, the user interface device providing user inputs to the processing element indicative of movement through the array, the processing element selecting permissible camera output based on the user input.

80. The device of claim 79 wherein the processing element is configured to disregard user input corresponding to a impermissible camera.

81. The device of claim 80 wherein the processing element is configured to disregard user input corresponding to a impermissible camera by awaiting new user input corresponding to a permissible camera.

82. The device of claim 80 wherein the processing element is configured to disregard user input corresponding to a impermissible camera by selecting a permissible camera output.

83. The device of claim 73 further including a data device other than a camera having an output coupled to the at least one processing element, the permissible cameras based on the data device output.

84. The device of claim 83 wherein the data device is a motion sensor.

85. The device of claim 83 wherein the data device is a microphone.

86. A method for providing a user with views of an environment from a plurality of cameras in accordance with user inputs, the method comprising:

electronically receiving user inputs indicative of a view from a desired camera;

identifying permissible cameras, the views from which are capable of being provided to the user; and

determining whether the desired camera is one of the permissible cameras; and

if the desired camera is not one of the permissible cameras, preventing the user from obtaining the view from the desired camera.

87. The method of claim 86 wherein the permissible cameras change while the user views the environment.

88. The method of claim 86 wherein identifying permissible cameras is based on time of day.

89. The method of claim 86 further including noting when the user begins to view the environment and wherein the permissible cameras are based on a period of time from when the user begins to view the environment.

90. The method of claim 86 wherein the environment is a performance and the permissible cameras are based on the performance.

91. A device for providing a user with a display of an environment in response to user inputs, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view;

memory storing an additional source output; and

at least one processing element coupled to the memory for receiving the additional source output, the processing element configured to interpret user inputs and select, based on the user inputs, a camera output to provide to the user, the additional source output to provide to the user, or both a camera view and the additional source output to provide to the user.

92. The device of claim 91 wherein the processing element is configured to mix the additional source output with camera output.

93. The device of claim 92 wherein the processing element is configured to mix the additional source output and camera output by switching from the camera output to the additional source output.

94. The system of claim 91 further including a plurality of user interface devices coupled to the processing element, each associated with a different user and having user inputs associated with movement among cameras in the array.

95. The system of claim 91 wherein the array of cameras includes a plurality of modular rails having a male end and a female end, the male end of one rail connectable to the female end of another rail, and a plurality of cameras connectable to each rail.

96. The system of claim 91 wherein the array includes at least one user navigable path of cameras and the user inputs include an indication of a speed of movement through the path.

97. The system of claim 91 wherein the processing element is further configured to override user inputs and select at least one predetermined camera output, thereby directing the user to a predetermined view.

98. The system of claim 97 wherein the processing element is configured to select the predetermined output based on a navigation control factor.

99. A method of providing a user with a view of an environment, the method comprising:

receiving electronic images of the environment from an array of cameras;

receiving via a telecommunication link user inputs indicative of movement through the array;

selecting camera images based on the user inputs;

retrieving additional source output from memory based on user input indicating a desire to view the additional source output; and

providing the user with an additional source output at one or more points in the array.

100. A system for remote seamless viewing of an environment from an array of cameras, each having an output representing an image of the environment, the device comprising:

an interface device having inputs for selecting a path through at least a portion of the array from which to view the environment, the path including a sequence of cameras, each camera in the sequence having a different point perspective and a field of view that overlaps that of an adjacent cameras; and

a display device for sequentially displaying the image from each camera in the sequence, thereby providing the user a seamless view of the environment.

101. The system of claim 100 wherein the images from the selected cameras are displayed at a rate of approximately twenty-four images per second.

102. The system of claim 100 wherein the user interface device includes at least one of the following devices:

a keyboard;

a joystick;

a mouse;

a touchscreen;

a biofeedback device; and

a voice recognition device.

103. The system of claim 100 wherein the images displayed by the display device are mixed.

104. The system of claim 103 wherein the images displayed are mixed by continuously switching from the image of a current camera in the sequence to the image of a next camera in the sequence.

105. The system of claim 104 wherein the inputs are for selecting a speed of progressing through the path and the images are switched in accordance with the selected speed.

106. The system of claim 103 wherein the images displayed are mixed by continuously compositing the image from a current camera in the sequence with the image of a next camera in the array.

107. The system of claim 100 wherein the inputs are for selecting a speed of progressing through the path.

108. The system of claim 100 wherein the display device displays a plurality of images stereoscopically.

109. A method for seamless viewing of an environment, the method comprising:

receiving electronically a first image from an array of cameras, the first image having a first field of view;

receiving electronically a second image from the array, the second image having a second field of view that overlaps the first field of view;

receiving electronically a third image from the array, the third image having a third field of view that overlaps the second field of view;

displaying the first, second, and third images in sequence to obtaining a seamless view through the environment.

110. The method of claim 109 wherein the displaying includes switching from the first image to the second image to the third image at a given speed.

111. The method of claim 110 further including selecting a speed of movement through the array, wherein the given speed is based on the selected speed.

112. The method of claim 109 wherein the displaying includes applying a given frame refresh rate to the images.

113. The method of claim 109 wherein the displaying includes compositing the first image with the second image and then compositing the second image with the third image.

114. The method of claim 109 wherein the displaying includes dissolving the first image into the second image and then dissolving the second image into the third image.

115. The method of claim 109 further including selecting an additional source output to be displayed and mixing the additional source output with the third image.

116. The method of claim 109 wherein the first, second and third images correspond to first, second, and third cameras, respectively.

117. The method of claim 116 wherein the second camera is adjacent to the first and third cameras.

118. The method of claim 109 wherein at least one of the images is a stereoscopic image obtained from a plurality of cameras in the array.

119. A telepresence system for providing a user with a display of an environment, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view;

an electronic storage device; and

at least one processing element coupled to the array for receiving camera outputs, the processing element also coupled to the electronic storage device, the processing element configured to electronically store outputs from cameras and an indication of which camera each output is associated with on the electronic storage device, thereby allowing subsequent retrieval of the stored outputs and viewing of the environment.