Navigable telepresence method and system utilizing an array of cameras

• US 6,535,226 B1
• Filed: 04/01/1999
• Issued: 03/18/2003
• Est. Priority Date: 04/02/1998
• Status: Expired due to Term

• Current Assignee: Kewazinga Corporation
• Original Assignee: Kewazinga Corporation
• Inventors: Weber, Andrew H., Worley, David C., Sorokin, Scott
• Primary Examiner(s): Cabeca, John
• Assistant Examiner(s): Hailu, Tadesse
• Application Number:
  US09/283,413
• Time in Patent Office: 1,447 Days
• Field of Search: 345/744, 345/716, 345/427, 345/848, 345/850, 345/419, 345/726, 348/159, 348/218, 348/716, 348/719, 348/518, 348/36-39, 386/118, 386/119, 382/154, 600/111
• US Class Current: 715/723
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A telepresence method and system for providing users the ability to navigate through an array of cameras capturing an environment. The array of cameras includes one or more series of cameras, wherein each series of cameras defines a path through the environment and wherein the cameras in each series have a progressively different perspective of the environment. Users interactively navigate through series of cameras, thereby simulating movement along paths.

Claims

1. A telepresence system for allowing a first user, in response to at least one first input, to navigate along a first path through an environment and a second user, in response to at least one second input, to navigate along a second path through the environment independently of the first user, the system comprising:

an array of cameras including a first series of cameras defining the first path through the environment, wherein the cameras in the first series have progressively different perspectives of the environment along the first path, and a second series of cameras defining the second path through the environment, wherein the cameras in the second series have progressively different perspectives of the environment along the second path;

at least one processing element coupled to the array, the processing element configured to select outputs of cameras in the first series, based on the first inputs, and to cause the selected outputs of cameras in the first series to be sequentially provided to the first user, thereby allowing the first user to progressively navigate through the environment along the first path, the at least one processing element also configured to select outputs of cameras in the second series, based on the second inputs, and to cause the selected outputs of cameras in the second series to be sequentially provided to the second user, thereby allowing the first user and second user to navigate simultaneously and independently through the environment along paths defined by cameras.

2. The system of claim 1 further including:

a first display device and first user interface device associated with the first user and coupled to the processing element; and

a second display device and second user interface device associated with the second user and coupled to the processing element,

wherein the processing element is further configured to select a first output based on first inputs received from interface device and send output to display device, the first display device configured to display the selected first output, and

select a second output based on second inputs received from the second interface device and send the second output to the second display device, the second display device configured to display the selected second output.

3. The system of claim 2 wherein the processing element is configured to select outputs in real time and the display devices are configure to display views in real time.

4. The system of claim 2 wherein the first user interface device and the first display device are coupled to the processing element by different types of communication links.

5. The system of claim 1 wherein the array includes a plurality of navigable camera paths and the first path is different than the second path.

6. The system of claim 1 wherein the processing element includes a control processing element and a plurality of mixing processing elements.

7. The system of claim 1 wherein the processing element is also coupled to the array.

8. The system of claim 1 wherein the processing element includes a plurality of processing elements.

9. The system of claim 1 wherein the user interface device is coupled to the processing element by one of the following communication links:

an internet link, a microwave link, a satellite link, a wireless link, or a fiber optics link.

10. The system of claim 1 wherein the first user interface device and second user interface device are coupled to the processing element by different communication links.

11. The system of claim 1 wherein the first user interface device and second user interface device are different types of devices.

12. The system of claim 1 wherein the array of cameras includes a plurality of modular rails having a male end and a female end, the male end of one rail connectable to the female end of another rail, and a plurality of cameras connectable to each rail.

13. The system of claim 1 wherein the array of cameras includes hollow ball joints, each of which houses a camera, and hollow rods interconnecting the ball joints, the rods housing communication paths for the outputs.

14. The system of claim 1 wherein the cameras in the array are positioned at a density of approximately one camera every inch.

15. The system of claim 1 wherein each camera in the first series has a view of the environment that overlaps the view of an adjacent camera in the first series.

16. The system of claim 1 wherein the first series of cameras includes a first camera having a first output and a second camera having a second output, the first and second cameras being adjacent one another, and wherein the processing element is further configured to mix the first and second outputs in accordance with the first input.

17. The system of claim 16 wherein the first inputs indicate a desire to move from the first camera to the second camera, and the processing element is configured to mix the first and second outputs by electronically switching between the first and second outputs at a switching speed.

18. The system of claim 17 wherein the processing element is further configured to mix the outputs by adjusting the switching speed.

19. The system of claim 16 wherein the first user inputs include an indication of relative speed of movement through the series of cameras and the processing element is configured to mix the first output and second output by switching from the first output to the second output at a switching speed, the processing element further configured to adjust the switching speed in accordance with the indication of relative speed.

20. The system of claim 16 wherein the first camera and record camera each have an associated view of the environment and the processing element is further configured to mix the outputs by dissolving the view associated with the first camera with the view of the second camera.

21. The system of claim 16 further comprising a first user display device associated with the first user and coupled to the processing element and wherein the first display device has a frame refresh rate associated therewith and the processing element is configured to mix the outputs by adjusting the frame refresh rate.

22. The system of claim 16 wherein the processing element is configured to mix the outputs by creating a composite output from the first and second outputs.

23. The system of claim 16 wherein the first series of cameras further includes a third camera adjacent the second camera and wherein the processor is configured to mix, in response to first inputs associated with moving along the first path from the first camera to the third camera, the outputs of the first camera and the second camera, and then mix the outputs of the second camera and the third camera.

24. The system of claim 16 wherein the processing element is configured to mix outputs by adding motion blur.

25. The system of claim 1 wherein the first inputs include an indication of a speed of movement through the first path series of cameras.

26. The system of claim 25 wherein the processing element is further configured to compare the indication of speed to a predetermined threshold and to select a number of outputs corresponding to fewer than all cameras in the series upon determining the indication of speed exceeds the predetermined threshold.

27. The system of claim 25 wherein the number of outputs is inversely proportional to the speed.

28. The system of claim 25 wherein the indication of relative speed of movement through the array includes an indication of moving relatively slower through the array and the processing element adjusts the switching speed by causing the view associated with the first camera to be displayed for a relatively longer period of time in accordance with the indication of moving slower.

29. The system of claim 1 wherein the user interface device includes a keyboard, wherein user inputs are keystrokes.

30. The system of claim 1 wherein the user interface device includes a joystick wherein user inputs are movements of the joystick.

31. The system of claim 1 wherein the user interface device includes a mouse, wherein user inputs are movements of the mouse.

32. The system of claim 1 wherein the user interface device includes a touchscreen.

33. The system of claim 1 wherein the user interface device includes a biofeedback device.

34. The system of claim 1 wherein the user interface device includes a voice recognition device.

35. The system of claim 34 wherein the user inputs further include moving forward and backward.

36. The system of claim 1 wherein the user inputs include an indication of a particular camera in the array.

37. The system of claim 1 wherein the user inputs include moving up, down, to the right, and to the left.

38. The system of claim 1 wherein the processing element operates in accordance with a navigation control algorithm, the navigation control algorithm distinguishing between permissible movement and impermissible movement in the array.

39. The system of claim 38 further including navigation control factors, wherein the navigation control algorithm controls movement through the array in accordance with the navigation control factors.

40. The system of claim 39 wherein the navigation control factors include one of the group of factors consisting of:

time of day, time duration, speed of navigation through the array, output of a device in the array, output of a motion sensor, and output of a microphone.

41. The system of claim 1 wherein the processing element is further configured to disregard a certain one or more of the fist imprints or the second imprints.

42. The system of claim 41 wherein the processing element is further configured to disregard the certain input by selecting at least one predetermined camera output, thereby directing the user to a predetermined view of the environment.

43. The system of claim 42 wherein the processing element is configured to select the predetermined output based on time of day.

44. The system of claim 42 wherein the processing element is configured to select the predetermined output based on a duration of time.

45. The system of claim 41 wherein the processing element is further configured to disregard the certain input by waiting for a permissible input.

46. The system of claim 41 further including an operator input device coupled to the processing element for receiving from the operator input device an operator input, and wherein the processing element is controlled to disregard the certain input based on the operator input.

47. The system of claim 1 wherein the system further includes a memory storing additional source output, wherein the first inputs include an indication of viewing the additional source output, and the processing element is further configured to mix camera output and the additional source output upon receiving the indication to view the additional source output.

48. The system of claim 47 wherein the indication of viewing the additional source output is available to the first user in connection with at least one predetermined camera.

49. The system of claim 47 wherein the processing element is configured to mix camera output and the additional source output by compositing camera output and the additional source output.

50. The system of claim 47 wherein the processing element is configured to mix camera output and the additional source output by switching from camera output to the additional source output.

51. The system of claim 47 wherein the additional source output includes output from the group of outputs consisting of:

computer graphic imagery, virtual world imagery, applets, film clips, and animation.

52. The system of claim 1 further including a plurality of microphones, each microphone having an output, wherein the processing element is further configured to select a microphone output based on the first input for transmission to the first user.

53. The system of claim 52 further including a memory coupled to the processing element, the memory correlating cameras and microphones, the processing element selecting a microphone by using the memory to correlate the selected camera output with a microphone output.

54. A method of providing users with views of a remote environment, the method comprising:

receiving electronic images of the environment captured from an array of cameras, the array including at least a first series of cameras defining a first path through the environment, the first series of cameras having progressively different perspectives of the environment along the first path;

receiving a first input from a first user interface device associated with a first user, the first input indicating movement through the environment along the first path;

providing to the first user images from the first series of cameras in sequence along the first path, thereby simulating movement by the first user through the environment along the first path.

55. The method of claim 54 wherein the receiving images including receiving images from a second series of cameras in the array, the second series defining a second path through the environment, the second series of cameras having progressively different perspectives on the environment along the second path and wherein the method further includes:

receiving a second input from a second user interface device associated with a second user, the second input indicating movement through the environment along the second path;

providing to the second user images from the second series of cameras in sequence along the second path, thereby simulating movement by the second user through the environment along the second path, independently of simulating movement by the first user through the environment.

56. The method of claim 55 wherein the first path differs from the second path.

57. The method of claim 54 wherein the providing of the images to the first user includes obtaining a first mixed image by mixing the image from a first camera in the first series with the image from a second camera in the first series and providing the first mixed image to the first user.

58. The method of claim 57 wherein mixing the first image with the second image includes compositing the first image with the second image.

59. The method of claim 58 wherein mixing the third image with the fourth image includes switching between the third image and the fourth image.

60. The method of claim 57 wherein mixing the first image with the second image includes dissolving the first image into the second image.

61. The method of claim 57 wherein the first image is from a first camera, the second image is from a second camera, the array includes another camera interposed between the first and second cameras, and mixing the first image with the second image includes providing the first user with an image from the interposed camera.

62. The method of claim 57 further including receiving a third input from the first user interface device and overriding the third input.

63. The method of claim 62 wherein overriding the received third input includes presenting the first user with a predetermined image.

64. The method of claim 62 wherein overriding the received third input includes continuing to provide the first user with the first mixed image and awaiting a fourth input.

65. The method of claim 57 wherein receiving the first input includes receiving an indication of mixing an additional source output, the method further including obtaining a third mixed output by mixing the second image with the additional source output.

66. The method of claim 65 wherein mixing the second image with the additional source output includes switching from the second image to the additional source output.

67. The method of claim 54 wherein mixing the first image with the second image includes switching from the first image to the second image at a first switching speed, the first mixed image including the second image, and mixing the third image with the fourth image includes switching from the third image to the fourth image at a second switching speed, the second mixed image including the fourth image.

68. The method of claim 67 wherein receiving the first input includes receiving an indication of a first speed of movement along the first path and wherein the first switching speed is based on the indication of the first speed of movement.

69. The method of claim 68 wherein receiving the second input includes receiving an indication of a second speed of movement along the second path, the second speed greater than the first speed, and wherein the second switching speed is based on the indication of the second speed of movement, the second switching speed greater than the first switching speed.

70. The method of claim 69 wherein mixing the third and fourth images involves providing the second user the third image for a relatively longer duration than the first image is provided to the first user.

71. The method of claim 69 wherein mixing the third and fourth images involves providing the second user the fourth image for a relatively longer duration than the second image is provided to the first user.

72. The method of claim 69 further including obtaining a third mixed image by mixing the second image with a fifth image, the fifth image from a camera in the first series, in accordance with the first input and providing the first user the third mixed image.

73. A device for providing a user with a display of an environment, the device comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view; and

at least one processing element coupled to the array, the processing element configured to identify permissible cameras, the outputs of which may be provided to the user, and impermissible cameras, the outputs of which are not provided to the user, despite receiving a request from the user to receive the output of an impermissible camera.

74. The device of claim 73 wherein the output of a certain impermissible camera is not provided to the user despite a request from the user to receive the output of the certain impermissible camera.

75. The device of claim 74 wherein the permissible cameras change based on the time of day.

76. The device of claim 73 wherein the environment being viewed is a performance and the permissible cameras change based on the duration of time from the start of the performance.

77. The device of claim 73 wherein the permissible cameras are predetermined.

78. The device of claim 77 further including memory coupled to the processing element, the memory storing an indication of the predetermined permissible cameras.

79. The device of claim 73 further including a user interface device associated with the user and coupled to the processing element, the user interface device providing user inputs to the processing element indicative of movement through the array, the processing element selecting permissible camera output based on the user input.

80. The device of claim 79 wherein the processing element is configured to disregard user input corresponding to a impermissible camera.

81. The device of claim 80 wherein the processing element is configured to disregard user input corresponding to a impermissible camera by awaiting new user input corresponding to a permissible camera.

82. The device of claim 80 wherein the processing element is configured to disregard user input corresponding to a impermissible camera by selecting a permissible camera output.

83. The device of claim 73 further including a data device other than a camera having an output coupled to the at least one processing element, the permissible cameras based on the data device output.

84. The device of claim 83 wherein the data device is a motion sensor.

85. The device of claim 83 wherein the data device is a microphone.

86. A method for providing a user with views of an environment from a plurality of cameras in accordance with user inputs,the method comprising:

electronically receiving user inputs indicative of a view from a camera;

identifying permissible cameras, the views from which are capable of being provided to the user; and

determining whether the camera is one of the permissible cameras; and

if the camera is not one of the permissible cameras, preventing the user from obtaining the view from the camera, wherein the permissible cameras change over time.

87. The method of claim 86 wherein the permissible cameras change while the user views the environment.

88. The method of claim 86 wherein identifying permissible cameras is based on time of day.

89. The method of claim 86 further including noting when the user begins to view the environment and wherein the permissible cameras are based on a period of time from when the user begins to view the environment.

90. The method of claim 86 wherein the environment is a performance and the permissible cameras are based on the performance.

91. A device for providing a user with a display of an environment in response to user inputs, the system comprising:

an array of cameras including one or more series of cameras in one or more paths, each camera in a series has a progressively different associated view of the environment along a path and an associated camera output representing the associated view;

memory storing an additional source output, the additional source output including other than the output of the array of cameras; and

at least one processing element coupled to the memory for receiving the additional source output, the processing element configured to interpret user inputs and select, based on the user inputs, a sequence of outputs of cameras in a path, or the additional source output to provide to the user.

92. The device of claim 91 wherein the processing element is configured to mix the additional source output with camera output.

93. The device of claim 92 wherein the processing element is configured to mix the additional source output and camera output by switching from the camera output to the additional source output.

94. The system of claim 91 further including a plurality of user interface devices coupled to the processing element, each associated with a different user and having user inputs associated with movement among cameras in the array.

95. The system of claim 91 wherein the array of cameras includes a plurality of modular rails having a male end and a female end, the male end of one rail connectable to the female end of another rail, and a plurality of cameras connectable to each rail.

96. The system of claim 91 wherein the array includes at least one user navigable path of cameras and the user inputs include an indication of a speed of movement through the path.

97. The system of claim 91 wherein the processing element is further configured to override user inputs and select at least one predetermined camera output, thereby directing the user to a predetermined view.

98. The system of claim 97 wherein the processing element is configured to select the predetermined output based on a navigation control factor.

99. A method of providing a user with a view of an environment, the method comprising:

receiving electronic images of the environment from an array of cameras, the array including one or more series of cameras in one or more paths, each camera in a series having a progressively different associated view of the environment along a path;

receiving via a telecommunication link user inputs indicative of movement through the array;

selecting a sequence of camera images in a path based on the user inputs;

retrieving additional source output from memory based on user input indicating a desire to view the additional source output, the additional source output including other than output of the array of cameras; and

providing the user with the additional source output at one or more points in the array.

100. A system for remote seamless viewing of an environment from an array of cameras, each having an output representing an image of the environment, the system comprising:

an interface device having inputs for selecting a path through at least a portion of the array from which to view the environment, the path including a sequence of cameras, each camera in the sequence having a different point perspective and a field of view that overlaps that of an adjacent camera; and

a display device for sequentially displaying at least a portion of the images selected by the processing element, thereby providing the user a substantially seamless view of the environment along the path.

101. The system of claim 100 wherein the images from the selected cameras are displayed at a rate of approximately twenty-four images per second.

102. The system of claim 100 wherein the user interface device includes at least one of the following devices:

a keyboard;

a joystick;

a mouse;

a touchscreen;

a biofeedback device; and

a voice recognition device.

103. The system of claim 100 wherein the images displayed by the display device are mixed.

104. The system of claim 103 wherein the images displayed are mixed by continuously switching from the image of a current camera in the path to the image of a next camera in the path.

105. The system of claim 104 wherein the inputs are for selecting a speed of progressing through the path and the images are switched in accordance with the selected speed.

106. The system of claim 103 wherein the images displayed are mixed by continuously compositing the image from a current camera in the sequence with the image of a next camera in the array.

107. The system of claim 100 wherein the inputs are for selecting a speed of progressing through the path.

108. The system of claim 100 wherein the display device displays a plurality of images stereoscopically.

109. A method for a user experiencing movement through an environment along a path of cameras in an array, the cameras having progressively different point perspectives of the environment, the method comprising:

receiving electronically a first image captured from the array of cameras, the first image having a first field of view from a first point perspective;

receiving electronically a second image captured from the array, the second image having a second field of view from a second point perspective, the second field of view overlapping the first field of view;

receiving electronically a third image captured from the array, the third image having a third field of view from a third point perspective, the third field of view overlapping that overlaps the second field of view, wherein the first, second and third point perspectives are on the path;

displaying the first, second, and third images in sequence to the user, thereby obtaining an experience of seamless movement through the environment along the path.

110. The method of claim 109 wherein the displaying includes switching from the first image to the second image to the third image at a given speed.

111. The method of claim 110 further including selecting a speed of movement through the array, wherein the given speed is based on the selected speed.

112. The method of claim 109 wherein the displaying includes applying a given frame refresh rate to the images.

113. The method of claim 109 wherein the displaying includes compositing the first image with the second image and then compositing the second image with the third image.

114. The method of claim 109 wherein the displaying includes dissolving the first image into the second image and then dissolving the second image into the third image.

115. The method of claim 109 further including selecting an additional source output to be displayed and mixing the additional source output with the third image.

116. The method of claim 109 wherein the first, second and third images correspond to first, second, and third cameras, respectively.

117. The method of claim 116 wherein the second camera is adjacent to the first and third cameras.

118. The method of claim 109 wherein at least one of the images is a stereoscopic image obtained from a plurality of cameras in the array.

119. A telepresence system for providing multiple users with simulated independent movement through an environment, the system comprising:

an array of cameras including a series of cameras, each camera in the series having an associated view of the environment from a progressively different point perspective and an associated camera output representing the associated view;

an electronic storage device for sharing outputs among multiple users; and

at least one processing element coupled to the electronic storage device, the processing element configured to electronically store outputs from cameras in the series and an indication of which camera each output is associated with on the electronic storage device, thereby allowing subsequent simultaneous retrieval of the stored outputs by multiple users; and

the processing element also configured to sequentially retrieve for multiple users from the electronic storage device.stored outputs from cameras in the series, thereby simulating progressive movement along the different point perspectives of the views associated with the retrieved outputs.