Navigable telepresence method and system utilizing an array of cameras

• US 6,522,325 B1
• Filed: 10/15/1999
• Issued: 02/18/2003
• Est. Priority Date: 04/02/1998
• Status: Expired due to Term

• Current Assignee: Kewazinga Corporation
• Original Assignee: Kewazinga Corporation
• Inventors: Weber, Andrew H., Worley, David C., Sorokin, Scott
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Hailu, Tadesse
• Application Number:
  US09/419,274
• Time in Patent Office: 1,222 Days
• Field of Search: 345/744, 345/716-726, 345/427, 345/737, 345/854, 345/848, 345/738, 345/419, 345/420, 348/159, 348/218, 348/36-39, 348/42, 348/51, 348/46-48, 382/154
• US Class Current: 345/427
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A telepresence system for providing a first user with a first display of an environment and simultaneously and independently providing a second user with a second display of the environment. In certain embodiments, the system includes a plurality of arrays of cameras, with each arrays are situated at varying lengths from the environment. In one embodiment, the cameras situated around the environment are removed after the cameras in the array are have transmitted the camera output to an associated storage node. A first user interface device has first user inputs associated with movement along a first path in the array, and a second user interface device has second user inputs associated with movement along a second path. The system interprets the first inputs and selects outputs from the storage nodes in the first path, and interprets second inputs and selects outputs from the storage nodes in the second path independently of the first inputs, thereby allowing a first user and a second user to navigate simultaneously and independently through the environment. In another embodiment, a telepresence system includes various techniques for mixing images of cameras along each path, such as, mosaicing and tweening, for effectuating seamless motion along such paths.

Claims

1. A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the array including at least one camera path wherein each path is defined by a series of cameras having progressively different perspectives of the environment;

a first user interface device associated with the first user having first user inputs associated with movement along a first path in the array;

a second user interface device associated with the second user having second user inputs associated with movement along a second path in the array;

at least one processing element coupled to the user interface devices for receiving user inputs, the processing element configured to;

interpret received first user inputs and select outputs of cameras in the first path, mix the outputs of cameras in the first path in accordance with the received first user inputs by sequentially mosaicing the selected outputs of cameras in the first path and

interpret received second inputs and select outputs of cameras in the second path independently of the first inputs, mix the outputs of cameras in the second path in accordance with the received second user inputs by sequentially mosaicing the selected outputs of cameras in the second path, thereby allowing the first user and second user to navigate simultaneously and independently through the array.

2. The system of claim 1 wherein the system further includes a memory storing additional source output, wherein the additional source output includes other than output of the environment and wherein the user inputs include an indication of viewing the additional source output, and the processing element is further configured to mix camera output and the additional source output upon receiving the indication to view the additional source output by mosaicing the camera output and the additional source output.

3. The system of claim 2 wherein the additional source output includes output from the group of outputs including:

computer graphic imagery, virtual world imagery, applets, film clips, and animation.

4. A device for providing a user with a display of an environment in response to user inputs, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the array having one or more paths wherein each path is defined by a series of cameras having progressively different perspectives of the environment;

memory storing an additional source output the additional source output including other than output of the environment; and

at least one processing element coupled to the memory for receiving the additional source output, the processing element configured to interpret user inputs and select, based on the user inputs, a camera output to provide to the user, the additional source output to provide to the user, or both a camera view and the additional source output to provide to the user, and the processing element is configured to mix the additional source output with camera output by mosaicing from the camera output to the additional source output.

5. A system for remote seamless viewing of an environment from an array of cameras, each having an output representing an image of the environment, the device comprising:

an interface device having inputs for selecting a path through at least a portion of the array from which to view the environment, the path including a sequence of cameras, each camera in the sequence having a different point perspective and a field of view that overlaps that of an adjacent cameras; and

a display device for sequentially displaying the image from each camera in the sequence by mosaicing the image of a current camera in the sequence to the image of a next camera in the sequence, thereby providing the user a seamless view of the environment.

6. A method for seamless viewing of an environment, the method comprising:

receiving electronically a first image from an array of cameras having progressively different perspectives of the environment, the first image having a first field of view;

receiving electronically a second image from the array, the second image having a second field of the view that overlaps the first field of view;

receiving electronically a third image from the array, the third image having a third field of view that overlaps the second field of view;

mosaicing the first image with the second image and then mosaicing the second image with the third image; and

displaying the first, second, third and mosaic images in sequence to obtain a seamless view through the environment.

7. The method of claim 6 further including selecting an additional source input to be displayed and mosaicing the additional source output with the third image, wherein the additional source output includes other than output of the environment.

8. The method of claim 6 wherein the first, second and third images correspond to first, second, and third cameras, respectively.

9. The method of claim 8 wherein the second camera is adjacent to the first and third cameras.

10. A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the array including at least one camera path, wherein each path is defined by a series of cameras having progressively different perspectives of the environment;

a first interface user device associated with the first user having first user inputs associated with movement along a first path in the array;

a second user interface device associated with the second user having second user inputs associated with movement along a second path in the array;

at least one processing element coupled to the user interface devices for receiving user inputs, the processing element configured to;

interpret received first user inputs and select outputs of cameras in the first path, mix the outputs of cameras in the first path in accordance with the received first user inputs by sequentially tweening the selected outputs of cameras in the first path, and

interpret received second inputs and select outputs of cameras in the second path independently of the first inputs, mix the outputs of cameras in the second path in accordance with the received second user inputs by sequentially tweening the selected outputs of cameras in the second path, thereby allowing the first user and second user to navigate simultaneously and independently through the array.

11. The system of claim 10 wherein the system further includes a memory storing additional source output, wherein the additional source output includes other than output of the environment and wherein the user inputs include an indication of viewing the additional source output, and the processing element is further configured to mix camera output and the additional source output upon receiving the indication to view the additional source output by tweening the camera output and the additional source output.

12. The system of claim 11 wherein the additional source output includes output from the group of outputs including:

computer graphic imagery, virtual world imagery, applets, film clips, and animation.

13. A device for providing a user with a display of an environment in response to user inputs, the system comprising:

an array of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the array having one or more paths wherein each path is defined by a series of cameras having progressively different perspectives of the environment;

memory storing an additional source output, the additional source output including output other than of the environment; and

at least one processing element coupled to the memory for receiving the additional source output, the processing element configured to interpret user inputs and select, based on the user inputs, a camera output to provide to the user, the additional source output to provide to the user, or both a camera view and the additional source output to provide to the user, and the processing element is configured to mix the additional source output with camera output by tweening from the camera output to the additional source output.

14. A system for remote seamless viewing of an environment from an array of cameras, each having an output representing an image of the environment, the device comprising:

an interface device having inputs for selecting a path through at least a portion of the array from which to view the environment, the path including a sequence of cameras, each camera in the sequence having a different point perspective and a field of view that overlaps that of an adjacent cameras; and

a display device for sequentially displaying the image from each camera in the sequence by tweening the image of a current camera in the sequence to the image of a next camera in the sequence, thereby providing the user a seamless view of the environment.

15. A method for seamless viewing of an environment, the method comprising:

receiving electronically a first image from an array of cameras having progressively different perspectives of the environment, the first image having a first field of view;

receiving electronically a second image from the array, the second image having a second field of view that overlaps the first field of view;

receiving electronically a third image from the array, the third image having a third field of view that overlaps the second field of view;

tweening the first image with the second image to obtain a first tweened image and then tweening the second image with the third image to obtain a second tweened image; and

displaying the first image, first tweened image, second image, second tweened image, and third image in sequence to obtain a seamless view through the environment.

16. The method of claim 15 further including selecting an additional source output to be displayed and tweening the additional source output with the third image, wherein the additional source output includes other than output of the environment and the displaying includes displaying the additional source output after the third image.

17. The method of claim 15 wherein the first, second and third images correspond to first, second, and third cameras, respectively.

18. The method of claim 17 wherein the second camera is adjacent to the first and third cameras.

19. A method of providing users with real time views of a remote environment, the method comprising:

receiving electronic images of the environment from an array of cameras, the array including at least one camera path through the environment, each path including a series of cameras having progressively different perspectives of the environment;

receiving a first input from a first user interface device associated with a first user, the first input indicating movement along a first path;

receiving a second input from a second user interface device associated with a second user, the second input indicating movement along a second path;

obtaining a first mixed image by mosaicing or tweening, with a first processing element, a first image with a second image in accordance with the first input;

obtaining a second mixed image by mosaicing or tweening, with a second processing element, a third image with a fourth image in accordance with the second input;

providing the first user with the first mixed image in substantially real time, thereby simulating movement along the first path; and

providing the second user with the second mixed image substantially in real time and simultaneous to providing the first user with the first mixed image, thereby independently simulating movement along the second path.

20. The method of claim 19 wherein receiving the first input includes receiving an indication of mosaicing or tweening an additional source output, the method further including obtaining a third mixed output by mosaicing or tweening the second image with the additional source output, wherein the additional source output includes other than output of the environment.

21. The method of claim 19 further including obtaining a third mixed image by mosaicing or tweening the second image with a fifth image in accordance with the first input and providing the first user the third mixed image.

22. A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment, the system comprising:

a plurality of removable arrays of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view;

at least one storage device including a plurality of storage nodes wherein the output of each camera is stored in an associated storage node, the storage nodes are accessible to permit at least one path for viewing the environment;

a first user interface device associated with the first user having first user inputs associated with movement along a first path in the environment, the first path including views from cameras in multiple arrays;

a second user interface device associated with the second user having second user inputs associated with movement along a second path in the environment;

at least one processing element coupled to the user interface devices for receiving user inputs including moving up down, clockwise around an environment, counter-clockwise around an environment, forward and backward indicative of movement through the environment, the processing element configured to interpret received first inputs and select outputs of the storage nodes forming the first path, and interpret received second inputs and select outputs of storage nodes forming the second path independently of the first inputs, thereby allowing the first user and second user to navigate simultaneously and independently through the environment, wherein each removable array is situated at different lengths from the environment and the first path includes a first view from a first array and a second view from a second array, thereby allowing the first user to simulate movement forwards and backwards in the environment.

23. The telepresence system of claim 22 wherein the outputs of the cameras are accessible by the processing element.

24. The telepresence system of claim 23 wherein each array is removed after the cameras in the array have transmitted the output to the associated storage node.

25. The telepresence system of claim 24 wherein each array is of cylindrical shape and of a varying diameter.

26. A telepresence system for providing a first user with a first display of an environment and a second user with a second display of the environment, the system comprising:

a plurality of removable arrays of cameras, each camera having an associated view of the environment and an associated camera output representing the associated view, the arrays situated at varying lengths from the environment and including at least one path for viewing the environment, each array capable of being removed after the cameras in the array have transmitted the output to an associated storage node;

at least one storage device including a plurality of storage nodes wherein the output of each camera is stored in an associated storage node, the storage nodes are accessible to permit at least one path for viewing the environment;

a first user interface device associated with the first user having first user inputs associated with movement along a first path in the environment;

a second user interface device associated with the second user having second user inputs associated with movement along a second path in the environment;

at least one processing element coupled to the user interface devices for receiving user inputs indicative of movement through the environment, the processing element configured to interpret received first inputs and select outputs of the storage node forming the first path, and interpret received second inputs and select outputs of the storage node forming the second path independently of the first inputs, thereby allowing the first user and second user to navigate simultaneously and independently through the environment, including movement forwards and backwards in the environment.

27. A method of providing a user with views of a remote environment, the method comprising:

receiving electronic images of the environment from a first array of cameras;

removing the first array from the environment;

receiving electronic images of the environment from a second array of cameras, wherein the first and second arrays capture images from different places in the environment;

storing images of the environment from at least the first array for retrieval;

receiving a first input from a first user interface device associated with a first user, the first input indicating movement along a first path, the first path including a first image from the first array and a second image from the second array;

sequentially providing the first user with the first image, and the second image, thereby simulating movement along the first path forward or backward in the environment.

28. The method of claim 27 wherein the arrays are situated concentric in the environment.

29. The method of claim 27 further comprising mixing the first image with the second image to produce a mixed image and sequentially providing the user with the first image, the mixed image and the second image.

30. The method of claim 28 wherein the path is defined by cameras each having a progressively different perspective of the environment.