Multi-user and multi-surrogate virtual encounters

• US 10,223,821 B2
• Filed: 04/25/2017
• Issued: 03/05/2019
• Est. Priority Date: 04/25/2017
• Status: Active Grant

• Current Assignee: Beyond Imagination Inc.
• Original Assignee: Beyond Imagination Inc.
• Inventors: Kurzweil, Raymond C.
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/496,213
• Publication Number:
  US 20180308268A1
• Time in Patent Office: 679 Days
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A virtual reality encounter system is described. A first surrogate supporting at least one first camera that captures image data from a first physical location and a second surrogate supporting at least one second camera that captures second image data from the first physical location. Aliasing substitution processing has a computing system including a processor receive the first image data and detect an image of the second surrogate in the first image data and replace the image data of the second surrogate in the first physical location, with image data of a user in the first physical location to form a transformed image that substitutes the image data of the user for the image data of the second surrogate.

Claims

1. A virtual reality encounter system comprising,

a first surrogate comprising a body and supporting at least one first camera that captures image data from a first physical location in which the first surrogate is disposed to produce a first image signal;

a second surrogate comprising a body and supporting at least one second camera that captures second image data from the first physical location in which the second surrogate is disposed to produce a second image signal;

a processor configured to;

receive the first image signal;

detect an image of the second surrogate in the first image signal;

replace the image data of the second surrogate in the first physical location, with image data of a user in the first physical location to form a transformed image that substitutes the image data of the user for the image data of the second surrogate; and

a user device in a second, different location, with the user device comprising a display, with the display configured to receive the transformed image.

2. The system of claim 1, further comprising:

an adapter to send the transformed image having the replaced image data to a communications network.

3. The system of claim 1 wherein the user device is a set of goggles to render the second video image, which goggles are in the second, different physical location, with the set of goggles including a pair of displays and a transducer.

4. The system of claim 1, further comprising:

a third surrogate supporting at least one third camera that captures third image data from the second, different physical location in which the third surrogate is disposed to produce a third image signal.

5. The system of claim 1 wherein the processor is configured to:

receive image frames from the output of a camera carried by the first surrogate; and

process the image frames to detect the second surrogate in a field of view of the camera by applying data received from electronic compasses attached to the first surrogate and the second surrogate to determine orientations and relative positions of the first surrogate and the second surrogate.

6. The system of claim 1 wherein the processor is configured to:

receive image frames from the output of a camera carried by first surrogate;

processed the image frames to detect the second surrogate in a field of view of the camera by applying recognition processing to determine whether and where in the image the second surrogate appears; and

replace the image of the second surrogate with the image of the user.

7. The system of claim 6 wherein the recognition processing is facial recognition and processor is configured to:

apply facial recognition processing that compares selected features in the image to stored facial features of the second surrogate.

8. The system of claim 1, further comprising:

cameras arranged in the location, and the processor is configured to;

capture in real time a moving digital image of the user moving through the first location using one or more of the cameras arranged in the first location; and

select which camera to supply real-time moving image data to replace the image data of the second surrogate in the first physical location to form the transformed image.

9. The system of claim 8 wherein compass data is used to select the camera to supply the moving digital image.

10. The system of claim 1, further comprising:

cameras arranged in the first location, and wherein the processor is configured to;

determine whether the second surrogate is within a field of view of one or more of the cameras;

determine where in the image frames is the image of the second surrogate; and

generate a modified image that depicts movement of the user substituted for second surrogate in the image frames.

11. The system of claim 1 wherein the surrogates are robots.

12. The system of claim 1 wherein the processor is further configured to:

receive image data as a plurality of image frames of data; and

apply image recognition to the plurality of image frames of data to detect the image of the second surrogate and location of the image of the second surrogate in the plurality of image frames.

13. The system of claim 12 wherein the processor is further configured to:

retrieve image data of the user in the first location;

scale the retrieved image data of the user; and

generate a modified image frame that is transformed by substitution of the image of second surrogate at an identified location in the image frame with the scaled, image of the user.

14. The system of claim 4 wherein the processor configured to detect the image is further configured to:

receive electronic compass data from the first surrogate and the second surrogate, the electronic compass data including orientation and relative position of the first surrogate and the second surrogate; and

determining from a field of view of the at least one first camera and the electronic compass data from the first surrogate and the second surrogate, location data of the second surrogate within the field of view of the at least one first camera.

15. The system of claim 14 wherein the processor is further configured to:

determine based on the received image data and the determined location data, a location of the image of the second surrogate in the image data;

retrieve image data of the user; and

scale the retrieved image data to replace the image of the second surrogate by substituting the scaled retrieved image data of the user for the image data of the second surrogate.

16. The system of claim 1 wherein the first and second surrogates are first and second robots, the user device that receives the transformed image is a first user device that receives the transformed image transformed from the image sent by the first robot, and the system further comprises:

a third robot supporting at least one third camera that captures third image data from the second, different physical location in which the third surrogate is disposed to produce a third image signal;

a second user device comprising a display and transducer, the third user device disposed in the second location and configured to receive image data from the second robot;

a third user device comprising a display and transducer, the third user device disposed in the first location and configured to receive image data from the third robot.

17. The system of claim 16, further comprising:

a fourth robot supporting at least one fourth camera that captures fourth image data from the second, different physical location in which the fourth surrogate is disposed to produce a fourth image signal; and

a fourth user device comprising a display and transducer, the fourth user device disposed in the first location, and configured to receive image data from the fourth robot.

18. The system of claim 1, further comprising:

a first computing system in the first location with the first computing system including the processor; and

a second computing system in the second location, the second computing system connected to the first computing system via a network.

19. A virtual reality encounter method comprises:

producing by a camera, supported on a first surrogate comprising a body, image data from a first physical location in which the first surrogate is disposed to produce a first image signal;

producing by a camera, supported on a second surrogate comprising a body, image data from the first physical location in which the second surrogate is disposed to produce a second image signal;

receiving by a computer system the first image signal;

detecting by the computer system an image of the second surrogate in the first image signal;

replacing by the computer system the image data of the second surrogate in the first physical location, with image data of a user in the first physical location to form a transformed image that substitutes the image data of the user for the image data of the second surrogate; and

receiving by a user device in a second, different location, the transformed image with the user device comprising a display configured to receive the transformed image.

20. The method of claim 19, further comprising:

sending the transformed image having the replaced image data to the user device over a communications network.

21. The method of claim 19 further comprising:

receiving audio from the first location, and wherein the user device is a set of goggles to render the second video image, which goggles are in the second, different physical location, with the set of goggles including a pair of displays and an audio transducer that renders the received audio.

22. The method of claim 19 further comprising:

producing by a camera, supported on a third surrogate comprising a body, image data from the second physical location in which the third surrogate is disposed to produce a third image signal.

23. The method of claim 19 further comprising:

receiving image frames from the output of a camera carried by first surrogate;

processing the image frames to detect the second surrogate in a field of view of the camera by applying recognition processing to determine whether and where in the image the second surrogate appears; and

replacing the image of the second surrogate with the image of the user.

24. The method of claim 23 wherein the recognition processing is facial recognition and the method further comprises:

applying facial recognition processing that compares selected features in the image to stored facial features of the second surrogate.

25. The method of claim 19 further comprising:

capturing in real time a moving digital image of the user moving through the first location using one or more cameras arranged in the first location; and

selecting which camera to supply real-time moving image data to replace the image data of the second surrogate in the first physical location to form the transformed image.

26. The method of claim 19 further comprising:

arranging cameras in the first location;

determining whether the second surrogate is within a field of view of one or more of the cameras;

determining where in the image frames is the image of the second surrogate; and

generating a modified image that depicts movement of the user substituted for second surrogate in the image frames.

27. The method of claim 19 wherein the surrogates are robots.

28. The method of claim 19 further comprising:

receiving image data as a plurality of image frames of data; and

applying image recognition to the plurality of image frames of data to detect the image of the second surrogate and location of the image of the second surrogate in the plurality of image frames.

29. The method of claim 19 further comprising:

retrieving image data of the user in the first location;

scaling the retrieved image data of the user; and

generating a modified image frame that is transformed by substitution of the image of second surrogate at an identified location in the image frame with the scaled, image of the user.

30. The method of claim 19 wherein the first and second surrogates are first and second robots, the user device that receives the transformed image is a first user device that receives the transformed image transformed from the image sent by the first robot, and the method further comprises:

producing by a camera, supported on a third robot comprising a body, image data from a the second physical location in which the third surrogate is disposed to produce a third image signal.

31. A computer program product stored on non-transitory hardware storage devices, the computer program product comprises instructions to cause a machine to:

receive a first image signal from a first camera that captures image data from a first physical location in which a first surrogate is disposed;

detect an image of a second surrogate in the first image signal;

replace the image data of the second surrogate in the first physical location, with image data of a user in the first physical location to form a transformed image that substitutes the image data of the user for the image data of the second surrogate;

receive image frames from the output of the camera carried by the first surrogate; and

process the image frames to detect the second surrogate in a field of view of the camera by applying recognition processing to determine whether and where in the image the second surrogate appears; and

replace the image of the second surrogate with the image of the user.

32. The product of claim 31 wherein the recognition processing is facial recognition and the product further includes instructions to:

apply facial recognition processing that compares selected features in the image to stored facial features of the second surrogate.

33. The product of claim 31 further includes instructions to:

capture in real time a moving digital image of the user moving through the first location using one or more of the cameras arranged in the first location; and

select which camera to supply real-time moving image data to replace the image data of the second surrogate in the first physical location to form the transformed image.

34. The product of claim 31 further includes instructions to:

determine whether the second surrogate is within a field of view of one or more cameras;

determine where in the image frames is the image of the second surrogate; and

generate a modified image that depicts movement of the user substituted for second surrogate in the image frames.