Systems And Methods For Providing Real-Time Composite Video From Multiple Source Devices Featuring Augmented Reality Elements

• US 20180365904A1
• Filed: 08/29/2018
• Published: 12/20/2018
• Est. Priority Date: 09/02/2016
• Status: Active Grant

• Current Assignee: Russell Holmes
• Original Assignee: Russell Holmes
• Inventors: Holmes, Russell
• Granted Patent: US 10,636,215 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: H04M 11/085 using a television receiver, e.g. viewdata system
• CPC Class Codes: H04M 2201/50 Telephonic communication in combination with video communication
• CPC Class Codes: H04M 2203/359 Augmented reality
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Systems and methods for superimposing the human elements of video generated by computing devices, wherein a first user device and second user device capture and transmit video to a central server which analyzes the video to identify and extract human elements, superimpose these human elements upon one another, adds in at least one augmented reality element, and then transmits the newly created superimposed video back to at least one of the user devices.

Claims

1. A computer-implemented method of superimposing video carried out by a processor, the method comprising the steps of:

receiving a first live video from a first user device, the first live video including video of a first user captured by a rear facing camera of the first user device;

receiving a second live video from a second user device, the second live video including video of a second user captured by a front facing camera of the second user device;

identifying and extracting, on a continuous basis, a first human element of the first user from the first live video using a detection algorithm;

combining the first human element of the first user and a portion or all of the second live video in real-time to create a superimposed video including a frame perimeter within which the superimposed video includes the first human element of the first user captured by the rear facing camera of the first user device from the first live video and a second human element of the second user captured by the front facing camera of the second user device from the second live video, wherein, within the superimposed video, the first human element of the first user overlaps and the second human element of the second user; and

transmitting the superimposed video to at least one of the first user device and the second user device.

2. The computer-implemented method of superimposing video of claim 1, wherein in response to real-time movement by the first human element of the first user relative to the first user device and the second human element of the second user relative to the second user device, contact is simulated between the first human element of the first user and the second human element of the second user in the superimposed video.

3. The computer-implemented method of superimposing video of claim 1, the method further comprising the steps of:

wherein, in response to movement of the first human element of the first user relative to the first user device, the first human element of the first user obscures at least a portion of the second human element of the second user in the transmitted superimposed video; and

further wherein the location of the extracted first human element of the first user within the superimposed video is directly controlled in real-time by the position of the first human element of the first user relative to the first user device.

4. The computer-implemented method of superimposing video of claim 1, wherein the first user device is a mobile computing device.

5. The computer-implemented method of superimposing video of claim 1, wherein at least one of the rear facing camera of the first user device and the front facing camera of the second device is a depth sensing camera.

6. The computer-implemented method of superimposing video of claim 1, wherein the superimposed video is communicated between the first user device and the second user device via a peer-to-peer network communication.

7. The computer-implemented method of superimposing video of claim 1, wherein the superimposed video includes an augmented reality graphic.

8. The computer-implemented method of superimposing video of claim 1, wherein the detection algorithm includes one or more of foreground and background separation, proximity recognition, Chroma keying, hand-arm/body detection, background subtraction, and a Kalman filter.

9. The computer-implemented method of superimposing video of claim 1, wherein the step of identifying and extracting, on a continuous basis, a first human element of the first user from the first live video using a detection algorithm further comprises automatically selecting the detection algorithm to optimize processor utilization.

10. The computer-implemented method of superimposing video of claim 1, wherein the step of identifying and extracting, on a continuous basis, a first human element of the first user from the first live video using a detection algorithm further comprises automatically selecting the detection algorithm to optimize image quality within the superimposed video.

11. A computer-implemented system for superimposing video, comprising:

a first user device featuring a camera, processor, memory, and networking interface;

a second user device featuring a camera, processor, memory, and networking interface;

wherein one of the first user device'"'"'s processor and the second user device'"'"'s processor;

receives a first live video from a first user device, the first live video including video of a first user captured by a rear facing camera of the first user device and a second live video from a second user device, the second live video including video of a second user captured by a front facing camera of the second user device;

identifies and extracts, on a continuous basis, a first human element of the first user from the first live video using a detection algorithm;

combines the first human element of the first user and a portion or all of the second live video in real-time to create a superimposed video including a frame perimeter within which the superimposed video includes the first human element of the first user captured by the rear facing camera of the first user device from the first live video and a second human element of the second user captured by the front facing camera of the second user device from the second live video, wherein, within the superimposed video, the first human element of the first user overlaps the second human element of the second user; and

transmits the superimposed video to at least one of the first user device and the second user device.

12. The computer-implemented system for superimposing video of claim 11, wherein in response to real-time movement by the first human element of the first user relative to the first user device and the second human element of the second user relative to the second user device, contact is simulated between the first human element of the first user and the second human element of the second user in the superimposed video.

13. The computer-implemented system for superimposing video of claim 11, wherein one of the first user device'"'"'s processor and the second user device'"'"'s processor:

wherein, in response to movement of the first human element of the first user relative to the first user device, the first human element of the first user obscures at least a portion of the second human element of the second user in the transmitted superimposed video; and

further wherein the location of the extracted first human element of the first user within the superimposed video is directly controlled in real-time by the position of the first human element of the first user relative to the first user device.

14. The computer-implemented system of superimposing video of claim 11, wherein the first user device is a mobile computing device.

15. The computer-implemented system of superimposing video of claim 11, wherein at least one of the rear facing camera of the first user device and the front facing camera of the second device is a depth sensing camera.

16. The computer-implemented system for superimposing video of claim 11, wherein the superimposed video is communicated between the first user device and the second user device via a peer-to-peer network communication.

17. The computer-implemented system for superimposing video of claim 11, wherein the superimposed video includes an augmented reality graphic.

18. The computer-implemented system for superimposing video of claim 11, wherein the detection algorithm includes one or more of foreground and background separation, proximity recognition, Chroma keying, hand-arm/body detection, background subtraction, and a Kalman filter.

19. The computer-implemented system for superimposing video of claim 11, wherein the detection algorithm is automatically selected by one of the central server, the first user device'"'"'s processor, and the second user device'"'"'s processor to optimize processor utilization.

20. The computer-implemented system for superimposing video of claim 11, wherein the detection algorithm is automatically selected by one of the central server, the first user device'"'"'s processor, and the second user device'"'"'s processor to optimize image quality within the superimposed video.