Video streaming augmenting

• US 10,306,315 B2
• Filed: 03/29/2016
• Issued: 05/28/2019
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Roblox Corp.
• Original Assignee: International Business Machines Corporation
• Inventors: Bailey, Matthew C., Daniel, Andrew J., Jones, Philip, Pilot, Richard W.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/083,545
• Publication Number:
  US 20170289623A1
• Time in Patent Office: 1,155 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2219/00 Indexing scheme for manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2219/20 Indexing scheme for editing of 3D models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/2187 : Live feed
• CPC Class Codes: H04N 21/4318 : by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/4542 : Blocking scenes or portions of the received content, e.g. censoring scenes
• CPC Class Codes: H04N 21/4545 : Input to filtering algorithms, e.g. filtering a region of the image
• CPC Class Codes: H04N 21/475 : End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data
• CPC Class Codes: H04N 21/8146 : involving graphical data, e.g. 3D object, 2D graphics

Abstract

Augmenting a video stream of an environment is provided, the environment containing a private entity to be augmented. Video of the environment is processed in accordance with an entity recognition process to identify the presence of at least part of an entity in the environment. It is determined whether the identified entity is to be augmented based on information relating to the identified entity and the private entity. Based on determining that the identified entity is to be augmented, the video stream is modified to replace at least a portion of the identified entity with a graphical element adapted to obscure the portion of the identified entity in the video stream. By modifying the video stream to obscure an entity, private or personal information in the environment may be prevented from being displayed to a viewer of the video stream.

Claims

1. A computer-implemented method of providing a live video stream of a real environment, the method comprising:

predefining a private entity and storing private entity information relating to the private entity;

processing in real-time the live video stream of the real environment for communication to a potentially un-trusted third-party remotely located from the real environment, the processing comprising;

evaluating the video stream in accordance with an entity recognition process to identify presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity substantially matches the private entity based on the generated identified entity information and the private entity information relating to the private entity; and

based on determining that the identified entity substantially matches the private entity, modifying the video stream to replace an imaged entity in, on, or under the identified entity within the live video stream with a graphical element adapted to obscure the imaged entity in the video stream, the modifying providing in real-time a modified live video stream of the real environment with the graphical element incorporated therein; and

providing the modified live video stream for display to the potentially un-trusted third-party located remotely from the environment.

2. The method of claim 1, wherein the predefining comprises:

receiving a user input describing the private entity to be augmented; and

generating the private entity information based on the received user input.

3. The method of claim 1, wherein the private entity information comprises at least one of:

private entity location information representative of a location of the private entity;

private entity orientation information representative of an orientation of the private entity;

private entity geometric information representative of at least one geometric property of the private entity;

private entity color information representative of at least one color of the private entity; and

private entity 3D coordinate information representative of a position of the private entity in a 3D coordinate space.

4. The method of claim 1, wherein the identified entity information comprises at least one of:

viewing direction information representative of a viewing direction of a video capture device adapted to capture the video stream;

viewing position information representative of a position of the video capture device;

identified entity location information representative of a location of the identified entity;

identified entity orientation information representative of an orientation of the identified entity;

identified entity geometric information representative of at least one geometric property of the identified entity;

identified entity color information representative of at least one color of the identified entity; and

identified entity 3D coordinate information representative of a position of the identified entity in a 3D coordinate space.

5. The method of claim 1, wherein modifying the video stream comprises generating the graphical element such that it is representative of a simplified 3D model of the imaged entity.

6. The method of claim 1, wherein the graphical element comprises a masked or blurred version of the portion of the imaged entity.

7. The method of claim 1, further comprising determining whether the imaged entity is to be augmented by:

determining a viewing privilege or access right of a target viewer of the video stream; and

determining that the imaged entity is to be augmented where the viewing privilege or access right of the target viewer does not meet a predetermined requirement.

8. The method of claim 1, further comprising displaying the modified video stream to a person remotely located from the environment.

9. A computer program product for providing a live video stream of a real environment, the computer program product comprising:

a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by a processing unit to cause the processing unit to perform a method comprising;

predefining a private entity and storing private entity information relating to the private entity;

processing in real-time the live video stream of the real environment for communication to a potentially un-trusted third-party remotely located from the real environment, the processing comprising;

evaluating the video stream in accordance with an entity recognition process to identify presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity substantially matches the private entity based on the generated identified entity information and the private entity information relating to the private entity; and

based on determining that the identified entity substantially matches the private entity, modifying the video stream to replace an imaged entity in, on, or under the identified entity within the live video stream with a graphical element adapted to obscure the imaged entity in the video stream, the modifying providing in real-time a modified live video stream of the real environment with the graphical element incorporated therein; and

providing the modified live video stream for display to the potentially un-trusted third-party located remotely from the environment.

10. A system for providing a live video stream of a real environment, the system comprising:

a memory; and

a processor communicatively coupled to the memory, the system performing a method comprising;

predefining a private entity and storing private entity information relating to the private entity;

processing in real-time the live video stream of the real environment for communication to a potentially un-trusted third-party remotely located from the real environment, the processing comprising;

evaluating the video stream in accordance with an entity recognition process to identify presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity substantially matches the private entity based on the generated identified entity information and the private entity information relating to the private entity; and

based on determining that the identified entity substantially matches the private entity, modifying the video stream to replace an imaged entity in, on, or under the identified entity within the live video stream with a graphical element adapted to obscure the imaged entity in the video stream, the modifying providing in real-time a modified live video stream of the real environment with the graphical element incorporated therein; and

providing the modified live video stream for display to the potentially un-trusted third-party located remotely from the environment.

11. The system of claim 10, wherein the predefining further comprises:

receiving a user input describing the private entity to be augmented, and generating the private entity information based on the received user input.

12. The system of claim 10, wherein the private entity information comprises at least one of:

private entity location information representative of a location of the private entity;

private entity orientation information representative of an orientation of the private entity;

private entity geometric information representative of at least one geometric property of the private entity;

private entity color information representative of at least one color of the private entity; and

private entity 3D coordinate information representative of a position of the private entity in a 3D coordinate space.

13. The system of claim 10, wherein the identified entity information comprises at least one of:

viewing direction information representative of a viewing direction of a video capture device adapted to capture the video stream;

viewing position information representative of a position of the video capture device;

identified entity location information representative of a location of the identified entity;

identified entity orientation information representative of an orientation of the identified entity;

identified entity geometric information representative of at least one geometric property of the identified entity;

identified entity color information representative of at least one color of the identified entity; and

identified entity 3D coordinate information representative of a position of the identified entity in a 3D coordinate space.

14. The system of claim 10, wherein modifying the video stream comprises generating the graphical element such that it is representative of a simplified 3D model of the imaged entity.

15. The system of claim 10, wherein the graphical element comprises a masked or blurred version of the portion of the imaged entity.