VIDEO STREAM AUGMENTING

• US 20170289623A1
• Filed: 03/29/2016
• Published: 10/05/2017
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Roblox Corp.
• Original Assignee: International Business Machines Corporation
• Inventors: BAILEY, Matthew C., DANIEL, Andrew J., JONES, Philip, PILOT, Richard W.
• Granted Patent: US 10,306,315 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2219/00 Indexing scheme for manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2219/20 Indexing scheme for editing of 3D models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/2187 : Live feed
• CPC Class Codes: H04N 21/4318 : by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/4542 : Blocking scenes or portions of the received content, e.g. censoring scenes
• CPC Class Codes: H04N 21/4545 : Input to filtering algorithms, e.g. filtering a region of the image
• CPC Class Codes: H04N 21/475 : End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data
• CPC Class Codes: H04N 21/8146 : involving graphical data, e.g. 3D object, 2D graphics

Abstract

Augmenting a video stream of an environment is provided, the environment containing a private entity to be augmented. Video of the environment is processed in accordance with an entity recognition process to identify the presence of at least part of an entity in the environment. It is determined whether the identified entity is to be augmented based on information relating to the identified entity and the private entity. Based on determining that the identified entity is to be augmented, the video stream is modified to replace at least a portion of the identified entity with a graphical element adapted to obscure the portion of the identified entity in the video stream. By modifying the video stream to obscure an entity, private or personal information in the environment may be prevented from being displayed to a viewer of the video stream.

Claims

1. A computer-implemented method of augmenting a video stream of an environment, the environment containing a private entity to be augmented, the method comprising:

processing the video stream in accordance with an entity recognition process to identify the presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity is to be augmented based on the generated identified entity information and private entity information relating to the private entity; and

based on determining that the identified entity is to be augmented, modifying the video stream to replace at least a portion of the identified entity with a graphical element adapted to obscure the portion of the identified entity in the video stream.

2. The method of claim 1, further comprising:

receiving a user input describing the private entity to be augmented; and

generating the private entity information based on the received user input.

3. The method of claim 1, wherein the private entity information comprises at least one of:

private entity location information representative of a location of the private entity;

private entity orientation information representative of an orientation of the private entity;

private entity geometric information representative of at least one geometric property of the private entity;

private entity color information representative of at least one color of the private entity; and

private entity 3D coordinate information representative of a position of the private entity in a 3D coordinate space.

4. The method of claim 1, wherein the identified entity information comprises at least one of:

viewing direction information representative of a viewing direction of a video capture device adapted to capture the video stream;

viewing position information representative of a position of the video capture device;

identified entity location information representative of a location of the identified entity;

identified entity orientation information representative of an orientation of the identified entity;

identified entity geometric information representative of at least one geometric property of the identified entity;

identified entity color information representative of at least one color of the identified entity; and

identified entity 3D coordinate information representative of a position of the identified entity in a 3D coordinate space.

5. The method of claim 1, wherein the determining whether the identified entity is to be augmented comprises:

determining whether the identified entity either substantially matches the private entity or is situated in, on, under or in close proximity to the private entity; and

based on determining that the identified entity substantially matches the private entity or is situated in, on, under or in close proximity to the private entity, determining that the identified entity is to be augmented.

6. The method of claim 5, wherein modifying the video stream comprises generating the graphical element based on whether it is determined that the identified entity substantially matches the private entity or is situated in, on, under or in close proximity to the private entity.

7. The method of claim 1, wherein modifying the video stream comprises generating the graphical element such that it is representative of a simplified 3D model of the identified entity.

8. The method of claim 1, wherein the graphical element comprises a masked or blurred version of the portion of the identified entity.

9. The method of claim 1, wherein the determining whether the identified entity is to be augmented comprises:

determining a viewing privilege or access right of a target viewer of the video stream; and

determining that the identified entity is to be augmented where the viewing privilege or access right of the target viewer does not meet a predetermined requirement.

10. The method of claim 1, further comprising displaying the modified video stream to a person remotely located from the environment.

11. A computer program product for augmenting a video stream of an environment, the environment containing a private entity to be augmented, the computer program product comprising:

a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by a processing unit to cause the processing unit to perform a method comprising;

processing the video stream of the environment in accordance with an entity recognition process to identify the presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity is to be augmented based on the generated identified entity information and private entity information relating to the private entity; and

based on determining that the identified entity is to be augmented, modifying the video stream to replace at least a portion of the identified entity with a graphical element adapted to obscure the portion of the identified entity in the video stream.

12. An augmentation system for augmenting a video stream of an environment, the environment containing a private entity to be augmented, the system comprising:

a memory; and

a processor communicatively coupled to the memory, the system performing a method comprising;

processing the video stream in accordance with an entity recognition process to identify the presence of at least part of an entity in the environment;

generating identified entity information relating to at least one property of the identified entity;

determining whether the identified entity is to be augmented based on the generated identified entity information and private entity information relating to the private entity; and

based on determining that the identified entity is to be augmented, modifying the video stream to replace at least a portion of the identified entity with a graphical element adapted to obscure the portion of the identified entity in the video stream.

13. The system of claim 12, wherein the system further comprises:

receiving a user input describing the private entity to be augmented, and generating the private entity information based on the received user input.

14. The system of claim 12, wherein the private entity information comprises at least one of:

private entity location information representative of a location of the private entity;

private entity orientation information representative of an orientation of the private entity;

private entity geometric information representative of at least one geometric property of the private entity;

private entity color information representative of at least one color of the private entity; and

private entity 3D coordinate information representative of a position of the private entity in a 3D coordinate space.

15. The system of claim 12, wherein the identified entity information comprises at least one of:

viewing direction information representative of a viewing direction of a video capture device adapted to capture the video stream;

viewing position information representative of a position of the video capture device;

identified entity location information representative of a location of the identified entity;

identified entity orientation information representative of an orientation of the identified entity;

identified entity geometric information representative of at least one geometric property of the identified entity;

identified entity color information representative of at least one color of the identified entity; and

identified entity 3D coordinate information representative of a position of the identified entity in a 3D coordinate space.

16. The system of claim 12, wherein the determining whether the identified entity is to be augmented comprises:

determining whether the identified entity either substantially matches the private entity or is situated in, on, under or in close proximity to the private entity; and

based on determining that the identified entity substantially matches the private entity or is situated in, on, under or in close proximity to the private entity, determining that the identified entity is to be augmented.

17. The system of claim 16, wherein modifying the video stream comprises generating the graphical element based on whether it is determined that the identified entity substantially matches the private entity or is situated in, on, under or in close proximity to the private entity.

18. The system of claim 12, wherein modifying the video stream comprises generating the graphical element such that it is representative of a simplified 3D model of the identified entity.

19. The system of claim 12, wherein the graphical element comprises a masked or blurred version of the portion of the identified entity.

20. The system of claim 12, wherein the video stream comprises a live video stream of the environment.