Methods and systems for real-time image and signal processing in augmented reality based communications

• US 10,580,040 B2
• Filed: 04/04/2017
• Issued: 03/03/2020
• Est. Priority Date: 04/03/2016
• Status: Active Grant

• Current Assignee: Integem Incorporated
• Original Assignee: Integem Incorporated
• Inventors: Du, Eliza Y.
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/479,269
• Publication Number:
  US 20170287226A1
• Time in Patent Office: 1,064 Days
• Field of Search: 345419
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Disclosed herein are methods and systems for real-time image and signal processing in an augmented reality environment; for example, for video conferencing in a virtual environment of the participants'"'"' choice. In particular, image information of a real life object is extracted by separating the object from its actual environment in one or more images that are captured in real-time using a comprehensive characteristic-based mechanism. The extracted real life object is then integrated with a virtual environment based on image relations between each pixel of the image information of the real life object and a corresponding pixel of each image of the plurality of images of the virtual environment through a pixel-by-pixel integration approach. The image relations comprise at least a depth relation or a transparency relation.

Claims

1. A method of integrating a real life object with a virtual environment, comprising:

providing, at a computer device and in real-time, multi-dimensional image information of a real life object, wherein the image information is extracted by separating the real life object from its actual environment in one or more images that are captured in real-time using a comprehensive characteristic-based mechanism;

receiving, at the computer device and in real-time, a selection of a virtual environment constructed from a plurality of images, said virtual environment comprising a plurality of elements;

providing, at the computer device and in real-time, image relations between each pixel of the image information of the real life object and a corresponding pixel of each image of the plurality of images of the virtual environment, wherein the image relations comprise at least a depth relation or a transparency relation;

integrating, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations to create the plurality of integrated images; and

rendering, at the computer device and in real-time, a plurality of integrated images, wherein each pixel in an integrated image is divided into multiple layers, and wherein each layer is determined using corresponding pixels in the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations;

wherein in a region of an integrated image corresponding to the real life object,

at least one pixel associated with the real life object is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment,

at least one pixel associated with the virtual environment is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment, and

at least a portion of the real life object interacts with one or more elements of the virtual environment,

wherein the comprehensive characteristic-based mechanism is based on at least one visual characteristic of either the real life object or the actual environment, and wherein the visual characteristic is one of;

captured by the computer device, a real-time learned characteristic, or a pre-learned feature.

2. The method of claim 1, further comprising:

extracting in real-time, at the computer device or a different computer device, the multiple dimensional image information of the real life object from the one or more images of the real life object in its actual environment.

3. The method of claim 1, wherein comprehensive characteristic-based mechanism is based on one or more differences in a characteristic between the real life object and the actual environment, the characteristic comprising a visual characteristic of the real life object or the actual environment captured in the video clip, a real-time learned characteristic of the real life object or the actual environment, or a pre-learned feature relating to the real life object or the actual environment.

4. The method of claim 3, wherein the visual characteristic comprises a spatial characteristic, a size characteristic, a shape characteristics, a motion characteristics, a color characteristic, a light illuminance and reflectance characteristic, a temporal characteristic, or a transparency characteristic, a depth characteristic, a material characteristic, or a combinations thereof.

5. The method of claim 4, wherein the spatial characteristic comprises a three-dimensional spatial characteristic.

6. The method of claim 3, wherein the real-time learned characteristic comprises color, shape, edge, light reflectance, light illuminance, exposure, brightness, shadow, highlight, contrast, blackpoint, motion, depth, material, contrast, or combinations thereof.

7. The method of claim 3, wherein the pre-learned feature comprises color, shape, edge, light reflectance, light illuminance, exposure, brightness, shadow, highlight, contrast, blackpoint, motion, depth, material, contrast, or combinations thereof.

8. The method of claim 2, wherein the exacting comprises a round of rough extraction and a round of fine extraction.

9. The method of claim 1, further comprising:

constructing, at the computer device or a different computer device, the virtual environment from the plurality of images.

10. The method of claim 1, wherein at least one image of the plurality of images is modified when being used to construct the virtual environment.

11. The method of claim 10, wherein the at least one image is modified in size, shape, image quality, color, light, perspective, visual effect, or combinations thereof.

12. The method of claim 1, further comprising:

integrate, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object, the new image information of the second real life object, and each image of the plurality of images of the virtual environment based on the image relations and new image relations to create a plurality of new integrated images.

13. The method of claim 1, wherein the multiple layers of a pixel comprises a foreground layer, an intermediate layer, and a background layer.

14. The method of claim 1, wherein the foreground layer is determined before the intermediate layer or background layer.

15. The method of claim 1, wherein one or more advertisement elements are embedded in the plurality of integrated images.

16. The method of claim 15, wherein the one or more advertisement elements are associated with the extracted image information of the real life object, the virtual environment, or combinations thereof.

17. The method of claim 1, further comprising:

determining, on a pixel-by-pixel basis, a first portion of the real life object that obscures a corresponding portion of one element of the virtual environment, and

determining, on a pixel-by-pixel basis, a second portion of the real life object that is obscured by a corresponding portion of the one element of the virtual environment,

such that the real life object is rendered partially in front of and partially behind the one element of the virtual environment.

18. A method of integrating two real life objects with a virtual environment, comprising:

providing, at a computer device and in real-time, multi-dimensional image information of a first real life object, wherein the image information is extracted by separating the first real life object from a first actual environment in one or more images that are captured in real-time using a comprehensive characteristic-based mechanism;

receiving, at the computer device and in real-time, a selection of a virtual environment constructed from a plurality of images, said virtual environment comprising a plurality of elements;

providing, at the computer device and in real-time, image relations between each pixel of the image information of the real life object and a corresponding pixel of each image of the plurality of images of the virtual environment, wherein the image relations comprise at least a depth relation or a transparency relation;

providing, at the computer device and in real-time, extracted multi-dimensional image information of a second real life object, and new image relations between each pixel of the image information of the second real life object and a corresponding pixel of each image of the plurality of images of the virtual environment, wherein the new image relations comprise at least a depth relation or a transparency relation;

integrating, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object, the new image information of the second real life object, and each image of the plurality of images of the virtual environment based on the image relations and new image relations to create a plurality of new integrated images; and

rendering, at the computer device and in real-time, a plurality of integrated images, wherein each pixel in an integrated image is divided into multiple layers, and wherein each layer is determined using corresponding pixels in the image information of the first real life object, the second real life object, and each image of the plurality of images of the virtual environment based on the image relations and new image relations;

wherein in a region of an integrated image corresponding to the real life object,

at least one pixel associated with the real life object is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment,

at least one pixel associated with the virtual environment is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment, and

at least a portion of the real life object interacts with one or more elements of the virtual environment;

wherein the comprehensive characteristic-based mechanism is based on at least one visual characteristic of either the real life object or the actual environment, and wherein the visual characteristic is one of;

captured by the computer device, a real-time learned characteristic, or a pre-learned feature.

19. The method of claim 18, wherein one or more advertisement elements are embedded in the plurality of integrated images.

20. The method of claim 19, wherein the one or more advertisement elements are associated with the extracted image information of the first real life object, the extracted image information of the second real life object, the virtual environment, or combinations thereof.

21. A computer system comprising:

one or more processors; and

a memory accessible to the one or more processors, the memory storing instructions executable by the one or more processors to;

provide, at a computer device and in real-time, multi-dimensional image information of a real life object, wherein the image information is extracted by separating the real life object from its actual environment in one or more images that are captured in real-time using a comprehensive characteristic-based mechanism;

receive, at the computer device and in real-time, a selection of a virtual environment constructed from a plurality of images, said virtual environment comprising a plurality of elements;

provide, at the computer device and in real-time, image relations between each pixel of the image information of the real life object and a corresponding pixel of each image of the plurality of images of the virtual environment, wherein the image relations comprise at least a depth relation or a transparency relation;

integrate, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations to create the plurality of integrated images; and

render, at the computer device and in real-time, a plurality of integrated images,

wherein each pixel in an integrated image is divided into multiple layers,

wherein each layer is determined using corresponding pixels in the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations

wherein in a region of an integrated image corresponding to the real life object,

at least one pixel associated with the real life object is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment,

at least one pixel associated with the virtual environment is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment, and

at least a portion of the real life object interacts with one or more elements of the virtual environment;

wherein the comprehensive characteristic-based mechanism is based on at least one visual characteristic of either the real life object or the actual environment, and

wherein the visual characteristic is one of;

captured by the computer device, a real-time learned characteristic, or a pre-learned feature.

22. The computer system of claim 21, wherein the memory stores further instructions executable by the one or more processors to:

extract in real-time, at the computer device or a different computer device, the multiple dimensional image information of the real life object from the one or more images of the real life object in its actual environment.

23. The computer system of claim 21, wherein the memory stores further instructions executable by the one or more processors to:

construct, at the computer device or a different computer device, the virtual environment from the plurality of images.

24. The computer system of claim 21, wherein the memory stores further instructions executable by the one or more processors to:

provide, at the computer device and in real-time, new image relations between each pixel of the image information of a second real life object and a corresponding pixel of each image of the plurality of images, wherein the new image relations comprise at least a depth relation or a transparency relation.

25. The computer system of claim 21, wherein the memory stores further instructions executable by the one or more processors to:

integrate, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object, the new image information of the second real life object, and each image of the plurality of images of the virtual environment based on the image relations and new image relations to create a plurality of new integrated images.

26. A non-transitory computer-readable medium containing instructions that, when executed by a computer processor, cause the computer processor to:

provide, at a computer device and in real-time, multi-dimensional image information of a real life object, wherein the image information is extracted by separating the real life object from its actual environment in one or more images that are captured in real-time using a comprehensive characteristic-based mechanism;

receive, at the computer device and in real-time, a selection of a virtual environment constructed from a plurality of images, said virtual environment comprising a plurality of elements;

provide, at the computer device and in real-time, image relations between each pixel of the image information of the real life object and a corresponding pixel of each image of the plurality of images of the virtual environment, wherein the image relations comprise at least a depth relation or a transparency relation;

integrate, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations to create the plurality of integrated images; and

render, at the computer device and in real-time, a plurality of integrated images, wherein each pixel in an integrated image is divided into multiple layers, and wherein each layer is determined using corresponding pixels in the image information of the real life object and each image of the plurality of images of the virtual environment based on the image relations,

wherein in a region of an integrated image corresponding to the real life object,

at least one pixel associated with the real life object is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment,

at least one pixel associated with the virtual environment is rendered invisible based on the determination based on the image relations between the real life object and the plurality of images of the virtual environment, and

at least a portion of the real life object interacts with one or more elements of the virtual environment;

wherein the comprehensive characteristic-based mechanism is based on at least one visual characteristic of either the real life object or the actual environment, and

wherein the visual characteristic is one of;

captured by the computer device, a real-time learned characteristic, or a pre-learned feature.

27. The non-transitory computer-readable medium of claim 26, wherein the non-transitory computer-readable medium contains further instructions executable by the one or more processors to:

extract in real-time, at the computer device or a different computer device, the multiple dimensional image information of the real life object from the one or more images of the real life object in its actual environment.

28. The non-transitory computer-readable medium of claim 26, wherein the non-transitory computer-readable medium contains further instructions executable by the one or more processors to:

construct, at the computer device or a different computer device, the virtual environment from the plurality of images.

29. The non-transitory computer-readable medium of claim 26, wherein the non-transitory computer-readable medium contains further instructions executable by the one or more processors to:

provide, at the computer device and in real-time, new image relations between each pixel of the image information of a second real life object and a corresponding pixel of each image of the plurality of images, wherein the new image relations comprise at least a depth relation or a transparency relation.

30. The non-transitory computer-readable medium of claim 26, wherein the non-transitory computer-readable medium contains further instructions executable by the one or more processors to:

integrate, at the computer device or a different computer device in real-time and on a pixel by pixel basis, the image information of the real life object, the new image information of the second real life object, and each image of the plurality of images of the virtual environment based on the image relations and new image relations to create a plurality of new integrated images.