Method for image transformation, augmented reality, and teleperence

• US 8,982,110 B2
• Filed: 04/15/2014
• Issued: 03/17/2015
• Est. Priority Date: 03/01/2005
• Status: Active Grant

• Current Assignee: Eyesmatch, Ltd.
• Original Assignee: Eyesmatch, Ltd.
• Inventors: Saban, Ofer, Vilcovsky, Nissi
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/253,831
• Publication Number:
  US 20140225978A1
• Time in Patent Office: 336 Days
• Field of Search: 345204-215, 348/333.01, 434/395
• US Class Current: 345/204
• CPC Class Codes: G02B 5/08 Mirrors vehicle mirrors involving special optical features B60R1/08
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09F 19/16 involving the use of mirrors
• CPC Class Codes: G09F 27/00 Combined visual and audible advertising or displaying, e.g. for public address
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/622 with simulation on a subsidiary picture reproducer
• CPC Class Codes: H04N 2005/2726 for simulating a person's appearance, e.g. hair style, glasses, clothes
• CPC Class Codes: H04N 21/00 : Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 5/2624 : for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/144 : camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

Computerized method for image transformation, augmented reality and telepresence. A camera generates a stream of images of the user; a processor detects the presence of the user in the stream of images and applies adaptive transformation mapping to the stream of images captured by the camera to generate modified images that appear to be captured from a different point of view of the camera'"'"'s actual point of view. The modified images are displayed on a local and/or remote monitor for telepresence. The processor can also generate an avatar having body characteristics corresponding to body characteristics of a user appearing in the images. The processor may generate the avatar by analyzing registration pointers on the images of the camera. The processor may reside in the remote location, and the camera images transmitted over a network to the processor, and the modified images transmitted back from the processor to the monitor.

Claims

1. A computerized method for operating a display system, the display system having a camera, a monitor, and a processor, the method comprising:

operating the camera to generate a stream of images of the user;

operating the processor to detect the presence of the user in the stream of images;

operating the processor to apply adaptive transformation mapping to the stream of images captured by the camera to generate modified images that appear to be captured from a different point of view of the camera'"'"'s actual point of view; and

,

displaying the modified images on the monitor;

wherein the different point of view corresponds to an image taken from a different angle and a different distance than the actual images captured by the camera; and

wherein during mirror tracking mode the processor utilizes the angle of the images captured by the camera and applies parametric input for a mapping transformation that changes imaging perspective of the images captured by the camera.

2. The method of claim 1, wherein the mapping transformation comprises a combination of at least two of the following transformations:

vertical flips about a vertical axis, a projection transformation, rotation, resize, two-dimensional offset, optic distortion correction, and polynomial mapping on part or entire captured image.

3. The method of claim 1, wherein during idle mode the camera continuously sends captured images to the processor, and the user presence is identified by the processor detecting a body part in the images.

4. The method of claim 3, wherein the body part comprises one of legs, face, eyes, and hands.

5. The method of claim 1, further comprising receiving user input and operating the processor to vary image magnitude or field-of-view of the modified images according to the user input.

6. The method of claim 5, wherein the processor varies the image magnitude or field-of-view mapping to display a partial or full body of the user in the modified images.

7. The method of claim 1, further comprising operating the processor to generate an avatar having body characteristics corresponding to body characteristics of a user appearing in the images.

8. The method of claim 7, further comprising operating the processor to display on the monitor the modified images and an image of the avatar.

9. The method of claim 7, further comprising operating the processor to augment the modified images by overlaying virtual objects on the avatar.

10. The method of claim 1, further comprising operating the processor to augment the modified images by overlaying virtual objects on a user'"'"'s image appearing within the modified images.

11. The method of claim 1, further comprising operating the processor to transmit the modified image over a network.

12. The method of claim 1, further comprising displaying the modified image on a remote screen for video telepresence.

13. The method of claim 1, further comprising providing data from a depth sensor to the processor, and operating the processor to measure a texture of the image from the data.

14. The method of claim 13, further comprising operating the processor to generate an avatar using the data from the depth sensor.

15. The method of claim 13, wherein the processor generate the avatar by analyzing registration pointers on the images of the camera.

16. The method of claim 1, wherein the processor operates on the images from the camera to generate background segmentation, and replaces segments on the images by replacement segments.

17. The method of claim 16, wherein the replacement segments comprise virtual background or background that was captured in a different setting.

18. The method of claim 1, wherein the processor resides in the remote location, and the method proceeds by transmitting the images from the camera over a network to the processor, and transmitting the modified images back from the processor to the monitor over the network.