Holographic video capture and telepresence system

• US 9,813,673 B2
• Filed: 01/20/2017
• Issued: 11/07/2017
• Est. Priority Date: 01/20/2016
• Status: Active Grant

• Current Assignee: Gerard Dirk Smits
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Smits, Gerard Dirk
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US15/411,959
• Publication Number:
  US 20170208292A1
• Time in Patent Office: 291 Days
• Field of Search: 348 1401- 1416, 700 94
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/0103 comprising holographic elements
• CPC Class Codes: G02B 30/56 by projecting aerial or floating images
• CPC Class Codes: G03H 1/0005 Adaptation of holography to specific applications holographic optical element G02B5/32; holographic scanner G02B26/106; recognition using holographic mask G06V10/88; holographic memories G11B7/0065, G11C13/042
• CPC Class Codes: G03H 2001/0088 for video-holography, i.e. integrating hologram acquisition, transmission and display
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

The invention is directed to recording, transmitting, and displaying a three-dimensional image of a face of a user in a video stream. Reflected light from a curved or geometrically shaped screen is employed to provide multiple perspective views of the user'"'"'s face that are transformed into the image, which is communicated to remotely located other users. A head mounted projection display system is employed to capture the reflective light. The system includes a frame, that when worn by a user, wraps around and grips the user'"'"'s head. Also, at least two separate image capture modules are included on the frame and generally positioned relatively adjacent to the left and right eyes of a user when the system is worn. Each module includes one or more sensor components, such as cameras, that are arranged to detect at least reflected non-visible light from a screen positioned in front of the user.

Claims

1. A system for capturing video images of a user, comprising:

a curved screen;

a head mounted projection display apparatus that includes;

a frame that is adapted to wrap around a portion of a head of a user;

two or more image capture modules that are separately located at opposite ends of the frame which are adjacent to a left eye and a right eye of the user, wherein each image capture module includes one or more sensors that detect non-visible light that is reflected from the curved screen when it is positioned in front of a face of the user; and

a client computer that is coupled to the frame and arranged to capture the detected reflections of the non-visible light which is employed to provide a plurality of real time perspective views of the face of the user to a server computer; and

the server computer that is employed to generate a real time three dimensional video image of the face of the user based on the plurality of perspective views, wherein the three-dimensional video image is provided for display to one or more remotely located users.

2. The system of claim 1, wherein each image capture module includes one or more light scanners that are arranged to direct non-visible light onto a curved screen located in front of a face of the user.

3. The system of claim 1, wherein each image capture module includes one or more light scanners that direct visible light onto a curved screen located in front of a face of the user.

4. The system of claim 1, further comprising one or more other sensors to detect visible light reflected from the curved screen, wherein the one or more other sensors are included in one or more of the two or more image capture modules.

5. The system of claim 1, wherein the curved screen is retroreflective.

6. The system of claim 1, wherein the client computer performs further actions, including:

when the face of the user is positioned within a conversation focal zone in front of the curved screen, providing the plurality of real time perspective views of the face of the user to the server computer; and

when the face of the user is positioned outside the conversation focal zone in front of the curved screen, pausing the providing of the plurality of real time perspective views of the face of the user to the server computer.

7. The system of claim 1, wherein the curved screen further comprises a shape that is one of concave, elliptical parabolic, toroid, or triptych.

8. A head mounted projection display apparatus for capturing video images of a user, that includes:

a frame that is adapted to wrap around a portion of a head of a user;

two or more image capture modules that are separately located at opposite ends of the frame which are adjacent to a left eye and a right eye of the user, wherein each image capture module includes one or more sensors that detect non-visible light that is reflected from a curved screen when it is positioned in front of a face of the user; and

a client computer that is coupled to the frame and arranged to capture the detected reflections of the non-visible light which is employed to provide a plurality of real time perspective views of the face of the user to a server computer, wherein the server computer is employed to generate a real time three-dimensional video image of the face of the user based on the plurality of perspective views, and wherein the three-dimensional video image is provided for display to one or more remotely located users.

9. The apparatus of claim 8, wherein each image capture module includes one or more light scanners that are arranged to direct non-visible light onto a curved screen located in front of a face of the user.

10. The apparatus of claim 8, wherein each image capture module includes one or more light scanners that direct visible light onto a curved screen located in front of a face of the user.

11. The apparatus of claim 8, further comprising one or more other sensors to detect visible light reflected from the curved screen, wherein the one or more other sensors are included in one or more of the two or more image capture modules.

12. The apparatus of claim 8, wherein the curved screen is retroreflective.

13. The apparatus of claim 8, wherein the client computer performs further actions, including:

when the face of the user is positioned within a conversation focal zone in front of the curved screen, providing the plurality of real time perspective views of the face of the user to the server computer; and

when the face of the user is positioned outside the conversation focal zone in front of the curved screen, pausing the providing of the plurality of real time perspective views of the face of the user to the server computer.

14. The apparatus of claim 8, wherein the curved screen further comprises a shape that is one of concave, elliptical parabolic, toroid, or triptych.

15. A method for capturing video images of a user, comprising:

employing a head mounted projection display apparatus to capture the video images, wherein the apparatus includes;

a frame that is adapted to wrap around a portion of a head of a user;

two or more image capture modules that are separately located at opposite ends of the frame which are adjacent to a left eye and a right eye of the user, wherein each image capture module includes one or more sensors that detect non-visible light that is reflected from a curved screen when it is positioned in front of a face of the user; and

employing a client computer that is coupled to the frame and arranged to capture the detected reflections of the non-visible light which is employed to provide a plurality of real time perspective views of the face of the user to a server computer; and

employing the server computer to generate a real time three dimensional video image of the face of the user based on the plurality of perspective views, wherein the three-dimensional video image is provided for display to one or more remotely located users.

16. The method of claim 15, wherein each image capture module includes one or more light scanners that are arranged to direct non-visible light onto the curved screen located in front of a face of the user.

17. The method of claim 15, wherein each image capture module includes one or more light scanners that direct visible light onto the curved screen located in front of a face of the user.

18. The method of claim 15, further comprising one or more other sensors to detect visible light reflected from the curved screen, wherein the one or more other sensors are included in one or more of the two or more image capture modules.

19. The method of claim 15, wherein the curved screen is retroreflective.

20. The method of claim 15, wherein the client computer performs further actions, including:

when the face of the user is positioned within a conversation focal zone in front of the curved screen, providing the plurality of real time perspective views of the face of the user to the server computer; and

when the face of the user is positioned outside the conversation focal zone in front of the curved screen, pausing the providing of the plurality of real time perspective views of the face of the user to the server computer.

21. The method of claim 15, wherein the curved screen further comprises a shape that is one of concave, elliptical parabolic, toroid, or triptych.