Methods, systems, and computer readable media for unified scene acquisition and pose tracking in a wearable display

• US 10,365,711 B2
• Filed: 05/17/2013
• Issued: 07/30/2019
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Fuchs, Henry, Dou, Mingsong, Welch, Gregory, Frahm, Jan-Michael
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): Almeida, Cory A
• Application Number:
  US14/401,833
• Publication Number:
  US 20150138069A1
• Time in Patent Office: 2,265 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

Methods, systems, and computer readable media for unified scene acquisition and pose tracking in a wearable display are disclosed. According to one aspect, a system for unified scene acquisition and pose tracking in a wearable display includes a wearable frame configured to be worn by a user. Mounted on the frame are: at least one sensor for acquiring scene information for a real scene proximate to the user, the scene information including images and depth information; a pose tracker for estimating the user'"'"'s head pose based on the acquired scene information; a rendering unit for generating a virtual reality (VR) image based on the acquired scene information and estimated head pose; and at least one display for displaying to the user a combination of the generated VR image and the scene proximate to the user.

Claims

1. A system for unified scene acquisition and pose tracking in a wearable display, the system comprising:

a wearable frame configured to be worn on the head of a user, the frame having;

at least one camera mounted to the wearable frame for acquiring scene information for a real scene proximate to the user, the scene information including images and depth information, the scene information including positions of real objects separate from the user in the real scene local to the user;

at least one sensor mounted to the wearable frame for acquiring images of gestures and body poses of the user;

a pose tracker mounted to the wearable frame for generating, based on the scene information, a 3D model of the scene, generating, based on the images of gestures and body poses of the user acquired by the at least one sensor, a 3D model of the user, and estimating a position and orientation of the user in relation to the 3D model of the scene based on the images and depth information acquired by the at least one camera mounted to the frame and the images of gestures and body poses of the user acquired by the at least one sensor;

a rendering unit mounted to the wearable frame for generating a virtual reality (VR) image based on the scene information acquired by the at least one camera and the estimated position and orientation of the user in relation to the 3D model of the scene, wherein the rendering unit receives, from a location remote from the user, images and depth information of real objects acquired in a remote scene, the images and depth information of real objects including an image and depth information of a virtual participant in a meeting, wherein the rendering unit receives the positions of the real objects in the scene local to the user, and determines, based on the positions of the real objects and a perceived location of the virtual participant, portions of the image of the virtual participant to occlude in the VR image, wherein image of the virtual participant comprises an image of a human participant captured by a camera local to the human participant and remote from the user; and

at least one display mounted to the frame for displaying to the user a combination of the generated VR image and the scene local to the user, wherein the VR image includes the image of the virtual participant with the portions occluded as determined by the rendering unit.

2. The system of claim 1 wherein the wearable frame comprises an eyeglasses frame.

3. The system of claim 1 wherein the scene information is used to generate the 3D model of the scene local to the user.

4. The system of claim 1 wherein the pose tracker includes an inertial sensor that aids the pose tracker in estimating the position and orientation of the user in relation to the 3D model of the scene local to the user.

5. The system of claim 1 wherein the VR image is at least one of:

an artificial image and a real, remote image.

6. The system of claim 1 wherein the rendering unit is configured to occlude a portion of the VR image based on a perceived location of the VR image in the scene local to the user.

7. The system of claim 1 wherein the at least one display comprises at least one of:

an optical see-though display and a video see-through display.

8. The system of claim 1 wherein the at least one sensor is configured to acquire:

an image of the user'"'"'s face, hands, feet, or body;

a hand gesture of the user; and

a body pose of the user.

9. The system of claim 8 wherein the user information is communicated to at least one of:

the rendering unit and a remote receiver.

10. The system of claim 8 wherein the user information is used for gesture recognition.

11. The system of claim 1 comprising a communications unit for communicating with a remote receiver.

12. The system of claim 11 wherein the communications unit communicates via a wireless local network.

13. The system of claim 11 wherein the communications unit receives the images and depth information of the real objects in the remote scene.

14. The system of claim 11 comprising a plurality of wearable displays, each wearable display being worn by one of a plurality of users, wherein each of the plurality of wearable displays communicate with at least one of:

the remote receiver and another of the plurality of wearable displays.

15. The system of claim 14 wherein scene information is generated from scene data acquired by the plurality of wearable displays.

16. The system of claim 11 wherein the remote receiver is a component of a telepresence application.

17. The system of claim 1 wherein the system further comprises an illumination projector for illuminating a first portion of the scene local to the user and for leaving as unlit a second portion of the scene local to the user that corresponds to an apparent location of the virtual participant from a point of view of the user viewing the image of the virtual participant displayed on the display to create an impression that the virtual participant is present within the scene local to the user.

18. A method for unified scene acquisition and pose tracking in a wearable display, the method comprising:

acquiring, from at least one camera that is mounted to a wearable display frame configured to be worn on the head of a user, scene information for a scene local to the user, the scene information including images and depth information, the scene information including positions of real objects separate from the user in the real scene local to the user;

acquiring, using at least one sensor mounted to the wearable frame, images of gestures and body poses of the user;

generating, by a pose tracker that is mounted to the wearable display frame and based on the scene information, a 3D model of the scene, generating, by the pose tracker, from the images of gestures and body poses of the users acquired by the at least one sensor, a 3D model of the user, and estimating, by the pose tracker, a position and orientation of the user in relation to the 3D model of the scene based on the images and depth information acquired by the at least one camera and the images of the gestures and body poses of the user acquired by the at least one sensor;

generating, by a rendering unit that is mounted to the wearable display frame, a virtual reality (VR) image based on the scene information acquired by the at least one camera and the estimated position and orientation of the user in relation to the 3D model of the scene, wherein the rendering unit receives, from a location remote from the user, images and depth information of real objects acquired in a remote scene, the images and depth information of real objects including an image and depth information of a virtual participant in a meeting, wherein the rendering unit receives the positions of the real objects in the scene local to the user and determines, based on the positions of the real objects in the scene local to the user and a perceived location of the virtual participant, portions of the image of the virtual participant to occlude in the VR image, wherein image of the virtual participant comprises an image of a human participant captured by a camera local to the human participant and remote from the user; and

displaying to the user a combination of the generated VR image and the scene local to the user using at least one display that is mounted to the wearable display frame, wherein the VR image includes the image of the virtual participant with portions occluded as determined by the rendering unit.

19. The method of claim 18 wherein the display frame comprises an eyeglasses frame.

20. The method of claim 18 wherein the acquired scene information is used to generate the 3D model of the scene local to the user.

21. The method of claim 18 wherein estimating the position and orientation of the user in relation to the 3D model of the scene local to the user includes using inertial information provided by an inertial sensor that is mounted to the display frame.

22. The method of claim 18 wherein generating a VR image comprises generating at least one of:

an artificial image and a real, remote image.

23. The method of claim 18 wherein generating a VR image comprises occluding a portion of the VR image based on a perceived location of the VR image in the scene local to the user.

24. The method of claim 18 wherein using at least one display comprises using at least one of:

an optical see-though display and a video see-through display.

25. The method of claim 18 comprising acquiring, using the at least one sensor:

an image of the user'"'"'s face, hand, feet, or body;

a hand gesture of the user; and

a body pose of the user.

26. The method of claim 25 wherein acquiring user information comprises using at least one of:

a sensor that is mounted to the display frame; and

a sensor that is separate from the display frame.

27. The method of claim 25 comprising communicating the user information to at least one of:

the rendering unit and a remote receiver.

28. The method of claim 25 comprising using the user information for gesture recognition.

29. The method of claim 18 comprising communicating with a remote receiver.

30. The method of claim 29 wherein communicating with a remote receiver comprises communicating via a wireless local network.

31. The method of claim 29 wherein communicating with a remote receiver comprises receiving, from the remote receiver, the 3D images of the real objects in the remote scene.

32. The method of claim 18 wherein acquiring the scene information comprises acquiring the scene information via a plurality of wearable displays, each wearable display being worn by one of a plurality of users, wherein each of the plurality of wearable displays communicate with at least one of:

the remote receiver and another of the plurality of wearable displays.

33. The method of claim 32 comprising using the acquired scene information to build the 3D model of the scene local to the user.

34. The method of claim 18 wherein acquiring the scene information comprises acquiring information about the plurality of users, the information including at least one of image, pose, and position of the plurality of users.

35. The method of claim 18 comprising, using an illumination projector, illuminating a first portion of the scene local to the user and for leaving as unlit a second portion of the scene local to the user that corresponds to an apparent location of the virtual participant from a point of view of the user viewing the image of the virtual participant displayed on the display to create an impression that the virtual participant is present within the scene local to the user.

36. A non-transitory computer readable medium having stored thereon executable instructions that when executed by the processor of a computer control the computer to perform steps comprising:

acquiring, from at least one camera that is mounted to a wearable display frame configured to be worn on the head of a user, scene information for a scene local to a user, the scene information including images and depth information, the scene information including positions of real objects separate from the user in the real scene local to the user;

acquiring, using at least one sensor mounted to the wearable frame, gestures and body poses of the user;

generating, by a pose tracker that is mounted to the wearable display frame and based on the scene information, a 3D model of the scene, generating, by the pose tracker, from the images of gestures and body poses of the users acquired by the at least one sensor, a 3D model of the user, and estimating, by the pose tracker, a position and orientation of the user in relation to the 3D model of the scene based on the images and depth information acquired by the at least one camera and the images of gestures and body poses of the user acquired by the at least one sensor;

generating, by a rendering unit that is mounted to the wearable display frame, a virtual reality (VR) image based on the scene information acquired by the at least one camera and the estimated position and orientation of the user in relation to the 3D model of the scene local to the user, wherein the rendering unit receives, from a location remote from the user, images and depth information of real objects acquired in a remote scene, the images and depth information including an image and depth information of a virtual participant in a meeting, wherein the rendering unit receives the positions of the real objects in the scene local to the user and determines, based on the positions of the real objects and a perceived location of the virtual participant, portions of the image of the virtual participant to occlude in the VR image, wherein image of the virtual participant comprises an image of a human participant captured by a camera local to the human participant and remote from the user; and

displaying to the user a combination of the generated VR image and the scene proximate to the user, wherein the VR image includes the image of the virtual participant with the portions occluded as determined by the rendering unit using at least one display that is mounted to the wearable display frame.

37. The non-transitory computer readable medium of claim 36 comprising controlling an illumination projector to illuminate a first portion of the scene local to the user and to leave as unlit a second portion of the scene local to the user that corresponds to an apparent location of the virtual participant from a point of view of the user viewing the image of the virtual participant displayed on the display to create an impression that the virtual participant is present within the scene local to the user.