Managing virtual content displayed to a user based on mapped user location

• US 10,503,351 B2
• Filed: 07/21/2017
• Issued: 12/10/2019
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Reavire Inc.
• Original Assignee: Reavire Inc.
• Inventors: Gullicksen, Jeff
• Primary Examiner(s): Nguyen, Anh-Tuan V
• Application Number:
  US15/656,500
• Publication Number:
  US 20180061127A1
• Time in Patent Office: 872 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A technique for rendering virtual content to a user stores map data of features in a physical environment of the user and measures the location of the user with stationary sensors placed at respective locations within the environment. A server provides the location of the user and portions of the map data to a headset worn by the user. The headset is thus enabled to render virtual content at apparent locations that are based on the measured location of the user and the features described by the map data.

Claims

1. A method of managing virtual content to be displayed to users via three-dimensional imaging headsets, the method comprising:

measuring locations of a user in a physical space as the user moves through the physical space, by a server apparatus receiving inputs from multiple stationary sensors positioned at respective sensor locations within the physical space and processing the inputs to generate the locations of the user;

storing map data that describes a map of the physical space;

specifying a set of holograms that have apparent locations that are defined relative to the map data; and

providing the measured locations of the user and at least a portion of the map data to a headset worn by the user, to enable the headset to render the set of holograms at the apparent locations relative to the map data and from a user perspective based on the measured locations of the user,

wherein the method further comprises generating an avatar of the user, the avatar providing a virtual representation of the user, wherein generating the avatar includes;

directing the user to apply wearable sensors to respective parts of a body of the user;

with the user wearing the wearable sensors, measuring locations of the wearable sensors and locations of the headset as the user performs a set of predetermined physical movements; and

generating a skeletal model of the user based on the measured locations of the wearable sensors and of the headset,

wherein, when generating the avatar, measuring the locations of the wearable sensors and the locations of the headset includes (i) directing the user to assume multiple predetermined physical postures and, (ii) for each physical posture, generating a three-dimensional map of the locations of the wearable sensors and the location of the headset while the user is assuming the respective physical posture, and

wherein generating the skeletal model includes estimating a set of joint locations and a set of limb lengths of the user based on changes in three-dimensional maps that result from the user assuming different physical postures.

2. The method of claim 1, wherein, when generating the avatar, measuring the locations of the wearable sensors includes processing, by the server apparatus, data generated by a set of the stationary sensors in response to each of the set of stationary sensors detecting signals transmitted by the wearable sensors.

3. The method of claim 1,

wherein measuring the locations of the headset includes processing, by the server apparatus, inputs received from a set of the stationary sensors in response to each of the set of stationary sensors detecting signals transmitted by the headset, and

wherein measuring the locations of the wearable sensors includes, for each wearable sensor, (i) establishing an initial location of the wearable sensor relative to the headset, and (ii) after the initial location has been established, tracking subsequent changes in the location of the wearable sensor using internal measurement circuitry within the wearable sensor.

4. The method of claim 3, further comprising:

generating a first measurement of a location of a particular one of the wearable sensors using the internal measurement circuitry within the particular wearable sensor;

generating a second measurement of the location of the particular wearable sensor by processing inputs received from a set of the stationary sensors, the second measurement establishing a three-dimensional bounding region predicted to contain the particular wearable sensor; and

performing a wearable-sensor-retraining operation based at least in part on the first measurement of the location of the particular wearable sensor falling outside the three-dimensional bounding region established by the second measurement.

5. The method of claim 4, wherein performing the wearable-sensor-retraining operation includes:

generating, based on (i) a current location of the headset and (ii) the skeletal model when arranged in a predetermined posture, an expected location of the particular wearable sensor when the user assumes the predetermined posture;

while the user assumes the predetermined posture, obtaining a test measurement of the location of the particular wearable sensor using the internal measurement circuitry within the particular wearable sensor;

calculating a three-dimensional offset between the expected location and the location obtained by the test measurement; and

going forward, applying the three-dimensional offset as a spatial correction to measurements of the location of the particular wearable sensor made using the internal measurement circuitry within the particular wearable sensor.

6. The method of claim 5, further comprising:

defining, by the map data, a virtual feature in a particular area within the physical space, to enable the headset to render the virtual feature as a hologram to the user, the user having been directed to assume the predetermined posture when entering the particular area where the virtual feature is defined; and

detecting that the user has physically entered the particular area where the virtual feature is defined,

wherein obtaining the test measurement is performed in response to detecting that the user has entered the particular area.

7. The method of claim 1, further comprising:

detecting multiple predetermined patterns of changes in locations and/or orientations in a set of the wearable sensors, based on output from internal measurement circuitry of the set of wearable sensors; and

for each of the multiple predetermined patterns detected, (i) mapping the detected predetermined pattern to a respective user command and (ii) executing the user command to which the detected predetermined pattern of changes is mapped.

8. The method of claim 1, further comprising:

preparing to render a hologram by the headset;

rendering, by the headset, pixels of the hologram in display areas of the headset that do not intersect with an appendage of the user as viewed through the headset; and

masking, by the headset, pixels of the hologram in display areas of the headset that do intersect with the appendage of the user as viewed through the headset, such that the hologram is made to appear partially blocked by the appendage of the user, the appendage of the user including any body parts that form the appendage as well as any objects held by the appendage.

9. The method of claim 1, wherein the user is a first user, and wherein the method further comprises:

repeatedly updating the avatar of the first user to reflect movement of the first user as indicated by the wearable sensors;

transmitting a real-time version of the avatar of the first user to a second headset worn by a second user, such that the second headset is enabled to render the avatar of the first user to the second user as the avatar of the first user is updated to reflect the movement of the first user.

10. The method of claim 9,

wherein the physical space is a first physical space,

wherein the second user is located in a second physical space distinct from the first physical space, and

wherein providing the real-time version of the avatar of the first user to the second headset includes transmitting, by the server apparatus in the first physical space, the avatar of the first user over a computer network to a second server apparatus running in the second physical space.

11. The method of claim 10, further comprising transmitting at least a portion of the map data over the computer network to the second server apparatus running in the second physical space, such that the second headset is enabled to (i) receive the transmitted map data from the second server apparatus, (ii) render the first physical space as a virtual space, and (iii) render the avatar of the first user in locations of the virtual space that correspond to locations of the first user within the first physical space.

12. The method of claim 11, further comprising, prior to transmitting the map data, filtering the map data to avoid conveying private information about the first physical space to the second server apparatus.

13. The method of claim 12, further comprising transmitting, over the computer network to the second server apparatus running in the second physical space, artificial content representing virtual features of the first physical space at specified locations within the first physical space, such that the second headset is enabled to receive the artificial content and to render the virtual space with the artificial content shown at the respective locations.

14. The method of claim 13, further comprising providing the artificial content to the headset worn by the first user, such that the headset worn by the first user is enabled to render holograms that display the artificial content at the respective locations.

15. The method of claim 11, further comprising:

directing the second headset to render a scene within the second physical space, the scene including a set of holograms that establish a virtual environment of the scene and a set of virtual interconnects at respective locations within the virtual environment of the scene, each virtual interconnect occupying a respective area that users may enter and exit; and

in response to detecting that the second user has entered a virtual interconnect, directing the second headset to present a choice to the second user that enables the second user to change the scene or a characteristic thereof.

16. The method of claim 15, wherein the method further comprises reorienting the second user relative to the scene by directing the second headset to virtually rotate the scene around the second user, such that, upon the second user exiting the virtual interconnect, a physical direction of movement of the second user differs from a virtual direction of movement of the second user within the scene.

17. The method of claim 15,

wherein the set of holograms includes a set of barrier holograms that represent one of (i) virtual barriers within the scene or (ii) illegal areas in the scene, and

wherein the method further comprises, in response to detecting that the second user has physically crossed from a first location on a first side of a barrier hologram to a second location on a second side of the barrier hologram, continuing to render an avatar of the second user at the first location on the first side of the barrier hologram, thereby enabling the second user and the avatar of the second user to become separated when the second user crosses the barrier hologram.

18. The method of claim 17, wherein the set of barrier holograms is arranged to form a virtual path on which the second user must remain to avoid becoming separated from the avatar of the second user.

19. The method of claim 11, further comprising:

directing the headset worn by the first user to render a VR (virtual reality) environment around the first user, the VR environment being a synthetic environment not based on the first physical environment;

while rendering the VR environment in the headset, (i) monitoring changes in locations and/or orientations of the wearable sensors and of the headset, (ii) mapping predefined patterns of the changes in locations and/or orientations of the wearable sensors and the headset to respective VR control commands, and (iii) executing the VR control commands to enable the first user to interact with the VR environment.

20. The method of claim 19, further comprising:

defining an area of floor space within the physical space;

enabling VR control commands within the defined area of floor space; and

accepting VR control commands from the first user only when the first user remains within the defined area of floor space.

21. The method of claim 1, wherein generating the skeletal model further includes directing the headset to project an image of the skeletal model of the user as the user assumes one or more of the physical postures.

22. A server apparatus comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:

measure locations of a user in a physical space as the user moves through the physical space, based on receipt of inputs from multiple stationary sensors positioned at respective sensor locations within the physical space;

store map data that describes a map of the physical space;

specify a set of holograms that have apparent locations that are defined relative to the map data; and

provide the measured locations of the user and at least a portion of the map data to a headset worn by the user, to enable the headset to render the set of holograms at the apparent locations relative to the map data and from a user perspective based on the measured locations of the user,

wherein the control circuitry is further constructed and arranged to generate an avatar of the user, the avatar providing a virtual representation of the user, the control circuitry constructed and arranged to;

direct the user to apply wearable sensors to respective parts of a body of the user;

with the user wearing the wearable sensors, measure locations of the wearable sensors and locations of the headset as the user performs a set of predetermined physical movements; and

generate a skeletal model of the user based on the measured locations of the wearable sensors and of the headset,

wherein the control circuitry is further constructed and arranged to;

generate a first measurement of a location of a particular one of the wearable sensors using internal measurement circuitry within the particular wearable sensor;

generate a second measurement of the location of the particular wearable sensor by processing inputs received from a set of the stationary sensors, the second measurement establishing a three-dimensional bounding region predicted to contain the particular wearable sensor; and

perform a wearable-sensor-retraining operation based at least in part on the first measurement of the location of the particular wearable sensor falling outside the three-dimensional bounding region established by the second measurement.

23. The server apparatus of claim 22, wherein the control circuitry constructed and arranged to perform the wearable-sensor-retraining operation is further constructed and arranged to:

generate, based on (i) a current location of the headset and (ii) the skeletal model when arranged in a predetermined posture, an expected location of the particular wearable sensor when the user assumes the predetermined posture;

while the user assumes the predetermined posture, obtain a test measurement of the location of the particular wearable sensor using the internal measurement circuitry within the particular wearable sensor;

calculate a three-dimensional offset between the expected location and the location obtained by the test measurement; and

going forward, apply the three-dimensional offset as a spatial correction to measurements of the location of the particular wearable sensor made using the internal measurement circuitry within the particular wearable sensor.

24. The server apparatus of claim 23, wherein the control circuitry is further constructed and arranged to:

define, by the map data, a virtual feature in a particular area within the physical space, to enable the headset to render the virtual feature as a hologram to the user, the user having been directed to assume the predetermined posture when entering the particular area where the virtual feature is defined; and

detect that the user has physically entered the particular area where the virtual feature is defined,

wherein obtaining the test measurement is performed in response to a detection that the user has entered the particular area.

25. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry, cause the control circuitry to perform a method of managing virtual content to be displayed to users, the method comprising:

measuring locations of a user in a physical space as the user moves through the physical space, by a server apparatus receiving inputs from multiple stationary sensors positioned at respective sensor locations within the physical space and processing the inputs to generate the locations of the user;

storing map data that describes a map of the physical space;

specifying a set of holograms that have apparent locations that are defined relative to the map data; and

providing the measured locations of the user and at least a portion of the map data to a headset worn by the user, to enable the headset to render the set of holograms at the apparent locations relative to the map data and from a user perspective based on the measured locations of the user,

wherein the method further comprises generating an avatar of the user, the avatar providing a virtual representation of the user, wherein generating the avatar includes;

directing the user to apply wearable sensors to respective parts of a body of the user;

with the user wearing the wearable sensors, measuring locations of the wearable sensors and locations of the headset as the user performs a set of predetermined physical movements; and

generating a skeletal model of the user based on the measured locations of the wearable sensors and of the headset,

wherein the user is a first user, and wherein the method further comprises;

repeatedly updating the avatar of the first user to reflect movement of the first user as indicated by the wearable sensors;

transmitting a real-time version of the avatar of the first user to a second headset worn by a second user, such that the second headset is enabled to render the avatar of the first user to the second user as the avatar of the first user is updated to reflect the movement of the first user,

wherein the physical space is a first physical space,

wherein the second user is located in a second physical space distinct from the first physical space, and

wherein providing the real-time version of the avatar of the first user to the second headset includes transmitting, by the server apparatus in the first physical space, the avatar of the first user over a computer network to a second server apparatus running in the second physical space, and

wherein the method further comprises;

directing the second headset to render a scene within the second physical space, the scene including a set of holograms that establish a virtual environment of the scene and a set of virtual interconnects at respective locations within the virtual environment of the scene, each virtual interconnect occupying a respective area that users may enter and exit; and

in response to detecting that the second user has entered a virtual interconnect, directing the second headset to present a choice to the second user that enables the second user to change the scene or a characteristic thereof.

26. The computer program product of claim 25, wherein the method further comprises reorienting the second user relative to the scene by directing the second headset to virtually rotate the scene around the second user, such that, upon the second user exiting the virtual interconnect, a physical direction of movement of the second user differs from a virtual direction of movement of the second user within the scene.

27. The computer program product of claim 25,

wherein the set of holograms includes a set of barrier holograms that represent one of (i) virtual barriers within the scene or (ii) illegal areas in the scene, and

wherein the method further comprises, in response to detecting that the second user has physically crossed from a first location on a first side of a barrier hologram to a second location on a second side of the barrier hologram, continuing to render an avatar of the second user at the first location on the first side of the barrier hologram, thereby enabling the second user and the avatar of the second user to become separated when the second user crosses the barrier hologram.

28. The computer program product of claim 27, wherein the set of barrier holograms is arranged to form a virtual path on which the second user must remain to avoid becoming separated from the avatar of the second user.

29. The computer program product of claim 25, wherein the method further comprises:

directing the headset worn by the first user to render a VR (virtual reality) environment around the first user, the VR environment being a synthetic environment not based on the first physical space;

while rendering the VR environment in the headset, (i) monitoring changes in locations and/or orientations of the wearable sensors and of the headset, (ii) mapping predefined patterns of the changes in locations and/or orientations of the wearable sensors and the headset to respective VR control commands, and (iii) executing the VR control commands to enable the first user to interact with the VR environment.

30. The computer program product of claim 29, further comprising:

defining an area of floor space within the physical space;

enabling VR control commands within the defined area of floor space; and

accepting VR control commands from the first user only when the first user remains within the defined area of floor space.