Safety for wearable virtual reality devices via object detection and tracking
First Claim
1. A method of detecting an imminent collision with a physical object during a virtual reality experience, the method including:
- using at least one forward facing camera mounted to a head mounted virtual reality device (HMVRD) to capture a sequence of images of a real world environment ahead of a user wearing the HMVRD;
using features in the images to track proximity of at least one physical object ahead of the user;
correlating multiple images to detect relative motion between at least one approaching physical object and the user; and
responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user;
at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object;
causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in a virtual environment of the HMVRD;
at a second time t1, sensing, in the real world environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the real world environment and the attached sensor due to body movement of the user, wherein the physical object has not moved in the real world environment between t0 and t1; and
causing display of a second virtual representation of the physical object at an actual second position, including;
sensing motion of the attached sensor and calculating a second reference frame that accounts for repositioning of the attached sensor;
calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and
transforming the first and second positions of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed second position is transformed to the actual second position.
11 Assignments
0 Petitions
Accused Products
Abstract
The technology disclosed can provide improved safety by detecting potential unsafe conditions (e.g., collisions, loss of situational awareness, etc.) confronting the user of a wearable (or portable) sensor configured to capture motion and/or determining the path of an object based on imaging, acoustic or vibrational waves. Implementations can enable improved safety to users of virtual reality for machine control and/or machine communications applications using wearable (or portable) devices, e.g., head mounted displays (HMDs), wearable goggles, watch computers, smartphones, and so forth, or mobile devices, e.g., autonomous and semi-autonomous robots, factory floor material handling systems, autonomous mass-transit vehicles, automobiles (human or machine driven), and so forth, equipped with suitable sensors and processors employing optical, audio or vibrational detection.
-
Citations
20 Claims
-
1. A method of detecting an imminent collision with a physical object during a virtual reality experience, the method including:
-
using at least one forward facing camera mounted to a head mounted virtual reality device (HMVRD) to capture a sequence of images of a real world environment ahead of a user wearing the HMVRD; using features in the images to track proximity of at least one physical object ahead of the user; correlating multiple images to detect relative motion between at least one approaching physical object and the user; and responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user; at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object; causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in a virtual environment of the HMVRD; at a second time t1, sensing, in the real world environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the real world environment and the attached sensor due to body movement of the user, wherein the physical object has not moved in the real world environment between t0 and t1; and causing display of a second virtual representation of the physical object at an actual second position, including; sensing motion of the attached sensor and calculating a second reference frame that accounts for repositioning of the attached sensor; calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and transforming the first and second positions of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed second position is transformed to the actual second position. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A non-transitory computer readable medium impressed with instructions for detecting an imminent collision with a physical object during a virtual reality experience, which instructions when executed by one or more processors cause the processors to perform the following:
-
using at least one forward facing camera mounted to a head mounted virtual reality device (HMVRD) to capture a sequence of images of a real world environment ahead of a user wearing the HMVRD; using features in the images to track proximity of at least one physical object ahead of the user; correlating multiple images to detect relative motion between at least one approaching physical object and the user; and responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user; at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object; causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in a virtual environment of the HMVRD; at a second time t1, sensing, in the real world environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the real world environment and the attached sensor due to body movement of the user, wherein the physical object has not moved in the real world environment between t0 and t1; and causing display of a second virtual representation of the physical object at an actual second position, including; sensing motion of the attached sensor and calculating a second reference frame that accounts for repositioning of the attached sensor; calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and transforming the first and second positions of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed second position is transformed to the actual second position.
-
-
20. A system, including:
-
a head mounted virtual reality device (HMVRD); at least one forward facing camera mounted to the head mounted virtual reality device (HMVRD) that captures a sequence of images of a real world environment ahead of a user wearing the HMVRD; and a computer readable medium including instructions for detecting an imminent collision with a physical object during a virtual reality experience, which instructions when executed by one or more processors cause the processors to perform the following; using features in the images to track proximity of at least one physical object ahead of the user; correlating multiple images to detect relative motion between at least one approaching physical object and the user; responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user; at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object; causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in a virtual environment of the HMVRD; at a second time t1, sensing, in the real world environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the real world environment and the attached sensor due to body movement of the user, wherein the physical object has not moved in the real world environment between t0 and t1; and causing display of a second virtual representation of the physical object at an actual second position, including; sensing motion of the attached sensor and calculating a second reference frame that accounts for repositioning of the attached sensor; calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and transforming the first and second positions of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed second position is transformed to the actual second position.
-
Specification