WEARABLE AUGMENTED REALITY DEVICES WITH OBJECT DETECTION AND TRACKING

• US 20170345219A1
• Filed: 08/18/2017
• Published: 11/30/2017
• Est. Priority Date: 05/20/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: HOLZ, David S.
• Granted Patent: US 10,600,248 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 5/18 Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12

Abstract

The technology disclosed can provide capabilities to view and/or interact with the real world to the user of a wearable (or portable) device using a sensor configured to capture motion and/or determining the path of an object based on imaging, acoustic or vibrational waves. Implementations can enable improved user experience, greater safety, greater functionality to users of virtual reality for machine control and/or machine communications applications using wearable (or portable) devices, e.g., head mounted devices (HMDs), wearable goggles, watch computers, smartphones, and so forth, or mobile devices, e.g., autonomous and semi-autonomous robots, factory floor material handling systems, autonomous mass-transit vehicles, automobiles (human or machine driven), and so forth, equipped with suitable sensors and processors employing optical, audio or vibrational detection.

Claims

1. A system, including:

a head mounted device (HMD) including a presentation mechanism to provide an augmented hybrid experience to a user wearing the head mounted device;

a sensor coupled to the HMD including at least one camera to capture a sequence of images of a physical real environment;

a controller including a processor and memory storing instructions that when executed provide smooth transitioning between an immersive virtual environment and a convergent mixed reality environment during the augmented hybrid experience, wherein the augmented hybrid experience includes selecting between;

(i) an immersive virtual environment including virtual presentation consisting of virtual objects;

wherein some virtual objects correspond to sensed physical objects sensed by the sensor; and

(ii) a convergent mixed reality environment including a near real time display of the physical real environment surrounding the user; and

responsive to a first command input received, automatically interrupting the immersive virtual environment being presented to the user and superimposing at least one virtual object of the immersive virtual environment into a generated data stream including near real time physical real environment to generate data representing a mixed reality environment during the augmented hybrid experience; and

responsive to a second command input received, automatically interrupting the mixed reality environment being presented to the user and resuming the immersive virtual environment during the augmented hybrid experience;

wherein the first command input and the second command input are determined using positions of a physical object sensed at time t0 and time t1 by the sensor coupled to the HMD and distinguishing motion made by the physical object sensed from motion of the sensor coupled to the HMD and interpreting the motion made by the physical object sensed to indicate a command.

2. The system of claim 1, wherein a command input is automatically triggered in response to a free-form gesture.

3. The system of claim 1, wherein a command input is automatically triggered in response to an audio signal.

4. The system of claim 1, wherein a command input is automatically triggered in response to a vibrational signal.

5. The system of claim 1, wherein a command input is automatically triggered in response to an optical signal.

6. The system of claim 1, further including, simultaneously, manipulating in responsive a command input at least one virtual object of the immersive virtual environment and at least one physical object of the physical real environment during the augmented hybrid experience.

7. A method of smoothly transitioning between an immersive virtual environment and a convergent mixed reality environment during an augmented hybrid experience, the method comprising:

using a sensor having at least one camera coupled to a head mounted device (HMD) to capture a sequence of images of a physical real environment in convergence with presenting to a user wearing the HMD an immersive virtual environment overlaid upon the physical real environment during an augmented hybrid experience; and

responsive to a first command input received, automatically, interrupting the immersive virtual environment being presented to the user and superimposing at least one virtual object of the immersive virtual environment in a generated data stream including near real time physical real environment to generate data representing a mixed reality environment during the augmented hybrid experience; and

responsive to a second command input received, automatically interrupting the mixed reality environment being presented to the user and resuming the immersive virtual environment during the augmented hybrid experience;

wherein the first command input and the second command input are determined using positions of a physical object sensed at time t0 and time t1 by the sensor coupled to the HMD and distinguishing motion made by the physical object sensed from motion of the sensor coupled to the HMD and interpreting the motion made by the physical object sensed to indicate a command.

8. The method of claim 7, wherein the mixed reality environment includes at least one virtual object of the immersive virtual environment and at least one physical object of the physical real environment.

9. The method of claim 7, wherein a command input is automatically triggered in response to a free-form gesture.

10. The method of claim 7, wherein a command input is automatically triggered in response to an audio signal.

11. The method of claim 7, wherein a command input is automatically triggered in response to a vibrational signal.

12. The method of claim 7, wherein a command input is automatically triggered in response to an optical signal.

13. A method of smoothly transitioning between an immersive virtual environment and a convergent physical real environment during an augmented hybrid experience generated by a head mounted device (HMD), the method comprising:

using at least one camera mounted to a head mounted device (HMD) to capture a sequence of images of a physical real environment in convergence with an immersive virtual environment during an augmented hybrid experience generated by the HMD; and

automatically triggering a pass through mode of the HMD in response to a command input, wherein the pass through mode interrupts the immersive virtual environment and substitutes a live feed of the physical real environment in the augmented hybrid experience.

14. The method of claim 13, further including:

at a first time t0, using a sensor attached to the HMD, sensing a first position for at least one physical object in a first reference frame of the physical real environment, including tracking portions of the physical object;

causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in the immersive virtual environment of the HMD;

at a second time t1, sensing, in the physical real environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the physical real environment and the sensor due to body movement of a user wearing the HMD, wherein the physical object has not moved in the physical real environment between t0 and t1; and

causing display of a second virtual representation of the physical object at an actual second position.

15. The method of claim 14, wherein causing display of a second virtual representation of the physical object at an actual second position further includes:

sensing motion of the sensor and calculating a second reference frame that accounts for repositioning of the sensor;

calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and

transforming the first position and the second position of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the second position sensed is transformed to the actual second position.

16. The method of claim 15, wherein the common reference frame is a world reference frame that does not change as the sensor is repositioned.

17. The method of claim 15, wherein the common reference frame is the second reference frame.

18. The method of claim 15, wherein the transforming the first position and second position of the physical object into the common reference frame further includes applying an affine transformation.

19. The method of claim 15, further including determining an orientation of the physical object at the first position with respect to the first reference frame and causing the display of the physical object accordingly.

20. The method of claim 15, further including determining an orientation of the physical object at the second position with respect to the second reference frame and causing the display of the physical object accordingly.