Augmented reality with motion sensing

• US 10,349,036 B2
• Filed: 08/07/2015
• Issued: 07/09/2019
• Est. Priority Date: 08/08/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S., Roy, Neeloy, He, Hongyuan
• Primary Examiner(s): Philippe, Gims S
• Assistant Examiner(s): Hasan, Mainul
• Application Number:
  US14/821,499
• Publication Number:
  US 20160044298A1
• Time in Patent Office: 1,432 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 For headphones

Abstract

The technology relates to a motion sensory and imaging device capable of acquiring imaging information of the scene and providing at least a near real time pass-through of imaging information to a user. The sensory and imaging device can be used stand-alone or coupled to a wearable or portable device to create a wearable sensory system capable of presenting to the wearer the imaging information augmented with virtualized or created presentations of information.

Claims

1. A device, including:

a plurality of imaging sensors, including a first set of infrared pixels sensitive to infrared light and a second set of red-green-blue pixels sensitive to visible light;

wherein imaging sensors are arranged to provide stereoscopic imaging information for a scene in front of a user wearing a coupled wearable or portable device that blocks the scene from being viewed including at least one hand, said at least one hand belonging to the user wearing the coupled wearable or portable device and moving within the scene, and wherein said at least one hand is blocked from being viewed by the user;

one or more illumination sources arranged to direct illumination towards the at least one hand; and

a controller coupled to the imaging sensors and illumination sources to control operation thereof, including;

operating the imaging sensors and illumination sources to acquire imaging information of the scene, including capturing imaging information for at least one hand within view of the imaging sensors;

wherein at least part of the at least one hand is illuminated by the one or more illumination sources;

separating imaging information received from infrared pixels sensitive to infrared light from imaging information received from red-green-blue pixels sensitive to visible light;

processing imaging information from infrared pixels to determine gesture information of a user indicating a command to a machine under control;

processing imaging information from red-green-blue pixels to be provided as a live video feed via a presentation interface; and

providing at least;

i) gestural information of the at least one hand determined using the imaging information from infrared pixels for the at least one hand; and

ii) a display of the live video feed including imaging information of the scene for presentation to a user via a coupled wearable or portable device worn by a user;

wherein the user wearing the coupled wearable or portable device is devoid of visual information from the scene surrounding the user other than from the live video feed.

2. The device of claim 1, wherein the processing imaging information from infrared pixels further includes:

extracting fine features of a real world space using infrared pixels that capture infrared components of illumination in the scene.

3. The device of claim 2, wherein fine features of the real world space include surface texture of the real world space.

4. The device of claim 2, wherein fine features of the real world space include edges of the real world space.

5. The device of claim 2, wherein fine features of the real world space include curvatures of the real world space.

6. The device of claim 2, wherein fine features of the real world space include surface texture of objects in the real world space.

7. The device of claim 2, wherein fine features of the real world space include edges of objects in the real world space.

8. The device of claim 2, wherein fine features of the real world space include curvatures of objects in the real world space.

9. The device of claim 1,

wherein the controller further provides:

determining ambient lighting conditions;

adjusting display of output based upon the ambient lighting conditions determined, including;

dynamically switching display between video information captured using red-green-blue pixels sensitive to visible light and video information captured using infrared pixels sensitive to infrared light for display depending upon ambient conditions.

10. The device of claim 1, further including:

a motion sensor; and

wherein the controller further provides;

determining difference information between a first and a second positional information from the motion sensor; and

computing movement information for the device with respect to a fixed point based upon the difference information.

11. The device of claim 1, further including:

one or more fasteners that fasten the imaging sensors and the illumination sources to a mounting surface in a wearable presentation device.

12. The device of claim 1, further including:

one or more fasteners that fasten the imaging sensors and the illumination sources to a cavity in a wearable presentation device.

13. The device of claim 1, further including:

one or more fasteners that fasten the imaging sensors and the illumination sources to a mounting surface in a portable presentation device.

14. The device of claim 1, further including:

one or more fasteners that fasten the imaging sensors and the illumination sources to a cavity in a portable presentation device.

15. The device of claim 1, further including the display of the live video feed including images of the hand grasping a real object.

16. The device of claim 15, further including the display of the live video feed including images of the hand grasping a virtual object.

17. The device of claim 1, further including the plurality of image sensors being disposed co-planar on a head mounted display.

18. The device of claim 17, the head mounted display including an optical assembly that displays a surrounding environment or a virtual environment to the user.

19. The device of claim 1, further including detecting a position and shape of a user'"'"'s hand and projecting onto a display of a head-mounted device;

wherein the user is presented with gestures in the display of the live video feed that reflecting hand motions of the user captured by the infrared pixels and interactively control objects in a virtual environment.