Drift cancelation for portable object detection and tracking

• US 10,007,329 B1
• Filed: 02/11/2015
• Issued: 06/26/2018
• Est. Priority Date: 02/11/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S.
• Primary Examiner(s): Haley, Joseph
• Assistant Examiner(s): Frank, Emily
• Application Number:
  US14/620,093
• Time in Patent Office: 1,231 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The technology disclosed can provide capabilities such as using motion sensors and/or other types of sensors coupled to a motion-capture system to monitor motions within a real environment. A virtual object can be projected to a user of a portable device integrated into an augmented rendering of a real environment about the user. Motion information of a user body portion is determined based at least in part upon sensory information received from imaging or acoustic sensory devices. Control information is communicated to a system based in part on a combination of the motion of the portable device and the detected motion of the user. The virtual device experience can be augmented in some implementations by the addition of haptic, audio and/or other sensory information projectors.

Claims

1. A method of operating a movable sensor system including at least one sensor, the method including:

determining a first positional information of the sensor with respect to a fixed point in space at a first time;

determining a second positional information of the sensor with respect to the fixed point in space at a second time;

determining from the first positional information and the second positional information, positional difference information between the first positional information and the second positional information;

computing movement information for the sensor with respect to the fixed point in space based upon the positional difference information;

applying the movement information for the sensor to apparent environment information sensed by the sensor to remove motion of the sensor therefrom to yield actual environment information; and

communicating the actual environment information.

2. The method of claim 1, wherein the applying further includes:

capturing positional information of an object portion at the first time and the second time; and

computing object portion movement information relative to the fixed point in space at the first time and the second time based upon the difference information and the movement information for the sensor.

3. The method of claim 2, further including:

repeatedly determining movement information for the sensor and the object portion at successive times; and

analyzing a sequence of movement information to determine a path of the object portion with respect to the fixed point in space.

4. The method of claim 3, further including comparing the path to a plurality of path templates and identifying a template that best matches the path.

5. The method of claim 4, further including determining from the template that best matches the path, control information to a system;

and

providing the control information to the system.

6. The method of claim 3, further including constructing a 3D model of the object portion and tracking movement thereof over a region of space through which image sensors of a movable device are directed.

7. The method of claim 3, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the object portion.

8. The method of claim 3, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of a body portion.

9. A method of creating a virtual device experience, the method comprising:

projecting a virtual object to a user using a portable presentation mechanism;

determining for a moving reference frame of a head mounted display moving reference frame information for motion of the moving reference frame and the head mounted display relative to a fixed point on a human body;

capturing body portion movement information;

extracting control information based partly on the body portion movement information with respect to the moving reference frame information; and

communicating control information to a system.

10. The method of claim 9, wherein the projecting includes projecting an image of a virtualized object in an augmented rendering of a real environment surrounding a user wearing a portable presentation device.

11. The method of claim 10, wherein the image includes an image of a book.

12. The method of claim 9, wherein the extracting includes constructing a 3D model of the body portion and tracking movement thereof based on a sequence of images.

13. The method of claim 9, wherein the extracting includes recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the body portion.

14. The method of claim 9, wherein the extracting includes recognizing at least one of an image sequence or an audio signal that indicates a gesture.

15. The method of claim 14, wherein the gesture indicates a change in operational mode.

16. A portable sensor system, including:

one or more motion sensors;

a sensory system to capture positional information of an object portion; and

a control module for tracking a position of an object portion, in response to detection of the object by the sensory system, by;

determining a first positional information of at least one of the motion sensors with respect to a fixed point in space at a first time;

determining a second positional information of at least one of the motion sensors with respect to the fixed point in space at a second time;

determining from the first positional information and the second positional information, positional difference information between the first positional information and the second positional information;

computing movement information for the sensor with respect to the fixed point in space based upon the difference information;

capturing positional information of an object portion at the first time and the second time;

computing object portion movement information relative to the fixed point in space at the first time and the second time based upon the difference information and the movement information for the sensor; and

communicating the actual environment information.

17. The system of claim 16, the sensory system further comprising:

an imaging system configured to capture a sequence of images; and

wherein the control module is further configured to calculate a path of the object by;

repeatedly determining movement information responsive to successive captures of the object; and

analyzing a sequence of movement information to determine a path of the object portion with respect to the fixed point in space.

18. The system of claim 17, further configured to construct a 3D model of the object and tracking movement thereof based on the sequence of images.

19. The system of claim 17, further configured to recognize at least one of an image sequence or an audio signal that indicates a trajectory of the object.

20. The system of claim 17, further configured to recognize at least one of an image sequence or an audio signal that indicates a gesture of the object.

21. The system of claim 16, wherein the one or more motion sensors comprises at least one accelerometer.

22. The system of claim 16, further comprising at least one presentation mechanism to render a virtual object based at least in part upon the object portion.

23. The system of claim 22, wherein the at least one presentation mechanism comprises a head mounted display.