User-defined virtual interaction space and manipulation of virtual configuration

• US 10,126,822 B2
• Filed: 12/16/2014
• Issued: 11/13/2018
• Est. Priority Date: 12/16/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Cohen, Isaac, Sills, Maxwell
• Primary Examiner(s): Sun, Li
• Assistant Examiner(s): Filosi, Terri
• Application Number:
  US14/572,704
• Publication Number:
  US 20150169176A1
• Time in Patent Office: 1,428 Days
• Field of Search: 715852
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H05K 999/99 dummy group

Abstract

The technology disclosed relates to creating user-defined interaction spaces and modalities in a three dimensional (3D) sensor space in response to control gestures. It also relates to controlling virtual cameras in the 3D sensor space using control gestures and manipulating controls of the virtual cameras through the control gestures. In particular, it relates to defining one or more spatial attributes of the interaction spaces and modalities in response to one or more gesture parameters of the control gesture. It also particularly relates to defining one or more visual parameters of a virtual camera in response to one or more gesture parameters of the control gesture.

Claims

1. A computer implemented method of defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including:

detecting circular sweeps in response to control gestures of a control object in a three dimensional (3D) sensor space, wherein each of a circular sweep of the detected circular sweeps has a center point and a direction vector;

constructing multiple virtual cameras for the 3D sensor space using the detected circular sweeps, wherein each of a virtual camera of the multiple virtual cameras is situated at a respective center point of a respective circular sweep and has a field of view in a direction indicated by a respective direction vector of the respective circular sweep;

assigning each of the virtual cameras a virtual camera checkpoint, that includes information for setting a camera perspective and designating a particular virtual camera selector, from an array of virtual camera selectors displayed in the 3D sensor space using one or more control gestures;

selecting a particular virtual camera in response to selection of a corresponding virtual camera selector in the 3D sensor space; and

controlling the particular virtual camera by modifying the information included in the checkpoint assigned to the particular virtual camera using one or more control gestures.

2. The computer implemented method of claim 1, the method further including:

detecting a lateral movement of the control object at least initiated at a starting point proximate to a center point of the particular virtual camera in the three dimensional (3D) sensor space;

defining a virtual spring having a static length that is determined from a length of the lateral movement;

defining a spring constant for the virtual spring, wherein the virtual spring is determined from a length of a radius of a circular sweep made by the control object when defining the virtual camera, wherein the virtual spring is defined to stretch between the center point of the circular sweep made by the control object when defining the virtual camera and a movable position of the virtual camera when grasped and moved by the control object;

using the static length and the spring constant, computing a virtual force to be applied to the virtual camera by the virtual spring when the virtual camera is released from the control object grasping the virtual camera; and

determining motion of the virtual camera in the 3D sensor space based upon interactions of the virtual spring with the virtual force computed.

3. The computer implemented method of claim 1, the method further including:

detecting at least one circular sweep around a virtual object;

calculating a radius of the circular sweep responsive to a found point that is equidistant to a plurality of points defined on contour of at least one control gesture;

determining a focal length of a virtual camera towards the virtual object responsive to the radius of the circular sweep by;

constructing a virtual sphere in the 3D sensor space that is in proportion to the radius of the circular sweep;

defining a vector from the virtual camera to a center of the virtual sphere; and

determining a point of intersection between the virtual sphere and the vector;

defining a field of view and orientation of the virtual camera responsive to orientation of the control object; and

interpolating the virtual camera through time to a new position that coincides with the point of intersection.

4. The computer implemented method of claim 1, the method further including:

calculating a radius of at least one circular sweep responsive to a found point that is equidistant to a plurality of points defined on contour of at least one control gesture, wherein

a radius of the circular sweep defines a spring constant of a virtual camera launcher of a virtual camera; and

a first distance between center of the circular sweep and the virtual camera defines a static length of a spring movement;

detecting a backward pull of the virtual camera launcher to a second distance in response to a second control gesture of the control object in the 3D space; and

accelerating the virtual camera through time responsive to releasing the virtual camera launcher by a third control gesture.

5. The method of claim 4, wherein the control object is a hand and orientation of the virtual camera is responsive to orientation of at least one finger of the hand.

6. The method of claim 1, wherein the virtual camera checkpoint further includes at least a position, an orientation, a focal length, a deviation relative to at least one virtual camera, or a maximum aperture.

7. The method of claim 1, further including:

linking a virtual camera selector to one or more real cameras in a physical space; and

selecting and controlling a particular real camera in response to selection of a corresponding virtual camera selector.

8. The computer implemented method of claim 1, wherein the control gestures are made by one or more hands, the method further including:

determining a focal length of at least one virtual camera in a three dimensional (3D) sensor space responsive to at least one of;

a radius of a circular sweep of one or more hands;

a distance between midpoints of the one or more hands;

a scale of curling of fingers of the one or more hands; and

a degree of freedom between fingers;

defining a field of view and an orientation for the virtual camera responsive to orientation of the one or more hands;

constructing a virtual sphere along a plurality of points on a non-intersecting contour of the one or more hands;

defining a view vector from a center of the virtual sphere to a point on a surface of the virtual sphere that is equidistant to a plurality of points on the one or more hands; and

manipulating the virtual camera by at least rotating, translating, compressing, or scaling the view vector responsive to subsequent control gestures of the one or more hands.

9. The computer implemented method of claim 1, the method further including:

detecting a first control gesture of a control object that defines a starting point of at least one virtual camera in a three dimensional (3D) sensor space;

detecting a second control gesture of the control object that defines a continuous contour through time in the 3D sensor space;

detecting a third control gesture of the control object that defines a finishing point of the virtual camera in the 3D sensor space; and

moving the virtual camera along the continuous contour between the starting point and the finishing point.

10. The method of claim 9, wherein the control object comprises a hand, the method further including determining a focal length of the virtual camera responsive to a distance of a finger of the hand from the continuous contour.

11. The method of claim 10, further including defining a field of view and an orientation for the virtual camera responsive to an orientation of the finger.

12. The method of claim 9, further including mapping the continuous contour to a straight line and moving the virtual camera along the straight line.

13. The method of claim 9, further including defining a plurality of points on the continuous contour to construct a Bezier curve responsive to respective sizes and directions of the points.

14. The method of claim 1, further including iterating through two or more virtual camera checkpoints represented using a visual interaction element, comprising selecting using a keyboard or mouse at least one virtual camera selector representing the virtual camera checkpoints.

15. A non-transitory, computer-readable medium having computer executable instructions for performing a method including:

defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including;

detecting circular sweeps in response to control gestures of a control object in a three dimensional (3D) sensor space, wherein each of a circular sweep of the circular detected sweeps has a center point and a direction vector;

constructing multiple virtual cameras for the 3D sensor space using the detected circular sweeps, wherein each of a virtual camera is situated at a respective center point of a respective circular sweep and has a field of view in a direction indicated by a respective direction vector of the respective circular sweep;

assigning each of the virtual cameras to a virtual camera checkpoint, that includes information for setting a camera perspective and designating a particular virtual camera selector, from an array of virtual camera selectors displayed in the 3D sensor space using one or more control gestures;

selecting a particular virtual camera in response to selection of a corresponding virtual camera selector in the 3D sensor space; and

controlling the particular virtual camera by modifying the information included in the checkpoint assigned to the particular virtual camera using one or more control gestures.

16. A computer system with one or more processors adapted to perform a method including:

defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including;

defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including;

detecting circular sweeps in response to control gestures of a control object in a three dimensional (3D) sensor space, wherein each of a circular sweep of the circular detected sweeps has a center point and a direction vector;

constructing multiple virtual cameras for the 3D sensor space using the detected circular sweeps, wherein each of a virtual camera is situated at a respective center point of a respective circular sweep and has a field of view in a direction indicated by a respective direction vector of the respective circular sweep;

assigning each of the virtual cameras to a virtual camera checkpoint, that includes information for setting a camera perspective and designating a particular virtual camera selector, from an array of virtual camera selectors displayed in the 3D sensor space using one or more control gestures;

selecting a particular virtual camera in response to selection of a corresponding virtual camera selector in the 3D sensor space; and

controlling the particular virtual camera by modifying the information included in the checkpoint assigned to the particular virtual camera using one or more control gestures.

17. A sensor device with one or more processors and coupled with an interface adapted to perform a method including:

defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including;

defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including;

detecting circular sweeps in response to control gestures of a control object in a three dimensional (3D) sensor space, wherein each of a circular sweep of the circular detected sweeps has a center point and a direction vector;

constructing multiple virtual cameras for the 3D sensor space using the detected circular sweeps, wherein each of a virtual camera is situated at a respective center point of a respective circular sweep and has a field of view in a direction indicated by a respective direction vector of the respective circular sweep;

assigning each of the virtual cameras to a virtual camera checkpoint, that includes information for setting a camera perspective and designating a particular virtual camera selector, from an array of virtual camera selectors displayed in the 3D sensor space using one or more control gestures;

selecting a particular virtual camera in response to selection of a corresponding virtual camera selector in the 3D sensor space; and

controlling the particular virtual camera by modifying the information included in the checkpoint assigned to the particular virtual camera using one or more control gestures.

18. A computer implemented method of defining and controlling multiple virtual cameras in a three dimensional (3D) sensor space, the method including:

detecting at least one circular sweep made by a control object around a virtual object;

calculating a radius of the circular sweep responsive to a found point that is equidistant to a plurality of points defined on contour of at least one control gesture;

determining a focal length of a virtual camera towards the virtual object responsive to the radius of the circular sweep by;

constructing a virtual sphere in the 3D sensor space that is in proportion to the radius of the circular sweep;

defining a vector from the virtual camera to a center of the virtual sphere; and

determining a point of intersection between the virtual sphere and the vector;

defining a field of view and orientation of the virtual camera responsive to orientation of the control object; and

interpolating the virtual camera through time to a new position that coincides with the point of intersection;

wherein the virtual camera is assigned to a virtual camera checkpoint defined for detected sweeps in response to control gestures made by the control object in a three dimensional (3D) sensor space, wherein the circular sweeps have respective center points and direction vectors; and

wherein the virtual camera checkpoint is selected from among other virtual camera checkpoints in an array upon receiving a selection of a virtual camera selector associated with a selected one of the virtual camera checkpoints; and

wherein the virtual camera is controlled using visual parameters stored with the virtual checkpoint.