CONTROLLING OBJECTS USING VIRTUAL RAYS

• US 20180059901A1
• Filed: 07/20/2017
• Published: 03/01/2018
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Reavire Inc.
• Original Assignee: Gullicksen Brothers LLC
• Inventors: Gullicksen, Jeff
• Granted Patent: US 11,269,480 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A technique for controlling objects includes a handheld controller that provides output for generating a virtual ray. The virtual ray has a location and a direction. The location of the virtual ray is measured as a location of the handheld controller, based on input from sensors in the surrounding space. As a user moves and/or rotates the handheld controller, the virtual ray sweeps through space, intersecting various objects. When the virtual ray intersects with the location of a controllable object whose identity and location are known, the handheld controller identifies the object and accepts remote-control input from the user for controlling that object. Using the handheld controller, the user may control any controllable object having a known identity and location in space, and may control virtual as well as physical objects.

Claims

1. A method of controlling objects in a 3-D (three-dimensional) space using a handheld controller, the method comprising:

storing an object database that associates multiple controllable objects with respective control locations of the controllable objects in the 3-D space, each control location representing a respective region in the 3-D space to which the handheld controller can be pointed to control the respective controllable object;

measuring a location of the handheld controller in the 3-D space;

measuring an orientation of the hand-held controller in the 3-D space;

combining the measured location of the handheld controller with the measured orientation of the handheld controller to construct a virtual ray, the virtual ray having an origin placed at the measured location of the handheld controller and a direction based on the measured orientation of the hand-held controller; and

in response to the handheld controller pointing such that the virtual ray intersects a control location of a controllable object in the object database, (i) accepting user input by the handheld controller for controlling the controllable object, and (ii) transmitting control instructions from the handheld controller for controlling the controllable object based on the user input.

2. The method of claim 1, wherein measuring the location of the handheld controller includes:

transmitting, by the handheld controller, a wireless signal, the wireless signal propagating through space and encountering multiple antennas, the antennas each processing the wireless signal to produce respective outputs; and

processing the outputs from the multiple antennas to generate the location of the handheld controller in the 3-D space.

3. The method of claim 2, further comprising:

while the virtual ray intersects the control location of the controllable object, receiving, by the handheld controller, a capture command from the user, the capture command directing the handheld controller to capture an identity of the controllable object and enabling subsequent control of the controllable object using the handheld controller, regardless of where the handheld controller points after the capture command is received,

the handheld controller thereafter receiving user input for controlling the controllable object and transmitting control instructions to control the controllable object when the virtual ray no longer intersects the control location of the controllable object.

4. The method of claim 3, further comprising, in response to the virtual ray intersecting control locations of a plurality of the controllable objects identified in the object database:

displaying, by the handheld controller, a list of indicators defined for the plurality of controllable objects; and

while the virtual ray intersects the control locations of the plurality of controllable objects, receiving, by the handheld controller, another capture command, the other capture command directing the handheld controller to capture the list of indicators of the plurality of controllable objects and enabling subsequent control of any of the plurality of controllable objects, regardless of where the handheld controller points after the other capture command is received,

the handheld controller thereafter receiving user input for selecting one of the plurality of controllable objects and for controlling the selected controllable object when the virtual ray no longer intersects the control location of the selected controllable object.

5. The method of claim 3, further comprising:

(A) storing a history of controllable objects previously controlled by the handheld controller;

(B) receiving, by the handheld controller, user input to control a controllable object selected from the history; and

(C) transmitting control instructions from the handheld controller for controlling the selected controllable object,

wherein the acts (B) and (C) are performed without the virtual ray intersecting the control location of the selected controllable object.

6. The method of claim 3, further comprising establishing a control location for a particular controllable object at a user-defined region in the 3-D space, the control location for the particular controllable object providing a location to which the handheld controller is pointed to control the particular controllable object.

7. The method of claim 6, wherein the particular controllable object is electrically connected to an electronic controller, the electronic controller occupying a location in the 3-D space, and wherein the control location for the particular controllable object does not intersect with the location of the electronic controller.

8. The method of claim 6, wherein establishing the control location of the particular controllable object includes the handheld controller executing a training procedure to associate the control location of the particular controllable object in the 3-D space with the particular controllable object.

9. The method of claim 8, wherein executing the training procedure includes:

receiving, by the handheld controller, a user selection of the particular controllable object for training;

acquiring, by the handheld controller, values of the virtual ray when the handheld controller is pointed to the control location of the particular controllable object from multiple different locations in the 3-D space; and

computing the control location of the particular controllable object as a region of intersection in the 3-D space of the acquired values of the virtual ray.

10. The method of claim 8, wherein executing the training procedure includes:

receiving, by the handheld controller, a user selection of the particular controllable object for training; and

with the handheld controller physically located at the control location for the particular controllable object, receiving, by the handheld controller, user input that directs the handheld controller to assign its current location in the 3-D space as the control location of the particular controllable object.

11. The method of claim 8, wherein the control location of the particular controllable object extends across a particular region in space, and wherein executing the training procedure includes:

acquiring, by the handheld controller, locations in the 3-D space of multiple vertices of the particular region in space; and

computing a 3-D region of the control location as a smallest convex shape that contains the multiple vertices.

12. The method of claim 8, wherein the control location of the particular controllable object includes multiple distinct regions at respective locations in the 3-D space, and wherein executing the training procedure includes:

acquiring, by the handheld controller, locations in the 3-D space of each of the multiple distinct regions; and

aggregating the acquired 3-D locations of the distinct regions as the control location of the particular controllable object, such that the control location of the controllable object corresponds to any of the multiple distinct regions in 3-D space.

13. The method of claim 3, wherein the object database stores a control location in the 3-D space of a virtual controllable object, and wherein the method further comprises:

establishing electronic communication between the handheld controller and a 3-D headset configured to render the virtual controllable object as a hologram; and

in response to the handheld controller pointing such that the virtual ray intersects the control location of the virtual controllable object, accepting user input by the handheld controller for controlling the virtual controllable object.

14. The method of claim 13, further comprising:

transmitting values of the virtual ray to the 3-D headset to enable the 3-D headset to render a holographic cursor projected to appear along the virtual ray.

15. The method of claim 14, further comprising:

receiving, by the handheld controller, user input to adjust a depth of the holographic cursor as rendered by the 3-D headset along the virtual ray.

16. The method of claim 14, further comprising:

receiving, by the handheld controller, user input to select a hologram pointed to by the holographic cursor.

17. The method of claim 16, further comprising:

receiving, by the handheld controller, user input to rotate the selected hologram.

18. The method of claim 16, further comprising:

receiving, by the handheld controller, user input to move the selected hologram.

19. The method of claim 16, further comprising:

receiving, by the handheld controller, user input to display a list of options for interacting with the selected hologram.

20. The method of claim 16, further comprising:

receiving, by the handheld controller, user input to generate a slicing plane orthogonal to the virtual ray, the slicing plane enabling the handheld controller to select holograms based on their locations relative to the slicing plane.

21. The method of claim 1, wherein measuring the orientation of the handheld controller is performed by measurement circuitry within the handheld controller 120, and wherein measuring the location of the handheld controller is performed by a server apparatus separate from the handheld controller.

22. A handheld controller for controlling objects, the handheld controller comprising:

a user interface;

a wireless communication interface;

measurement circuitry configured to measure an orientation of the handheld controller in 3-D (three-dimensional) space;

a set of processors; and

memory,

the set of processors and the memory together forming control circuitry constructed and arranged to;

direct the measurement circuitry to produce measurements of orientation of the handheld controller;

direct the wireless interface to transmit wireless signals, including signals that carry the measurements of orientation of the handheld controller, to multiple sensors (i) to enable locations of the handheld controller to be measured based on the wireless signals as received by the multiple sensors and (ii) to generate virtual rays, each virtual ray having an origin at a respective measured location of the handheld controller and a direction based on a respective measured orientation of the handheld controller;

identify a controllable object intersected by one of the virtual rays; and

direct the wireless interface to transmit control instructions for controlling the identified controllable object.

23. The handheld controller of claim 22, further comprising a set of accelerometers, wherein the control circuitry is further constructed and arranged to:

vary a rate at which the wireless interface is directed to transmit wireless packets based on output from the set of accelerometers, the rate increasing for higher levels of acceleration and decreasing for lower levels of acceleration.

24. The handheld controller of claim 23, wherein the handheld controller has an outward appearance that resembles one of (i) a gun and (ii) a sword.