Virtual/augmented reality input device

• US 10,545,584 B2
• Filed: 05/17/2017
• Issued: 01/28/2020
• Est. Priority Date: 05/17/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Tome, Basheer, Raffle, Hayes S., Li, Chun Yat Frank
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US15/597,776
• Publication Number:
  US 20170336882A1
• Time in Patent Office: 986 Days
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A controller configured to control a pointer in a virtual reality environment includes a multi-axis magnetic field sensor, a multi-axis accelerometer, a gyroscope, a touchpad, and a wireless communications circuit. The controller can also include a processor and a memory storing instructions that when executed by the processor, cause the processor to obtain geomagnetic field data from the multi-axis magnetic field sensor, obtain acceleration data describing a direction and a magnitude of force affecting the controller from the multi-axis accelerometer, and obtain angular velocity data describing a rotational position of the controller from the gyroscope. The processor can communicate movement data to a computing device configured to generate a rendering of the virtual reality environment, the movement data describing an orientation of the controller wherein the movement data is based on at least one of the geomagnetic field data, the acceleration data, or the angular velocity data.

Claims

1. A controller configured to control a pointer in an augmented reality environment, the controller comprising:

a multi-axis accelerometer;

a gyroscope;

a touchpad;

a wireless communications circuit;

a processor; and

a memory storing instructions that when executed by the processor cause the processor to;

receive, from the multi-axis accelerometer, acceleration data describing a direction and a magnitude of force affecting the controller;

receive, from the gyroscope, angular velocity data describing a rotational position of the controller;

generate movement data describing a six degree of freedom position and orientation of the controller, the movement data based on at least one of the acceleration data or the angular velocity data; and

communicate, via the wireless communications circuit, the movement data to a computing device configured to generate a rendering of the augmented reality environment,

wherein the instructions further cause the processor to use a swipe boundary line to recognize gestures performed on the touchpad, the swipe boundary line being based on testing.

2. The controller of claim 1, further comprising a multi-axis magnetic field sensor, wherein the instructions further cause the processor to receive geomagnetic field data from the multi-axis magnetic field sensor, and wherein the movement data is based also on the geomagnetic field data.

3. The controller of claim 1, wherein the instructions further cause the processor to detect touch events on the touchpad and generate touchpad input data responsive to the detected touch events.

4. The controller of claim 3, wherein the instructions further cause the processor to communicate, via the wireless communications circuit, the touchpad input data to the computing device.

5. The controller of claim 4, wherein the instructions further cause the processor to initiate communication of the touchpad input data when the touchpad input data reflects a change in position of the detected touch events satisfies a touchpad threshold position value.

6. The controller of claim 1, further comprising a first input button disposed below the touchpad.

7. The controller of claim 6, further comprising a second input button.

8. The controller of claim 7, wherein the second input button is a menu button, the controller further comprising a third input button configured to perform an action on an object within the augmented reality environment.

9. The controller of claim 1, wherein the instructions further cause the processor to initiate communication of the movement data when at least one of the acceleration data or the angular velocity data satisfies a threshold value.

10. The controller of claim 9, wherein the instructions further cause the processor to adjust the threshold value based on a rate of change of the acceleration data or the angular velocity data.

11. The controller of claim 1, wherein the instructions further cause the processor to define a virtual beam in the augmented reality environment using the movement data, the virtual beam angled relative to a zero line of the controller.

12. The controller of claim 1, further comprising a pair of input buttons configured for adjusting a range value in the augmented reality environment.

13. The controller of claim 1, wherein the movement data includes the acceleration data and the angular velocity data.

14. An augmented reality computing system comprising:

a display screen configured for rendering an augmented reality environment and for being disposed within a head-mounted display (HMD);

a multi-axis accelerometer;

a gyroscope;

a wireless communications circuit;

a processor; and

a memory storing instructions that when executed by the processor cause the processor to;

receive, from the multi-axis accelerometer, acceleration data describing a direction and a magnitude of force affecting the augmented reality computing system;

receive, from the gyroscope, angular velocity data describing a rotational position of the augmented reality computing system;

receive, via the wireless communications circuit, first movement data from a controller for the augmented reality computing system, the first movement data describing an orientation of the controller;

determine second movement data describing an orientation of the augmented reality computing system relative to the controller, the second movement data based on the acceleration data, the angular velocity data and the first movement data;

render on the display screen a pointer based on the first movement data and the second movement data;

render a calibration user interface element at a calibration location on the display screen; and

determine a calibration value based on the first movement data, the second movement data, and the calibration location.

15. The augmented reality computing system of claim 14, wherein the instructions further cause the processor to detect an input event triggered by the controller representing a selection event performed by the controller at the calibration location.

16. The augmented reality computing system of claim 14, wherein the virtual reality computing system is a mobile computing device disposed within a HMD housing.

17. The augmented reality computing system of claim 14, further comprising a multi-axis magnetic field sensor, wherein the instructions further cause the processor to receive geomagnetic field data from the multi-axis magnetic field sensor, and wherein the second movement data is based also on the geomagnetic field data.

18. The augmented reality computing system of claim 17, wherein the instructions further cause the processor to receive, via the wireless communications circuit, touchpad input data describing one or more touch events detected by the controller on a touchpad of the controller.

19. The augmented reality computing system of claim 18, wherein the instructions further cause the processor to modify a virtual object at a position of the pointer based on the received touchpad input data.

20. A method for manipulating a virtual object in a virtual environment, the method comprising:

receiving, by a computing system, first movement data from a controller of the virtual environment, the first movement data describing a first orientation of the controller;

receiving, by the computer system, acceleration data and angular velocity data from the controller;

determining, by the computer system, second movement data based on the acceleration data, the angular velocity data and the first movement data;

rendering on a display screen of the computing system, a pointer at a pointer position within the virtual environment, the pointer position based on the received first movement data and the second movement data;

setting, based on the pointer position, an anchor point on a virtual object rendered at the pointer position;

receiving first input event data from the controller;

modifying the virtual object based on the received first input event data;

render a calibration user interface element at a calibration location on the display screen; and

determine a calibration value based on the first movement data, the second movement data, and the calibration location.

21. The method of claim 20, further comprising receiving second input event data from the controller.

22. The method of claim 21 wherein setting the anchor point is done responsive to receiving the second input event data.

23. The method of claim 22, wherein the second input event data corresponds to depression of an input button of the controller.

24. The method of claim 23, wherein the input button of the controller is disposed under a touchpad of the controller.

25. The method of claim 21, further comprising receiving third input event data from the controller.

26. The method of claim 25, wherein the third input event data includes second movement data describing a second orientation of the controller.

27. The method of claim 20, wherein the first input event data includes touch input event data corresponding to a touch event captured by a touchpad of the controller.

28. The method of claim 27, wherein the touch input event data describes at least one selected from the group consisting of a swipe event, a touch-and-drag event, and a zoom event.

29. The method of claim 27, wherein the first movement data describes the first orientation of the controller relative to a magnetic field affecting the controller and the computing system.