VIRTUAL REALITY AND AUGMENTED REALITY CONTROL WITH MOBILE DEVICES

• US 20160232715A1
• Filed: 03/31/2016
• Published: 08/11/2016
• Est. Priority Date: 02/10/2015
• Status: Abandoned Application

• Current Assignee: Fangwei Lee
• Original Assignee: Fangwei Lee
• Inventors: Lee, Fangwei
• Application Number:
  US15/087,131
• Publication Number:
  US 20160232715A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

Systems and methods for generating an action in a virtual reality or augmented reality environment based on position or movement of a mobile device in the real world are disclosed. A particular embodiment includes: displaying an optical marker on a display device of a motion-tracking controller; receiving a set of reference data from the motion-tracking controller; receiving captured marker image data from an image capturing subsystem of an eyewear system; comparing reference marker image data with the captured marker image data, the reference marker image data corresponding to the optical marker; generating a transformation matrix using the reference marker image data and the captured marker image data, the transformation matrix corresponding to a position and orientation of the motion-tracking controller relative to the eyewear system; and generating an action in a virtual world, the action corresponding to the transformation matrix.

Claims

1. A system comprising:

a motion-tracking controller configured to be held by a user, the motion-tracking controller including a display device for displaying an optical marker; and

an eyewear system configured to be worn by the user, the eyewear system including;

a data processor;

an image capturing subsystem;

a tracking subsystem in data communication with the data processor and the image capturing subsystem, the tracking subsystem including reference marker image data corresponding to the optical marker, the tracking subsystem being configured to;

receive captured marker image data from the image capturing subsystem;

compare the reference marker image data with the captured marker image data;

generate a transformation matrix using the reference marker image data and the captured marker image data, the transformation matrix corresponding to a position and orientation of the motion-tracking controller relative to the eyewear system; and

a virtual rendering subsystem in data communication with the data processor and the tracking subsystem, the virtual rendering subsystem being configured to generate an action in a virtual world, the action corresponding to the transformation matrix.

2. The system of claim 1 wherein the motion-tracking controller is a device of a type from the group consisting of:

a hand-held mobile device, a mobile phone, and a smartphone.

3. The system of claim 1 wherein the motion-tracking controller includes an accelerometer, a gyroscope, and a compass.

4. The system of claim 1 wherein the motion-tracking controller being further configured to determine its absolute orientation and position in the real world.

5. The system of claim 1 wherein the tracking subsystem being further configured to scan a field of view of the image capturing subsystem for a unique pattern corresponding to the optical marker.

6. The system of claim 1 wherein the tracking subsystem being further configured to generate a second transformation matrix using the reference marker image data and the captured marker image data, the second transformation matrix corresponding to a position and orientation of the eyewear system.

7. The system of claim 1 wherein the tracking subsystem being further configured to recognize the reference marker image data as a spatial frequency, a light wave frequency, or a temporal frequency.

8. The system of claim 1 wherein the tracking subsystem being further configured to estimate a position of the motion-tracking controller if the motion-tracking controller is out of a field of view of the image capturing subsystem.

9. The system of claim 1 wherein the action in the virtual world is the movement or manipulation of a virtual object or the manipulation of a virtual user interface.

10. The system of claim 1 wherein the action in the virtual world corresponds to control of a real world device.

11. A method comprising:

displaying an optical marker on a display device of a motion-tracking controller;

receiving captured marker image data from an image capturing subsystem of an eyewear system;

comparing reference marker image data with the captured marker image data, the reference marker image data corresponding to the optical marker;

generating a transformation matrix using the reference marker image data and the captured marker image data, the transformation matrix corresponding to a position and orientation of the motion-tracking controller relative to the eyewear system; and

generating an action in a virtual world, the action corresponding to the transformation matrix.

12. The method of claim 11 wherein the motion-tracking controller is a device of a type from the group consisting of:

a hand-held mobile device, a mobile phone, and a smartphone.

13. The method of claim 11 wherein the motion-tracking controller includes an accelerometer, a gyroscope, and a compass.

14. The method of claim 11 including determining the absolute orientation and position of the motion-tracking controller in the real world.

15. The method of claim 11 including scanning a field of view of the image capturing subsystem for a unique pattern corresponding to the optical marker.

16. The method of claim 11 including generating a second transformation matrix using the reference marker image data and the captured marker image data, the second transformation matrix corresponding to a position and orientation of the eyewear system.

17. The method of claim 11 including recognizing the reference marker image data as a spatial frequency, a light wave frequency, or a temporal frequency.

18. The method of claim 11 including estimating a position of the motion-tracking controller if the motion-tracking controller is out of a field of view of the image capturing subsystem.

19. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive captured marker image data from an image capturing subsystem of an eyewear system;

compare reference marker image data with the captured marker image data, the reference marker image data corresponding to an optical marker displayed on a display device of a motion-tracking controller;

generate a transformation matrix using the reference marker image data and the captured marker image data, the transformation matrix corresponding to a position and orientation of the motion-tracking controller relative to the eyewear system; and

generate an action in a virtual world, the action corresponding to the transformation matrix.

20. The instructions embodied in the machine-useable storage medium of claim 19 wherein the action in the virtual world is the movement or manipulation of a virtual object or the manipulation of a virtual user interface.