VIRTUAL REALITY AND AUGMENTED REALITY CONTROL WITH MOBILE DEVICES

• US 20160232713A1
• Filed: 06/20/2015
• Published: 08/11/2016
• Est. Priority Date: 02/10/2015
• Status: Abandoned Application

• Current Assignee: Fangwei Lee
• Original Assignee: Fangwei Lee
• Inventors: Lee, Fangwei
• Application Number:
  US14/745,414
• Publication Number:
  US 20160232713A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems and methods for converting a physical input from a user into an action in a virtual reality or augmented reality environment are disclosed. A particular embodiment includes: receiving image data from an image capturing subsystem, the image data including at least a portion of the at least one reference image, the at least one reference image representing a portion of a set of reference data; receiving position and orientation data of the image capturing subsystem, the position and orientation data representing another portion of the reference data; measuring, by use of a data processor, a change in spatial relation relative to the reference data when a physical input is applied to a tracking subsystem; and generating an action in a virtual world, the action corresponding to the measured change in spatial relation.

Claims

1. An apparatus comprising:

a processor;

an image capturing subsystem;

a tracking subsystem including at least one reference image; and

a data conversion subsystem in data communication with the processor and the image capturing subsystem, the data conversion subsystem to;

receive image data from the image capturing subsystem, the image data including at least a portion of the at least one reference image, the at least one reference image representing a portion of a set of reference data;

receive position and orientation data of the image capturing subsystem, the position and orientation data representing another portion of the reference data;

measure a change in spatial relation relative to the reference data when a physical input is applied to the tracking subsystem; and

generate an action in a virtual world, the action corresponding to the measured change in spatial relation.

2. The apparatus of claim 1 wherein the data conversion subsystem being further configured to determine whether the change in spatial relation falls within a predetermined threshold range and to generate the action in the virtual world, if the change in spatial relation falls within the predetermined threshold range.

3. The apparatus of claim 1 wherein the tracking subsystem includes one or more markers, the data conversion subsystem being further configured to measure the change in spatial relation relative to the one or more markers when a physical input is applied to the tracking subsystem.

4. The apparatus of claim 1 wherein the physical input applied to the tracking subsystem is a rotation or a translation of a portion of the tracking subsystem.

5. The apparatus of claim 1 wherein the physical input applied to the tracking subsystem is a rotation or a translation of the tracking subsystem itself.

6. The apparatus of claim 1 wherein the change in spatial relation is measured in two-dimensional space or three-dimensional space.

7. The apparatus of claim 1 wherein the image capturing subsystem includes a plurality of image capturing devices and the data conversion subsystem being further configured to receive image data from each of the plurality of image capturing devices and to synchronize the image data received from the plurality of image capturing devices.

8. The apparatus of claim 1 wherein the action in the virtual world is the movement or manipulation of a virtual object or the manipulation of a virtual user interface.

9. The apparatus of claim 1 wherein the action in the virtual world corresponds to control of a real world device.

10. The apparatus of claim 1 wherein the reference data includes accelerometer data from a user device.

11. A method comprising:

receiving image data from an image capturing subsystem, the image data including at least a portion of at least one reference image, the at least one reference image representing a portion of a set of reference data;

receiving position and orientation data of the image capturing subsystem, the position and orientation data representing another portion of the reference data;

measuring, by use of a data processor, a change in spatial relation relative to the reference data when a physical input is applied to the tracking subsystem; and

generating an action in a virtual world, the action corresponding to the measured change in spatial relation.

12. The method of claim 11 including determining whether the change in spatial relation falls within a predetermined threshold range and generating the action in the virtual world, if the change in spatial relation falls within the predetermined threshold range.

13. The method of claim 11 including measuring a change in spatial relation relative to one or more markers when a physical input is applied to the tracking subsystem.

14. The method of claim 11 wherein the physical input applied to the tracking subsystem is a rotation or a translation of the tracking subsystem itself or a portion of the tracking subsystem.

15. The method of claim 11 wherein the reference data includes accelerometer data from a user device.

16. The method of claim 11 wherein the change in spatial relation is measured in two-dimensional space or three-dimensional space.

17. The method of claim 11 including receiving image data from each of a plurality of image capturing devices and to synchronize the image data received from the plurality of image capturing devices.

18. The method of claim 11 wherein the action in the virtual world corresponds to the movement or manipulation of a virtual object or the manipulation of a virtual user interface or control of a real world device.

19. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from the image capturing subsystem, the image data including at least a portion of the at least one reference image, the at least one reference image representing a portion of a set of reference data;

receive position and orientation data of the image capturing subsystem, the position and orientation data representing another portion of the reference data;

measure a change in spatial relation relative to the reference data when a physical input is applied to the tracking subsystem; and

generate an action in a virtual world, the action corresponding to the measured change in spatial relation.

20. The instructions embodied in the machine-useable storage medium of claim 19 being further configured to determine whether the change in spatial relation falls within a predetermined threshold range and generating the action in the virtual world, if the change in spatial relation falls within the predetermined threshold range.