Systems and methods for augmenting an appearance of a hilt to simulate a bladed weapon

• US 9,846,971 B2
• Filed: 01/19/2016
• Issued: 12/19/2017
• Est. Priority Date: 01/19/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Goslin, Michael P., Ryan, Matthew, Olson, Joseph Logan
• Primary Examiner(s): Demeter, Hilina K
• Application Number:
  US15/001,160
• Publication Number:
  US 20170206709A1
• Time in Patent Office: 700 Days
• Field of Search: None
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

This disclosure relates to systems and methods for augmenting an appearance of a hilt to simulate a bladed weapon. A hilt may be augmented with a blade of a bladed weapon by detecting a landmark associated with the hilt, determining a position and/or an orientation of the hilt, determining an overlay image comprising the blade of the bladed weapon, wherein the blade is placed within the overlay image according to the determined position and/or the determined orientation of the hilt, and displaying the overlay image so that the blade of the bladed weapon appears to be attached to the hilt.

Claims

1. A system for augmenting an appearance of a hilt to simulate a bladed weapon, the system comprising:

a display configured to display an overlay image;

an image sensor configured to generate visual output signals conveying visual information within a field of view of the image sensor;

a housing of the hilt;

a landmark carried by the housing, the landmark indicating a reference point for the hilt that facilitates determination of a position and orientation of the housing;

a first inertial measurement unit carried by the display, the first inertial measurement unit separate from the image sensor, the first inertial measurement unit configured to generate first motion and/or orientation output signals conveying motion and/or orientation information of the display such that the first motion and/or orientation output signals are separate and independent from the visual output signals, wherein the first motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor;

a second inertial measurement unit carried by the housing, the second inertial measurement unit separate from the image sensor, the second inertial measurement unit configured to generate second motion and/or orientation output signals conveying motion and/or orientation information of the housing such that the second motion and/or orientation output signals are separate and independent from the visual output signals, wherein the second motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor; and

one or more processors configured by machine readable instructions to;

when the landmark is within the field of view of the image sensor;

detect the landmark based on the visual output signals;

determine a position and/or an orientation of the landmark; and

determine a position and/or an orientation of the housing based on the determined position and/or the determined orientation of the landmark;

when the landmark is not within the field of view of the image sensor, determine the position and/or the orientation of the housing with respect to the display based on the first motion and/or orientation output signals and the second motion and/or orientation output signals;

determine the overlay image comprising a blade of the bladed weapon, wherein the blade is placed within the overlay image according to the determined position and/or the determined orientation of the housing such that the blade is placed within the overlay image based on the visual output signals when the landmark is within the field of view of the image sensor and the blade is placed within the overlay image based on the first motion and/or orientation output signals and the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor; and

effectuate displaying of the overlay image on the display so that the blade appears to be attached to the hilt.

2. The system of claim 1, wherein the one or more processors are further configured to:

detect an object based on the visual output signals;

determine when the blade of the bladed weapon touches the object;

determine the overlay image to further include a visual effect based on the blade of the bladed weapon touching the object.

3. The system of claim 1, further comprising a haptic generator, wherein the one or more processors are further configured to effectuate operation of the haptic generator in response to a movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals.

4. The system of claim 1, further comprising a speaker, wherein the one or more processors are further configured to effectuate operation of the speaker in response to a movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals to simulate one or more sounds of the bladed weapon such that the one or more sounds of the bladed weapon are simulated based on the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor.

5. The system of claim 1, wherein the landmark includes a light emitting diode.

6. The system of claim 1, wherein the first inertial measurement unit and the second inertial measurement unit include one or more of an accelerometer, a gyroscope, and/or a magnetometer used to generate the first motion and/or orientation output signals and the second motion and/or orientation output signals.

7. The system of claim 1, wherein the one or more processors are further configured to select the blade based on a user input received through an input device.

8. The system of claim 1, wherein the one or more processors are further configured to:

identify a given type of the landmark, the given type of the landmark associated with a given type of blade; and

select the blade based on the given type of the landmark, the blade being of the given type of blade.

9. The system of claim 1, wherein the one or more processors are further configured to select the blade based on remotely received information.

10. The system of claim 1, wherein the display comprises one or more of a head-mounted display, an optical head-mounted display, a see-through display, an optical see-through display, a video see-through display, a visor, eyeglasses, sunglasses, a smartphone, a tablet, a mobile device, and/or a projector.

11. The system of claim 1, wherein the image sensor and the one or more processors are carried on the display, and wherein the field of view of the image sensor is a function of the position and the orientation of the display.

12. A method for augmenting an appearance of a hilt to simulate a bladed weapon, the method comprising:

generating visual output signals conveying visual information within a field of view of an image sensor;

generating first motion and/or orientation output signals conveying motion and/or orientation information of a display, the first motion and/or orientation output signals generated by a first inertial measurement unit carried by the display, the first inertial measurement unit separate from the image sensor, wherein the first motion and/or orientation output signals are separate and independent from the visual output signals and the first motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor;

generating second motion and/or orientation output signals conveying motion and/or orientation information of a housing of the hilt, the second motion and/or orientation output signals generated by a second inertial measurement unit carried by the housing, the second inertial measurement unit separate from the image sensor, wherein the second motion and/or orientation output signals are separate and independent from the visual output signals and the second motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor;

when a landmark is within the field of view of the image sensor;

detecting the landmark based on the visual output signals, wherein the landmark is carried by the housing and indicates a reference point for the hilt that facilitates determination of a position and orientation of the housing;

determining a position and/or an orientation of the landmark; and

determining a position and/or an orientation of the housing based on the determined position and/or the determined orientation of the landmark;

when the landmark is not within the field of view of the image sensor, determining the position and/or the orientation of the housing with respect to the display based on the first motion and/or orientation output signals and the second motion and/or orientation output signals;

determining an overlay image comprising a blade of the bladed weapon, wherein the blade is placed within the overlay image according to the determined position and/or the determined orientation of the housing such that the blade is placed within the overlay image based on the visual output signals when the landmark is within the field of view of the image sensor and the blade is placed within the overlay image based on the first motion and/or orientation output signals and the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor; and

effectuating displaying of the overlay image on the display so that the blade appears to be attached to the hilt.

13. The method of claim 12, further comprising:

detecting an object based on the visual output signals;

determining when the blade of the bladed weapon touches the object;

determining the overlay image to further include a visual effect based on the blade of the bladed weapon touching the object.

14. The method of claim 12, further comprising effectuating operation of a haptic generator in response to a movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals.

15. The method of claim 12, further comprising effectuating operation of a speaker in response to a movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals to simulate one or more sounds of the bladed weapon such that the one or more sounds of the bladed weapon are simulated based on the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor.

16. The method of claim 12, wherein the landmark includes a light emitting diode.

17. The method of claim 12, further comprising selecting the blade based on a user input received through an input device.

18. The method of claim 12, further comprising:

identifying a given type of the landmark, the given type of the landmark associated with a given type of blade; and

selecting the blade based on the given type of the landmark, the blade being of the given type of blade.

19. The method of claim 12, further comprising selecting the blade based on remotely received information.

20. A system for augmenting an appearance of a hilt to simulate a bladed weapon, the system comprising:

a display configured to display an overlay image;

a haptic generator;

a speaker;

an image sensor configured to generate visual output signals conveying visual information within a field of view of the image sensor;

a housing of the hilt;

a landmark carried by the housing, the landmark indicating a reference point for the hilt that facilitates determination of a position and orientation of the housing;

a first inertial measurement unit carried by the display, the first inertial measurement unit or separate from the image sensor, the first inertial measurement unit configured to generate first motion and/or orientation output signals conveying motion and/or orientation information of the display such that the first motion and/or orientation output signals are separate and independent from the visual output signals, wherein the first motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor;

a second inertial measurement unit carried by the housing, the second inertial measurement unit separate from the image sensor, the second inertial measurement unit configured to generate second motion and/or orientation output signals conveying motion and/or orientation information of the housing such that the second motion and/or orientation output signals are separate and independent from the visual output signals, wherein the second motion and/or orientation output signals facilitate determination of the position and orientation of the housing when the landmark is not within the field of view of the image sensor; and

one or more processors configured by machine readable instructions to;

when the landmark is within the field of view of the image sensor;

detect the landmark based on the visual output signals;

determine a position and/or an orientation of the landmark; and

determine a position and/or an orientation of the housing based on the determined position and/or the determined orientation of the landmark;

when the landmark is not within the field of view of the image sensor, determine the position and/or the orientation of the housing with respect to the display based on the first motion and/or orientation output signals and the second motion and/or orientation output signals;

determine the overlay image comprising a blade of the bladed weapon, wherein the blade is placed within the overlay image according to the determined position and/or the determined orientation of the housing such that the blade is placed within the overlay image based on the visual output signals when the landmark is within the field of view of the image sensor and the blade is placed within the overlay image based on the first motion and/or orientation output signals and the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor;

effectuate displaying of the overlay image on the display so that the blade appears to be attached to the hilt;

effectuate operation of the haptic generator in response to a movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals

effectuate operation of the speaker in response to the movement of the hilt based on one or more of the visual output signals and/or the second motion and/or orientation output signals to simulate one or more sounds of the bladed weapon such that the one or more sounds of the bladed weapon are simulated based on the second motion and/or orientation output signals when the landmark is not within the field of view of the image sensor;

detect an object based on the visual output signals;

determine when the blade of the bladed weapon touches the object; and

determine the overlay image to further include a visual effect based on the blade of the bladed weapon touching the object.