REMOTE CONTROLLED VEHICLE WITH AUGMENTED REALITY OVERLAY

• US 20180028931A1
• Filed: 09/01/2017
• Published: 02/01/2018
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: MONKEYmedia, Inc.
• Original Assignee: MONKEYmedia, Inc.
• Inventors: Bear, Eric Justin Gould, Strickland, Rachel M., McKee, Jim
• Granted Patent: US 9,919,233 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63H 2200/00 Computerized interactive toys, e.g. dolls
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: A63H 33/22 Optical, colour, or shadow toys puzzles with optical elements A63F9/0613; kaleidoscopes G02B27/08
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 1/1688 the I/O peripheral being integrated loudspeakers
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/046 : by electromagnetic means
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 15/20 : Perspective computation
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 21/41407 : embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 : involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 : for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/47 : End-user applications
• CPC Class Codes: H04N 21/8106 : involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 5/445 : for displaying additional information H04N5/50 takes precedence
• CPC Class Codes: H04N 7/142 : Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/157 : defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

In some embodiments, extemporaneous control of remote objects can be made more natural using the invention, enabling a participant to pivot, tip and aim a head-mounted display apparatus to control a remote-controlled toy or full-sized vehicle, for example, hands-free. If the vehicle is outfitted with a camera, then the participant may see the remote location from first-person proprioceptive perspective.

Claims

1. One or more non-transitory computer readable storage media comprising instructions that when executed by a computer having one or more sensors and associated with a head-mounted display apparatus and a remote controllable vehicle having an on-board camera, are capable of causing the computer to.

a. generate a signal to control the movement of the remote vehicle in the physical world;

b. generate a signal to establish an orientation of the remote vehicle'"'"'s camera;

c. generate a signal to update the location of the remote vehicle in response to data from at least one of the sensors of the head-mounted apparatus relating to a first axis of movement of the head-mounted display apparatus;

d. generate a signal to update the orientation of the remote vehicle'"'"'s camera in response to data from at least one of the sensors of the head-mounted apparatus relating to a second axis of movement of the head-mounted display apparatus independent from the first axis of movement of the head-mounted display apparatus.

e. generate a signal to display video of the physical world captured by the remote vehicle'"'"'s camera; and

f. generate a signal to overlay one or more augmented visual elements over a display of the video of the physical world captured by the remote vehicle'"'"'s camera, wherein movement of the one or more augmented visual elements tracks with movement of the physical world captured by the remote vehicle'"'"'s camera.

2. The computer readable storage media of claim 1, wherein at least one of the augmented visual elements comprises graphical and/or textual indicia relating to one or more locations in the physical world.

3. The computer readable storage media of claim 2, wherein at least one of the graphical and/or textual indicia comprises a name of a location or street in the physical world.

4. The computer readable storage media of claim 2, wherein at least one of the graphical and/or textual indicia comprises a rendered simulation of an architectural structure.

5. The computer readable storage media of claim 1, wherein at least one of the augmented visual elements comprises fictional content.

6. The computer readable storage media of claim 5, wherein the fictional content comprises one or more game elements.

7. The computer readable storage media of claim 6, wherein at least one of the game elements is animated.

8. The computer readable storage media of claim 1, wherein at least one of the augmented visual elements comprises video.

9. The computer readable storage media of claim 1, wherein at least a portion of one of the augmented visual elements obscures at least a portion of an object in the physical world captured by the remote vehicle'"'"'s camera.

10. The computer readable storage media of claim 1, wherein at least a portion of one of the augmented visual elements is obscured by at least a portion of an object in the physical world captured by the remote vehicle'"'"'s camera.

11. A method performed by a computer having one or more sensors and associated with a head-mounted display apparatus and a remote controllable vehicle having an on-board camera, comprising the acts of:

a. generating a signal to control the movement of the remote vehicle in the physical world;

b. generating a signal to establish an orientation of the remote vehicle'"'"'s camera;

c. generating a signal to update the location of the remote vehicle in response to data from at least one of the sensors of the head-mounted display apparatus relating to a first axis of movement of the head-mounted display apparatus;

d. generating a signal to update the orientation of the remote vehicle'"'"'s camera in response to data from at least one of the sensors of the head-mounted display apparatus relating to a second axis of movement of the head-mounted display apparatus independent from the first axis of movement of the head-mounted display apparatus;

e. generating a signal to display video of the physical world captured by the remote vehicle'"'"'s camera; and

f. generating a signal to overlay one or more augmented visual elements over a display of the video of the physical world captured by the remote vehicle'"'"'s camera, wherein movement of the one or more augmented visual elements tracks with movement of the physical world captured by the remote vehicle'"'"'s camera.

12. The method of claim 11, wherein at least one of the augmented visual elements comprises graphical and/or textual indicia relating to one or more locations in the physical world.

13. The method of claim 12, wherein at least one of the graphical and/or textual indicia comprises a name of a location or street in the physical world.

14. The method of claim 12, wherein at least one of the graphical and/or textual indicia comprises a rendered simulation of an architectural structure.

15. The method of claim 11, wherein at least one of the augmented visual elements comprises fictional content.

16. The method of claim 15, wherein the fictional content comprises one or more game elements.

17. The method of claim 16, wherein at least one of the game elements is animated.

18. The method of claim 11, wherein at least one of the augmented visual elements comprises video.

19. The method of claim 11, wherein at least a portion of one of the augmented visual elements obscures at least a portion of an object in the physical world captured by the remote vehicle'"'"'s camera.

20. The method of claim 11, wherein at least a portion of one of the augmented visual elements is obscured by at least a portion of an object in the physical world captured by the remote vehicle'"'"'s camera.