Interactivity With A Mixed Reality

• US 20140059437A1
• Filed: 11/03/2013
• Published: 02/27/2014
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Cohen, Ronald H.
• Granted Patent: US 10,463,961 B2
• Time in Patent Office: Days
• US Class Current: 715/727
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/18 : Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72427 : for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

Claims

1. A method for enabling a user of a mobile computing device to use audio sensor data, the method comprising the steps of:

obtaining audio sensor data related to an object via an audio sensor of a mobile device;

identifying at least one attribute associated with the object by using the audio sensor data obtained from the audio sensor of the mobile device via at least one of the mobile device and a server communicatively coupled to the mobile device;

using the at least one attribute to interact with software that determines an appropriate command set as a function of the at least one attribute; and

enabling the mobile device to receive a user selection of a command from the command set, wherein the command is executable by the mobile device to interact with a network address associated with the command.

2. The method of claim 1, wherein the step of identifying at least one attribute comprises identifying at least one attribute associated with the object by using the audio sensor data and time information.

3. The method of claim 1, wherein the mobile device comprises a cell phone.

4. The method of claim 1, wherein the audio sensor data comprises captured audio.

5. The method of claim 1, wherein the command comprises a URL.

6. The method of claim 1, wherein the at least one attribute comprises a plurality of attributes.

7. The method of claim 1, wherein the step of using the at least one attribute to interact with software that determines the appropriate command set comprises consulting a database storing a plurality of command sets.

8. The method of claim 1, wherein the object comprises a media object.

9. The method of claim 8, wherein the media object comprises electronically provided entertainment.

10. The method of claim 9, wherein the electronically provided entertainment comprises television screen content.

11. The method of claim 9, wherein the electronically provided entertainment comprises computer screen content.

12. A system for using audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets;

wherein the mobile device comprises an audio sensor configured to capture audio sensor data related to a object;

wherein at least one of the server and the mobile device is configured to identify at least one attribute associated with the object by using the audio sensor data obtained by the audio sensor of the mobile device;

wherein the server is configured to use the at least one attribute to interact with software that determines an appropriate command set from the database as a function of the at least one attribute; and

wherein the mobile device is configured to receive a user selection of a command from the command set, and wherein the command is executable by the mobile device to interact with a network address associated with the command.

13. The system of claim 12, wherein at least one of the server and the mobile device is further configured to identify the at least one attribute by using the audio sensor data and time information.

14. The system of claim 12, wherein the mobile device comprises a cell phone.

15. The system of claim 12, wherein the audio sensor data comprises captured audio.

16. The system of claim 12, wherein the command comprises a URL.

17. The system of claim 12, wherein the at least one attribute comprises a plurality of attributes.

18. The system of claim 12, wherein the object comprises a media object.

19. The system of claim 18, wherein the media object comprises electronically provided entertainment.

20. The system of claim 19, wherein the electronically provided entertainment comprises television screen content.

21. The system of claim 19, wherein the electronically provided entertainment comprises computer screen content.

22. A system for using audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets;

wherein the mobile device comprises an audio sensor configured to capture audio sensor data related to a object;

wherein at least one of the server and the mobile device is configured to identify at least one attribute associated with the object by using the audio sensor data obtained by the audio sensor of the mobile device and at least one of GPS sensor data and location information;

wherein the server is configured to use the at least one attribute identified by using audio sensor data and at least one of GPS sensor data and location information to interact with software that determines an appropriate command set from the database as a function of the at least one attribute; and

wherein the mobile device is configured to receive a user selection of a command from the command set, and wherein the command is executable by the mobile device to interact with a network address associated with the command.

23. A system for using audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets;

wherein the mobile device comprises an audio sensor configured to capture audio sensor data related to a object;

wherein at least one of the server and the mobile device is configured to identify at least one attribute associated with the object by using the audio sensor data obtained by the audio sensor of the mobile device and GPS sensor data;

wherein the server is configured to use the at least one attribute to interact with software that determines an appropriate command set from the database as a function of the at least one attribute; and

wherein the mobile device is configured to receive a user selection of a command from the command set, and wherein the command is related to a time and executable by the mobile device to interact with a network address associated with the command.

24. A system for using audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets;

wherein the mobile device comprises an audio sensor configured to capture audio sensor data related to a object;

wherein at least one of the server and the mobile device is configured to identify at least one attribute associated with the object by using the audio sensor data obtained by the audio sensor of the mobile device;

wherein the server is configured to use the at least one attribute to interact with software that determines an appropriate command set from the database as a function of the at least one attribute; and

wherein the mobile device is configured to receive a user selection of a command from the command set, and wherein the command is related to a time executable by the mobile device to interact with a network address associated with the command.