Interactivity with a mixed reality

• US 8,633,946 B2
• Filed: 07/20/2009
• Issued: 01/21/2014
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Cohen, Ronald
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US12/505,726
• Publication Number:
  US 20100017722A1
• Time in Patent Office: 1,646 Days
• Field of Search: 345630-632, 463 37- 42, 715/863
• US Class Current: 345/632
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/18 : Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72427 : for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

Claims

1. A method of interacting with a mixed reality, comprising:

providing a mobile device having a physical interface;

using the mobile device to electronically capture image data relating to one or more real-world objects;

using the image data to identify attributes associated with the one or more real-world objects;

using the attributes to interact with software being operated at least in part remote to the mobile device to obtain a first command set for a first real-world object and a second command set for a second real-world object from a distal database storing a plurality of command sets for multiple, mixed reality objects, where the command sets comprise network API addresses corresponding to commands that control the first and the second real-world objects, respectively;

presenting a virtual interface comprising the first and the second command sets to a user via the physical interface, where at least one of the command sets comprising a non-visible interactive command set is superimposed on an image of a corresponding real-world interface part of the corresponding real-world object and is configured to stay in proper relation to the corresponding real-world object within a mixed reality view; and

allowing execution of at least one command from the non-visible interactive command set upon the user interacting directly with the image of the corresponding real-world interface part by issuing a call over a network from the mobile device to a corresponding network API address of the at least one command targeting the corresponding real-world object.

2. The method of claim 1, wherein the first real-world object is an appliance.

3. The method of claim 1, wherein the first real-world object is physically distinct from the second real-world object.

4. The method of claim 1, wherein the first real-world object is a component of a consumer electronic appliance.

5. The method of claim 1, wherein the first real-world object is an element of an electronic game.

6. The method of claim 5, wherein the attributes indicates the first real-world object has a value in the game.

7. The method of claim 5, wherein the attributes indicates the first real-world object is a goal object of the game.

8. The method of claim 1 wherein, the first real-world object and the second real-world object are parts of a multi-part object.

9. The method of claim 1, wherein the attributes comprises at least one of the following a name, a class, and an environmental characteristic.

10. The method of claim 1, wherein the command is one of an “

on”

command and an “

off”

command.

11. The method of claim 1, wherein the execution of the command results in changing a channel.

12. The method of claim 1, wherein the execution of the command results in playing music.

13. The method of claim 1, wherein the execution of the command inputs data associated with a virtual weapon.

14. The method of claim 1, wherein the execution of the command inputs data associated with virtual travel.

15. The method of claim 1, wherein the image data comprises a still image.

16. The method of claim 1, wherein the image data comprises a video image.

17. The method of claim 1, further comprising using an additional sensor associated with the mobile device to interact with the software.

18. The method of claim of claim 17, wherein the sensor provides information associated with a physical location.

19. The method of claim 18, wherein the sensor provides information associated with at least one of an orientation and an acceleration of the mobile device.

20. The method of claim 1, further comprising superimposing the virtual interface on the physical interface where the virtual interface relates to at least one of the real-world objects.