Interactivity With A Mixed Reality

• US 20140055493A1
• Filed: 11/04/2013
• Published: 02/27/2014
• Est. Priority Date: 08/29/2005
• Status: Abandoned Application

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Cohen, Ronald H.
• Application Number:
  US14/070,591
• Publication Number:
  US 20140055493A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/18 : Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72427 : for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 : including functional features of a camera
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

Claims

1. A method of interacting with a mixed reality, the method comprising the steps of:

obtaining user identity information from a mobile device via a wireless network, the user identity information corresponding to an identity of a user of the mobile device;

capturing image data of an object via a camera of the mobile device, wherein the image data comprises a video image;

deriving at least one attribute associated with the image data of the object from the video image via decomposition of the video image into quantifiable salient parameters;

using the at least one attribute and geographic information related to the image data to consult a database storing a plurality of command sets for multiple, mixed reality objects; and

obtaining at least one of the mixed reality objects via the mobile device including data necessary for rendering a virtual object on a display of the mobile device, wherein the at least one mixed reality object is associated with an identity of a user of the mobile device.

2. The method of claim 1, further comprising the step of superimposing the virtual object over an image of the object.

3. The method of claim 1, wherein the user identity information comprises personal information of the user.

4. The method of claim 3, wherein the personal information comprises at least one of a password and an image of the user.

5. The method of claim 1, wherein the at least one of the quantifiable salient parameters represents at least one of a shape, a color, and a symbol.

6. The method of claim 1, further comprising the step of applying a filter to the image data to at least partially remove a distortion.

7. The method of claim 6, wherein the filter is applied using distributed processing between the mobile device and a server.

8. The method of claim 1, wherein the video image comprises a streamed video.

9. The method of claim 1, wherein the video image comprises a recorded video.

10. The method of claim 1, wherein user identity information comprises at least one of a password, a user identity, and a user location.

11. A mixed reality system, comprising:

a mobile device configured to capture image data of an object via a camera of the mobile device, wherein the image data comprises a video image;

a database storing a plurality of command sets for multiple, mixed reality objects;

an engine configured to receive user identity information from a mobile device via a wireless network, the user identity information corresponding to an identity of a user of the mobile device;

an image matching engine configured to derive at least one attribute associated with the image data of the object from the video image via decomposition of the video image into quantifiable salient parameters;

a virtual world server configured to use the at least one attribute and geographic information related to the image data to consult the database and supply at least one of the mixed reality objects to the mobile device including data necessary for rendering a virtual object on a display of the mobile device, wherein the at least one mixed reality object is associated with an identity of a user of the mobile device.

12. The system of claim 11, wherein the virtual object is superimposed over an image of the object.

13. The system of claim 11, wherein the user identity information comprises personal information of the user.

14. The system of claim 13, wherein the personal information comprises at least one of a password and an image of the user.

15. The system of claim 11, wherein the at least one of the quantifiable salient parameters represents at least one of a shape, a color, and a symbol.

16. The system of claim 15, wherein the image matching engine is further configured to apply a filter to the image data.

17. The system of claim 16, wherein the image matching engine is configured to apply the filter to the image data using distributed processing between the mobile device and a server.

18. The system of claim 11, wherein the video image comprises a streamed video.

19. The system of claim 11, wherein the video image comprises a recorded video.

20. The system of claim 11, wherein user identity information comprises at least one of a password, a user identity, and a user location.

21. The system of claim 11, wherein the object comprises a license plate.

22. The system of claim 11, wherein the object comprise a electronically provided programming.

23. A mixed reality system, comprising:

a mobile device configured to capture image data of an object via a camera of the mobile device, wherein the image data comprises a video image;

a database storing a plurality of command sets for multiple, mixed reality objects;

an engine configured to receive user identity information from a mobile device via a wireless network, the user identity information corresponding to an identity of a user of the mobile device;

an image matching engine configured to derive at least one attribute associated with the image data of the object from the video image via decomposition of the video image into quantifiable salient parameters;

a virtual world server configured to use the at least one attribute, and at least one of location and position information to consult the database and supply at least one of the mixed reality objects to the mobile device including data necessary for rendering a virtual object on a display of the mobile device, wherein the at least one mixed reality object is associated with an identity of a user of the mobile device.

24. A mixed reality system, comprising:

a mobile device configured to capture image data of an object via a camera of the mobile device, wherein the image data comprises a video image;

a database storing a plurality of command sets for multiple, mixed reality objects;

an engine configured to receive user identity information from a mobile device via a wireless network, the user identity information corresponding to an identity of a user of the mobile device;

an image matching engine configured to derive at least one attribute associated with the image data of the object from the video image via decomposition of the video image into quantifiable salient parameters;

a virtual world server configured to use the at least one attribute and time data to consult the database and supply at least one of the mixed reality objects to the mobile device including data necessary for rendering a virtual object on a display of the mobile device, wherein the at least one mixed reality object is associated with an identity of a user of the mobile device.

25. A mixed reality system, comprising:

a mobile device configured to capture image data of an object via a camera of the mobile device, wherein the image data comprises a video image;

a database storing a plurality of command sets for multiple, mixed reality objects;

an engine configured to receive user identity information from a mobile device via a wireless network, the user identity information corresponding to an identity of a user of the mobile device;

an image matching engine configured to derive at least one attribute associated with the image data of the object from the video image via decomposition of the video image into quantifiable salient parameters;

a virtual world server configured to use the at least one attribute, time data and at least one of location and position information to consult the database and supply at least one of the mixed reality objects to the mobile device including data necessary for rendering a virtual object on a display of the mobile device, wherein the at least one mixed reality object is associated with an identity of a user of the mobile device.