Interactivity With A Mixed Reality

• US 20140055492A1
• Filed: 11/01/2013
• Published: 02/27/2014
• Est. Priority Date: 08/29/2005
• Status: Abandoned Application

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Cohen, Ronald H.
• Application Number:
  US14/070,028
• Publication Number:
  US 20140055492A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/18 : Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72427 : for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

Claims

1. A mixed reality system for multiple users, comprising:

an image processing server configured to receive image data related to an object from each of first and second mobile devices over a wireless network, and derive at least one attribute associated with the object from the image data related to the object received from the first and second mobile devices wherein the first and second mobile devices are in physical proximity to each other wherein the first and second mobile devices are equipped with a video camera configured to capture image data related to the object and wherein the first server is coupled with an object database configured to store information related to a plurality of objects; and

a mixed reality server coupled to the image processing server and configured to use the at least one attribute to consult a database storing a plurality of command sets for multiple, mixed reality objects, wherein the mixed reality server is configured to return information associated with at least one of the mixed reality objects to the first mobile device including data necessary for rendering a virtual object on a first display of the first mobile device, and wherein the mixed reality server is configured to return information associated with at least one of the mixed reality objects to the second mobile device including data necessary for rendering a virtual object on a second display of the second mobile device.

2. The system of claim 1, wherein at least one of the first and second mobile devices comprises a global positioning system (GPS) sensor configured to provide information associated with a physical location.

3. The system of claim 1, further comprising a first sensor configured to provide information associated with an orientation.

4. The system of claim 3, wherein the orientation comprises an orientation of the mobile device.

5. The system of claim 1, wherein the data necessary for rendering a virtual object on the first display of the first device comprises data necessary for superimposing the virtual object over an image of the object.

6. The system of claim 1, wherein at least one of the first and second mobile devices comprises a cell phone.

7. The system of claim 1, wherein the mixed reality server is further configured to analyze the image data to recognize attributes associated with the object.

8. The system of claim 7, wherein the image data comprises a video image.

9. The system of claim 1, wherein the at least one attribute comprises a name.

10. The system of claim 1, wherein the at least one attribute comprises a class.

11. The system of claim 1, wherein the object comprises a tag.

12. The system of claim 1, wherein the object lacks a tag.

13. The system of claim 1, wherein the image processing server comprises the mixed reality server.

14. A method of interacting with a mixed reality, comprising:

receiving image data related to an object via first and second mobile devices equipped with a video camera, wherein the first and second mobile devices are in physical proximity to each other;

identifying at least one attribute of the object from the image data received from the first and second mobile devices;

using the at least one attribute to consult a database storing a plurality of command sets for multiple, mixed reality objects;

returning information associated with at least one of the mixed reality objects to the first mobile device including data necessary for rendering a virtual object on a first display of the first mobile device; and

returning information associated with at least one of the mixed reality objects to the second mobile device including data necessary for rendering a virtual object on a second display of the second mobile device.

15. The method of claim 14, further comprising the step of allowing a user to select a command from at least one of the command sets stored in the database.

16. The method of claim 14, further comprising the step of determining a desired command set from the plurality of command sets, and wherein the at least one of the mixed reality objects is related to the desired command set.

17. The method of claim 16, further comprising the step of receiving GPS data from at least one of the first and second mobile devices and using the GPS data to aid in determining a desired command set from the plurality of command sets.

18. The method of claim 16, further comprising the step of receiving orientation data from at least one of the first and second mobile devices and using the orientation data to aid in determining a desired command set from the plurality of command sets.

19. The method of claim 14, wherein the data necessary for rendering a virtual object on the first display of the first mobile device comprises data necessary for superimposing the virtual object over an image of the object.

20. The method of claim 14, wherein at least one of the first and second mobile devices comprises a cell phone.

21. The method of claim 14, further comprising the step of analyzing the image data.

22. The method of claim 14, wherein the image data comprises a video image.

23. The method of claim 14, wherein the at least one attribute comprises a name.

24. The method of claim 14, wherein the at least one attribute comprises a class.

25. The method of claim 14, wherein the object comprises a tag.

26. A method of interacting with a mixed reality, comprising:

capturing image data related to an object in a video image using first and second mobile devices each equipped with a video camera and a display;

deriving at least one attribute associated with the object from the image data captured using the first and second mobile devices;

consulting a database having storing a plurality of command sets for multiple, mixed reality objects;

receiving information associated with at least one of the mixed reality objects via the first and second mobile devices, wherein the information includes data necessary for rendering a virtual object on a display of the first mobile device and a display of the second mobile device; and

displaying an image of a mixed reality world that includes a portion of the image data and the virtual object on the display of the first mobile device and the display of the second mobile device.

27. The method of claim 26, further comprising the step of allowing a user to select a command from at least one of the command sets stored in the database.

28. The method of claim 26, further comprising the step of determining a desired command set from the plurality of command sets, and wherein the at least one of the mixed reality objects is related to the desired command set.

29. The method of claim 28, further comprising the step of receiving GPS data from the first and second mobile devices and using the GPS data to aid in determining a desired command set from the plurality of command sets.

30. The method of claim 28, further comprising the step of receiving orientation data from the first and second mobile devices and using the orientation data to aid in determining a desired command set from the plurality of command sets.

31. The method of claim 26, wherein the data necessary for rendering a virtual object on the display of the first mobile device and the display of the second mobile device comprises data necessary for superimposing the virtual object over an image of the object.

32. The method of claim 26, wherein at least one of the first and second mobile devices comprises a cell phone.

33. The method of claim 26, further comprising the step of analyzing the image data.

34. The method of claim 26, wherein the image data comprises a video image.

35. The method of claim 26, wherein the at least one attribute comprises a name.

36. The method of claim 26, wherein the at least one attribute comprises a class.

37. The method of claim 26, wherein the object comprises a tag.

38. A mixed reality system, comprising:

a first mobile device configured to capture image data related to an object and

transmit the image data to a computer, and a second mobile device configured to capture image data related to the object and transmit the image data to a computer;

wherein at least one of the first and second mobile devices and the computer is configured to derive at least one attribute associated with the object from the image data transmitted by the first and second mobile devices;

wherein the at least one attribute is used to interact with a database having a plurality of command sets for multiple, mixed reality objects; and

wherein the first mobile devices is configured to obtain data necessary for rendering a virtual object on a first display of the first mobile device, and wherein the second mobile devices is configured to obtain data necessary for rendering a virtual object on a second display of the second mobile device.

39. The system of claim 38, wherein the first and second mobile devices each comprises a global positioning system (GPS) sensor configured to provide information associated with a physical location.

40. The system of claim 38, further comprising a first sensor configured to provide information associated with an orientation.

41. The system of claim 40, wherein the orientation comprises an orientation of at least one of the first and second mobile device.

42. The system of claim 38, wherein the data necessary for rendering a virtual object on at least one of the first and second displays comprises data necessary for superimposing the virtual object over an image of the object.

43. The system of claim 38, wherein the first and second mobile devices each comprise a cell phone.

44. The system of claim 38, wherein at least one of the first and second mobile devices and the computer is configured to analyze the image data.

45. The system of claim 38, wherein the image data comprises a video image.

46. The system of claim 38, wherein the at least one attribute comprises at least one of a name and a class.

47. The system of claim 38, wherein the object comprises a tag.

48. The system of claim 38, wherein the object lacks a tag.