Interactivity with a mixed reality

• US 10,463,961 B2
• Filed: 11/03/2013
• Issued: 11/05/2019
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Cohen, Ronald H.
• Primary Examiner(s): Kumar, Anil N
• Application Number:
  US14/070,571
• Publication Number:
  US 20140059437A1
• Time in Patent Office: 2,193 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/18 : Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72427 : for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

Claims

1. A method for enabling a user of a mobile computing device to use image sensor data and audio sensor data for interactions, the method comprising the steps of:

capturing, from an image sensor of a mobile device, image sensor data relating to a plurality of real-world objects;

identifying, by the mobile device in cooperation with a server, a first set of attributes associated with a first real-world object and a second set of attributes associated with a second real-world object from the plurality of real-world objects, based on the captured image sensor data;

providing, by the mobile device to the server, the first set of attributes and the second set of attributes;

determining, by the server, a first command set based on the first set of attributes and a second command set based on the second set of attributes from a database storing a plurality of command sets corresponding to a plurality of real-world objects, wherein the first and second command sets comprise commands that control functions of the first and the second real-world objects, respectively;

supplying, by the server, the first command set and the second command set to the mobile computing device;

making accessible, by the mobile computing device, the first command set and the second command set to the user; and

receiving, by the mobile computing device, a user-selected command from one of the first command set and the second command set, wherein the user-selected command is executable by the mobile computing device to control an accessible function of the respective first or second real-world object.

2. The method of claim 1, wherein the mobile computing device comprises a cell phone.

11. The method of claim 1, wherein at least one of the first real-world object and the second real-world object comprises a vending machine.

16. The method of claim 1, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.

3. A system for using image sensor data and audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets corresponding to a plurality of real-world objects;

wherein the mobile device comprises an image sensor configured to capture image sensor data and an audio sensor configured to capture audio sensor data;

wherein the mobile device is configured to;

identify, in cooperation with the server, a first set of attributes associated with a first real-world object and a second set of attributes associated with a second real-world object by using the image sensor data captured by the image sensor of the mobile device; and

provide the first set of attributes and second set of attributes to the server;

wherein the server is configured to determine a first command set based on the first set of attributes and a second command set based on the second set of attributes from a database storing a plurality of command sets corresponding to a plurality of real-world objects, wherein the first and second command sets comprise commands that control functions of the first and the second real-world objects, respectively; and

wherein the mobile device is further configured to;

receive the first command set and the second command set from the server;

make the first command set and the second command set accessible to a user; and

receive a user-selected command from one of the first command set and second command set, wherein the user-selected command is executable by the mobile device to control an accessible function of the respective first or second real-world object.

4. The system of claim 3, wherein the mobile device comprises a cell phone.

12. The system of claim 3, wherein at least one of the first real-world object and the second real-world object comprises a vending machine.

17. The system of claim 3, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.

18. The system of claim 3, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.

5. A system for using image sensor data and audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets corresponding to a plurality of real-world objects;

wherein the mobile device comprises an image sensor configured to capture image sensor data and an audio sensor configured to capture audio sensor data;

wherein the mobile device is configured to;

identify, in cooperation with the server, a first set of attributes associated with a first real-world object and a second set of attributes associated with a second real-world object by using the image sensor data captured by the image sensor of the mobile device and at least one of GPS sensor data and location information; and

provide the first set of attributes and the second set of attributes to the server;

wherein the server is configured to determine a first command set based on the first set of attributes and the at least one of GPS sensor data and location information and a second command set based on the second set of attributes and the at least one of GPS sensor data and location information from a database storing a plurality of command sets corresponding to a plurality of real-world objects, wherein the first and second command sets comprise commands that control functions of the first and the second real-world objects, respectively; and

wherein the mobile device is further configured to;

receive the first command set and second command set from the server;

make the first command set and the second command set accessible to the user; and

receive a user-selected command from one of the first command set and second command set, wherein the user-selected command is executable by the mobile to control an accessible function of the respective first or second real-world object.

8. The system of claim 5, wherein the mobile device comprises a cell phone.

13. The system of claim 5, wherein at least one of the first real-world object and the second real-world object comprises a vending machine.

19. The system of claim 5, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.

6. A system for using image sensor data and audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets corresponding to a plurality of objects;

wherein the mobile device comprises an image sensor configured to capture image sensor data and an audio sensor configured to capture audio sensor data;

wherein the mobile device is configured to;

identify, in cooperation with the server, a first set of attributes associated with a first real-world object and a second set of attributes associated with a second real-world object by using the image sensor data captured by the image sensor of the mobile device and GPS sensor data; and

provide the first set of attributes and the second set of attributes to the server;

wherein the server is configured to determine a first command set based on the first set of attributes and a second command set based on the second set of attributes from a database storing a plurality of command sets corresponding to a plurality of real-world objects, wherein the first and second command sets comprise commands that control functions of the first and the second real-world objects, respectively; and

wherein the mobile device is further configured to;

receive the first command set and the second command set from the server;

make the first command set and the second command set accessible to the user; and

receive a user-selected command from one of the first command set and second command set, wherein the user-selected command is related to a time and executable by the mobile device to control an accessible function of the respective first or second real-world object.

9. The system of claim 6, wherein the mobile device comprises a cell phone.

14. The system of claim 6, wherein at least one of the first real-world object and the second real-world object comprises a vending machine.

20. The system of claim 6, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.

7. A system for using image sensor data and audio sensor data for interactions, the system comprising:

a server communicatively coupled with a mobile device and a database storing a plurality of command sets corresponding to a plurality of objects;

wherein the mobile device comprises an image sensor configured to capture image sensor data and an audio sensor configured to capture audio sensor data;

wherein the mobile device is configured to;

identify, in cooperation with the server, a first set of attributes associated with a first real-world object and a second set of attributes associated with a second real-world object by using the image sensor data captured by the image sensor of the mobile device; and

provide the first set of attributes and the second set of attributes to the server;

wherein the server is configured to determine a first command set based on the first set of attributes and a second command set based on the second set of attributes from a database storing a plurality of command sets corresponding to a plurality of real-world objects, wherein the first and second command sets comprise commands that control functions of the first and the second real-world objects, respectively; and

wherein the mobile device is further configured to;

receive the first command set and the second command set from the server;

make the first command set and the second command set accessible to the user; and

receive a user-selected command from one of the first command set and second command set, wherein the user-selected command is related to a time executable by the mobile device to control an accessible function of the respective first or second real-world object.

10. The system of claim 7, wherein the mobile device comprises a cell phone.

15. The system of claim 7, wherein at least one of the first real-world object and the second real-world object comprises a vending machine.

21. The system of claim 7, wherein at least one of the first real-world object and the second real-world object comprises a stereo system, a printer, a kitchen appliance, a consumer electronics device, a set-top box, or a television.