Augmentation of tangible objects as user interface controller

• US 9,514,570 B2
• Filed: 03/15/2013
• Issued: 12/06/2016
• Est. Priority Date: 07/26/2012
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Keating, Virginia Walker, Gervautz, Michael, Nielsen, Per O.
• Primary Examiner(s): Li, Tracy Y
• Application Number:
  US13/815,813
• Publication Number:
  US 20140028850A1
• Time in Patent Office: 1,362 Days
• Field of Search: 348/158
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Method, computer program product, and apparatus for providing interactions of tangible and augmented reality objects are disclosed. In one embodiment, a method of controlling a real object using a device having a camera comprises receiving a selection of at least one object, tracking the at least one object in a plurality of images captured by the camera, and causing control signals to be transmitted from the device to the real object via a machine interface based at least in part on the tracking.

Claims

1. A method of controlling a real object using a device having a camera, comprising:

receiving a selection of at least one object, wherein the real object, the device, and the at least one object comprise separate objects, and wherein each of the real object, the device, and the at least one object is a physical object;

tracking the at least one object in a plurality of images captured by the camera;

rendering augmentation of a user-defined control mechanism for the at least one object based at least in part on the tracking;

displaying the user-defined control mechanism overlaying the at least one object on a display of the device;

using motions of the device to emulate operations of the user-defined control mechanism; and

causing control signals to be transmitted from the device to the real object via a machine interface.

2. The method of claim 1, wherein the tracking comprises performing 3-dimensional tracking comprising:

determining a relative pose of the at least one object with respect to the device; and

updating states of the at least one object using the relative pose.

3. The method of claim 2, wherein determining a relative pose comprises:

detecting the relative pose of the at least one object with respect to a previously captured image of the at least one object.

4. The method of claim 1, wherein the receiving comprises:

identifying a zooming in to obtain a close up view of the at least one object.

5. The method of claim 1, wherein the receiving comprises:

determining that the at least one object has been placed at center screen of the device.

6. The method of claim 1, wherein the machine interface comprises at least one of:

Bluetooth;

Wi-Fi; and

IR.

7. The method of claim 1, wherein at least one image of the plurality of images is displayed on a touchscreen of the device, and wherein receiving a selection of at least one object comprises receiving information regarding a user input on the touchscreen identifying the at least one object.

8. The method of claim 1, further comprising:

tracking a radio in view of the device;

providing an augmentation of a song playing by the radio; and

adjusting a volume of the song based on determining that the radio has been spun.

9. The method of claim 1, further comprising:

tracking a television in view of the device;

determining that an object has been placed near the television; and

triggering an augmentation of video playing on the television based on the object having been placed near the television.

10. The method of claim 1, wherein:

the causing comprises determining that the at least one object has been rotated, and wherein the control signals comprise signals instructing the real object to change channel or change volume.

11. A device for controlling a real object, comprising:

a camera;

a control unit including processing logic, the processing logic comprising;

logic configured to receive a selection of at least one object, wherein the real object, the device, and the at least one object comprise separate objects, and wherein each of the real object, the device, and the at least one object is a physical object;

logic configured to track the at least one object in a plurality of images captured by the camera;

logic configured to render augmentation of a user-defined control mechanism for the at least one object based at least in part on the tracking;

logic configured to display the user-defined control mechanism overlaying the at least one object on a display of the device;

logic configured to use motions of the device to emulate operations of the user-defined control mechanism; and

logic configured to cause control signals to be transmitted from the device to the real object via a machine interface.

12. The device of claim 11, wherein the logic configured to track the at least one object comprises performing 3-dimensional tracking comprising:

logic configured to determine a relative pose of the at least one object with respect to the device; and

logic configured to update states of the at least one object using the relative pose.

13. The device of claim 12, wherein logic configured to determine relative poses comprises:

logic configured to detect the relative pose of the at least one object with respect to a previously captured image of the at least one object.

14. The device of claim 12, wherein logic configured to receive the selection comprises:

logic configured to identify a zoom in to a close up view of the at least one object.

15. The device of claim 11, wherein logic configured to receive the selection comprises:

logic configured to determine that the at least one object has been placed at center screen of the device.

16. The device of claim 11, wherein the machine interface comprises at least one of:

Bluetooth;

Wi-Fi; and

IR.

17. The device of claim 11, wherein at least one image of the plurality of images is displayed on a touchscreen of the device, and wherein logic configured to receive a selection of at least one object comprises logic configured to receive information regarding a user input on the touchscreen identifying the at least one object.

18. The device of claim 11, further comprising:

logic configured to track a radio in view of the device;

logic configured to provide an augmentation of a song playing by the radio; and

logic configured to adjust a volume of the song based on determining that the radio has been spun.

19. The device of claim 11, further comprising:

logic configured to track a television in view of the device;

logic configured to determine that an object has been placed near the television; and

logic configured to trigger an augmentation of video playing on the television based on the object having been placed near the television.

20. The device of claim 11, wherein:

the logic configured to cause control signals to be transmitted comprises logic configured to determine that the at least one object has been rotated, and wherein the control signals comprise signals instructing the real object to change channel or change volume.

21. A non-transitory medium storing instructions for execution by one or more computer systems for controlling a real object using a device having a camera, the instructions comprising:

instructions for receiving a selection of at least one object, wherein the real object, the device, and the at least one object comprise separate objects, and wherein each of the real object, the device, and the at least one object is a physical object;

instructions for tracking the at least one object in a plurality of images captured by the camera;

instructions for rendering augmentation of a user-defined control mechanism for the at least one object based at least in part on the tracking;

instructions for displaying the user-defined control mechanism overlaying the at least one object on a display of the device;

instructions for using motions of the device to emulate operations of the user-defined control mechanism; and

instructions for causing control signals to be transmitted from the device to the real object via a machine interface.

22. A system for controlling a real object using a device having a camera, comprising:

means for receiving a selection of at least one object, wherein the real object, the device, and the at least one object comprise separate objects, and wherein each of the real object, the device, and the at least one object is a physical object;

means for tracking the at least one object in a plurality of images captured by the camera;

means for rendering augmentation of a user-defined control mechanism for the at least one object based at least in part on the tracking;

means for displaying the user-defined control mechanism overlaying the at least one object on a display of the device;

means for using motions of the device to emulate operations of the user-defined control mechanism; and

means for causing control signals to be transmitted from the device to the real object via a machine interface.

23. The system of claim 22, wherein the tracking comprises means for performing 3-dimensional tracking comprising:

means for determining a relative pose of the at least one object with respect to the device; and

means for updating states of the at least one object using the relative pose.

24. The system of claim 23, wherein means for determining a relative pose comprises:

means for detecting the relative pose of the at least one object with respect to a previously captured image of the at least one object.

25. The system of claim 22, wherein the receiving comprises:

means for identifying a zooming in to obtain a close up view of the at least one object.

26. The system of claim 22, wherein the receiving comprises:

means for determining that the at least one object has been placed at center screen of the device.

27. The system of claim 22, wherein the machine interface comprises at least one of:

Bluetooth;

Wi-Fi; and

IR.

28. The system of claim 22, wherein at least one image of the plurality of images is displayed on a touchscreen of the device, and wherein means for receiving a selection of at least one object comprises means for receiving information regarding a user input on the touchscreen identifying the at least one object.

29. The system of claim 22, further comprising:

means for tracking a radio in view of the device;

means for providing an augmentation of a song playing by the radio; and

means for adjusting a volume of the song based on determining that the radio has been spun.

30. The system of claim 22, further comprising:

means for tracking a television in view of the device;

means for determining that an object has been placed near the television; and

means for triggering an augmentation of video playing on the television based on the object having been placed near the television.

31. The system of claim 22, wherein:

the causing comprises determining that the at least one object has been rotated, and wherein the control signals comprise signals instructing the real object to change channel or change volume.