Blow tracking user interface system and method

• US 8,786,698 B2
• Filed: 09/23/2010
• Issued: 07/22/2014
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Chen, Ruxin, Osman, Steven
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US12/889,347
• Publication Number:
  US 20120075462A1
• Time in Patent Office: 1,398 Days
• Field of Search: 348/135, 348/699, 374/15, 374/104, 374/135, 382/192, 382/208
• US Class Current: 348/135
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A blow tracking user interface method and apparatus may detect an orientation of blowing of a user'"'"'s breath and a magnitude of blowing of the user'"'"'s breath. A blow vector may be generated from the orientation and magnitude of the blowing of the user'"'"'s breath. The blow vector may be used as a control input in a computer program.

Claims

1. A blow tracking interface method for control of a computer program, comprising:

detecting an orientation of blowing of a breath, wherein detecting the orientation includes determining a three-dimensional direction towards which the blowing of the breath is pointed other than whether the breath is directed inward or outward;

detecting a magnitude of blowing of the breath;

generating a blow vector from the orientation and the magnitude, wherein the orientation of the blow vector includes the determined direction, wherein generating the blow vector includes generating a three-dimensional geometric vector from the determined three-dimensional direction; and

using the blow vector as a control input in a user interface of the computer program.

2. The method of claim 1, further comprising determining a timing of the blowing of the breath, wherein the timing of the blowing of the breath determines timing of an action in the computer program.

3. The method of claim 1, wherein detecting the orientation and/or magnitude of the blowing of the breath includes obtaining one or more thermal infrared images of a user with one or more thermographic cameras, analyzing the one or more infrared images, and identifying one or more patterns in the one or more images corresponding to the blowing of breath from the one or more thermal infrared image.

4. The method of claim 1 wherein detecting the orientation of blowing of the breath includes determining towards which direction a blowing of the breath through a user'"'"'s nose is pointed, wherein generating a blow vector includes generating a nose breath vector from the determined direction of the breath through the user'"'"'s nose and determining how the user'"'"'s face is orientated from the determined direction in the nose breath vector.

5. The method of claim 1, wherein detecting the orientation and/or magnitude of the blowing of the breath includes detecting sounds of the blowing of the breath with a microphone.

6. The method of claim 1 wherein detecting the orientation of the breath includes determining a source of the blowing of the breath.

7. The method of claim 1, further comprising mapping the timing and intensity of a user'"'"'s breathing to simulated breathing of an avatar controlled by a user so that the avatar appears to breathe in synchronization with user.

8. The method of claim 1 wherein detecting orientation of a breath includes determining whether a user is blowing breath through the user'"'"'s mouth or nose.

9. The method of claim 1 wherein detecting the orientation of a breath further includes determining whether a blowing of the breath is inward or outward.

10. The method of claim 1 wherein detecting orientation or magnitude of a breath includes determining an intended temperature of the breath.

11. The method of claim 1 wherein using the blow vector as a control input includes using the determined direction of the breath as a directional command in the computer program.

12. The method of claim 1 wherein using the blow vector as a control input includes using the magnitude of the breath to determine the magnitude of an action taken by the computer program.

13. The method of claim 1 wherein the orientation of the breath is determined relative to a video display.

14. The method of claim 1 wherein generating the blow vector includes using a past breath history to improve an estimation of the blow vector.

15. The method of claim 1, wherein said using the blow vector as the control input in the user interface includes using the control input to control a graphic display presented to the user.

16. The method of claim 1, wherein said using the blow vector as the control input in the user interface includes using the control input to control a sound playback presented to the user.

17. The method of claim 2, further comprising synchronizing timing of actions in the computer program to the timing of the blowing of breath.

18. The method of claim 3 wherein detecting the orientation and/or magnitude of the breath includes analyzing thermal contours in the one or more thermal infrared images, identifying one or more patterns characteristic of the blowing of the breath from the thermal infrared image, and determining the orientation from the one or more patterns.

19. The method of claim 3, wherein the thermographic camera is mounted to a pair of glasses worn by the user.

20. The method of claim 3, wherein generating the blow vector includes augmenting information obtained from the one or more thermal infrared images with information relating to the breath from one or more auxiliary sensors.

21. The method of claim 3, wherein the one or more thermographic cameras include one or more three-dimensional thermographic cameras, wherein obtaining the one or more thermal infrared images includes obtaining one or more stereoscopic infrared images of blowing of the breath, and wherein analyzing the one or more infrared images includes determining from the one or more stereoscopic infrared images a three-dimensional orientation of a thermal trail of breath in the one or more infrared images.

22. The method of claim 20, wherein the one or more auxiliary sensors includes a microphone or camera.

23. The method of claim 20, wherein the one or more auxiliary sensors includes an additional thermographic camera mounted to a pair of glasses worn by the user.

24. The method of claim 6 wherein determining the source of the breath includes determining which user from among a plurality of users is the source of the blowing of the breath.

25. The method of claim 6 wherein determining the source of the breath includes determining whether the source of the blowing of the breath is a user'"'"'s mouth or nose.

26. A blow tracking interface apparatus for control of a computer program, comprising:

a blow detector configured to detect one or more signals corresponding to blowing of a breath; and

a processor coupled to the blow detector, wherein the processor is configured to determine an orientation and a magnitude of the blowing of the breath from the one or more signals, wherein the processor is configured to determine the orientation by determining a three-dimensional direction towards which the blowing of the breath is pointed other than whether the breath is inward or outward, and generate a blow vector from the orientation and the magnitude, wherein the orientation of the blow vector includes the determined direction, wherein generating the blow vector includes generating a three-dimensional geometric vector from the determined three-dimensional direction, wherein the processor is configured to use the blow vector as a control input in a user interface of the computer program.

27. The apparatus of claim 26 wherein the breath detector includes one or more thermographic cameras.

28. The apparatus of claim 26 wherein the blow detector includes one or more microphones.

29. The apparatus of claim 27, wherein the processor is configured to detect the orientation and/or magnitude of the breath by analyzing thermal contours in one or more infrared images captured by the one or more thermographic cameras to identify one or more patterns characteristic of the breath from the thermal infrared image.

30. The apparatus of claim 27 wherein the one or more thermographic cameras include a thermographic camera mounted to a video display that is coupled to the processor.

31. The apparatus of claim 27 wherein the one or more thermographic cameras include a thermographic camera mounted to a pair of glasses.

32. A non-transitory computer readable medium having computer executable instructions embodied therein, wherein the instructions are configured to implement a blow tracking interface method for control of a computer program when executed by a computer processor, wherein the method implemented by the execution of the instructions comprises:

detecting an orientation of blowing of a breath, wherein detecting the orientation includes determining a three-dimensional direction towards which the blowing of the breath is pointed other than whether the breath is directed inward or outward;

detecting a magnitude of blowing of the breath;

generating a blow vector from the orientation and magnitude, wherein the orientation of the blow vector includes the determined direction, wherein generating the blow vector includes generating a three-dimensional geometric vector from the determined three-dimensional direction; and

using the blow vector as a control input in a user interface of the computer program.