User interface system and method using thermal imaging

• US 8,638,364 B2
• Filed: 12/15/2010
• Issued: 01/28/2014
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Chen, Ruxin, Osman, Steven
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US12/969,501
• Publication Number:
  US 20120075463A1
• Time in Patent Office: 1,140 Days
• Field of Search: 348/135, 600/549, 600/529, 382/128, 463/37, 463 1- 2, 463/31, 463/39
• US Class Current: 348/135
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A thermal imaging interface for control of a computer program may obtain one or more thermal infrared images of one or more objects with one or more thermographic cameras. The images may be analyzed to identify one or more characteristics of the objects. Such characteristics may be used as a control input in the computer program.

Claims

1. A thermal imaging interface method for control of a computer program, comprising:

a) obtaining one or more thermal infrared images of one or more objects with one or more thermographic cameras;

b) analyzing the one or more thermal infrared images;

c) identifying one or more characteristics of the one or more objects from the one or more thermal infrared images, wherein the one or more objects include one or more users, wherein the one or more characteristics include one or more vital signs of the one or more of the users which are determined by measuring a temperature or temperature change in the one or more thermal infrared images of the one or more users; and

d) using the one or more characteristics as a control input in the computer program.

2. The method of claim 1, wherein the computer program uses the one or more vital signs to control a pace of actions executed by the program.

3. The method of claim 2, wherein the pace of actions includes a tempo of music.

4. The method of claim 1, wherein the computer program uses the one or more vital signs to control one or more graphics executed by the computer program.

5. The method of claim 1, wherein the computer program uses the one or more vital signs to measure a performance of the one or more users in conjunction with the computer program.

6. The method of claim 1, wherein the computer program uses the one or more vital signs to distinguish living objects from inanimate objects.

7. The method of claim 1, wherein the computer program uses the one or more vital signs to control a response speed of one or more user interfaces executed by the computer program.

8. The method of claim 1, wherein the one or more vital signs is mapped to trigger an action of a virtual object.

9. The method of claim 1, wherein the one or more vital signs is sampled and used by the computer program to generate a sound.

10. The method of claim 1, wherein the one or more vital signs is a heart beat rate, respiration rate, body temperature, or blood pressure.

11. The method of claim 10, further comprising analyzing the one or more vital signs and determining a health condition of a user from results of analysis of the vital signs.

12. The method of claim 1, wherein the one or more characteristics in c) include a respiration pattern for one or more users determined by measuring the airflow and temperature change around the nose and mouth of the one or more users.

13. The method of claim 12, wherein the respiration pattern for the one or more users is used to identify a source of sounds originating amongst the one or more users.

14. The method of claim 12, wherein the respiration pattern for the one or more users is used to determine whether the one or more of the users is an authorized user.

15. The method of claim 1, wherein the one or more characteristics in c) include a depth of the one or more objects in relation to the one or more thermographic cameras determined by infrared reflection coupled with interpolation using the one or more thermal infrared images.

16. The method of claim 15, wherein d) includes using the one or more characteristics for each of the one or more users in combination with one or more normal images of the one or more users to track the movement of one or more body parts associated with the one or more users.

17. The method of claim 1, wherein the one or more characteristics in c) include a segmentation cue that distinguishes the one or more objects from their background.

25. The method of claim 1, wherein the computer program is a computer entertainment program.

26. The method of claim 25, wherein the control input causes a change of state in a video game.

18. An apparatus for control of a computer program, comprising:

one or more thermographic cameras;

one or more processors coupled to the one or more thermographic cameras; and

instructions executable by the one or more processors configured to;

a) obtain one or more thermal infrared images of one or more objects with the one or more thermographic cameras;

b) analyze the one or more thermal infrared images;

c) identify one or more characteristics of the one or more objects from the one or more thermal infrared images, wherein the one or more objects include one or more users, wherein the one or more characteristics include one or more vital signs of the one or more of the users which are determined by measuring a temperature or temperature change in the one or more thermal infrared images of the one or more users;

d) use the one or more characteristics as a control input for a computer program.

19. The apparatus of claim 18, wherein the one or more characteristics in c) include a respiration rate for each of the one or more users determined by measuring the airflow and temperature change around the nose and mouth of the one or more users.

20. The apparatus of claim 18, wherein the one or more characteristics in c) include a depth of the one or more objects in relation to the one or more thermographic cameras determined by infrared reflection coupled with interpolation using the one or more thermal infrared images.

21. The apparatus of claim 18, wherein the one or more thermographic cameras include a thermographic camera mounted to a video display that is coupled to the processor.

22. The apparatus of claim 18, wherein the one or more processors include one or more dedicated processors, wherein each of the one or more thermographic cameras is coupled to a corresponding dedicated processor.

23. The apparatus of claim 22, wherein b), c) and d) are implemented by the one or more dedicated processors.

24. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer readable program code embodied in said medium for controlling a computer program using a thermal image interface, said computer program product having;

a) computer readable program code means for obtaining one or more thermal infrared images of one or more objects with one or more thermographic cameras;

b) computer readable program code means for analyzing the one or more thermal infrared images;

c) computer readable program code means for identifying one or more characteristics of the one or more objects from the one or more thermal infrared images, wherein the one or more objects include one or more users, wherein the one or more characteristics include one or more vital signs of the one or more of the users which are determined by measuring a temperature or temperature change in the one or more thermal infrared images of the one or more users; and

d) computer readable program code means for using the one or more characteristics as a control input in the computer program.