Methods for directing pointing detection conveyed by user when interfacing with a computer program

• US 8,568,230 B2
• Filed: 11/10/2009
• Issued: 10/29/2013
• Est. Priority Date: 09/15/2003
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Entertainment Computer Incorporated
• Inventors: Marks, Richard L., Deshpande, Hrishikesh R.
• Primary Examiner(s): PIERCE, DAMON JOSEPH
• Application Number:
  US12/616,117
• Publication Number:
  US 20100056277A1
• Time in Patent Office: 1,449 Days
• Field of Search: 463/36, 463/39, 345/156, 382/103, 382/312, 348/135, 348/136, 348/137, 348/139, 348/140, 348/141, 348/142, 348/143, 348/169
• US Class Current: 463/36
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device. The computing device being interfaced with an image capture device. The method includes detecting a human head of a person in the image taken with the image capture device and assigning the human head a head location. And, detecting an object held by the person in the image and assigning the object an object location. The method determines a relative position in space between the head location and the object location from the capture location, such that the relative position is used to identify a pointing direction of the object.

Claims

1. A method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device, the computing device being interfaced with an image capture device, comprising:

(a) detecting a human head of a person in an image taken with the image capture device and assigning the human head a head location;

(b) detecting an object held by the person in the image and assigning the object an object location;

(c) determining a relative position in space between the head location and the object location from a capture location of the image capture device, the relative position used to identify a pointing direction of the object;

(d) displaying the pointing direction of the object on a display screen; and

(e) updating the pointing direction of the object on the display screen based on a change in the relative position between the head location and the object location, wherein the pointing direction of the object is updated on the display screen when the head location changes independent of a change in the object location,

wherein the method is executed by one or more processors.

2. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, further comprising:

capturing a plurality of frames through the image capture device, each of the frames being analyzed to process (a)-(c); and

determining changes in the pointing direction of the object over the plurality of analyzed frames.

3. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 2, wherein the determined changes in the pointing direction of the object are translated to detected motion of the pointing direction of the object over the plurality of frames captured through the image capture device.

4. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein the pointing direction of the object is translated as a pointing location on the display screen, the display screen being located below the location of the image capture device.

5. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein the relative position is identified by computing a first angle and a second angle between the head location and the object location in relation to the capture location.

6. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein the display screen is configured to render interactive graphics.

7. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, further comprising:

enabling interactivity with particular interactive graphics presented on the display screen using a graphic indicator of the pointing direction of the object.

8. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 7, wherein the interactivity includes selection of a graphic, shooting of a graphic, touching a graphic, moving of a graphic, activation of a graphic, triggering of a graphic, acting upon or with a graphic, or a combination thereof.

9. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein detecting the human head is processed using face detection.

10. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein detecting the object is facilitated using colors, lights, or a combination thereof, or a combination of identifying differences in colors, lights, and shapes, or identifying on or off states.

11. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein the computer program is a video game.

12. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 1, wherein the computer program is either local to the image capture device or networked over the internet.

13. A method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device, the computing device being interfaced with an image capture device, comprising:

(a) detecting a human head of a person in an image taken with the image capture device and assigning the human head a head location;

(b) detecting a body part of the person, other than the human head, in the image and assigning the body part a body part location;

(c) determining a relative position in space between the head location and the body part location from a capture location of the image capture device, the relative position used to identify a pointing direction of the body part;

(d) displaying the pointing direction of the body part on a display screen; and

(e) updating the pointing direction of the body part on the display screen based on a change in the relative position between the head location and the body part location, wherein the pointing direction of the body part is updated on the display screen when the head location changes independent of a change in the body part location,

wherein the method is executed by one or more processors.

14. The method for detecting direction when interfacing with a computer program executed on a computing device as recited in claim 13, further comprising:

capturing a plurality of frames through the image capture device, each of the frames being analyzed to process (a)-(c); and

determining changes in the pointing direction of the body part over the plurality of analyzed frames.

15. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 14, wherein the determined changes in the pointing direction of the body part are translated to detected motion of the pointing direction of the body part over the plurality of frames captured through the image capture device.

16. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, wherein the pointing direction of the body part is translated as a pointing location on the display screen, the display screen being located below the location of the image capture device.

17. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, wherein the relative position is identified by computing a first angle and a second angle between the head location and the body part in relation to the capture location.

18. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, further comprising enabling interactivity with particular interactive graphics presented on the display screen using a graphic indicator of the pointing direction of the body part.

19. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 18, wherein the interactivity includes selection of a graphic, shooting of a graphic, touching a graphic, moving of a graphic, activation of a graphic, movement of a graphic, triggering of a graphic, acting upon or with a graphic, or a combination thereof.

20. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, wherein detecting the human head is processed using face detection and detecting the body part is facilitated by identifying shapes.

21. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, wherein the computer program is a video game or an interactive communication program.

22. The method for detecting direction conveyed by a user when interfacing with a computer program executed on a computing device as recited in claim 13, wherein the computer program is either local to the image capture device or networked over the internet.