Control device for communicating visual information

• US 8,287,373 B2
• Filed: 04/17/2009
• Issued: 10/16/2012
• Est. Priority Date: 12/05/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard, Mikhailov, Anton, Huang, Ennin, Zalewski, Gary M.
• Primary Examiner(s): Such, Matthew W
• Assistant Examiner(s): Carpenter, Robert
• Application Number:
  US12/426,186
• Publication Number:
  US 20100144436A1
• Time in Patent Office: 1,278 Days
• Field of Search: 463/36
• US Class Current: 463/36
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 2203/0333 Ergonomic shaped mouse for one hand
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Methods and systems for interfacing a control device with a computer program executing at a base computing device are presented. The method generates a visual cue at a spherical section of the control device and captures an image of the visual cue using an image capture device connected to the base computing device. Further, the method determines whether the visual cue is user feedback or input for the computer program, and processes the visual cue at the base computing device when the visual cue is an input. Additionally, a state of an object being processed is updated by the computer program in response to the input to drive interactivity with the computer program via the control device.

Claims

1. A method for interfacing a control device with a computer program executing at a base computing device, the method comprising:

transmitting a command from the base computing device to the control device;

generating a visual cue at a spherical section of the control device based on the transmitted command to provide feedback for a user;

capturing an image of the control device at the base computing device;

determining whether the captured image contains the visual cue that is the feedback for the user or input for the computer program;

processing the visual cue in the captured image at the base computing device; and

updating a state of an object being processed by the computer program when the captured image contains the input, the updating used to drive interactivity with the computer program via the control device.

2. The method as recited in claim 1, further including,

establishing a game session for a period of time, wherein the feedback for the user is alternated with input for the computer program during the game session.

3. The method as recited in claim 1, wherein a plurality of control devices interface with the computer program simultaneously, each of the control devices using a visual cue of a different color for input to the computer program.

4. The method as recited in claim 1, wherein the visual cue includes lighting up the spherical section.

5. The method as recited in claim 4, wherein brightness when lighting up the spherical section is adjusted according to a value of a parameter in the computer program.

6. The method as recited in claim 1, wherein the visual cue includes lighting up a portion of the spherical section.

7. The method as recited in claim 1, further including,

creating a sound at the control device associated with the feedback for the user.

8. The method as recited in claim 1, further including,

providing vibrotactile feedback associated with the feedback for the user.

9. The method as recited in claim 1, wherein the visual cue is input to the computer program to determine a location of the control device, wherein processing the visual cue further includes processing the captured image to locate the spherical section in the captured image.

10. The method as recited in claim 9, wherein the visual cue is used simultaneously for feedback for the user and for input.

11. A method for interfacing a control device with a computer program executing at a base computing device, the method comprising:

transmitting a command from the base computing device to the control device;

generating a visual cue request based on the transmitted command, the visual cue request being one of feedback for a user or input for the computing device or both;

producing a feedback visual cue at a spherical section of the control device when the visual cue request is for feedback for the user; and

receiving the input at the base computing device when the visual cue request is for input, receiving the input including,

producing an input visual cue at the spherical section,

capturing an image of the input visual cue,

determining whether the captured image contains the visual cue that is the feedback for the user or input for the computer program, and

processing the image of the input visual cue at the base computing device to update an object used by the computer program to drive interactions between the control device and the computer program.

12. The method as recited in claim 11, further including,

combining the visual cue with status information to determine an orientation of the control device, the status information being acquired by at least one of an accelerometer, a gyroscope or a magnetometer in the control device.

13. The method as recited in claim 12, wherein the orientation of the control device is used to operate the control device as a pointer.

14. The method as recited in claim 12, wherein the orientation of the control device enables using the control device as a flashlight to illuminate a section of a display connected to the base computing device.

15. The method as recited in claim 12, wherein the orientation of the control device enables using the control device as a shooting device into a display connected to the base computing device.

16. The method as recited in claim 11, wherein the visual cue for feedback for the user is selected from a group consisting of a gun shooting, an amount of life left for a player in a game, an entity associated with the control device is hit in the game, a spell is cast on the entity, and the spherical section being occluded from view from an image capturing device.

17. A system for using visual cues for feedback for a user and input to a computer program, the system comprising:

a base computing device having a processor executing the computer program;

a control device with a spherical section that generates a visual cue to provide user feedback based on a transmitted command from the base computing device; and

an image capture device connected to the base computing device, the image capture device capturing an image of the control device;

wherein program instructions from the computer program when executed by the processor cause the processor to,

determine whether the captured image contains the visual cue that is the feedback for the user or input for the computer program,

process the visual cue in the captured image at the base computing device, and

update a state of an object being processed by the computer program in response to the input, the updating used to drive interactivity with the computer program via the control device.

18. The system of claim 17, wherein the base computing device transmits a signal to the control device to set a color of the spherical section.

19. The system of claim 17, wherein pushing a button in the control device causes the spherical section to change colors, the base computing device establishing the pushing of the button by detecting the color change.

20. The system of claim 17, wherein a calibration procedure causes, the control device to light the spherical section with a predetermined color, and the processor to,

analyze an image from the image capture device to detect the spherical section, and

calibrate the image capture device based on the detected spherical section.

21. The system of claim 17, wherein the control device emits sound to provide additional feedback for the user.

22. The system of claim 17, wherein the control device emits sound that is detected by at least one of a plurality of microphones connected to the base computing device, the sound emitted used to track movement of the control device.

23. A system for using visual cues for user feedback and input to a computer program, the system comprising:

a base computing device having a processor executing the computer program;

a control device with a spherical section that generates a visual cue; and

an image capture device connected to the base computing device, the image capture device capturing an image of the visual cue;

wherein program instructions from the computer program when executed by the processor cause the processor to,

determine whether the visual cue is user feedback or input for the computer program,

process the visual cue at the base computing device when the visual cue is an input, and

update a state of an object being processed by the computer program in response to the input, the updating used to drive interactivity with the computer program via the control device,

wherein a calibration procedure causes the control device to light the spherical section with a predetermined sequence of colors, and the processor to,

analyze a plurality of images from the image capture device to detect the colors of the spherical section, and

calibrate the image capture device based on the detected colors.

24. A system for using visual cues for user feedback and input to a computer program, the system comprising:

a base computing device having a processor executing the computer program;

a control device with a spherical section that generates a visual cue; and

an image capture device connected to the base computing device, the image capture device capturing an image of the visual cue;

wherein program instructions from the computer program when executed by the processor cause the processor to,

determine whether the visual cue is user feedback or input for the computer program,

process the visual cue at the base computing device when the visual cue is an input, and

update a state of an object being processed by the computer program in response to the input, the updating used to drive interactivity with the computer program via the control device,

wherein a calibration procedure causes the control device to light the spherical section with a predetermined color, and the processor to,

analyze an image from the image capture device to detect the spherical section, and

calibrate the image capture device based on the detected spherical section, wherein the computer program keeps a database of a plurality of calibration procedures performed under different environmental conditions, wherein the computer program selects a color for the spherical section based on the database and current environmental conditions.

25. The system of claim 24, wherein the computer program identifies problem areas within a field of vision for the image capture device based on the plurality of calibrations.

26. The system of claim 24, wherein a camera gain of the image capture device is adjusted based on the database and the current environmental conditions.