Control device for communicating visual information

• US 10,445,884 B2
• Filed: 10/24/2017
• Issued: 10/15/2019
• Est. Priority Date: 10/26/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Marks, Richard, Mikhailov, Anton, Huang, Ennin, Zalewski, Gary M.
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): Mosser, Robert E
• Application Number:
  US15/792,690
• Publication Number:
  US 20180047172A1
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 2203/0333 Ergonomic shaped mouse for one hand
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Methods and systems for processing input by a computing device are presented. One method includes operations for receiving images of a control device that includes an object section, and for determining a location of the control device utilizing image analysis for each captured image. Additionally, the movement of the control device is tracked based on the determined locations, where the tracking of the movement includes receiving inertial sensor information obtained by sensors in the control device, and determining an orientation of the control device based on the sensor information. Additionally, the method includes an operation for translating the movement and orientation of the control device into input for a game executing in the computing device, where the input is translated into a motion and orientation of an object in the game based on the movement of the control device.

Claims

1. A controller comprising:

a handle;

a visual tracking section connected to the handle, the visual tracking section having a shape that includes one or more light emitting devices;

inertial sensors being part of the controller;

a wireless transceiver being part of the controller, wherein the controller is operable to wirelessly transmit inertial sensor information from the inertial sensors to a computing device, wherein the computing device uses the inertial sensor information and images captured of the visual tracking section to define input for an application executed by the computing device, the input used as interactive input used by the application, the application is configured to output display data to a screen; and

a vibration device being part of the controller, the vibration device is configured activate to provide vibrotactile feedback responsive to said interactive input processed by the application.

2. The controller of claim 1, wherein the light emitting devices illuminate on the shape a pattern for the visual tracking section, the pattern is defined as an arc of a ring shape.

3. The controller of claim 2, wherein the light emitting devices produce light in one of an infrared or visible spectrum.

4. The controller of claim 1, further comprising,

a plurality of buttons, where each button is configured for providing a command to the application that is executed by the computing device.

5. The controller of claim 1, further comprising,

a joystick, where the joystick is configured for providing a command to the application that is executed by the computing device.

6. The controller of claim 1, wherein wireless transceiver is configured to communicate data in accordance with one of a radio signal, a WiFi signal, or a Bluetooth signal.

7. The controller as recited in claim 1, wherein the inertial sensors include an accelerometer for measuring acceleration and gravity forces on the controller.

8. The controller as recited in claim 1, wherein the inertial sensors include a magnetometer.

9. The controller as recited in claim 1, wherein the inertial sensors include an accelerometer and a magnetometer used together to obtain an inclination and an azimuth of the controller.

10. The controller as recited in claim 1, wherein the inertial sensors include a gyroscope for measuring orientation of the controller.

11. The controller as recited in claim 1, wherein the computing device monitors visual cues produced by the visual tracking section.

12. The controller as recited in claim 1, wherein the computing device determines an orientation of the controller to enable use of the controller as one of a pointing device, a selection device, or input device.

13. The controller as recited in claim 1, further including:

one or more speakers for providing audio feedback.

14. The controller of claim 1, wherein the visual tracking section is configured to be illuminated in different colors, or levels of brightness, or intermittently.

15. The controller as recited in claim 1, wherein the inertial sensors include an accelerometer for measuring acceleration and gravity forces on the controller and/or a magnetometer.

16. A controller comprising:

a handle;

a visual tracking section integrated with the handle, the visual tracking section having a shape that includes one or more light emitting devices that are viewable from a surface of the visual tracking section;

inertial sensors being part of the controller; and

a wireless transceiver being part of the controller, wherein the controller is operable to wirelessly transmit inertial sensor information from the inertial sensors to a computing device, wherein the computing device uses the inertial sensor information and one or more video images captured of the visual tracking section to provide input for an application executed by the computing device, the input used as interactive input by the application, the application is configured to output display data to a screen, the screen provides graphical content and at least some of the graphical content changes responsive to the interactive input;

wherein the light emitting devices illuminate a pattern on the visual tracking section, and the pattern is arranged along an arc.

17. The controller of claim 16, wherein the light emitting devices produce light in one of an infrared or visible spectrum.

18. The controller of claim 16, further comprising,

a vibration device being part of the controller, the vibration device is configured activate to provide vibrotactile feedback responsive to certain interactive input processed by the application.

19. The controller of claim 16, further comprising,

a plurality of buttons, where each button is configured for providing a command to the application that is executed by the computing device, and or a joystick, where the joystick is configured for providing a command to the application that is executed by the computing device.

20. The controller of claim 16, wherein wireless transceiver is configured to communicate data in accordance with one of a radio signal, a WiFi signal, or a Bluetooth signal.