Control device for communicating visual information

• US 9,799,121 B2
• Filed: 05/29/2015
• Issued: 10/24/2017
• Est. Priority Date: 10/26/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Marks, Richard, Mikhailov, Anton, Huang, Ennin, Zalewski, Gary M.
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): MOSSER, ROBERT E
• Application Number:
  US14/725,260
• Publication Number:
  US 20150294480A1
• Time in Patent Office: 879 Days
• Field of Search: 463 36
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 2203/0333 Ergonomic shaped mouse for one hand
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Methods and systems for processing input by a computing device are presented. One method includes operations for receiving images of a control device that includes an object section, and for determining a location of the control device utilizing image analysis for each captured image. Additionally, the movement of the control device is tracked based on the determined locations, where the tracking of the movement includes receiving inertial sensor information obtained by sensors in the control device, and determining an orientation of the control device based on the sensor information. Additionally, the method includes an operation for translating the movement and orientation of the control device into input for a game executing in the computing device, where the input is translated into a motion and orientation of an object in the game based on the movement of the control device.

Claims

1. A controller comprising:

a handle, wherein the controller is defined for single-hand use;

an object section at one end of the handle;

inertial sensors; and

a wireless transceiver, wherein the controller is operable to wirelessly transmit inertial sensor information to a computing device, wherein the computing device tracks a motion of the controller utilizing the inertial sensor information and images captured of the object section and translates the motion to an input for a game executing in the computing device, the input being translated into a motion of an object in the game based on the motion of the controller;

wherein when the computing device sends a command to the controller, the controller sets light emitted by the object section in a desired state identified in the command, wherein the computing device is configured to visually track the controller after setting the light, wherein the object section is configured to be illuminated with different colors to enable visual differentiation of the controller with other controllers.

2. The controller as recited in claim 1, wherein the inertial sensors include at least one accelerometer and at least one magnetometer for measuring acceleration and gravity forces of the controller when moved.

3. The controller as recited in claim 1, wherein the inertial sensors include an accelerometer and a magnetometer used together to obtain and inclination and an azimuth of the controller.

4. The controller as recited in claim 1, wherein the inertial sensors include a gyroscope for measuring orientation of the controller.

5. The controller as recited in claim 1, wherein the computing device monitors visual cues produced by the object section.

6. The controller as recited in claim 1, wherein the computing device determines an orientation of the controller to enable use of the controller as a pointing device.

7. The controller as recited in claim 1, further including:

one or more speakers for providing audio feedback.

8. The controller as recited in claim 1, wherein the object is an avatar.

9. A device comprising:

a body;

a plurality of objects coupled to the body and configured to provide visual cues for tracking a location of the device;

inertial sensors in the body; and

a wireless transceiver in the body, wherein the device is operable to wirelessly transmit inertial sensor information to a base computing device, wherein the base computing device is configured to analyze images captured of the plurality of objects to determine positions of the plurality of objects within the captured images, wherein the base computing device tracks the location of the device and a motion of the device utilizing the inertial sensor information and the determined positions of the plurality of objects;

wherein the inertial sensors include an accelerometer, a magnetometer and a gyroscope for measuring acceleration, gravitational forces and orientation of the device based on the motion.

10. The device as recited in claim 9, wherein when the base computing device tracks the location and motion of the device, the base computing device determines tilt, roll and yaw of the device.

11. The device as recited in claim 9, wherein the base computing device determines a distance of the plurality of object relative to an image capture device.

12. The device as recited in claim 9, wherein when the base computing device tracks the location and motion of the device, the base computing device detects an edge of the plurality of objects and defines lines, curves or polygons corresponding to the edges of the plurality of objects.

13. The device as recited in claim 9, wherein the base computing device determines orientation of the plurality of objects relative to an image capture device, and the base computing device calculates a three dimensional location of the device.

14. The device as recited in claim 9, wherein the plurality of objects are configured to be tracked in X, Y, and Z directions through color or shape.

15. The device as recited in claim 9, wherein a change in position of the plurality of objects in X, Y, and Z directions relative to an image capture device causes objects displayed on a display to be moved.

16. A device comprising:

a body;

a plurality of objects coupled to the body and configured to provide visual cues for tracking a location of the device;

inertial sensors in the body; and

a wireless transceiver in the body, wherein the device is operable to wirelessly transmit inertial sensor information to a base computing device, wherein the base computing device is configured to analyze images captured of the plurality of objects to determine positions of the plurality of objects within the captured images, wherein the base computing device tracks the location of the device and a motion of the device utilizing the inertial sensor information and the determined positions of the plurality of objects;

wherein the plurality of objects are configured to be illuminated in different colors, level of brightness, or intermittently, wherein visual cues are generated at the plurality of objects upon receiving a command transmitted from the base computing device.