Control system employing identification of a displayed image

• US 9,965,027 B2
• Filed: 06/22/2016
• Issued: 05/08/2018
• Est. Priority Date: 10/25/2004
• Status: Active Grant

• Current Assignee: I-Interactive LLC
• Original Assignee: I-Interactive LLC
• Inventors: Henty, David L.
• Primary Examiner(s): Daniels, Anthony J
• Application Number:
  US15/190,140
• Publication Number:
  US 20160299562A1
• Time in Patent Office: 685 Days
• Field of Search: 34820799, 345157
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0386 for light pen
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/93 Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: G08C 23/04 using light waves, e.g. infrared
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42225 characterized by types of remote control, e.g. universal remote control
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

An application control system and method is adapted for use with an entertainment system of a type including a display such as a monitor or TV and having display functions. A control device may be conveniently held by a user and employs an imager. The control system and method images the screen of the TV or other display to detect distinctive markers displayed on the screen. This information is transmitted to the entertainment system for control of an application or is used by the control device to control an application.

Claims

1. A control system for use in an environment having a display device, comprising:

a user-manipulated device, the user-manipulated device including a video camera;

the user-manipulated device further comprising control circuitry including one or more processors with associated programs implementing image processing and control processing functions under programmed control, the image processing implemented functions including receiving image information from the video camera in response to user operation of the video camera of the user-manipulated device, the image information corresponding to a captured scene which includes at least part of a display device in the field of view of the camera and also background around the display device, the image processing isolating the display image information from the background scene using known characteristics of the display, the image processing further identifying within the display portion of the image one or more displayed reference patterns, the control processing controlling an application using the reference patterns.

2. The system of claim 1, wherein the one or more processors comprise a digital signal processor and a control microprocessor and the control microprocessor implements controlling the application based on the identified patterns.

3. The system of claim 1, wherein the known characteristics of the display employed by the processor include display shape and brightness.

4. The system of claim 2, wherein the user-manipulated device further comprises a wireless transmitter and the control microprocessor provides application control signals to the display device using the transmitter to control a display application controlling displayed user interface information.

5. The system of claim 3, wherein the displayed reference patterns comprise a logo.

6. The system of claim 1, wherein the controlled application involves determining an object displayed on the screen of the display device that the user is pointing at with the user-manipulated device.

7. The system of claim 1, wherein the controlled application involves identifying information displayed on the screen of the display device based on the reference patterns.

8. A control system for use in an environment having a display device, comprising:

a user-manipulated device, the user-manipulated device including a video camera;

the user-manipulated device further comprising control circuitry including one or more processors with associated programs implementing image processing and control functions under programmed control, the image processing implemented functions including receiving image information from the video camera in response to user operation of the video camera of the user-manipulated device, the image information corresponding to a captured scene which includes at least part of a display device in the field of view of the camera including at least one reference field incorporated as graphical information displayed on a screen of the display device to provide reference information for controlling an application, the user-manipulated device processor implementing program instructions identifying the at least one reference field based on one or more colors of the at least one reference field, based on a shape of the at least one reference field, and based on a pattern of elements that comprise the at least one reference field, the control functions including controlling an application based on said reference information.

9. The system of claim 8, wherein the user-manipulated device further comprises a wireless transmitter and the control microprocessor provides application control signals to the display device using the transmitter to control a display application controlling displayed user interface information.

10. The system of claim 8, wherein the controlled application involves identifying information displayed on the screen of the display device based on the reference patterns.

11. A control system adapted for use in an environment having a display device presenting display information including predetermined objects, the system comprising:

a control device including a video camera associated therewith and movable so as to be pointed by the user to image a desired field of view;

the control device further comprising control circuitry including one or more processors with associated programs implementing processing functions under programmed control, the programmed processor implemented functions including;

an image control processing programmed to receive image information from the video camera, the image information capturing a scene in response to user operation of the video camera of the control device which includes at least part of a display device including a displayed pattern of reference elements;

an identification control processing to identify an object in said image information employing the displayed pattern of reference elements to provide object reference information; and

an application control processing to initiate control of an application based on the identified object reference information.

12. A system as set out in claim 11, wherein the display device is a TV.

13. A system as set out in claim 11, wherein the displayed pattern of reference elements is a logo.

14. A system as set out in claim 11, wherein the displayed pattern of reference elements is a pointer.

15. A system as set out in claim 11, wherein the application control processing provides object information to a user interface.

16. A system as set out in claim 11, wherein the application control processing accesses a remote location product information database to retrieve and provide product information to the user.

17. A system as set out in claim 16, wherein the application control processing initiates further object identification processing at said remote location.

18. A system as set out in claim 11, wherein the application control processing accesses a remote service provider information database to provide displayed service provider content information to the user.