Method and apparatus for improved handset multi-tasking, including pattern recognition and augmentation of camera images

• US 7,142,231 B2
• Filed: 12/29/2003
• Issued: 11/28/2006
• Est. Priority Date: 12/29/2003
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Nokia Corporation
• Inventors: Sugano, Makoto, Chipchase, Jan, Waris, Heikki, Mizobuchi, Sachi
• Primary Examiner(s): WOO, STELLA L
• Application Number:
  US10/748,100
• Publication Number:
  US 20050146600A1
• Time in Patent Office: 1,065 Days
• Field of Search: 348/14.02, 348/14.03, 348/14.07, 455/556.1, 455/556.2, 379/110.01
• US Class Current: 348/14.02
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

This invention provides a handset (10) and a method of operating a handset. The handset includes a user interface that contains a data entry device (20) and a visual display device (14), a camera (16) and a controller (12) coupled to the visual display device and to the camera. The controller operates under the control of a stored program (22A) for displaying to a user an image representative of at least a portion of an environment of the user as seen through the camera during a time that the user is interacting with the user interface. The controller may further operate under control of the stored program to process images generated by the camera to detect a potential for a collision with an object that is present in the environment of the user, and to warn the user of the potential for a collision.

Claims

1. A handset, comprising:

a user interface that comprises a data entry device and a visual display device comprising a display screen;

a camera; and

a controller coupled to the visual display device and to the camera, said controller operating under the control of a stored program for displaying to a user on the display screen an image representative of at least a portion of an environment of the user as seen through the camera during a time that the user is interacting with said user interface and viewing a result of the interaction with said user interface on the display screen.

2. A handset as in claim 1, where said controller further operates under control of the stored program to process images generated by the camera to detect a potential for a collision with an object that is present in the environment of the user.

3. A handset as in claim 2, where said controller is responsive to detecting a potential for a collision to generate a warning to the user.

4. A handset as in claim 3, where the warning is generated by one or more of a visual warning, an audio warning and a tactile warning.

5. A handset as in claim 2, where the object is a person, and where said controller is responsive to detecting a potential for a collision to generate a warning to the person.

6. A handset as in claim 2, where said camera comprises a zoom lens, and where said controller is responsive to detecting a potential for a collision with the camera lens in a zoomed position to generate a warning to the user.

7. A handset as in claim 2, where said camera comprises a fish-eye lens, and where said controller is responsive to detecting a potential for a collision with said fish-eye lens to generate a warning to the user.

8. A handset as in claim 1, further comprising a context sensor, and where said controller is responsive to said context sensor indicating that said handset is in motion, and responsive to the user entering data via said data entry device, for automatically initiating the display of the image representative of at least a portion of the environment of the user.

9. A handset as in claim 2, further comprising a context sensor, and where said controller is responsive to said context sensor indicating that said handset is in motion, and responsive to the user entering data via said data entry device, for automatically initiating the detection of the potential for a collision with an object that is present in the environment of the user.

10. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen as a background to data that is displayed on the display screen.

11. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen in a window in conjunction with data that is displayed on the display screen.

12. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen as a reduced color version of an original image as a background to data that is displayed on the display screen.

13. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen as a simplified version of an original image as a background to data that is displayed on the display screen.

14. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen as a reduced opacity version of an original image as a background to data that is displayed on the display screen.

15. A handset as in claim 1, where the image representative of at least a portion of the environment of the user is displayed on the display screen so as to enhance a contrast between the displayed image and data that is displayed over the displayed image on the display screen.

16. A handset as in claim 1, further comprising a radio frequency transceiver, where said handset functions as a cellular telephone.

17. A method to operate a handset having a camera and a user interface that comprises a data entry device and a visual display device that comprises a display screen, comprising:

generating images of at least a portion of an environment of the handset as seen through the camera; and

displaying to a user on the display screen at least one image that is representative of the environment during a time that the user is interacting with the user interface and viewing a result of the interaction with the user interface on the display screen.

18. A method as in claim 17, further comprising processing the images generated by the camera to detect a potential for a collision with an object that is present in the environment of the user.

19. A method as in claim 18, and responsive to detecting a potential for a collision, generating a warning to the user.

20. A method as in claim 19, where the warning is generated by one or more of a visual warning, an audio warning and a tactile warning.

21. A method as in claim 18, where the object is a person, and responsive to detecting a potential for a collision, generating a warning to the person.

22. A method as in claim 18, where said camera comprises a zoom lens, and responsive to detecting a potential for a collision with the zoom lens in a zoomed position, generating a warning to the user.

23. A method as in claim 18, where said camera comprises a fish-eye lens, and responsive to detecting a potential for a collision with said fish-eye lens, generating a warning to the user.

24. A method as in claim 17, where said handset further comprises a context sensor, and responsive to said context sensor indicating that said handset is in motion, and responsive to the user entering data via said data entry device, automatically initiating the display of the image representative of at least a portion of the environment of the user.

25. A method as in claim 18, where said handset further comprises a context sensor, and responsive to said context sensor indicating that said handset is in motion, and responsive to the user entering data via said data entry device, automatically initiating the detection of the potential for a collision with an object that is present in the environment of the user.

26. A method as in claim 17, where the displayed at least one image is displayed on the display screen as a background to data that is displayed on the display screen.

27. A method as in claim 17, where the displayed at least one image is displayed on the display screen in a scaled format in a window in conjunction with data that is displayed on the display screen.

28. A method as in claim 17, where the displayed at least one image is displayed on the display screen as a reduced color version of an original image as a background to data that is displayed on the display screen.

29. A method as in claim 17, where the displayed at least one image is displayed on the display screen as a simplified version of an original image as a background to data that is displayed on the display screen.

30. A method as in claim 17, where the displayed at least one image is displayed on the display screen as a reduced opacity version of an original image as a background to data that is displayed on the display screen.

31. A method as in claim 17, where the displayed at least one image is displayed on the display screen so as to enhance a contrast between the displayed image and data that is displayed over the displayed image on the display screen.

32. A method as in claim 17, where the handset further comprises a radio frequency transceiver and functions as a cellular telephone.

33. A method as in claim 17, further comprising displaying a keyboard aid to the user.

34. A method to operate a wireless communication device having a camera and a user interface that comprises a data entry device and a visual display device, comprising:

generating images of at least a portion of an environment of the handset as seen through the camera; and

processing the images generated by the camera to detect a potential for a collision with an object that is present in the environment.

35. A method as in claim 34, further comprising displaying to a user at least one image that is representative of at least a portion of the environment during a time that the user is interacting with the user interface to enter data into the device.

36. A method as in claim 34 where said device comprises a context sensor, and responsive to said context sensor indicating that said device is in motion, and responsive to a user entering data via said data entry device, automatically initiating the detection of the potential for a collision with an object that is present in the environment.

37. A method as in claim 35, where said device comprises a context sensor, and responsive to said context sensor indicating that said device is in motion, and responsive to the user entering data via said data entry device, automatically initiating the display of the at least one image.

38. A method as in claim 35, where the displayed at least one image is displayed as one of a background to data that is displayed or in a window in conjunction with data that is displayed.

39. A method as in claim 34, and responsive to detecting a potential for a collision, generating a warning to the user by at least one of a visual warning, an audio warning and a tactile warning.

40. A method as in claim 34, where the object is a person, and responsive to detecting a potential for a collision, generating a warning to the person.

41. A wireless communication device comprising one of a camera or a link to a camera and a user interface that comprises a data entry device and a visual display device, said camera generating images of at least a portion of an environment of the handset as seen through the camera and said handset further comprising a controller operating under the direction of a stored program to process the images generated by the camera to detect a potential for a collision with an object that is present in the environment.

42. A device as in claim 41, where said controller further operates under the direction of the stored program to display to a user at least one image that is representative of at least a portion of the environment during a time that the user is interacting with the user interface to enter data into the device.

43. A device as in claim 42, where the data that is entered is not directly related to a camera function.

44. A device as in claim 41, where said controller further operates under the direction of the stored program to display to a user at least one image that is representative of at least a portion of the environment during a time that the user is interacting with the user interface to read data from the visual display device.

45. A device as in claim 44, where the data that is read is not directly related to a camera function.

46. A device as in claim 41 further comprising a context sensor, where said controller further operates under the direction of the stored program to be responsive to said context sensor indicating that said device is in motion, and to be responsive to a user entering data via said data entry device, to automatically initiate the detection of the potential for a collision with an object that is present in the environment.

47. A device as in claim 42 further comprising a context sensor, where said controller further operates under the direction of the stored program to be responsive to said context sensor indicating that said device is in motion, and to be responsive to a user entering data via said data entry device, to automatically initiate the display of the at least one image.

48. A device as in claim 41, where said controller further operates under the direction of the stored program to display the image on said visual display device as one of a background to data that is displayed or in a window in conjunction with data that is displayed.

49. A device as in claim 41, where said controller further operates under the direction of the stored program to be responsive to detecting a potential for a collision to generate a warning to the user by at least one of a visual warning, an audio warning and a tactile warning.

50. A device as in claim 41, where the object is a person, and where said controller further operates under the direction of the stored program to be responsive to detecting a potential for a collision to generate a warning to the person.