Harmonizing a projected user interface

• US 10,122,978 B2
• Filed: 04/01/2014
• Issued: 11/06/2018
• Est. Priority Date: 04/01/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Mobile Communications Incorporated (Sony Group Corp.)
• Inventors: De Leon, David, Thorn, Ola, Midholt, Magnus
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Mathews, Crystal A
• Application Number:
  US14/428,092
• Publication Number:
  US 20160261835A1
• Time in Patent Office: 1,680 Days
• CPC Class Codes: G01N 2291/02827 Elastic parameters, strength or force
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: H04N 9/3182 Colour adjustment, e.g. white balance, shading or gamut white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present disclosure provides a projection device for adapting a property of a projected user interface. The device determines at least one property of a surface onto which a user interface is projected. The property of the projected user interface is changed based on the at least one property of the surface onto which the projected user interface is projected.

Claims

1. A method for adapting a property of an interactive projected user interface based on at least one property of a surface on to which the projected user interface is projected, the method comprising:

determining the at least one property of the surface;

determining a relevant user interface animation based on the determined at least one property of the surface, wherein the determined relevant user interface animation appears to match a physics of the surface; and

applying the determined relevant user interface animation based on the determined at least one property of the surface, such that the updating of the projected user interface in response to the user interaction with the elements of the projected user interface is changed according to the determined user interface animation.

2. The method of claim 1, wherein the at least one property of the surface is at least one of the coloring of the surface, the texture of the surface, the deformability of the surface, the orientation of the surface, the hardness of the surface, or the topography of the surface.

3. The method of claim 2, wherein, if the surface is determined to be deformable, the projected user interface is updated to increase the size of elements of the projected user interface.

4. The method of claim 1, wherein the at least one property of the surface is measured during user interaction with the projected user interface.

5. The method of claim 1, wherein the projected user interface is updated such that sound effects of the projected user interface appear to match the at least one property of the surface.

6. The method of claim 1, wherein the determined relevant user interface setting is applied such that the projected user interface appears substantially homogeneous across color variations of the surface or such that the contrast between the projected user interface and the surface is approximately constant across the surface despite color variations of the surface.

7. The method of claim 1, wherein the at least one property of the surface is determined using a camera.

8. The method of claim 7, wherein the camera measures at least one of visual distortion of the projected user interface during user input to determine deformability of the surface or coloring of the surface.

9. The method of claim 8, wherein the camera measures visual distortion of the projected user interface during user input by analyzing deformation of the projected user interface during user interaction with the projected user interface.

10. The method of claim 1, wherein the at least one property of the surface is determined using an acoustic sensor and the at least one property of the surface is determined based on the sound generated by the user interacting with the surface.

11. The method of claim 10, wherein the at least one property of the surface measured using the acoustic sensor is at least one of a type of material of the surface or a hardness of the surface.

12. The method of claim 1, wherein the at least one property of the surface is determined using a user worn device.

13. The method of claim 12, wherein the user worn device includes at least one of an accelerometer or a gyroscope configured to capture measurements of user movements during interaction with the projected user interface.

14. The method of claim 13, wherein a processor analyzes the captured measurements to determine at least one of the hardness of the surface or the deformability of the surface.

15. The method of claim 1, wherein the at least one property of the surface is determined based on geo-location coordinates of the projected user interface.

16. The method of claim 15, wherein the geo-location coordinates of the projected user interface are determined using at least one of near field communication, Bluetooth, Geo fencing, indoor Wi-Fi positioning, or GPS.

17. The method of claim 15, wherein the user manually selects the relevant user interface setting for the surface and a current location of the user is tagged such that the selected relevant user interface setting are automatically selected to be the relevant user interface setting for user interfaces projected at the current location.

18. The method of claim 1, wherein the determined at least one property of the surface includes the deformability of the surface.

19. The method of claim 1, wherein the determined at least one property of the surface includes the hardness of the surface.

20. A method for adapting a property of a projected user interface based on at least one property of a surface on to which the projected user interface is projected, the method comprising:

determining the at least one property of the surface;

determining a relevant user interface setting based on the determined at least one property of the surface; and

applying the determined relevant user interface setting to the projected user interface, such that the property of the projected user interface is changed according to the determined user interface setting,

wherein the projected user interface is updated such that at least one of animations or sound effects of the projected user interface appear to match the at least one property of the surface, wherein;

on a less deformable surface, a pressed button results in a hard surface animation; and

on a more deformable surface, a pressed button results in a soft surface animation.

21. The method of claim 20, wherein at least one of:

the hard surface animation results in an element of the user interface bouncing higher than the soft surface animation;

or

the hard surface animation is a hard scrolling animation that behaves differently than a soft scrolling animation by at least one of responding quicker, scrolling faster, or scrolling for a longer time.

22. A projecting device for projecting an interactive user interface onto a surface and for adapting a property of the projected user interface based on at least one property of the surface, the device comprising:

a projector configured to project the user interface onto the surface;

a sensor configured to determine the at least one property of the surface; and

a processor configured to;

determine a relevant user interface animation based on the determined at least one property of the surface, wherein the determined relevant user interface animation appears to match a physics of the surface; and

control the user interface projected by the projector such that the updating of the projected user interface in response to the user interaction with the elements of the projected user interface is applied according to the determined user interface animation based on the determined at least one property of the surface.

23. A system for projecting a user interface onto a surface and for adapting a property of the projected user interface based on at least one property of the surface, the system comprising:

the projecting device of claim 22; and

a computing device in communication with the projecting device.

24. The system of claim 23, wherein the computing device includes the projecting device.