www/ Accessibility/ Input

Use cases

Users who can only use a keyboard

Need keyboard shortcuts for all actions in applications.

Need to be able to drive the pointer for positioning actions

  • -> MouseKeys, currently implemented in mutter,

Users who can only use a joystick

Need to be able to drive the pointer

Users who can only drive the pointer, or just use a button

Need to use a virtual keyboard.

  • -> virtual keyboard application needs to be able to inject keyboard events

Users who can only move an eye

Need to use eye-tracking, e.g. with http://viacam.org , which needs to drive the pointer.

It also needs to detect when the user is using the traditional mouse, to disable the tracker when somebody else is assisting the user. Otherwise we can easily get stuck with eye-tracking and not be able to control the computer any more.

  • -> needs to be able to emit events for a virtual mouse (absolute positioning)
  • -> needs to be able to know when events happen from another mouse.

Users who can not afford any of these, but can speak

Need to use speech recognition.

Triggered actions should be keypresses, but also high-level actions, to avoid having to know and spell the keypresses to use to achieve the action, i.e. "copy", "paste", not "control-c", "control-v".

The actions can also be selecting e.g. an icon by its name.

  • -> needs to be able to emit keyboard events. high-level actions should probably go through at-spi.

Selecting a region

shouldn't require using a mouse. Mark & point method needs to be available.

  • should probably go through at-spi.

Half-keyboard

For people with one hand, using a toggle to "mirror" the keyboard layout. Some implementations use xkb: http://blog.xkcd.com/2007/08/14/mirrorboard-a-one-handed-keyboard-layout-for-the-lazy/ https://github.com/jorissteyn/xkb-halfqwerty, others use extensions: http://kieranbingham.co.uk/implementing-xhk/ , https://github.com/kbingham/xhk. There is also a kernel implementation https://github.com/eklhad/acsint. In principle this would be at the physical level, but depending on the layout details may vary. Also http://www.onehandkeyboard.org/linux-one-handed-keyboards/. A generic implementation would already be useful, though.

  • Could be implemented like AccessX

AccessX

Done in Xorg

Done in mutter

  • could be moved to a separate library to make it shared with other compositors?

  • StickyKeys: modifiers are sticky so the user does not have to be able to press several keys at the same time

  • MouseKeys: mouse can be driven from the keyboard
  • SlowKeys: only keys that are held for a long time produce a key event.
  • RepeatKeys: the repeat delay can be set to a very long time.
  • ToggleKeys: makes an audible feedback when a toggle (capslock, numlock) is enabled (beep once) or disabled (beep twice)

  • BounceKeys: ignores rapid, repeated key events for the same key

  • -> Basically, these intercept keyboard events inside the compositor, and deliver (possibly other) events for virtual keyboard/mouse devices (with relative positioning)

The user uses shortcuts to tell the screen reader specific actions (read the current line, the current character, spell the current word, detail the attributes of the text, get the content of the status bar, change feedback verbosity level, navigate the interface with global view, etc.)

To avoid interfering with usual shortcuts in some situations, some of these are done with a "screen reader" key that needs to be pressed like a modifier. Typically, insert or capslock are used. For instance, one types insert-v to change the verbosity level. Typing that "screen reader" key alone, or (if required technically) double-typing it, should however allow to achieve the original behavior.

  • Currently, this snooping is done within the toolkits, but this is getting pushed out, see https://gitlab.gnome.org/GNOME/gtk/issues/1739
  • This is a sensitive question because we don't want to allow keyloggers.
  • We do not want to include the screen reader within the compositor, at least just because it's written in Python.
  • We do not want to have to wait, for each keypress, for the screen reader to tell whether it wants it or not.
  • The compositor could expose an API for the screen reader to register the shortcuts it is interested in.
  • the screen reader key, in the end, should act either as a modifier (and thus be affected by StickyKeys for instance) to trigger screen reader shortcuts, or as a normal key (if it's pressed alone).

key typing feedback

Users needs to have audio feedback of what they are typing, i.e. not the physical key, but the label of the key that was pressed.

  • -> the screen reader needs to get a notification (can be async) of all keypresses
  • Same remarks as above for security. This event can be always async
  • -> The compositor could expose an API for the screen reader to get notifications of keypresses.

In the case of the Compose key, the resulting output should be spoken.

  • Only the toolkit really knows what composition rule was used,
  • -> so this (async) notification needs to be done from the toolkit, to provide the eventual string.

In the case of input methods, the resulting output should be spoken as well.

  • The input method and the toolkit know the eventual string.
  • -> It could be either of them to notify of the result.

Mouse review

User needs feedback on what is under the pointer, to be sure what s⋅he clicks on. The screen reader thus either needs the coordinates of the pointer to determine what widget is at these coordinates, or hover feedback from the widget itself.

  • Only the compositor knows where windows are and the absolute pointer position
  • toolkits only provide the relative positions of widgets within the window.
  • -> The compositor could expose an API for the screen reader to get notifications of mouse movement, over which window it currently is, and the position of the window.

Touchscreen review

User would use a touchscreen in review mode: while moving the finger around the screen, the screen reader says what the finger is on, and only double-taps makes an actual pointer click, as well as other kinds of gestures.

  • -> Perhaps this should go in libinput?
  • -> Or be handled within the compositor, to transform touchscreen events into normal events for a virtual mouse.
  • Then the screen reader can get movements just like with a normal mouse, to perform the review
  • It could however probably be useful to trigger the screen reader review mode when the touchscreen is getting used instead of the normal mouse.

Braille typing

Some Braille output devices have braille keys, numbered from dot 1 to dot 8. One would for instance type dot 1 to type "a", which is sent by the device to the computer as "a" (not as dot1), which needs to be interpreted as such by the application. Pressing dot 1 and dot 2 at the same time produces "b", which is sent as such by the device, and so one. This is delivered through the brlapi protocol.

  • -> either compositor or libinput could have a libbrlapi driver.
  • -> need to use the text protocol to deliver the text to applications.

Some people can not afford a Braille output device with such keys, and would use the keyboard to type Braille, using keys 'a', 's', 'd', 'f', 'j', 'k', 'l', ';' in a chord. For instance, the 'f' keyboard key is dot 1, and thus typing it alone produces "a". The 'd' keyboard key is dot 2, and thus typing both 'f' and 'd' at the same time produces "b".

Some people have only one hand, and thus need to type Braille pattern in two halves, typically typing dots 1237 first then 4568.

Keyboard shortcuts are for instance expressed through prefixes, for instance control is expressed by typing dot7+dot8+dot10 and then the letter (e.g. dot1+dot4 for c, thus control-c)

Braille contains abreviations. For instance, dot1 ("a") followed by dot1+dot2 ("b") is a contraction for "about". Typing dot1 alone then dot1 and dot2 together, then a word separator character, should expand the contraction to produce "about".

We want the screen reader to be able to provide feedback (optionally "a" and "b", and always "about")

Typing includes various kinds of cases:

  • simple press of physical keys, possibly with extra modifiers
  • press of key symbols: "a", "A", "À", possibly with extra modifiers

  • suite of Braille patterns to be uncontracted

  • Some of this can probably be implemented as an input method.

  • But some of this can not (e.g. typing keyboard shortcut such as control-c)

One way to see it is with several layers of translation, and depending on the kind of typing (PC keyboard, Braille keyboard with separate key events, Braille keyboard with whole chord, Braille keyboard providing directly the translated letter):

  • Pure Braille key presses, showing up as 0xfff1-8 braille dot keysyms.
  • Some processing turns this into braille pattern keysyms (currently handled in libX11, but could be around libinput), possibly with handling typing Braille by halves
  • Some processing turns some of this into keysyms (for keyboard shortcuts) to be delivered to applications
  • Others are passed a such to applications, to be translated by an input method.

Global positioning

The user uses e.g. a touchpad as a global positioning device within a virtual field, while another device provides tactile feedback of the virtual field. For instance, allows to get feedback on the frames of the windows of the desktop.

  • -> be able to get absolute coordinates from a touchpad, and prevent it from moving the pointer.
  • can't really stuff it all completely in the compositor (has to interact with at-spi etc.)

Touchpad gestures

It should be possible to make touchpad gestures act as key presses (e.g. arrows) or high-level actions

  • -> be able to interpret gestures and prevent the touch pad from moving the pointer.
  • be able to emit key presses
  • can't really stuff it all completely in the compositor (has to interact with at-spi for the high-level actions)

To be continued...

Examples of ways to achieve them

not necessarily what we can afford in the end, here just for the record !!

Mouse event injection

Orca needs this for clicking in flat review mode

See https://bugzilla.gnome.org/show_bug.cgi?id=709999

Mouse event feedback

See https://bugzilla.gnome.org/show_bug.cgi?id=710012

Absolute mouse coordinates event

Orca needs this for flat review

See https://bugzilla.gnome.org/show_bug.cgi?id=710012

Key injection

We need to be able to inject both untranslated key presses and translated keypresses, as some specific devices or handling happen to provide both of them.

  • keycode injection
    • Available in Xorg through XTest
  • keysym injection
    • Not available in Xorg, have to backtranslate.
    • Could be done by adding EV_KEYSYM and EV_UNICODE to evdev layer.

Wayland: this is completely missing apparently?

See https://gitlab.gnome.org/GNOME/mutter/issues/9

A bridge for dogtail was implemented on https://gitlab.gnome.org/ofourdan/gnome-ponytail-daemon but this is more of a hack

A macro daemon is currently in development on https://gitlab.freedesktop.org/whot/macrod/

A library for compositors is being developped on https://gitlab.freedesktop.org/libinput/libei

input stealing

We need to be able to steal input event, for instance:

  • Orca wants to eat all presses/releases of its "orca key" (insert or capslock, depending on the configuration), to be able to use it for its own shortcuts
  • Orca also wants to eat the shortcuts it uses for navigation etc.
  • feedback applications want to eat all mouse events from a given touchpad, getting absolute coordinates from it, to use it as a positioning device.

A macro daemon is currently in development on https://gitlab.freedesktop.org/whot/macrod/

PC braille keyboard