So last week I posted on lkml an old patch that we were carrying for a long time in the Linux community. It basically brings the multiple (old) video cards functionally again on Linux and X server (and this time doing on the right and beauty way). For the people that was following multiseat implementations, this is a HUGE step: we will finally be able to discard the old and ugly hack (a mix of Xorg, several Xephyr servers + evdev) and and go to a clean way, starting multiple X servers in parallel. Cool! Well, not that much, because it might take some time to be in your beloved distribution :)

It’s too early and I don’t know if it’s recommended to say this, but if you want to give a try basically you have to get all X components, this X server patches, my libpciaccess and Dave’s kernel patchset. Again: it’s a very unstable work!

If you’re concerned with the technical explanations then you can follow the nice memo that Dave wrote about this.

Unfortunately that model which I described some weeks ago to put the input event delivery of the X server in a separate thread wouldn’t be an advantage. I precipitated myself thinking that it could be feasible. Sorry :(

I started to implement all this but it showed a very boring task to grab all the globals variables which both threads touch and to lock it. So I decided to stop going in this way. It’s hard to program thinking in parallel. It’s even harder to debug a program with severals flows. More, the tools don’t help you (if you have lucky, gdb will work).

But the main reason I can argue to stop with this model is that the “main” event flow of execution (i.e. basically all the functions in {Swapped,}ProcVector) and the input delivery flow (ProcessInputEvents()) are very very tied. Both deal a lot with clients and we’d need to lock several globals, thus spending a lot of time in the management of the threads. It’s easy to see this acting: just put a breakpoint in TryClientEvents(). Every single request to deliver a given event to a given client involves this function. And both input and main event flow will call TryClientEvents(). So you will see a zillion of times this function being called. The contention of the eventual processing and main threads would be even greater if the client choose to receive MotionNotify event.

So yeah, it’s far from be clear how to put processing of input events inside another thread.

== Next ==

In the next days I’ll be traveling to CESol, Fortaleza here in Brazil. I was invited to talk about my work in X land. Latin America has a lot of promising countries concerning FOSS development however for some reason no one actively participate and contribute for the X development (why?). I’ll try to motivate people there somehow :)

In the next week I’ll put the generation thread in a shape good enough to eventually push this to upstream. Also I’ll try to write a good sumary of all my work given that GSoC is in the end.

Input events routed through another thread/process can have bad effects on latency because we can’t guarantee that it will get scheduled at the right moment. Although this is hard to see happening with the current X server threaded implementation, we must design something to avoid it. One way to improve the responsiveness is to give a high priority to the input thread and also adjust the CPU scheduling. (Note that this will not avoid problems related with page faults which usually happen in the X input flow.)

Linux uses 1:1 thread model and the scheduler handles every thread as a process. For now I don’t care about others systems. Both input generation and processing threads was designed to sleep after a relatively short CPU run. So we can give a priority to processes that are trusted to not hog the CPU. And given they are special time-critical applications I have no doubt in what policy to use: I set both input threads to use the real-time FIFO policy and to get the maximum priority (sched_get_priority_max()).

—

I’m sure that someone will complain telling that this would decrease a bit the main thread when used together with both input threads. In GUI we’re talking about better user experience. Latency variability must be avoided whenever possible in interactive situations. What the user see is what matters. For non-interactive processes (server scheduling workloads) the situation is totally different.

Xorg’s philosophy is to be portable so we have to take care when setting this kind of parameters. It is a complex issue and different systems do it in wildly different ways. I was using my Linux box (2.6.24) to design it all.

Given that GSoC ‘08 is getting close to the end, strategy number 2 showed more feasible to proceed my work. Strategy #3 would be a lot of fun but would imply a hell massive codification as well (also a little out of our scope). Unfortunately no-no for now.

GSoC summary #1 – July 29

The current implementation of X Window System relies in a signal scheme to manage the input event coming from hardware devices. This scheme frequently get blocked when lot of IO is occurring (for instance, when the process is swapping in/out). Get blocked means for instance a jumping cursor on the screen and in GUI is always desirable to prioritize the system responsiveness for end users. The human/computer interface should be smooth and this is the most user visible aspect of a system.

Besides the need for improvement in system responsiveness, the current design of the event stream has some oddities, probably due historical reasons, such as the cursor update done in user-space or the huge path that takes to draw the cursor instead just connect the mouse hardware directly with the cursor position update in-kernel. Moreover there is no fundamental reason to input drivers be dependent of DDX part of the X server. Therefore a design of the input subsystem must be carefully redone to improve such issues.

Our project try to solve all this problems. In summary the goal is: to get a path from hardware input event to client delivery that cannot be blocked by rendering or IO operations, meaning we always have very low latency on input events. Moreover, a redesign of such event stream could improve the overall X graphics stack, which must be considered as well.

So far three strategies were explored to achieve the goal:

1. put X input generation stage in a separate thread

2. put X input generation and processing stages others threads

3. shortcut the kernel input layer with drm to decrease the cursor update latency

Basically 1. and 2. tries to solve the issue of blocking signals and 3. would be a completely redesign in input infrastructure. Anyway, the 3. strategy would impact in 1. and 2. but these could be implemented in parallel with the third strategy. The following sections details each strategy.

== strategy #1 ==

Strategy 1 does not uses a signal handler anymore to wake up the event generation code. It simply poll for device’s socket and giving that this code is under a separate thread this is a win for the CPUs.

With the separate thread taking care only the input code, it was expected that the cursor footprint always lived on resident memory when the mouse stills in movement. Unfortunately this was not true. For some reason it swaps back to disk. Maybe some scheduler adjusts would help here. A memory lock scheme was tried to do lock the cursor footprint always in physical memory without success.

This strategy is basically what we’ve been done is the first GSoC. This is pretty much implemented. It would not require much trouble to push it to X server from upstream. The code is here:
http://cgit.freedesktop.org/~vignatti/xserver/

== strategy #2 ==

This strategy can be thought as an improvement of #1. It can be separated in two models of implementation:

Model one:

thread #1 deals with
- injection and processing of input events
thread #2 deals with
- requests from known clients
- new client that tries to connect

It would be very very nice to let both threads totally independents. But we cannot. The event delivery depends on window structure and the first thread must always wake up the second. Also, sometimes the processing of events take a while and the injection of events stays stucked in this model. So we came with this another:

Model two:

thread #1 deals with
- injection of input events from devices
thread #2 deals with
- processing of input events to clients
thread #3 deals with
- requests from known clients
- new client that tries to connect

With this model the first and the second thread become not so tied and given that we’re using non blocking fds to wake up each thread (through a pipe), CPU “enjoys” the effect of threads. For instance, under heavy drawing primitives only thread #3 would wake up.

We had a proof-of-concept of this last model and it workish (occasionally seeing some segfaults probably due of some critical regions we forgot to lock – now the only mutex that exists is inside the server queue of events).

It’s hard to imagine other threaded models mainly because the way X deals with clients are very tied in every piece of the server and it would require a lot of mutexes.

== strategy #3 ==

For sure this strategy is the most shocking one :) The idea is to connect the mouse hardware directly to the cursor position update function, all inside kernel. We’d then rewrite the event stream from the pointer device to an absolute position. Transform the relative mouse motion into an absolute screen position seems to be not that complicated, but this strategy would involve acceleration and cursor limits inside kernel as well (the current implementation of accel deals with floats, so we would have to adapt it to live in kernel).

It is a _very_ _large_ amount of codification. It would require changes to the X server, DDX driver and its corresponding kernel DRM drivers, drm library and kernel input drivers. A mini-input driver *inside* drm is also needed. We would add complexities of the connection between input device and output device to the kernel (in my proof-of-concept implementation evdev is dependent of drm. Yeah, really weird world). Moreover, we would have to avoid somehow two differents sets of the exact same code in different contexts in the case of sw cursors (think MPX). It’s a completely redesign. Things would have to go incrementally.

But why this strategy? Well, this would solve all the current issues with input latency. For instance with the current design of the kernel modesetting – which seems the future – the cursor is jumping a lot, much more than with current implementation. Try to call a xrandr instance and move the mouse with kernel modesetting. xrandr will do DDC communication which will blocked X in the kernel. So with the handling and update of the cursor inside the kernel all would work fine (and my proof-of-concept already showed this).

Moreover, I believe the current implementation remained until now due historical reasons. Ultrix systems placed the entire input subsystem in the kernel. What is the problem to do this in Linux (and others) as well (besides massive codification)?

and non-dri drivers? Should we forget them?

EOF

The current DRM kernel modesetting tree is already taking care to update the cursor registers and paint it to the screen. Very cool [0].

What I’ve done today is a shortcut between the kernel input layer and DRM to update the cursor directly on screen without the X server be notified always. Of course, a lot of issues raised up together. So let’s try to delegates the tasks again.

userspace app (X server):
- starts all this mechanism telling which is the device responsible for the cursor (input ddx drv)
- responsible for loading new cursor images and push to the DRM (video ddx drv)

kernel input layer (evdev driver):
- notify and send its relative coordinates events to DRM

DRM:
- transform relative motion into absolute
- takes care the cursor limits
- responsible for the acceleration computation
- responsible for the input transformation as well?
- touch the gfx registers.

Seems that a reasonable amount of code in ddx input drv (mainly ReadInput) and dix (mainly GetPointerEvents) would be “swallowed” by the DRM. The “event generation stage” of the server would deal with the event itself + xkb + Xi things (which eventually could be done in a dedicated thread) and will let to DRM the responsibility of paint the cursor on screen.

The communication between kernel input drv can be directly, calling a DRM function; the DRM and userspace can communicate basically using ioctls. Complains?

This would only works with DRM supporting OSes. What about the others?

[0]  Not so much. Seems this method to update the cursor is sending _a lot of_ ioctls and sometimes doing cursor jumps. But I have to double check to see if the problem is for sure with context switches.

I really don’t like to read large blog posts. Anyway…

What I did so far is a separated thread that takes care only the injection stage on the X server queue. Who is interested with the results, please read some past posts in my blog. It is currently in a very good shape (synced with post-mpx merge, all input devices are inside the thread and etc). The implementation looks like this:

thread #1 deals with
- injection of input events from devices
thread #2 deals with
- processing of input events to clients
- requests from known clients (rendering things)
- new client that tries to connect (pretty easy to do)


Now I am pondering the following:
Model one:

thread #1 deals with
- injection and processing of input events
thread #2 deals with
- requests from known clients
- new client that tries to connect


It would be very very nice to let both threads totally independents. But we cannot. The event delivery depends on the window structure and the first thread must always wake up the second. Also, sometimes the processing of events take a while and the injection of events stays stucked in this model. So I came with this another:

Model two:

thread #1 deals with
- injection of input events from devices
thread #2 deals with
- processing of input events to clients
thread #3 deals with
- requests from known clients
- new client that tries to connect


With this way the first and the second thread become not so tied and given that I’m using non blocking fds to wake up each thread (through a pipe), the server “enjoys” the effect of threads. For instance, under heavy drawing primitives only thread #3 would wake up.

Well, I had implemented both models here and it workish (occasionally seeing some segfaults probably due of some critical regions I forgot to lock — now the only mutex that exists is inside the server queue of events).

It’s hard to imagine other models mainly because the way X deals with clients are very tied in every piece of the server and it would require a lot of mutexes. But I’d love to hear anyone disagreeing with this and proposing something here!

=== What make things so hard ===

Debug a multi-thread program is really tedious, consequentially work in this kind of environment as well. And that could be a great argument to not accept an X server multi-threaded in upstream some day.

The thing that most pissing me off is how gdb sucks to debug the server. I’m using the last version of gdb (6.8 version and from today’s snapshot) and it often hard lock my machine (I’m sure I had to reboot my machine a least 20 times today…sigh.) Sometimes it consumes 99% of CPU and sometimes it completely hangs my machine. Yes, this is really boring. Anyone knows some magic parameters in gdb to avoid this hard lock up? Anyway, the goal for long term is to let an option — at compilation time — to not use the input thread.

Comments?

I was thinking how we could make remotely X clients totally connective with the server when both are behind a NAT/firewall.

We can imagine one big motivation to do this: a scenario where someone using his thin and poor machine wants to use the resources of some “fat” machines which he simply doesn’t know where they are seated. Those fat machines could be arranged through a P2P network of “X11 pool of resources” and the list of machines displayed to the user select his desired one (e.g. with minor lag/load). Someone more capitalist than me could go further and imagine a provider selling X11 resources to mobile devices. Or just open your home machine’s web browser in any place of the world. Well, the field of applications would be huge.

Maybe this would be a kick in the a** of the so called web-based applications. Now you have to learn how to program in html, php, ajax or another boring language to build something in this kind of environment. Instead, we could see this P2P network also as a web-based environment letting people to program apps in their preferred toolkit (and yes, I’m betting that the data transfer rates in the various networks will increase significantly in the next years).

- How?

There’s a technique to create TCP connections between machines behind NAT/firewall called “hole punching’ [0, 1]. With the help of a server (called rendezvous) two machines behind NAT/firewall open a ‘hole’ in their NAT/firewall to establish a connection. A similar technique — using UDP — is what Skype and others are relaying today.

Someone already mentioned [2] to wrap X11 protocol with jabber (or other protocol). It would be also an idea. But I don’t know how much the latency would increase in this case.

Certainly there are others techniques to achieve the X11 end-to-end transparency among machines that I’m missing.

- Why ssh -X and xfwp is not the way?

Simply because it doesn’t let end-to-end connectivity.

I didn’t thought what would be the impact in both client and server side. Probably all this will touch some aspects of authentication and
security. I don’t know.

So I would like to hear from you what do you think about this all (and eventually find a mentor to apply this in GSoC ;) )

Cheers,

[0]

http://www.usenix.org/event/imc05/tech/full_papers/guha/guha_html/index.html

[1] http://www.bford.info/pub/net/p2pnat/
[2] http://butterfeet.org/?p=23