WLAN and SPI: going native

[Werner Almesberger]

It's been a while since my last update. I have some good news and some
bad news. First, let me show an overview of the stack that handles the
communication between the AR6k driver and the AR6k SDIO module. Each
line represents a possible (compile-time) configuration of the stack:

AR6k(Atheros) -> HIF(Atheros) -> SDIO(Atheros) -> S3C SDIO(Samuel)
              -> HIF(Werner) -> SDIO(Linux) -> MMC-SPI(Linux) -> S3C SPI GPIO
							      -> S3C SPI (poll)
                                                              -> S3C SPI (int)
                                                              -> S3C SPI (dma)
					    -> S3C MMC/SD/SDIO
	      -> HIF(?) -> G_SPI ...

What the Openmoko kernel is using at the moment is the configuration
in the first line: The AR6k driver from Atheros connected to the
Atheros SDIO stack, and then the S3C SDIO driver Samuel Ortiz wrote
for Openmoko.

We're trying to find out if we could also attach WLAN via SPI instead
of SDIO, and - more importantly - how we can use the Linux mainline
SDIO stack instead of Atheros' stack (which would be unsuitable for
mainline inclusion).

A while ago, I succeeded in replacing the HIF interface between the
AR6k driver and SDIO with a version that connects to the Linux SDIO
stack instead of the Atheros stack, using SD(IO) SPI mode, and Ben
Dooks' SPI GPIO driver to talk to the WLAN chip [1].


The next step, using the SPI hardware in the S3C (using Ben Dooks'
driver), turned out to be much harder than expected. I found the
following three problems:

- when enabling the SPI hardware, it generates a stray interrupt that
  confused the driver's internal synchronization and made it overrun
  the SPI hardware. This is already fixed in upstream, but I missed
  the change when comparing the diffs :-(

- SPI always sends and receives one data unit (e.g., a byte) at the
  same time. The S3C's SPI hardware seems to have a flaw in its receive
  buffer than makes it deliver the received byte only after the next
  byte has been transmitted.

  I'm not sure if this happens only in the 2442 or if other S3C are
  affected as well.

  The S3C SPI driver assumes that the nth byte received is available
  after the nth byte has been transmitted, so it shifts the receive
  buffer by one byte. This is okay for many SDIO commands, but it
  causes the kernel to miss the response token after commands which
  are followed by a data block.

  A tentative fix is in [2].

- This allows me to use the native SPI driver for SDIO in SPI mode up
  an SPI clock rate of 12MHz. At speeds of 13MHz or above, the S3C
  SPI hardware seems to sample the MISO signal a bit too early, and
  byte sequences sent by the WLAN module thus appear to be shifted by
  one bit to the left.

  I'm not entirely sure if this is just an effect of my hardware setup
  or if it's a real bug in the 2442. However, on the scope, the timing
  of MISO seems impeccable. Changing SPI mode from 0 to 3 and removing
  the probes (capacitative load) did not change the results, i.e., it
  works at 12MHz but not at 13MHz.

  There's another mystery, and that's that I think I actually had SPI
  run at 25MHz without ill effects, but that was in interrupt mode,
  not poll mode (see below).

Using the S3C's SPI hardware to communicate with WLAN requires a
connection from the SPI0 port (which is available on test points near
the debug connector) to the connector of the WLAN module, which can
be accessed on test points under the LCM. This connection can be easily
made with a ribbon cable, as shown in [3]. I also wrote a script that
tests the connections [4], using the gpio utility [5].

When making this rework, the GPIOs also need to be reconfigured such
that SDIO and SPI don't end up driving the now shared lines. This is
accomplished by the patch in [6]. (If anyone wants to reproduce the
setup, there are some more patches you need. Please look at the whole
set in http://svn.openmoko.org/developers/werner/wlan-spi/patches/)

This was with the S3C SPI driver operating in interrupt mode. I
obtained a throughput of 100-110kB/s using SCP over a WEP-encrypted
link to my access point. This is much better than the ~20kB/s I got
with bit-banging S3C SPI GPIO, but still well below the ~1.2MB/s we
should be able to obtain over SDIO SPI at 12MHz.

A look at what's happening on the wire [7] (from the bottom up, we see
clock, MOSI, MISO, interrupt, SS, and the unused DATA2) makes it clear
why this is so slow: after each byte, the S3C SPI hardware generates an
interrupt, and transmission stops until the CPU handles that interrupt
and puts the next byte into the transmit register. This causes enormous
delays between bytes.

Rewriting the driver to use poll mode was easy [8], and the result
looks much friendlier [9]. Since we're now approaching speeds where the
CPU overhead of SCP makes itself felt, I switched to good old ttcp as
the testing tool. In interrupt mode, I now obtained about 190 MB/s
transferring data (that was immediately discarded). Poll mode delivered
a much more satisfying ~385kB/s. (*)

Can we go faster ? But of course. First, the driver could use DMA mode,
which gives the CPU time to do more useful things than busy-waiting
until the SPI hardware is done shifting bits. Since DMA mode has a
relatively high setup cost and also includes an interrupt, I wouldn't
expect it to yield a dramatic speed increase, but it should still make
things work a little better.

Second, the Linux SDIO stack uses a phantastically inefficient method
for detecting interrupts from the SDIO device: a kernel thread
periodically issues an SDIO command to read the device's interrupt
register and then calls the interrupt handlers according to the result.

This may make sense with SDIO controllers that don't properly support
the somewhat twisted SDIO protocol for signaling an interrupt, but in
SPI mode, we have a dedicated interrupt line for just this purpose ...

What's left ? SDIO in SPI mode is of no direct use for GTA02, so I'll
now look into making this work with the S3C MMC driver. I had already
tried the S3C MMC driver at the very beginning, but it didn't work.
Ben Dooks posted a number of S3C MMC patches at the end of last week,
and Christer Weinigel has a lot more improvement, so there's hope that
there is a working driver amidst all this :-)

Returning to the protocol stack variations I drew at the beginning of
this posting, there's still another variant left: SPI without an SDIO
stack. Atheros call this G_SPI. This is even a little more obscure
than SDIO in SPI mode at the moment, so we're not exploring into this
direction for now.

[1] http://lists.openmoko.org/pipermail/openmoko-kernel/2008-September/004955.html
[2] http://svn.openmoko.org/developers/werner/wlan-spi/patches/s3c-spi-fix-int-mode.patch
[3] http://svn.openmoko.org/developers/werner/wlan-spi/gta02-spi/rework.pdf
[4] http://svn.openmoko.org/developers/werner/wlan-spi/gta02-spi/verify.sh
[5] http://svn.openmoko.org/trunk/src/target/gpio/
[6] http://svn.openmoko.org/developers/werner/wlan-spi/patches/gta02-mmc-spi.patch
[7] http://people.openmoko.org/werner/wlan-spi/spi-int.png
[8] http://svn.openmoko.org/developers/werner/wlan-spi/patches/s3c-spi-from-int-to-poll.patch
[9] http://people.openmoko.org/werner/wlan-spi/spi-poll.png

(*) I didn't examine the maximum throughput possible in my test setup.
    So it may well be that access point or the choice of channel also
    contribute to limiting the data rate.

- Werner