More progress on the Teensy 3 keyboard

[dawnmist]

I've been doing a lot of work again the last month on my upgrade from using a Teensy 2++ with monochrome screen to a Teensy 3 with Colour TFT + SD Card + Touch, and it's now coming together nicely.

Data Transfers, SD Card access and a Java-based GUI

A while ago, I was trying to work out how to get access to the filesystem on the SD Card when I wanted to upload new logos for display, or to transfer an updated keymap configuration file. I could always just turn off the keyboard, unplug the SD Card, plug it into the computer, transfer files, then plug it back into the keyboard and plug the keyboard in - but the keyboard was already connected to the computer by a USB cable so it would be far nicer to not have to fiddle about like that. I considered things like providing SD Card access by adding it as a mass storage device...but this had various issues. When the computer OS mounts a drive as a mass storage device, it takes sole ownership of the filesystem on the drive...which would mean that the Teensy itself would no longer be abe to access it. If you tried to change keymaps at that point, you wouldn't be able to load the image file for the keymap, or save any changes to the keymap config back to the SD Card. Nor could the Teensy detect any changes you made while the SD Card was mounted by the computer's OS. It's the same issue as seen on mobile phones - and the reason behind things like MTP and PTP access to drives - it allows the "host" system (in this case, the Teensy or the phone) to continue to use the drive while allowing the computer to send updates to the host to handle and receive updates from the host.

For the Teensy 2, Paul Stoffregen (the person who designed and sells the Teensy) had created a "Keyboard + USB disk" type that the Teensy 2 could be programmed as. This seemed to allow the Teensy to still access the drive by using interupts to lock the computer out of making changes for the period of time that the Teensy needed access to the filesystem. However, I couldn't follow the code itself (went over my head), and I couldn't find much by way of documentation for how to use it, which meant I never really understood what it was doing. However, the Teensy 3 still being pretty new, this USB configuration hadn't been written/converted for use on the Teensy 3 yet. Between it not being there and me not understanding how it worked on the Teensy 2 and so having no idea what would be needed to port it across, it also wasn't an option.

The next thing I did was to take a look at the PTP/MTP filesystem access stuff. I mean, there's lots of devices out there that use it (though linux support for it is still pretty patchy), so there might be something I could use as a reference - or if I was really lucky, a library available for use. On looking at the documentation, these turned out to be fairly complex and hefty protocols - and I suspect somewhat beyond what I could make fit in the flash/ram of the Teensy 3, as well as being somewhat excessive for what I really needed. When all came down to it, what I needed was some way to query the Teensy for directory listings, and a way to transfer files to/from the teensy. Even better if I could include rename/delete of files/directories too.

I already had a Java-based GUI for editing the keymap files, which was mainly to make it easier to manage things like making sure that keymap indexes were consecutive (as if they're not, the Teensy will not load them properly - the index number is actually used as the section name in the config file). I'd worked out one serial command to set the time on the real time clock on the Teensy, which was sent via a terminal window. I suddenly realised that I could create a serial command interface that would send a command code followed by a specific format for that command if I could connect to the Teensy through the comm port via my Java GUI program. And while I was at it, I could set up a local config directory that was essentially a duplicate of the filesyetem on the Teensy, which gave the ability to back up the Teensy's configuration in the event that the SD Card died, etc.

I've still got a few commands I'd like to add, but I've now got set time, get time, list directory, get file, put file, rename file/dir and delete file/dir working (although I've restricted the delete directory to only permit the deletion of empty directories from the Java GUI side, as it'd be somewhat...dangerous otherwise). These operate on both the local filesystem AND (if it is currently connected) on the Teensy's filesystem as well. With these, I can show the filesystem in the GUI, and mark files that are new locally (exist on the computer but not on the keyboard), modified locally (newer timestamp on the computer), modified on the keyboard, and new on the keyboard (don't exist on the local computer). The only real issue with this is that I must install the Teensy serial inf file on any Windows machine that I want to be able to connect to the keyboard's comm port from (it just loads an already-existing Windows driver for the comm port, but it is a driver that is not otherwise loaded by default), which means the keyboard is not *quite* plug-in-and-it-just-works on Windows machines (the keyboard part will work fine, it's just the serial port comms between the Java GUI and the keyboard that won't). I can also store the jar file for the Java GUI on the Teensy's SD Card, so worst case is that on a new Windows machine you might need to remove the SD Card, grab the inf file and jar file off the card, then re-insert it into the keyboard. Linux and MAC (though MAC is untested at this stage) should be able to send a "getfile" command to retrieve the GUI jar file through a terminal window. :-D

I've also added the ability to import images from within the Java GUI. When you select a new image, it gets automatically resized to an appropriate size for display on the keyboard's TFT, has its background set to the keymap's current background if it was previously transparent, its filename adjusted to be 8.3 DOS name friendly, and saved to the local configuration directory (which is a subdirectory under ~/.config on linux, /AppData/Local on Windows, and in the application support area on MAC - though again I don't have a MAC so that bit is untested). While doing this, I had trouble for a while - some images were working fine, and some would get these odd lines across them and their colours were all screwy when displayed on the TFT screen attached to the Teensy. This seemed to only happen with the newly-imported bitmap that came from a transparent png file that got resized down to fit...it took about a week of me trying to work out what the java conversion process was doing wrong when saving this converted bitmap before I discovered that in bitmap files, the data for each row to be displayed is padded out to a multiple of 4 bytes...the files that worked previously had no padding needed, whereas the new one that didn't work I was treating the padding bytes as pixel data to be displayed (causing colour shift and the image to be slanted as the padding bytes added up). The conversion process was fine...it was the drawing process that had issues. The examples I'd seen previously didn't handle these padding bytes either - because they were dealing with fixed-sized images that didn't need them, so I hadn't realised they existed.

Getting NKRO working on the Teensy 3

NKRO = "N" Key Roll-over. This means that you can have "N" keys pressed at once time, and the keyboard reports ALL of those keys as pressed to the computer. Most keyboards have issues in their wiring such that they can only guarantee 2-key rollover (some combinations of 3 keys pressed together will lose one of the keys). Many gaming keyboards are better - but tend to be optimised around reporting multiple keys on the "wasd" left side of the keyboard (so the right hand side tends to be quite patchy/sucky - if you use a mouse left-handed means that keyboards "optimised for gaming" can sometimes actually be worse than standard ones for you!).

As well as this, in order for a computer to not have to have a full USB protocol stack in the bios (something that the new UEFI bios might be sorting out...it'll be interesting to see how that develops over time), usb keyboards that want to be bios-compatible must conform to a particular USB device definition when in "boot mode". This definition restricts the keyboard to reporting a maximum of 6 non-modifier keys pressed at one time (plus 8 modifiers - left ctrl, left shift, left alt, left win/gui key, then the right side versions of those). Bioses have been notoriously buggy though, so some of them would NOT correctly set/leave a keyboard in bios mode, so for maximum compatibility many USB keyboards stick to pretty much bios-mode-always. For a gaming keyboard where you're wanting to combine direction keys, speed modifiers, jump, dodge and skills, this 6-key-max limitation of bios-mode-keyboards can become a problem. A lot of games work around this by using the modifier keys for things like crouch/jump/etc, reducing the number of other keys that have to be pressed simultaneously, but when I started this project initially I wanted something that would give me true NKRO-over-usb. The keyboard controller is designed so that each key can be individually checked, and the Teensy reads in a series of shift registers that hold the complete state of all the keys, so the hardware fully supported NKRO. Searching through the GeekHack forums, there was a thread by Soarer that discussed his adventures in creating a PS/2 to USB converter using a Teensy that gave full NKRO (here), his compiled code for a Teensy 2 and testing in the thread here, and from there an open implementation by trk at github here. Using the github code as a reference, I originally got the NKRO working on the Teensy 2++ - so for the Teensy 3 I just had to port my Teensy 2++ code across (in the new style) to the Teensy 3's USB descriptors. After a weekend of breaking my Teensy 3 due to bugs in the descriptors, I finally got it working properly...and the Teensy 3 chip has now been the "brains" in use behind the keyboard instead of my older Teensy 2++ for the last week.

The NKRO USB descriptor is implemented as an additional keyboard interface (leaving the bios-compatible one alone) with the alternate report needed for NKRO-over-USB (it sends a 32 byte report, with the first byte being the normal modifier keys, and the other 31 bytes being a bitmap of keys from 0-(31*8 - 1). My experience with the Teensy 2++ was that Windows would understand 16 or 32 byte reports, but choked when I tried a 20 byte report (linux read those just fine!) - and to cover all the keycodes I wanted to be able to map, I needed at least 19 bytes of bitmap, hence the 32 byte report. (I never did test 24 bytes - that might also have worked, being a multiple of 8). Likewise, Windows would *not* read the bitmap report if it didn't start with the modifier byte first - so 1+15 would work, but 16 would not. I haven't gone back and retested that with the Teensy 3, mainly because it was working for what I needed at the 32-byte report. If you're not using the odder codes such as F13-F24, you may be able to drop that down to a 16 byte report. My current patch for NKRO on the Teensy 3 is here.

Touch, SdFat, and SPI

For a long time, I believed that the only way I'd be able to use the touch interface on my TFT would be to use it through the SPI port on the Teensy 3, and use the chip select pins to select between the SD Card and the touch. This was a nightmare... trying to read/follow the manual, I just got bogged down in details that didn't actually mean anything to me, since I hadn't worked with SPI before and thus had no frame of reference for the terminology used. The only examples I could find for using the Hardware SPI were pretty much the SdFat code itself, and a thread on the Teensy forums here. In that thread they warned that the SdFat library was doing a few odd things that meant it'd break using SPI for anything else at the same time, but I couldn't understand enough of what SdFat was actually doing and enough of what the differences were in their sample code to be able patch SdFat to behave better. If anyone else works it out, I'd appreciate hearing about it :).

As far as I remembered, I had 3 pins free - and I'd need 5 to put the touch interface on a separate set of pins. While trying (again) to see if I could hack my way through the SPI stuff to get touch and SdFat working together (and instead just hanging my Teensy waiting while the SPI buffers were full, or breaking the ability to actually read the sdcard), I finally noticed that in a spot where I thought I had 2 pins unused, I actually had 3...giving me 4 unused pins. I also had both a micro-SD Card reader and the TFT screen's SD Card reader connected to the Teensy (a legacy of when I was initially getting the SD Card access working) - dropping these back to just the TFT screen's SD Card socket freed one of the two chip select lines, finally giving me the full 5 lines needed to be able to bit-bang the touch interface without having to use the Teensy's SPI port. Every single digital IO port on my Teensy 3 is in use...but ALL the hardware is connected now, so there's nothing else I need any more ports for. :-D

Once I had that, getting the touch interface working was very simple. Henning Karlsen, the same person who wrote the UTFT library, also wrote a library called UTouch for exactly this type of touch controller interface. No code changes were needed to make this library work on the Teensy 3s ARM chip - it was literally a "drop in and use" library. :-)

Since then, the touch screen has been hooked into the menu system as it currently is. On the keymap display (the default screen when not in menus, tells you what the keys are currently assigned as), tapping the left third or right third of the screen will switch to prev|next keymap. Tapping the centre top will open the menus, while tapping the centre bottom will redraw the current keymap. I've been finding that I get a significant percentage of random redraw errors (probably 5-10% of screen redraws has an issue) that I still have to track down. I don't know yet if these are due to cable length, breadboard capacitance, an intermittent wire in the cable, a dodgy connection/termination somewhere, a timing issue in the Teensy or TFT controllers, or what - but as a result, having an "oops, that didn't show properly, please redraw it" option is...useful. ;-)

Physical Construction

Also known as "getting the bloody circuits to fit in the box!".

My husband found a case that could be used to mount the screen, combined with a bracket that we had lying around (I can't remember what it was for originally). This weekend we got the TFT mounted inside the box, then attached that to the keyboard box, using a mount that allows me to rotate the screen (in fixed steps) and adjust its vertical angle. Given that it sits off on my right, it's angled for the best front-on view when I'm gaming (which means it's angled compared to the keyboard itself).

The 40-pin connector can be attached/detached without taking the back of the box off, and there is a slot in the side for the SD Card. I'll need to add a strip to the card so I can reach it properly to remove it - I need to use tweezers to remove it at the moment, but that was part of why I didn't want to have to take it out of the keyboard to add new files to it in the first place.

We're down to needing to get the keyboard key-press detection board shrunk down to a circuit board with surface mount components and the Teensy itself mounted to a board that brings the pins out to a 40-pin connector (34 used) for the screen, a 6-pin connector for the keyboard keypress detection board (I know those add up to more than the number of digital IO, but those numbers include the power/gnd/etc lines as well), a battery mount for the RTC's coin cell, and a button connection for the reset button so it can be mounted on the rear panel of the keyboard (once it's in the keyboard, I won't be able to reach the reset button on the Teensy itself, of course).

Since the electronics side of things is my husband's area of expertise, the circuit board layout and fabrication will be done by him. In the mean time, I've still got a ton of software tweaks I want/need to make, those odd redraw issues to try to sort out, the edit menus on the Teensy itself to write (as at present all keymap editing is only able to be done on the Java GUI) and possibly changing a few things so that I use the Teensy's (emulated) EEPROM for recording the "current" keymap index instead of having to re-save the whole configuration file every time I switch keymaps (not that it takes more than a few hundred miliseconds, but still...it's distracting).

So there's still lots to keep me busy, but in the mean time - the basics are now all there and working together well enough to ditch the Teensy 2++ controller for the keyboard and move across to using the Teensy 3 controller full-time!