arek stopczynski

Recently I got BB Playbook from RIM (thanks Ash) to play around and see how research-friendly it is. Here in the lab we pay close attention to the developments around BB10: we like open, hackable and mainstreamed platforms. As folks at RIM were out of BB10 Dev units, they sent me Playbook, just to touch the base and check where the platform is and where is it going.

One of our large projects, the one where I do most of the coding, is SmartphoneBrainScanner2. It’s a framework for building real-time, multi-platform apps for EEG (check out other posts on this blog). So the very first thing I did with Playbook (after updating it…) was to try to compile and run an example SBS2 app on it.

The framework we use is entirely written in Qt4, the example app visualizing brain cortical activity uses OpenGL ES2.0 for rendering. The framework is growing fast, but as we try to keep the number of platform-specific hacks to minimum (and you do always have hacks in the mobile applications world…) the app compiled cleanly and nicely with BB10 native toolchain. There was some learning curve about the package structure (where to put the large files we use to keep math models or how to do icons), but otherwise it just worked ™: Brain on Playbook. Not much to write, code that was originally begun for Nokia N900 still compiles for Android and QNX (and hopefully it will remain like that for BB10).

BB Playbook and Nexus 7 running SmartphoneBrainScanner2

Although the software compiled pretty much out of the box, Playbook does not support USB hostmode which we use for connecting USB dongle for EEG neuroheadset. Streaming the data from the network however, the system worked perfectly fine. So if we ever see BB10 devices supporting USB hostmode (and HIDRAW), SmartphoneBrainScanner2 will automagically become available on the new platform. The power of Qt in its fullest.

My general impressions from playing with the NDK and putting stuff on the device were quite positive. Plus for using standards to do stuff (such as transfers over SSH), but the whole procedure could be more streamlined (keeping SSH session active in one terminal, transferring stuff in the other, such things).

For research we are interested mainly in two things: hackability and support for sensors. I’m hoping for BB10 to have a robust architecture and to behave as RTOS should. If we could do proper low-delay audio processing on it with integration with EEG, this would be fantastic.  And if we could deploy applications collecting (various) sensor data in a stable manner over weeks or months, we would probably buy a few hundred units right away (only recently we have purchased 200 Samsung Galaxy Nexus for an experiment).

For teaching mobile development, we need platforms that are fairly stable, offer streamlined development, support multi-platform SDK and give access to various stuff on the device (including raw sensor data, background processes etc.). Currently we teach Android for all these reasons, but to be perfectly honest, I’m growing so tired of the sometimes ridiculous bugs/features in Android Java (my favorite so far) and Google attitude of don’t-fix-just-run-forward (it may be possible that audio latency has been fixed for some devices in Android 4.1. Yup, after all these years, we may see decent audio arch in Android). Qt on Android is not yet ready to be taught in the classroom, but boy, I would love to go back to teaching QML on embedded devices. We tried it once with N9s and it was so much fun for everyone.

So RIM, please do it right. And think about your presence in research and teaching. You want to get to them while they are (professionally) young…

Last week, on October 9 I went  for a few hours to London to participate in UBhave conference. It was a kick-off event for UBhave project on the theme of “Making Multidisciplinary Research Work”.

UBhave is a really cool project (http://ubhave.org/) focused on mobile phones and social networks used for behavior  changing interventions. Can you make people exercise more, socialize, or generally improve their quality of life thanks to access to the data about them, ability to do almost-real-time research, and direct feedback channel (in a form of smartphone apps, sms, social network posts etc.)?

Some great speakers at the conference (http://ubhave.org/conference2012/), including Prof. Kevin Patrick from UCSD (advisor of Ernesto Ramirez @e_ramirez, a well-known Quantified Self life hacker) and Dr Niels Rosenquist from Mass General, co-author of one of the coolest social media analysis projects ever (Pulse of the Nation).

It was really fun to see the common ground for both my projects (Smartphone Brain Scanner and SensibleDTU, more about the latter soon). We are living in times when brain scanning people and social network people can (start to) talk the same language. Über-exciting.

Android doesn’t like USB hubs in general and I couldn’t make it talk to two Emotiv EPOC dongles (that use hidraw module) using any hubs I have tried. But where there’s a will there’s a way: using a USB keyboard that includes hub and connecting the dongles to the keyboard, before connecting it to the phone/tablet, makes all devices (keyboard and multiple usb dongles) get recognized and properly mounted in the system as /dev/hidrawX. Sun keyboard type 7 is easy to take apart and supports 3 devices. It ain’t pretty, but doesn’t require external power and works fine.

Smartphone Brain Scanner 2 now supports using multiple headsets on a single device. This can be used for synchronized data acquisition or gaming on a single device. And for some other stuff that I should be soon able to share.

Finally, a little peace in-between periods of craziness.

Two months ago SmartphoneBrainScanner2 went open source and found home at http://code.google.com/p/smartphonebrainscanner2/ The source is there, the wiki is there and even some bugs start appearing. It’s awesome to see how the project starts to be alive, escaping your own machine into the wild.

The code is open. As much as it can: there is a single binary blob, required if you want to use Emotiv Epoc hardware. Having all the algorithms and pipelines open is extremely important, not only because you want people to be able to contribute to your project. To be serious about the results, black-boxes are seldom a good idea, especially if software and methodology is far from being mature.

What can SmartphoneBrainScanner2 currently do? Well, it allows you to build applications in Qt (4.x but 5.x should also work) that will run on various platforms/devices. Our focus is on mobile devices, primarily Android. Yes, we use Qt for Android and except for occasional bugs we find, everything works amazingly well. With the intensification of work from Digia on Qt for Android and iOS, we are starting to feel that the initial decision about going Qt was the future-proof one.

How have those two months of having SBS2 on google code been? We went live July 25, and since then we had 971 visitors. Not bad for a, let’s be honest, niche project. And it seems that the attention is getting less dependent on the announcements (spikes in the early phases).

Of course, it is not really the website traffic that counts for the project, but how it is used. And lot of really fun stuff is currently being done with SBS2. Different hardware, new algorithms and applications. And I hope life will be calm enough to write about them.

Just one year ago our team at Technical University of Denmark (DTU) started Project Smartphone Brain Scanner. We obtained one Emotiv Epoc gaming neuroheadset and asked ‘can we make it fully mobile?’. This is how it has begun.

If you managed to escape all the hype we got about this in the media, check out in the press tab.

Emotiv Epoc and Nokia N900

The first platform of choice was the Nokia N900 mobile computer, a fully fledged portable Linux box (that can even make calls). Thanks to community efforts, a kernel with USB hostmode was already there, allowing us to utilize the Epoc dongle. I compiled a hidraw module and voilà: data was flowing in 32 bytes packages to /dev/hidraw1.

Data from the Emotiv Epoc is encrypted. It is part of the business model of the company, to sell different licenses (i.e. decryption keys) with different dongles, giving access to different levels of data (game controller, extracted features, raw EEG). As a university, our first and subsequent sets were purchased as Research editions, giving us access to everything in the signal stream (including raw EEG data). But only on Microsoft Windows.

Initially we used parts of the code from the Emokit project. Some enterprising folks broke the encryption scheme and published the keys. This allowed us to access decrypted and demultiplexed data on the phone, initially using Python and later C++. This was awesome, but just the beginning.

Raw data from EEG (here 14 channels, 128 Hz) is just… well, numbers flowing in. A wide range of EEG analysis methods exist (notably collected in Matlab toolbox EEGLAB.). How much could we do on the mobile phone and in real-time?

Example of raw eeg signal (4 seconds) from a single electrode and source reconstruction (from all 14 channels).

Carsten Stahlhut, my colleague (currently post-doc) does remarkable work in the field of source reconstruction of brain activity. To explain in very simple terms: the current we measure with an EEG electrode is a sum of all activations in the brain, propagated through the brain itself, scalp, skin etc,. So what we measure with electrode A doesn’t necessary originate in the place in the brain closest to this electrode. The main task is to reconstruct the sources of the signal in the brain, instead of focusing just on the measured current. Carsten’s thesis is a good read if this sounds interesting (and you don’t mind some badass math).

Source reconstruction is computationally expensive. We are talking a 1028-vertices brain model (every vertex is a potential source of the signal), matrices of 1028×1028 and larger, real-time AES decryption, recording data, visualization, and so on. We need to overclock the N900 (again, power kernel is an amazing community effort) as much as we can, going up to 1.15GHz. Some phones do not like it so much and become unstable; luckily we have plenty to choose from. Still, some simplification of the calculations are implemented, everything to pursuit this hard real-time operation of the system.

In the next installment I will try to explain why we pursued this mobile, real-time, and complex approach to EEG at all. Why not just settle either on simple 2-3 electrodes portable systems, or big-and-serious but not mobile solutions?  Stay tuned!