Framework for physical Audio User Interfaces

[^rooker]

Inspired by my work for serial-pyio (for monome devices) and problems I've personally encountered regarding user interfaces for musical applications, I was hoping to maybe contribute something useful back to the community.

This thread will contain ideas and document my progress regarding my diploma thesis at the technical university.

At first I was planning to focus on the problem of lacking "flow" when it comes to making electronic music.

Due to the large amount of articles and scientific papers regarding that topic, I thought: I probably won't be *the* genius who will solve that in such a short time.

Since I didn't want to give up on that subject, I thought about other ways of improving the current situation for musicians dealing with electronic music. My idea is to increase the availability of alternative devices for the public - but how?

I thought about what keeps me from building my own physical interface, or use an existing one differently (e.g. joystick, wii-mote, ...). The work, time and knowhow necessary to build the bridge between the device and an audio application of my choice is simply a lot.

Furthermore, I'm usually not even sure about the results of that work:
- will the new interface really solve some of my problems?
- will my code be reliable/stable enough for live performances?
- what about maintaining it?
- can I share my work?
- which audio applications will/should/can it work with?
- damn! so much docs to read for that one device...

[^rooker]

What about alternative interfaces that you can simply go out an buy?

1) Most of them are so expensive that I guess a lot of people think twice before buying one (e.g. Lemur).

2) Very often they're designed to work with certain applications. Only with these certain applications. Furthermore, the audio apps targeted by commerical suppliers of audio interfaces are usually "Mainstream only" - meaning windows, and sometimes Mac - but hardly or never really platform independent - and without any support for LInux.


3) Open interfaces (e.g. monome), offer amazing capabilities, but a lot of code provided by the community is too specifically designed for a certain software environment or application.

Examples:
- Programs for handling audio samples using a monome 40h have been written and are being used by a wide audience, but in Max/MSP. thus, requiring new programming work to do the same thing in a different application, although remapping of commands might do the job (e.g. PureData, SooperLooper, FreeWheeling, ...)

- There exist videos about people using the Wii-mote for musical performances (e.g. wii-tar), but often it is a cumbersome task setting up ones own system to even try it out (see "lab environment" problem, below).

What about alternative interfaces as a result of research?

Some interfaces that have been built while doing UI research are actually also unavailable to the public. why?

a) Commercial research:
Research being done by companies will probably never be made available to anyone, unless some executives think that it will yield some profit. That's why major companies are more likely to stick to "well known" interfaces / devices. And even if they do come up with something, they make it closed (like the tenori-on) with the highest degree of interoperability being MIDI.

b) Scientific research:
UIs that emerged in a more "open" context (e.g. universities), were often built in what I call "lab environments", making it cumbersome or almost impossible to get it running in another environment. I've experienced that with some of my own works already, which I couldn't get up and running again after eg. half a year, because the development setup (my "lab environment") was quite fragile and changing steadily... It simply was a prototype that was used only once.

Unfortunately, the setup of a proper lab environment (especially regarding software prerequisites) for a project is often lacking good documentation - or is too complicated for someone else to rebuild within reasonable time (e.g. Studierstube & PUC).

[^rooker]

Here's a short overview of my idea to improve the current situation:

Make a framework for audio user interface developers - Giving them a base to work on. As a programmer, I know that the hardest part is the start of a new project - if you don't have to start from scratch, but can improve existing stuff, it's less demotivating and due to a common ground, developers can exchange code and experiences more easily.

If that framework is done as I imagine, it should provide a number of reusable blocks (like lego blocks) making it possible even for non-programmers to link a hardware device to their favorite audio application (by using e.g. OSC or MIDI).


My framework idea consists of several parts:
(Entries marked with a * were contributed by Brian Crabtree (monome.org) - Thanks!)

1) Support different kinds of hardware interfaces, by logically splitting each device into its elements. e.g. buttons, leds, faders... and map each one of them to programming structures.

Examples:
- 2-axis, 4-button Joystick: 4 Buttons, 2 Faders.
- Wii-mote: 4 buttons, 3 Faders, 4 LEDs
- Monome 40h: 64 buttons, 64 LEDs

There are more elements than just buttons, leds and faders, but for now, I'll just focus on these most basic ones.


2) Value preprocessing:
Some devices - especially DIY ones (e.g. arduino based), require preprocessing of input / output values.

- Transformation (*)
e.g. some ADC returns values from 0-1024, but the target application expects a range from 0-100.

- Ramping (*)
e.g. Instead of immediately applying a new input/output value, one can define a ramp-time from the previous value to the new one.


3) These elements should be mappable to to UI-widgets, by a so called "semantic layer". This grouping is supposed to be possible across devices.

Examples:
a) elementary:
- several LEDs as a meter row.
- several pushbuttons as radio group.
- several LEDs in a 2D array as output display

b) state keeping:
- button A on: joystick X-axis = volume,
button A off: joystick X-axis = panning
- 2 pushbuttons for increasing/decreasing a value (*)


4) Mapping UI-widgets to OSC/MIDI messages.
Finally, each one of the previously defined virtual devices (=UI widgets) must be made available to audio applications. I think doing the communcation using OSC is preferrable, but MIDI might be necessary to provide compatibility with other apps, too.

[^rooker]

The "semantic layer" is my personal favorite.
I'd like to make the semantics of devices configurable using XML files. These will include the connection between a virtual UI-widget and the physical element (button, LED, ...).

It could even be imaginable to write an editor for such XMLs (similar to "mapd"). Although I'm not really a fan of frontends for config files, it could offer a possibility for absolute-non-coders to still use this framework.


Edit / Sidenote:
I've found it quite confusing when writing/talking/thinking about the semantic layer, because of its name - so from now on I'll refer to it as "widget layer".

[^rooker]

Use cases
In order to move on towards implementation design, I thought it might make sense to define some use cases that this framework should be able to handle while trying to keep things as simple as possible.

My examples will be quite specific to get some concrete ideas on how they could possibly be managed and to avoid too many "could's" and "should's".

Furthermore, I do not want to assume that the target audio application is scriptable, like pd, Max or ChucK - so I'd like to have preprocessing capabilities within the framework.
However that one's still a subject of discussion, since if done badly it could add unnecessary complexity to the framework, causing undesired additional work on programmer/musician side.

Here's some brainstorming:

1) 2 btn-Joystick to control parameters: speed (BPM), volume, effect (wet)
2) A series of LEDs as VU meter
3) A series of LEDs as position indicator
4) x buttons as radio group
5) A monome 40h displaying a 2D pattern


Even though a single user will end up with a 1(device):1(app) mapping, my idea is to make other profit from the work and effort that one user has already put into getting a certain device to work with some app - and make it reusable with other apps, too (e.g. loop handling in MaxMSP, enabling someone else to reuse sample position display in SooperLooper, etc...)

[^rooker]

How could one want meaningfully use a joystick to control some parameter values (e.g. effect parameters, speed, volume, etc...)?

Let's assume we have a default 2-axis, 4 button joystick with a coolie-hat.
(The coolie-hat is somewhat tricky, because when using an analogue gameport, it's read as a 3rd axis where each of the 4 positions of the hat is mapped to a certain value on that axis)

=== Things to consider: ===
a) Changing the value of only 1 axis at a time.
This causes some trouble, because it's quite hard to move a joystick in a single dimension only, without slightly affecting the other dimension, too.

b) Absolute vs. relative values:
absolute:
In some cases, one might want the value from an axis to absolute - For example, if we assume a 16bit signed integer value, that would be:
-32767 / 0 / +32767
snapping back to 0 whenever one releases the joystick. This is nice for e.g . pitch-bending.

PROs / CONs
++) This method is nice when continuously modifying a value without interruption.

--) A drawback of this approach is that one cannot "hold" a certain value - and even if you had a "lock" feature (e.g. press/release button), it's hard to return to the value you've locked in case you'd like to gradually change the value relative to its locked position.


relative:
So, what about a relative approach? This would require the push/release of a button so that an application knows when to ignore the movement of the joystick (because it'll snap back to its center position).

However, this approach is easier for values that one will change at certain points in time, having a pause in between - or when you don't have your hand on the joystick all the time.


=== Desired output: ===
Widgets:
(Note: I'm referring to axis as "fader" (see elements-theory in previous posts above), and I'm writing it down in some pseudo code so that I see which things will be necessary in the widget layer)

I'd like to have the fader-meaning depending on which button is pressed:

Code: Select all

a) /js1/speed: Fader[1] && (Button[1] == 1), mode=relative
b) /js1/pitch: Fader[1] && (Button[1] == 0), mode=absolute
c) /js1/pan: Fader[2] && (Button[2] == 0), mode=absolute
d) /js1/pan: Fader[2] && (Button[2] == 1), mode=relative

Damn. now there are overlapping conditions... I would have to define which button is used to trigger the "release" state in relative-mode. hm...

Furthermore, I'd like to have the coolie-hat as 4 additional buttons instead of 4 different fader-values. Something like this:

Code: Select all

d) /js1/coolie1: (Fader[3] >= -32000) && (Fader[3] <-16000)? 1 : 0
e) /js1/coolie2: (Fader[3] >= -16000) && (Fader[3] <0)? 1 : 0
f) /js1/coolie3: (Fader[3] >= 0) && (Fader[3] < 16000)? 1 : 0
g) /js1/coolie4: (Fader[3] >= 16000) && (Fader[3] < 32000)? 1 : 0

Idea: By looking at that example, I think such a case must be moved to the hardware/variables mapping layer, providing the coolie hat as fader[3] -and- button[5-8].

[^rooker]

[2007-11-03 09:51]
Here are some thoughts about which elements should have which attributes (for configuration and communication) - and which datatypes. It's still pretty rough tinkering though...


Fader - INPUT
== settings ==
- min
- max
- absolute / relative (needs toggling)
absolute & relative should not be on element layer, but on widget layer.

== send ==
- value


Button - INPUT
Can represent any form of contact that returns a discrete state.

== settings ==
- autofire: [0-255]
continously toggle state. The value defines the speed (higher = faster). 0 disables autofire.

== send ==
- state: [0-255]
The state of the button. Usually will be 0/1 only, but what about x-state things like a discrete turn knob? That's why it's a number, not a boolean.


LED - OUTPUT
== receive ==
- state: [0-1]


Char - OUTPUT
(e.g. 7 segment display)
== setting ==
- min
- max

== receive ==
- value: [0-9]


-----
OptoEncoder? Actually that one's only -/+ right? Should it be handled as its own element or as a fader in the end? hm...



= IDEAS =
1) state request by OSC.
2) "resend" attribute for all elements: Allow state to be re-sent in a certain interval (like keep-alives)
3) simple name=value mapping, even for numbers. example:
button.state_mapping = ([0] = "off", [1] = "on", [2] = "sleep")

[^rooker]

So here are some updates about my thoughts about the basic hardware elements of this audio framework - I'm trying to keep things as simple as possible, even if losing "some" functionality - the gain in simplicity should be worth it:

1) Faders:

Code: Select all

<fader index="0" min="0" max="1024"/>

2) Buttons:
Buttons should only have 2 states: on / off.
Discrete Turn knobs used for mode switching should rather be mapped to several buttons on the hardware layer (similar to hat switch).

Furthermore, I think that virtual functionality should reside in the widget layer. So here's the button:

Code: Select all

<button index="0"/>

3) LEDs:

Code: Select all

<led index="0"/>

4) character display (7 segment, LCD or whatever...)

Code: Select all

<char index="0"/>

I was thinking about the possibility to name each element - just for readability reasons. The reference should still be by its index to keep parsing simple. For referencing certain hardware elements within the widget layer however, it might be useful to reference by name. hm....

[^rooker]

Now I'm trying to show some practical examples of widgets the way I imagine them:

1) 8 LEDs as VU meter:
The declaration of devices is only necessary to give them a name and to provide connection details (port, bluetooth-ID?) - the elements are fixed per device, so no need to declare them here.

Code: Select all

<device name="thing1" model="arduino_4led">
    <serial port="/dev/ttyS0"/>
</device>

<device name="thing2" model="arduino_4led">
    <serial port="/dev/ttyS1"/>
</device

Now here's a simple example of an output widget, receiving values from 0 to 100 and mapping them to 8 LEDs on 2 devices:

Code: Select all

<output>
    <meter min="0" max="100">
       <osc recv="/test/volume"/>
       <thing1.led index="1"/>
       <thing1.led index="2"/>
       <thing1.led index="3"/>
       <thing1.led index="4"/>

       <thing2.led index="1"/>
       <thing2.led index="2"/>
       <thing2.led index="3"/>
       <thing2.led index="4"/>
    </meter>
</output

The order within the XML is important. The mapping should be read "in order of appearance within the XML" to the value "min to max".
e.g.: thing1.led.1 = 0, thing2.led.4 = 100

I'm still not sure about the xml structure: should it be <devicename.element> or rather <element device="xxx">? I guess the second version would be easier from the coders point of view, but slightly confusing for the user side....

The most XML-ish way would probably be:

Code: Select all

<device name="">
  <element index="x"/>
  ...
</device>

hm...

Actually it even looks better:

Code: Select all

<output>
    <meter min="0" max="100">
       <osc recv="/test/volume"/>

       <device name="thing1">
           <led index="1"/>
           <led index="2"/>
           <led index="3"/>
           <led index="4"/>
       </device>

       <device name="thing2">
           <led index="1"/>
           <led index="2"/>
           <led index="3"/>
           <led index="4"/>
       </device>
    </meter>
</output>

[^rooker]

Fortunately, I'm back on the subject. Can't wait to make some progress.
First of all, the updated working title is something like:

"Semantic Abstraction Layer for Hardware Interfaces with Focus on Electronic Music" (SALHIFEM)

News:
1) Directly code instead of XML?
A friend of mine questioned my approach to use XML instead of directly using Python code. I wanted to avoid coding on that level (widget assignment), but I started thinking about it:

[PROs]
* A non-coder won't be comfortable with XML anyway => necessity for XML-generating frontend.
* Reading/writing of XML and thus the translation between XML and the programming objects would be omitted. Partially saving time & work.
* A programmer could make use of the possibility of being "within the code" on that layer.

[CONs]
* XML is language independent, so it doesn't matter which programming language someone is familiar with.
* XML would introduce a clean separation between configuration and code. Too many possibilities make things complicated again.


Anyway:
The version of the python interpreter would have to be fixed for these files - otherwise we'd end up with someone writing e.g v2.5 code - failing on v2.4. That's something I want to avoid!

[^rooker]

Discussions
Within the monome community there's a lot of discussion about a framework/toolkit like this:

http://post.monome.org/comments.php?Dis ... 025&page=1
http://post.monome.org/comments.php?Dis ... 297&page=1

Again a sign that I should really allocate more time for my diploma thesis.


Similar Project?
xndr (monome forum) mentioned junXion by Steim:
http://www.steim.org/steim/junxion_v3.html

It sounds pretty similar to my idea, so I thought I'd mention it here - Unfortunately it's neither platform independent (OSX only), nor open, nor free, nor community, nor anything - so a no-go.

[^rooker]

[ keep-alive & auto-fire ]
* keep-alive:
Re-send the current value triggered by a timer instead of only once at state-change.

* auto fire:
Toggle the state of e.g. a button (1/0).
(Really necessary? Keep-alive might be enough)


[ ramping ]
Interpolate between 2 values.

Only makes sense with auto-fire-like re-sending, because of the intermediate values.


[ value transformation ]
auto-calibration (optional)
It would be possible to measure the min/max values coming from an element and automatically re-calibrate it.

* Scaling
Scale numeric values - Useful for scaling ADC inputs.

Example:

Code: Select all

input range = 64 - 1024		
output range = 0 - 100

* Offset
Apply a signed (+/-) numeric offset to values.

Example:
Offset = -20
Input value = 28
Output value = 8


[ threshold ]
* absolute (= noise gate)
Reduce signal noise by only allowing values above/below threshold through.

Code: Select all

if( value > threshold ) then send_value();

* relative
Current value must differ enough from previous value to be sent.

Code: Select all

diff = abs(old_value, new_value);
if( diff > threshold ) then send_value();

[^rooker]

I'm not too good with text, so here's a graphical representation of my idea:
Flow Chart v0.1 (08.Sept.2008)

It contains the most important parts and connections, but is lacking lots of details.

[^rooker]

Value transformation can also be used to convert between integer (discrete) and float (continous) values.

Here's an example:

Code: Select all

$FADER_1
input: 0-1024
output: 0-1
precision: 2 (=2 digits after the comma)

And LEDs to display the current value:

Code: Select all

$RADIOGROUP_1
contains 8 LEDs
input: 0-1

so, an input value of $FADER_1=734 would give the following results:

• 0.72 as fader output (734 * 100 / 1024)
  LED #6 lit (round(72 * 8 / 100))

[^rooker]

Here are more thoughts about auto-fire and keep-alives:

[ auto fire ]
Although the "real" autofire of a joystick actually toggles the button's value (0/1), but I think in our application it's better to just re-send the value as long as the input is in "pressed" state.
So this would only apply to a button as far as I can see, because e.g. a fader doesn't have clearly defined "on" state.

[ keep alive ]
There might probably be some cases where a value should be sent although no user input has taken place. This applies to *any* element, and is regardless of it's current "state" (e.g. pressed or released).
The sending interval should be configurable. So it can also be used when ramping a value to configure how 'smooth' the value update on the output will be.