On a second trial I was able to keep the three way connection active for about 15-20 seconds, but then one of the devices disconnected again.
We think it may be related to sending much data (right now we're pushing out data at maximum capacity, about 60KiB/s with two device p2p).
So far, this is really, really, really bad :-(
I'm debugging this today, so hopefully I've got more news later.
I have performed some more tests. Sending only a 5 byte unreliable ping every 3 seconds works very well with 3 devices. I've then upped the stakes and added another unreliable broadcast, sending 65 bytes every _frame_ (timer running at 120hz, but the frequency is probably closer to 60hz). So far it has worked without problems and I have experienced no disconnects. This leads to the conclusion that it's either a high bandwidth problem (bluetooth congested by too much data being sent), or it has something to do with the reliable sockets. Stay tuned!
I ran more experiments. As long as I do not send too much data (about 30KiB/s limit total data sent by the host), it works reliably and without problem (as long as the devices are very close). When I increase the bandwidth, I'm getting the same problem with the stalled connection. Also, when I'm moving away with a device, the bandwidth goes down naturally. It also increases the probability of a stalled connection drastically. Ie. walking away 3 meters or more usually stalls my 15KiB/s transfer.
Another very weird thing is that when the reliable or unreliable stream stalls, the other streams (unreliable or reliable, resp.) still continues at a normal pace. I also noticed that broadcasting is very stall-prone and should be avoided (iterating through the list of connected peers and sending the data manually is much more reliable in my tests).
All in all I'm getting the feeling that GameKit is extremely fragile and error prone, but I think I can make it work for some applications with the right assumptions and tricks. As long as no one walks away...