1Testing the Midi Kit 2==================== 3 4Most of the OpenBeOS source code has unit tests in the current/src/tests 5directory. I looked into building CppUnit tests for the midi2 kit, but 6decided that it doesn't really make much sense. Unit tests work best if 7you can test something in isolation, but in the case of the midi2 kit 8this is very hard to achieve. Because the classes from libmidi2.so 9always need to talk to the midi_server, the tests depend on too many 10external factors. The available endpoints, for example, will differ from 11system to system. The spray and hook functions are difficult to test 12this way, too. 13 14So instead of a CppUnit test suite, here is a list of manual tests that 15I performed when developing the midi2 kit: 16 17-------------- 18 19Registering the application 20--------------------------- 21 22*Required:* Client app that calls BMidiRoster::MidiRoster() 23 24- When a client app starts, it should first receive mNEW notifications 25 for all endpoints in the system (even unregistered remotes), followed 26 by mCON notifications for all connections in the system (even those 27 between two unregistered local endpoints from another app). 28 29- Send invalid Mapp message (without messenger). The midi_server 30 ignores the request, and the client app blocks forever. 31 32- Fake a delivery error for the mNEW notifications and the mAPP reply. 33 (Add a snooze() in the midi_server's OnRegisterApplication(). While 34 it is snoozing, Ctrl-C the client app. Now the server can't deliver 35 the message and will unregister the application again.) 36 37- Kill the server. Start the client app. It should realize that the 38 server is not running, and return from MidiRoster(); it does not 39 block forever. 40 41- Note: The server does not protect against sending two or more Mapp 42 messages; it will add a new app_t object to the roster and it will 43 also send out the mNEW and mCON notifications again. 44 45- Verify that when the client app quits, the BMidiRoster instance is 46 destroyed by the BMidiRosterKiller. The BMidiRosterLooper is also 47 destroyed, along with any endpoint objects from its list. We don't 48 destroy endpoints with a refcount > 0, but print a warning message on 49 stderr instead. 50 51- When the app quits before it has created a BMidiRoster instance, the 52 BMidiRosterKiller should do nothing. 53 54-------------- 55 56Creating endpoints 57------------------ 58 59*Required:* Client app that creates a new BMidiLocalProducer and/or 60BMidiLocalConsumer 61 62- Send invalid Mnew message (missing fields). The server will return an 63 error code. 64 65- Don't send reply from midi_server. The client receives a B_NO_REPLY 66 error. 67 68- If something goes wrong creating a new local endpoint, you still get 69 a new BMidiEndpoint object (but it is not added to 70 BMidiRosterLooper's internal list of endpoints). Verify that its ID() 71 function returns 0, and IsValid() returns false. Verify that you can 72 Release() it without crashing into the debugger (i.e. the reference 73 count of the new object should be 1). 74 75- Snooze in midi_server's OnCreateEndpoint() before sending reply to 76 client to simulate heavy processor load. Client should timeout. When 77 done snoozing, server fails to deliver the reply because the client 78 is no longer listening, and it unregisters the app. 79 80- Note: if you kill the client app with Ctrl-C before the server has 81 sent its reply, SendReply() still returns okay, and the midi_server 82 adds the endpoint, even though the corresponding app is dead. There 83 is not much we can do to prevent that (but it is not really a big 84 deal). 85 86- Start the test app from two different Terminals. Verify that the new 87 local endpoint of app1 is added to the BMidiRosterLooper's list of 88 endpoints, and that its "isLocal" flag is true. Verify that when you 89 start the second app, it immediately receives mNEW notifications for 90 the first app's endpoints. It should also create BMidiEndpoint proxy 91 objects for these endpoints with "isLocal" set to false, and add them 92 its own list. Vice versa for the endpoints that app2 creates. Verify 93 that the "registered" field in the mNEW notification is false, 94 because newly created endpoints are not registered yet. The 95 "properties" field should contain an empty message. 96 97- Start server. Start client app. The app makes new endpoints and the 98 server adds them to the roster. Ctrl-C the app. Start client app 99 again. The new client first receives mNEW notifications for the old 100 app's endpoints. When the new app tries to create its own endpoints, 101 the server realizes that the old app is dead, and sends mDEL 102 notifications for the now-defunct endpoints. 103 104- The test app should now create 2 endpoints. Let the midi_server 105 snooze during the second create message, so the app times out. The 106 server now unregisters the app and purges its first endpoint (which 107 was successfully created). 108 109- The test app should now create 3 endpoints. Let the midi_server 110 snooze during the second create message, so the app times out. (It 111 also times out when sending the create request for the 3rd endpoint, 112 because the server is still snoozing.) Because it cannot send a reply 113 for the 2nd create message, the server now unregisters the app and 114 purges its first endpoint (which was successfully created). Then it 115 processes the create request for the 3rd endpoint, but ignores it 116 because the app is now no longer registered with the server. 117 118- Purging endpoints. The test app should now create 2 endpoints. Let 119 the midi_server snooze during the \_fourth\_ create message. Run the 120 server. Run the test app. Run the test app again in a second 121 Terminal. The server times out, and unregisters the second app. The 122 first app should receive an mDEL notification. Repeat, but now the 123 test app should make 3 endpoints and the server fails on the 124 \_sixth\_ endpoint. The first app now receives 2 mDEL notifications. 125 126- You should be allowed to pass NULL into the BMidiLocalProducer and 127 BMidiLocalConsumer constructor. 128 129- Let the midi_server assign random IDs to new endpoints; the 130 BMidiRosterLooper should sort the endpoints by their IDs when it adds 131 them to its internal list. 132 133-------------- 134 135Deleting endpoints 136------------------ 137 138*Required:* client app that creates one or more endpoints and 139Release()'s them 140 141- Verify that Acquire() increments the endpoint's refcount and 142 Release() decrements it. When you Release() a local endpoint so its 143 refcount becomes zero, the client sends an Mdel request to the 144 server. When you Release() a local endpoint too many times, your app 145 jumps into the debugger. 146 147- Send an Mdel request with an invalid ID to the server. Examples of 148 invalid IDs: -1, 0, 1000 (or any other large number). 149 150- Start the test app from two different Terminals. Note that when one 151 of the apps Release()'s its endpoints, the other receives 152 corresponding mDEL notifications. 153 154- Snooze in midi_server's OnCreateEndpoint() before sending reply to 155 "create endpoint" request. The client will timeout and the server 156 will unregister the app. Now have the client Release() the endpoint. 157 This sends a "delete endpoint" request to the server, which ignores 158 the request because the app is no longer registered. 159 160- Override BMidiLocalProducer and BMidiLocalConsumer, and provide a 161 public destructor. Call "delete prod; delete cons;" from your code, 162 instead of using Release(). Your app should drop into the debugger. 163 164- Start the client app and let it make its endpoints. Kill the server. 165 Release() the endpoints. The server doesn't run, so the Mdel request 166 never arrives, but the BMidiEndpoint objects should be deleted 167 regardless. 168 169- Start the test app from two different Terminals, and let them make 170 their endpoints. Quit the apps (using the Deskbar's "Quit 171 Application" menu item). Verify that both clean up and exit 172 correctly. App1 removes its own endpoint from the BMidiRosterLooper's 173 list of endpoints and sends an 'mDEL' message to the server, which 174 passes it on to app2. In response, app2 removes the proxy object from 175 its own list and deletes it. Again, vice versa for the endpoint from 176 app2. 177 178- Start both apps again and wait until they have notified each other 179 about the endpoints. Ctrl-C app1, and restart it. Verify that app1 180 receives the 'mNEW' messages and creates proxies for these remote 181 endpoints. Both apps should receive an 'mDEL' message for app1's old 182 endpoint (because the midi_server realizes it no longer exists and 183 purges it), and remove it from their lists accordingly. 184 185-------------- 186 187Changing attributes 188------------------- 189 190*Required:* Client app that creates an endpoint and calls Register(), 191Unregister(), SetName(), and SetLatency() 192 193- Send an Mchg request with an invalid ID to the server. 194 195- Register() a local endpoint that is already registered. This does not 196 send a message to the server and always returns B_OK. Likewise for 197 Unregister()ing a local endpoint that is not registered. 198 199- Register() or Unregister() a remote endpoint, or an invalid local 200 endpoint. That should immediately return an error code. 201 202- Verify that BMidiRoster::Register() does the same thing as 203 BMidiEndpoint::Register(). Also for BMidiRoster::Unregister() and 204 BMidiEndpoint::Unregister(). 205 206- If you pass NULL into BMidiRoster::Register() or Unregister(), the 207 functions immediately return with an error code. 208 209- SetName() should ignore NULL names. When you call it on a remote 210 endpoint, SetName() should do nothing. SetName() does not send a 211 message if the new name is the same as the current name. 212 213- SetLatency() should ignore negative values. SetLatency() does not 214 send a message if the new latency is the same as the current latency. 215 (Since SetLatency() lives in BMidiLocalConsumer, you can never use it 216 on remote endpoints.) 217 218- Kill the server after making the new endpoint, and call Register(). 219 The client app should return an error code. Also for Unregister(), 220 SetName(), SetLatency(), and SetProperties(). 221 222- Snooze in the midi_server's OnChangeEndpoint() before sending the 223 reply to the client. Both sides will flag an error. No mCHG 224 notifications will be sent. The server unregisters the app and purges 225 its endpoints. 226 227- Verify that other apps will receive mCHG notifications when the test 228 app successfully calls Register(), Unregister(), SetName(), and 229 SetLatency(), and that they modify the corresponding BMidiEndpoint 230 objects accordingly. Since clients are never notified when they 231 change their own endpoints, they should ignore the notifications that 232 concern local endpoints. Latency changes should be ignored if the 233 endpoint is not a consumer. 234 235- Send an Mchg request with only the "midi:id" field, so no 236 "midi:name", "midi:registered", "midi:latency", or "midi:properties". 237 The server will still notify the other apps, although they will 238 obviously ignore the notification, because it doesn't contain any 239 useful data. 240 241- The Mchg request is overloaded to change several attributes. Verify 242 that changing one of these attributes, such as the latency, does not 243 overwrite/wipe out the others. 244 245- Start app1. Wait until it has created and registered its endpoint. 246 Start app2. During the initial handshake, app2 should receive an 247 'mNEW' message for app1's endpoint. Verify that the "refistered" 248 field in this message is already true, and that this is passed on 249 correctly to the new BMidiEndpoint proxy object. 250 251- GetProperties() should return NULL if the message parameter is NULL. 252 253- The properties of new endpoints are empty. Create a new endpoint and 254 call GetProperties(). The BMessage that you receive should contain no 255 fields. 256 257- SetProperties() should return NULL if the message parameter is NULL. 258 It should return an error code if the endpoint is remote or invalid. 259 It should work fine on local endpoints, registered or not. 260 SetProperties() does not compare the contents of the new BMessage to 261 the old, so it will always send out the change request. 262 263- If you Unregister() an endpoint that is connected, the connection 264 should not be broken. 265 266-------------- 267 268Consulting the roster 269--------------------- 270 271*Required:* Client app that creates several endpoints, and registers 272some of them (not all), and uses the BMidiRoster::FindEndpoint() etc 273functions to examine the roster. 274 275- Verify that FindEndpoint() returns NULL if you pass it: 276 277 - invalid ID (localOnly = false) 278 - invalid ID (localOnly = true) 279 - remote non-registered endpoint (localOnly = false) 280 - remote non-registered endpoint (localOnly = true) 281 - remote registered endpoint (localOnly = true) 282 283 | 284 285 Verify that FindEndpoint() returns a valid BMidiEndpoint object if 286 you pass it: 287 288 - local non-registered endpoint (localOnly = false) 289 - local non-registered endpoint (localOnly = true) 290 - local registered endpoint (localOnly = false) 291 - local registered endpoint (localOnly = true) 292 - remote registered endpoint (localOnly = false) 293 294 | 295 296- Verify that FindConsumer() works just like FindEndpoint(), but that 297 it also returns NULL if the endpoint with the specified ID is not a 298 consumer. Likewise for FindProducer(). 299 300- Verify that NextEndpoint() returns NULL if you pass it NULL. It also 301 returns NULL if no more endpoints exist. Otherwise, it returns a 302 BMidiEndpoint object, bumps the endpoint's reference count, and sets 303 the "id" parameter to the ID of the endpoint. NextEndpoint() should 304 never return local endpoints (registered or not), nor unregistered 305 remote endpoints. Verify that negative "id" values also work. 306 307- Verify that you can safely call the Find and Next functions without 308 having somehow initialized the BMidiRoster first (by making a new 309 endpoint, for example). The functions themselves should call 310 MidiRoster() and do the handshake with the server. 311 312- The Find and Next functions should bump the reference count of the 313 BMidiEndpoint object that they return. However, they should not 314 (inadvertently) modify the refcounts of any other endpoint objects. 315 316- Get a BMidiEndpoint proxy for a remote published endpoint. Release(). 317 Now it should not be removed from the endpoint list or even be 318 deleted, even though its reference count dropped to zero. 319 320- Start app1. Start app2. App2 gets a BMidiEndpoint proxy for a remote 321 endpoint from app1. Ctrl-C app1. Start app1 again. Now app2 receives 322 an mDEL message for app1's old endpoint. Verify that the endpoint is 323 removed from the endpoint list, but not deleted because its reference 324 count isn't zero. If app2 now Release()s the endpoint, the 325 BMidiEndpoint object should be deleted. Try again, but now Release() 326 the endpoint before you Ctrl-C; now it should be deleted and removed 327 from the list when you start app1 again. 328 329-------------- 330 331Making/breaking connections 332--------------------------- 333 334*Required:* Client app that creates a producer and consumer endpoint, 335optionally registers them, consults the roster for remote endpoints, and 336makes various kinds of connections. 337 338- Test the following for BMidiProducer::Connect(): 339 340 - Connect(NULL) 341 - Connect(invalid consumer) 342 - Connect() using an invalid producer 343 - Send Mcon request with invalid IDs 344 - Kill the midi_server just before you Connect() 345 - Let the midi_server snooze, so the connect request times out 346 - Have the midi_server return an error result code 347 - On successful connect, verify that the consumer is added to the 348 producer's list of endpoints 349 - Verify that you can make connections between 2 local endpoints, a 350 local producer and a remote consumer, a remote producer and a 351 local consumer, and two 2 remote endpoints. Test the local 352 endpoints both registered and unregistered. 353 - 2x Connect() on same consumer should give an error 354 - The other applications should receive an mCON notification, and 355 adjust their own local rosters accordingly 356 - If you are calling Connect() on a local producer, its Connected() 357 hook should be called. If you are calling Connect() on a remote 358 producer, then its own application should call the Connected() 359 hook. 360 361 | 362 363- Test the following for BMidiProducer::Disconnect(): 364 365 - Disconnect(NULL) 366 - Disconnect(invalid consumer) 367 - Disconnect() using an invalid producer 368 - Send Mdis request with invalid IDs 369 - Kill the midi_server just before you Disconnect() 370 - Let the midi_server snooze, so the disconnect request times out 371 - Have the midi_server return an error result code 372 - On successful disconnect, verify that the consumer is removed from 373 the producer's list of endpoints 374 - Verify that you can break connections between 2 local endpoints, a 375 local producer and a remote consumer, a remote producer and a 376 local consumer, and two 2 remote endpoints. Test the local 377 endpoints both registered and unregistered. 378 - Disconnecting 2 endpoints that were not connected should give an 379 error 380 - The other applications should receive an mDIS notification, and 381 adjust their own local rosters accordingly 382 - If you are calling Disconnect() on a local producer, its 383 Disconnected() hook should be called. If you are calling 384 Disconnect() on a remote producer, then its own application should 385 call the Disconnected() hook. 386 387 | 388 389- Make a connection on a local producer. Release() the producer. The 390 other app should only receive an mDEL notification. Likewise if you 391 have a connection with a local consumer and you Release() that. 392 However, now all apps should throw away this consumer from the 393 connection lists, invoking the Disconnected() hook of local 394 producers. The same thing happens if you Ctrl-C the app and restart 395 it. (Now the old endpoints are purged.) 396 397- BMidiProducer::IsConnected() should return false if you pass NULL or 398 an invalid consumer. 399 400- BMidiProducer::Connections() should return a new BList every time you 401 call it. The objects in this list are the BMidiConsumers that are 402 connected to this producer; verify that their reference counts are 403 bumped for every call to Connections(). 404 405-------------- 406 407Watching 408-------- 409 410*Required:* Client app that creates local consumer and producer 411endpoints, and calls Register(), Unregister(), SetName(), SetLatency(), 412and SetProperties(). It should also make and break connections. 413 414- When you call StartWatching(), you should receive B_MIDI_EVENT 415 notifications for all remote registered endpoints and the connections 416 between them. You will get no notifications for local endpoints, or 417 for any connections that involve unregistered endpoints. The 418 BMidiRosterLooper should make a copy of the BMessenger, so when the 419 client destroys the original messenger, you will still receive 420 notifications. Verify that calling StartWatching() with the same 421 BMessenger twice in a row will also send the initial set of 422 notifications twice. StartWatching(NULL) should be ignored and does 423 not remove the current messenger. 424 425- Run the client app from two different Terminals. Verify that you 426 receive properly formatted B_MIDI_EVENT notifications when the other 427 app changes the attributes of its *registered* endpoints with the 428 various Set() functions. You should also receive notifications if the 429 app Register()s or Unregister()s its endpoints. That app that makes 430 these changes does not receive the notifications. 431 432- Run the client app from two different Terminals. Verify that you 433 receive properly formatted B_MIDI_EVENT notifications when the apps 434 make and break connections. Every app receives these connection 435 notifications, whether the endpoints are published or not. The app 436 that makes and breaks the connections does not receive any 437 notifications. 438 439- StopWatching() should delete BMidiRosterLooper's BMessenger copy, if 440 any. Verify that you no longer receive B_MIDI_EVENT notifications for 441 remote endpoints after you have called StopWatching(). 442 443- If the client is watching, and the BMidiRosterLooper receives an mDEL 444 notification for a registered remote endpoint, it should also send an 445 "unregistered" B_MIDI_EVENT to let the client know that this endpoint 446 is no longer available. If the endpoint was connected to anything, 447 you'll also receive "disconnected" B_MIDI_EVENTs. 448 449- If you get a "registered" event, and you do FindEndpoint() for that 450 id, you'll get its BMidiEndpoint object. If you get an "unregistered" 451 event, then FindEndpoint() returns NULL. So the events are send 452 *after* the roster is modified. 453 454-------------- 455 456Event tests 457----------- 458 459*Required:* Several client apps that create and register consumer 460endpoints that override the various MIDI event hook functions, as well 461as producer endpoints that spray MIDI events. Also useful is a tool that 462lets you make connections between all these endpoints (PatchBay), and a 463tool that lets you monitor the MIDI events (MidiMonitor). 464 465- BMidiLocalProducer's spray functions should only try to send 466 something if there is one or more connected consumer. If the spray 467 functions cannot deliver their events, they simply ignore that 468 consumer until the next spray. (No connections are broken or 469 anything.) 470 471- All spray functions except SprayData() should set the atomic flag to 472 true, even SpraySystemExclusive(). 473 474- When you send a sysex message using SpraySystemExclusive(), it should 475 add 0xF0 in front of your data and 0xF7 at the back. When you call 476 SprayData() instead, no bytes are added to the MIDI event data. 477 478- Verify that all events arrive correctly and that the latency is 479 minimal, even when the load is heavy (i.e. many events are being 480 sprayed to many different consumers). 481 482- Verify that the BMidiLocalConsumer destructor properly destroys the 483 corresponding port and event thread before it returns. 484 485- BMidiLocalConsumer should ignore messages that are too small, 486 addressed to another consumer, or otherwise invalid. 487 488- BMidiLocalConsumer's Data() hook should ignore all non-atomic events. 489 The rest of the events, provided they contain the correct number of 490 bytes for that kind of event, are passed on to the other hooks. 491 492- Hook a producer up to a consumer and call all SprayXXX() functions 493 with a variety of arguments to make sure the correct hooks are being 494 called with the correct values. Call SprayData() and 495 SpraySystemExclusive() with NULL data and/or length 0. 496 497- Call GetProducerID() from one of BMidiLocalConsumer's hooks to verify 498 that this indeed returns the ID of the producer that sprayed the 499 event. 500 501- To test timeouts, first call SetTimeout(system_time() + 2000000), 502 spray an event to the consumer, and wait 2 seconds. The consumer's 503 Timeout() hook should now be called. Try again, but now spray 504 multiple events to the consumer. The Timeout() hook should still be 505 called after 2 seconds, measured from the moment the timeout was set. 506 Replace the call to SetTimeout() with SetTimeout(0). After spraying 507 the first event, you should immediately get the Timeout() signal, 508 because the target time was set in the past. Verify that calling 509 SetTimeout() only takes effect after at least one new event has been 510 received. 511 512-------------- 513 514Other tests 515----------- 516 517- Kill the server. Now run a client app. It should recognize that the 518 server isn't running, and return error codes on all operations. Also 519 kill the server while the test app is running. From then on, the 520 client app will return error codes on all operations. Also bring it 521 back up again while the test app is still running. Now the client 522 app's request messages will be delivered to the server again, but the 523 server will ignore them, because our app did not register with this 524 new instance of the server. 525 526- Start the midi_server and several client apps. Use PatchBay to make 527 and break a whole bunch of connections. Quit PatchBay. Start it 528 again. Now the same connections should show up. Run similar tests 529 with MidiKeyboard. Also install VirtualMidi (and run the old 530 midi_server for the time being) to get a whole bunch of fake MIDI 531 devices. 532 533- *Regression bug:* After you quit one client app, another app fails to 534 send request to the midi_server. 535 536 *Required:* Client app that creates a new endpoint and registers it. 537 In the app's destructor, it unregisters and releases the endpoint. 538 539 *How to reproduce:* Run the app from two different Terminals. Ctrl-C 540 app1. Start app1 again. From the Deskbar quit both apps at the same 541 time (that is possible because app1 and app2 both have the same 542 signature). When it tries to send the Unregister() request to the 543 midi_server, app2 gives the error "Cannot send msg to server". The 544 error code is "Bad Port ID", which means that the reply port is dead. 545 The Mdel message from Release() is sent without any problems, 546 however, because that expects no reply back. This is not the only way 547 to reproduce the problem, but it seems to be the most reliable one. 548 549 The reason this happens is because you kill app1. When app2 sends a 550 synchronous request to the midi_server, the server re-used that same 551 message to notify the other apps. (Because it already contained all 552 the necessary fields.) But app1 is dead, the notification fails, and 553 this (probably) wipes out the reply address in the message. I changed 554 the midi_server to create new BMessages for the notifications, and 555 was no longer able to reproduce the problem. 556