Hi Asle, are you still working on this one? I wanted to take a look at this earlier but real life kicked in in multiple unavoidable ways. Better late than never I hope.
I also examined memory and replay code with UAE debugger and I can confirm what you already know. The tunes are perfect Protracker modules, except that the pattern data (and only the pattern data) looks like garbage. Ok, there is also a curious "K.M." tag in place of the normal "M.K." tag. I dumped the entire memory and looked for anything that resembles pattern data, but there is no such thing in memory.
What I found interesting however is that the replay code looks also perfectly normal for Protracker: it picks the sample number, period and effect from their usual locations in memory. I experimented with breakpoints and memory dumps, and behold: a single line of the pattern data is decrypted in place for playback, and encrypted immediately after use again. At $00123b30 there seems to be code which reads the pattern data (pattern data base address is @a0, current offset from base is @d1). So I put a breakpoint there and investigated pattern data in memory.
>f 123b30
Breakpoint added
>g
Breakpoint at 00123B30
D0: 00000004 D1: 00000a30 D2: 00000004 D3: 00000008
D4: 00000010 D5: 00000020 D6: 00000040 D7: 00000080
A0: 0014843c A1: 00121a20 A2: 001483b8 A3: 0014800c
A4: 00148b66 A5: 00dff0a0 A6: 001249b6 A7: 000fff86
USP=0011fffc ISP=000fff86 MSP=00000000 VBR=00000000
T=00 S=1 M=0 X=0 N=0 Z=1 V=0 C=0 IMASK=3
FP0: 0 FP1: 0 FP2: 0 FP3: 0
FP4: 0 FP5: 0 FP6: 0 FP7: 0
N=0 Z=0 I=0 NAN=0
00123b30: 4a96 6604 6100 02e4 2cb0 TST.L (A6)
next PC: 00123b32
I then dump pattern data at base + offset, or $14843c + $a30 = $148e6c. 16 bytes per pattern line (4 channels * 4 bytes per channel). Dump three times after stepping the module forward three divisions (4 break + go rounds each, as there are 4 channels). See how nicely the pattern data comes and goes:
; round N
>m 148e6c
00148e6c 0000 0000 0000 0000 0000 0000 0000 0000 ................ <-- currently playing this line
00148e7c ff7a 3c1d 09b4 222d 0814 a0ad 0800 cebd .z<..."-........
00148e8c 0966 700d d2bc 0098 9680 5380 4a80 6e08 .fp.......S.J.n.
; round N+4 (1 round for each channel)
>m 148e6c
00148e6c 7400 2c6d 0170 41ed 09b4 43ed 09b8 4eae t.,m.pA...C...N.
00148e7c 00d6 1c30 0000 0000 01ac 3000 01ac 5c10 ...0......0...\. <-- currently playing this line
00148e8c 0966 700d d2bc 0098 9680 5380 4a80 6e08 .fp.......S.J.n.
; round N+8
>m 148e6c
00148e6c 7400 2c6d 0170 41ed 09b4 43ed 09b8 4eae t.,m.pA...C...N.
00148e7c ff7a 3c1d 09b4 222d 0814 a0ad 0800 cebd .z<..."-........
00148e8c 00d6 1c05 0000 0000 0000 0000 0000 0000 ................ <-- currently playing this line
I have yet to find and isolate the decryption code however. Perhaps it is easier to inject some extra code to copy decrypted pattern data into some stash and just overwrite the pattern data from the tunes you already ripped. l'll try to locate the decryption code with memory write watch first and see if it makes any sense.
Edit: clarified a little