Hello,
How would I programmatically sign an app or OS?

Moved to calc help since this isn't exclusively KOS-related

It takes an unsigned .8xu and signs it. For an idea of how it works (and what other tools you need), you can look at my build script here
Relevant lines:

..\tools\ostools-0.1\multihex 00 "Page $00.hex" 1D "Page $1D.hex" > os.hex
..\tools\ostools-0.1\packxxu os.hex -o os84.8xu -t 83p -q 0A -v 0.01 -h 255
..\tools\rabbitsign\rabbitsign -t 8xu -k ..\tools\keys\0A.key -K 0A -g -p -r os84.8xu

multihex (part of OS Tools) takes alternating page numbers and hex files for each page and generates a hex file for the whole OS.
packxxu, also from OS Tools, takes the unsigned hex file, a key ID, and a couple other things and generates an unsigned .8xu
rabbitsign, linked above, takes the key and the unsigned .8xu file and produces a signed one.

As in, to write a tool yourself? Rabbitsign is opensource (written in C), so you could look at the source. I don't know the details, sorry

I think it uses the rabin Algorithm, yes?

From BrandonW on IRC

[23:14:25] <+BrandonW> A signature is an RSA-encrypted MD5 hash of the data you're wanting to sign.
[23:14:54] <+BrandonW> So you just MD5 hash the contents, and then encrypt it with 512-bit RSA using the private key associated with the ID you're wanting to sign with.
[23:15:15] <+BrandonW> So for example, to sign an OS with the 04 key, you take the 04 private key and use it to encrypt the MD5 hash of the OS.

At the en of the file, iirc. I believe you can use rabbitsign to see if the hash is correct. Md5 hashes are implemented in most popular languages.

Thanks, will do.