things like kerm's Source Coder 2 or merth's TokenIDE are all well and good for converting an 8x program between plaintext and link formats, but there are a few limitations to both: namely that they require graphical environments and cannot be scripted. to achieve these two purposes, i've started writing a tokeniser/detokeniser of my own (github). it's already functional, with the only remaining tasks being adding in full token sets (right now it only contains a very small subset of the axe tokens which i was using for testing purposes), implementing the option to skip comment lines, optionally writing to stdout, and writing the detokeniser (which is much simpler than its counterpart). have a screenshit!

Great ! Will you add some directives like #include, #define etc. ?

what sorts of things would you want from a preprocessor (besides define, which might be doable)?

and thanks, lionel =)

#if calculator=84+ then (89? nspire?)
<code>
#else
<code>
#end

just a thought

this is only for the 8x series

If your tokenizer is modular enough, it could (assuming someone spends time on it, while there are few programmers for that platform anymore...) be used for TI-68k/AMS programs as well. The token lists for AMS are well known, and haven't evolved since 2005 (89T / V200 AMS 3.10).
As far as the Nspire is concerned... I performed enough reverse-engineering on the OS in 2011 to show that Nspire BASIC programs are still based on tokens similar to the AMS BASIC ones (and in general, the Nspire's CAS still has its foundation in the 92 CAS from 1996), but there are lots of undocumented things on the Nspire, and third parties are not making Nspire BASIC programs on the computer side.

There are, indeed, some new tokens for the 84+CSE, but I'm not the most knowledgeable about them - ask Kerm, BrandonW, Benjamin Moody or several others

Will this support XML files like Tokens (and even the same format)?

Ah ok I was mostly wondering since new commands could be added in apps like Grammer/Axe/etc, and cross-compatibility with TokenIDE XML files would prevent your app from falling behind if you ever got too busy to maintain it.

This is pretty neat. It's similar to what elfprince is doing, it seems. I've thought about just having a simple executable that doesn't require the editor and everything (it would be fairly simple), but no one expressed interest in it so I haven't done it--yours is probably better for that anyway since it's in C and therefore requires fewer dependencies.

One suggestion I would have it to make it so it can take TokenIDE-style XML files and use those for tokenization/detokenization. I see that you've mentioned that, but your solution of "anyone can just poke me with a new set to add to the source (or do it themselves) and compile it in" isn't really idea. Why make users recompile when they can just drop in an XML file? It also has the added bonus of making it so if someone makes a new token set for TokenIDE, it'll automatically work with yours and vice-versa. Standardization is, I think, a good thing.

There are, indeed, some new tokens for the 84+CSE, but I'm not the most knowledgeable about them - ask Kerm, BrandonW, Benjamin Moody or several others

Or Merth . The latest release of TokenIDE has the 84+CSE XML file with all the new/renamed tokens:
http://merthsoft.com/Tokens.zip
(Hopefully you don't think I'm trying to advertise, that's just where the xml file is, and you can use that for the new tokens.)