A few days ago, Spyro543 showed me his IRC bot that takes the command !tibasic [TOKEN], where [TOKEN] is a TI-BASIC token such as ClrHome and Pause. As far as I could tell, his bot code manually pairs a description with each valid [TOKEN], and that got me thinking.

We have many webapps/bots/references for TI-BASIC tokens in this community, and each was written and programmed by an individual, with his own format and style. There are the TI-BASIC Developer [tibd]Tokens[/tibd] tables and ASCII guides; KermMartian's SourceCoder, which does tokenizing and detokenizing of TI-BASIC programs; TI.ZEWAREN, which does roughly the same thing; my own TI-BASIC BBify'r, which detokenizes programs into BBCode syntax; jacobly's token converter, which also detokenizes; and now Spyro's bot, which outputs short descriptions for several tokens. All of them use similar routines, but all of them were created separately, with their own token databases, their own formats, and their own bugs.

Why don't we have a common system for all of these? We could have all the information about every token anyone ever needed stored in one comprehensive database, that anyone could update and from which anyone could retrieve data and use in his own app, bot, whatever. Anyone who finds a faster way to access, say, the sub( command could put his finding in the database, and anyone else's script that tells you where to find TI-BASIC commands would be updated silently and immediately. If TI created a new OS version that adds more tokens to the TI-83 Plus series, there'd be only one thing for us to update. Even better: all the bugs would be in one place!

And so I've started (you've been expecting this, haven't you?). The main page of the Unified Token Reference is just a simple token table, but what's really important is behind it. Put ?output=xml after the URL, and it'll give you a XML 1.0 database containing all the information in the table. The format is simple: <table> nodes contain a table of tokens, and <token> nodes, whose id attributes contain their TokenIDE-style ASCII equivalent, contain child nodes with whatever information there is about the token. Currently there's only the <description> tag implemented, but adding more is as simple as sending me a reasonable request (for example, to add a <syntax> tag for syntax information or <howto> tag for brief instructions on how to find a command).

To fetch a particular token point, use the parameter i. For example, ?output=xml&i=166 will return the <token> node for Horizontal, while ?output=xml&i=187 will return the <table> for the [tibd]miscellaneous tokens[/tibd]. For two-byte tokens, use both i and j; ?output=xml&i=187&j=1 returns npv(. If you need help on using XML in a program, Google is your friend. You can also ask around here if you'd like.

But what's the good of a complete database if nobody can edit it? Well, go back to the main page of the UTR, click on any box in the table, and double-click where it says (confusingly enough) "Double-click to edit." A text box is added, where you can change the information to whatever you want. As soon as it loses focus, the information you entered is sent to the server and updated immediately (through the power of AJAX).

As I said, the only information tag implemented right now is <description>, but you can request more (right in this thread). The <description> tag should be used for a clear, concise description of what a token does, in at most 140 characters. (The parenthetical insertion has 94 characters, to give you an idea of how long it should be.) Please don't vandalize; the script keeps logs of all edit activity, including your IPs.

What do you think of this idea? I'm always free to suggestions, but when this really gets going (if it ever gets going), it'll be harder to alter the more basic structures and stuff.