Omnimaga

Calculator Community => Other Calc-Related Projects and Ideas => TI Z80 => Topic started by: merthsoft on August 12, 2010, 12:26:36 am

Title: TokenIDE - TI-BASIC IDE
Post by: merthsoft on August 12, 2010, 12:26:36 am
I've had an idea for some PC-side calculator related things, so I've been messing around with some ideas. Yesterday I made a "detokenizer", a program that would read in a calc program and give a human-readable version of it. Today I have added the ability to tokenize text. For now the test program is pretty simple, you open a program, and it gives you four text boxes, which are (in order): raw hex, detokenized text, tokenized hex, detokenized text of the tokenized hex. This way you can see that the hex and the text match up.
(http://merthsoft.calcg.org/test_token.png)
So here you can see that they match up. Here's a download link if you want to test it out:
http://merthsoft.calcg.org/Tokens.zip (http://merthsoft.calcg.org/Tokens.zip)
Tomorrow I will be working on taking text that you actually edit yourself and turning it into a program that you can actually put on your calculator.

There are sometimes some inconsistencies, but they tend to be due to ambiguities in the Tokens.xml file, which you're able to edit to suit your needs.

Edit: Updated download link:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: Builderboy on August 12, 2010, 12:30:46 am
Nice!  So it will turn into a kind of On-Computer program editor of sorts?
Title: Re: Tokens
Post by: merthsoft on August 12, 2010, 12:32:41 am
Nice!  So it will turn into a kind of On-Computer program editor of sorts?
That's the idea. The goal is for this to become a pretty full-featured TI-BASIC IDE, but I'm going one step at a time.
Title: Re: Tokens
Post by: SirCmpwn on August 12, 2010, 08:53:47 am
Hey Merth, this looks like .NET, and I know you use C#.  If you like, I can help out a bit.
Title: Re: Tokens
Post by: TC01 on August 12, 2010, 10:04:44 am
It seems like everyone's making a TI-Basic IDE/Tokenizer these days- there's Kerm's DoorsCS-based on-calc one, there's Mighty Moose's on-calc one, there's your PC one, and there's mine on-68k-calc one. I guess there will be no shortage of z80 Basic editors/tokenizers in the future...

Nice work so far!
Title: Re: Tokens
Post by: merthsoft on August 12, 2010, 10:48:26 pm
Hey Merth, this looks like .NET, and I know you use C#.  If you like, I can help out a bit.
I think I'm alright, but I'll keep that in mind.
Title: Re: Tokens
Post by: merthsoft on August 14, 2010, 07:31:33 pm
And here we are, the latest release. You can now open or directly type in programs. Some neat features:
Right now, build only saves the file TEST.8xp as a program called "TEST", that's because this is still all testing stuff for the toolkit.
A lot of it is XML driven, so you can add things function/lib-wise, or adapt it for specific types of development (such as AXE). I'm including the schema in case you want to know how it's formatted.
Here's a screen shot:
(http://merthsoft.calcg.org/test_token2.png)
Note the "DrawTileMap(" and "DrawSprite(" functions. Those can by typed straight in and run like that.

And the download link:
http://merthsoft.calcg.org/Tokens.zip (http://merthsoft.calcg.org/Tokens.zip)
Please try and out and let me know how it works for you. It would be nice to know that there is actual interest in a product like this.

For the next step, I'll be switching gears to the map/sprite editors. You'll be able to make a sprite and insert it as hex or into a picture file. You'll be able to make maps and insert them as matrices into a program, or as strings. Basically, anything that has a specific format with these libs you'll be able to create and insert easily. As well as DCS GUI stuff.
Title: Re: Tokens
Post by: DJ Omnimaga on August 17, 2010, 07:59:24 am
Mhmm, interesting to see the HEX code to the right. Does it means if we copied it on calc and appended an AsmPrgm command at the beginning, it would let us squish the program with AsmComp() and it would let us turn it back into a basic program or something? Not that this would be convenient but I am just curious how that part works.

Also nice to see updates. I'll try this with one of my program when I have some more time.

As for interest I am sure there might be, because some people sometimes asked if there was a BASIC editor other than TI-Graph Link that was reliable and available for use with no internet connection (SourceCoder) and I usually couldn't figure out an answer. However, the majority of the 83+ programmers around here prefer to program on-calc, especially that most have restrictive parents limiting their time allowed on the computer. Plus Cooliojazz was working on a TI-BASIC editor as well a while ago, so you might have some competition. I would like to see this finished, though. Does this works on all versions of Windows, btw?

Also I moved this in calc projects and ideas because it was posted in the wrong sub-forum. (The non-calc computer projects section is for computer projects that absolutely have no relation with calc-related stuff)
Title: Re: Tokens
Post by: merthsoft on August 17, 2010, 11:46:55 am
Mhmm, interesting to see the HEX code to the right. Does it means if we copied it on calc and appended an AsmPrgm command at the beginning, it would let us squish the program with AsmComp() and it would let us turn it back into a basic program or something? Not that this would be convenient but I am just curious how that part works.
I don't think so, since this hex is token data, not opcodes.

As for interest I am sure there might be, because some people sometimes asked if there was a BASIC editor other than TI-Graph Link that was reliable and available for use with no internet connection (SourceCoder) and I usually couldn't figure out an answer. However, the majority of the 83+ programmers around here prefer to program on-calc, especially that most have restrictive parents limiting their time allowed on the computer. Plus Cooliojazz was working on a TI-BASIC editor as well a while ago, so you might have some competition. I would like to see this finished, though. Does this works on all versions of Windows, btw?
I'm alright with competition :) It'll motivate me more. It requires the .Net 4.0 framework, so whatever version of windows that supports, this will run on. It should also run under Linux with Mono, though I haven't tested that myself. I can when I get home.

Also I moved this in calc projects and ideas because it was posted in the wrong sub-forum. (The non-calc computer projects section is for computer projects that absolutely have no relation with calc-related stuff)
Thanks, I wasn't sure which forum this belonged in :)
Title: Re: Tokens
Post by: ztrumpet on August 17, 2010, 11:51:16 am
That looks cool!  I have a request:  Can you make an option be to choose if it's an Axe program or not, and if it is show the Axe tokens?
Regardless, it looks neat.  Good luck! ;D
Title: Re: Tokens
Post by: merthsoft on August 17, 2010, 12:13:59 pm
That looks cool!  I have a request:  Can you make an option be to choose if it's an Axe program or not, and if it is show the Axe tokens?
Regardless, it looks neat.  Good luck! ;D
You'll be able to change the token-set by picking which XML file is used, and if there's one for Axe, you can choose that one. I haven't made on for Axe yet, just DCS (which includes xLib/Celtic III/PicArc), but I plan on making one for Axe, though I don't know enough Axe yet to do that.
Title: Re: Tokens
Post by: DJ Omnimaga on August 17, 2010, 12:47:44 pm
Mhmm, interesting to see the HEX code to the right. Does it means if we copied it on calc and appended an AsmPrgm command at the beginning, it would let us squish the program with AsmComp() and it would let us turn it back into a basic program or something? Not that this would be convenient but I am just curious how that part works.
I don't think so, since this hex is token data, not opcodes.

As for interest I am sure there might be, because some people sometimes asked if there was a BASIC editor other than TI-Graph Link that was reliable and available for use with no internet connection (SourceCoder) and I usually couldn't figure out an answer. However, the majority of the 83+ programmers around here prefer to program on-calc, especially that most have restrictive parents limiting their time allowed on the computer. Plus Cooliojazz was working on a TI-BASIC editor as well a while ago, so you might have some competition. I would like to see this finished, though. Does this works on all versions of Windows, btw?
I'm alright with competition :) It'll motivate me more. It requires the .Net 4.0 framework, so whatever version of windows that supports, this will run on. It should also run under Linux with Mono, though I haven't tested that myself. I can when I get home.

Also I moved this in calc projects and ideas because it was posted in the wrong sub-forum. (The non-calc computer projects section is for computer projects that absolutely have no relation with calc-related stuff)
Thanks, I wasn't sure which forum this belonged in :)
Aaah ok and no problem ^^, people sometimes confuse both forums, since a calc project may be a computer one and think it might go in non-calc stuff due to being executable on a computer.

For Axe, a few commands use different tokens. For example, in the BASIC editor, Tangent() will become Bitmap() and SinReg will become Freq(. The original token name is given in the command index, for those who don't use a BASIC editor that is Axe-compatible or who are used to certain old tokens.
Title: Re: Tokens
Post by: merthsoft on August 19, 2010, 12:16:54 am
So I've started working on the sprite/hex editor. It allows you to create/edit/insert a hex sprite. So you can see here that I have a hex sprite highlighted:
(http://i35.tinypic.com/169qc76.png)
And then when I hit Tools->Hex Sprite Editor, I get this:
(http://i35.tinypic.com/126bsep.png)
So that's pretty slick. You can adjust the display options and stuff. You can also edit in that window, both at the pixel-level and the hex-level.
I'm not releasing this one yet for you guys to try, because there's still quite a bit I want to do to it, but that's the progress so far!
This example also does a pretty good job of showing off functions, we've got DetectDCS(, ClearScreen( (note the ',' after it because of the optional parameter), CloseGUIStack(, PushGUIStack(, and GUIMouse( all lookin' sweet :D
Title: Re: Tokens
Post by: TIfanx1999 on August 19, 2010, 01:59:32 am
Sprite editors are always a welcome addition. That looks very nice! :)
Title: Re: Tokens
Post by: Quigibo on August 19, 2010, 03:46:22 am
If you need a complete and current list of all Axe token conversions, the "commands.html" file on the left column shows all the tokens that change and what they change to.  Its included in all zip files, but I can send you a better formatted list if you really need one or I can write the XML myself if you have an sdk for it.
Title: Re: Tokens
Post by: ztrumpet on August 19, 2010, 10:07:47 am
Wow, that looks wonderful!  Awesome job Merth! ;D
Title: Re: Tokens
Post by: merthsoft on August 19, 2010, 12:37:16 pm
Thanks for the encouragement, everyone :)
If you need a complete and current list of all Axe token conversions, the "commands.html" file on the left column shows all the tokens that change and what they change to.  Its included in all zip files, but I can send you a better formatted list if you really need one or I can write the XML myself if you have an sdk for it.
The .xls file is included with the .zip I posted earlier, and that with the already present Tokens.xml should give you an idea of how it works. I plan on writing an actual tutorial closer to the release. If you could just give me a text file that is formatted something like OnCalcToken - Function, so, like:
DiagnosticOn - RunIndicatorOn()
DiagnosticOff - RunIndicatorOff()
Something like that, that would be great. They don't need the parentheses, they can be whatever.

Edit: It just occurred to me that the download link is down, so you can't get the .xls.
Title: Re: Tokens
Post by: DJ Omnimaga on August 21, 2010, 02:11:06 pm
Wow I'm glad to see a sprite editor built-in. Should make development much faster, not to mention the sprite editor is included in your software. Nice job.
Title: Re: Tokens
Post by: shmibs on August 21, 2010, 09:10:57 pm
this shall be very useful!
release it soon so i can get to work spriting for that one mega rpg that i actually want to make
Title: Re: Tokens
Post by: merthsoft on August 21, 2010, 09:19:45 pm
I've added resizing on-load. Here's what I say in my git commit:
Before, when passed in a hex string, the user would have to adjust from an 8x8 sprite to make it look right. Now, the editor will find the closest square sprite that is at least the same size as the passed in hex, align the width with the closest byte (higher or lower), and then trim the height down until it's the right size. While this isn't always perfect, it will, for the most part, get close to the right dimensions.
Download here http://madigan.boldlygoingnowhere.org/Tokens.zip (http://madigan.boldlygoingnowhere.org/Tokens.zip) if you want. There are still some bugs, and it might be a little crashy. I think next will be the DCS GUI designer. This'll probably be a pretty big chunk of work.
Title: Re: Tokens
Post by: DJ Omnimaga on August 22, 2010, 03:35:38 pm
Nice, in your sprite editor I like how we can drag the mouse while holding down the button instead of having to click every pixel one by one. Keep up the good work!
Title: Re: Tokens
Post by: merthsoft on October 18, 2010, 09:17:58 am
So I haven't updated here in a while, so this will be a pretty picture-heavy post :)
I've started working on a couple things at the same time for this. The first is the DCS GUI editor, right now you can drag checkboxes (I know they say textbox... oops!)!
(http://myserverathome.com/guidesign.png)
Results in:
(http://myserverathome.com/guioutput.png)
Code: [Select]
PushGUIStack(0,255
PushGUIStack(7,40,24,0,Unchecked,"
PushGUIStack(7,44,20,0,Unchecked,"
PushGUIStack(7,48,20,0,Unchecked,"
PushGUIStack(7,52,24,0,Unchecked,"
PushGUIStack(7,52,28,0,Unchecked,"
PushGUIStack(7,48,32,0,Unchecked,"
PushGUIStack(7,44,36,0,Unchecked,"
PushGUIStack(7,40,40,0,Unchecked,"
PushGUIStack(7,36,36,0,Unchecked,"
PushGUIStack(7,32,32,0,Unchecked,"
PushGUIStack(7,28,28,0,Unchecked,"
PushGUIStack(7,28,24,0,Unchecked,"
PushGUIStack(7,32,20,0,Unchecked,"
PushGUIStack(7,36,20,0,Unchecked,"
This code actually is wrong, but I've since fixed the output.
I've also made progress in the image editor. These are just some of the shapes that you'll be able to add. There will also be the ability to toggle a grid and set the size (so the grid can be every 8x8 pixels, etc.):
(http://i56.tinypic.com/10yfmfl.png)
There is also undo/redo, and importing. When you go to import a picture it brings up this crop window:
(http://i52.tinypic.com/rbdxf5.png)
The black rectangle is what you're selecting to be the picture. It does a simple algorithm to turn in to black and white, and gives you this:
(http://i54.tinypic.com/29e4xfa.png)
I might do an edge detection thing instead of just is this pixel color average closer to black or white, but that's what it does.

And then you can save it!
(http://i55.tinypic.com/2iv1gs4.png)
This obviously shouldn't be used to replace your current image editor, as I'm not in the business of reinventing the wheel, but it is a nice way to open and quickly convert/edit existing image. And lastly (for now), you can export the image to hex, either in one giant string, or broken up as you see fit. It doesn't do the best error detection, so don't go crazy. 8x8 and 16x16 are built in, the rest you have to go into a custom menu. It's not terribly intuitive, but it works. Here are some screenies:
(http://i52.tinypic.com/2yoz8ev.png)
(http://i56.tinypic.com/2u8vv9d.png)
(http://i51.tinypic.com/25iq99e.png)

You can download it here:
http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip)

Hope it isn't too crashy, though I guarantee nothing.
Title: Re: Tokens
Post by: DJ Omnimaga on October 18, 2010, 02:29:49 pm
Wow nice update! I love all the features you included in it :)
Title: Re: Tokens
Post by: kindermoumoute on October 18, 2010, 03:29:20 pm
This looks interesting, but even with the online translator, I can not understand what brings this software. DJ, can you explain briefly (by PM in french)?  ;D


EDIT : Error.. I need install a stuff :
(http://img143.imageshack.us/img143/5158/tokenserror.png)
Title: Re: Tokens
Post by: ztrumpet on October 18, 2010, 05:56:28 pm
Looks great Merth!  Nice job! ;D
Title: Re: Tokens
Post by: Deep Toaster on October 18, 2010, 07:12:20 pm
Looks great! This is to write BASIC prgms, right? What libs does it support?
Title: Re: Tokens
Post by: merthsoft on October 18, 2010, 09:14:35 pm
It's got Celtic, xLib, and DCS support, as in, it puts in the actually command names when you open a file. You should give it a try :)
Title: Re: Tokens
Post by: Deep Toaster on October 18, 2010, 09:17:36 pm
I'm on someone else's computer that doesn't have .NET :(
Title: Re: Tokens
Post by: Yeong on October 19, 2010, 09:01:09 am
Wow..TI-BASIC editor in PC..
Awesome!
Title: Re: Tokens
Post by: DJ Omnimaga on October 19, 2010, 01:29:59 pm
This looks interesting, but even with the online translator, I can not understand what brings this software. DJ, can you explain briefly (by PM in french)?  ;D


EDIT : Error.. I need install a stuff :
(http://img143.imageshack.us/img143/5158/tokenserror.png)
(Il s'agit d'un éditeur TI-BASIC pour PC ainsi qu'un éditeur d'image de calculatrice.)

Keep up the good work merthsoft :D
Title: Re: Tokens
Post by: kindermoumoute on October 19, 2010, 02:35:51 pm
Nice, I'll try It soon =)
Title: Re: Tokens
Post by: shmibs on October 19, 2010, 04:19:36 pm
this is amazing. i cant use it at the moment, but shall as soon as this computer issue is resolved.
thanks merth!
Title: Re: Tokens
Post by: merthsoft on October 19, 2010, 07:42:08 pm
Some people had mentioned ASM style outputs for the image editor as well, so I implemented that. It was pretty trivial. Here're the xlib sprites:
(http://i56.tinypic.com/34y77y9.png)
And here's the dino picture:
(http://i55.tinypic.com/30cub86.png)
Looks like dinos to me!

I suppose the right thing to do now would be to make the sprite editor able to do both hex and binary. I'll also be looking over some of the code Ben sent me to see how easy it would be to make this work with BBC BASIC as well. Ideally, this'll be able to do all languages. For ASM it'll just do the letters and numbers and stuff. I'd like to find out how Mimas saves its files. There will also, of course, be a way to save it in plain text. There's also some talks for getting it to integrate into SourceCoder over at Cemetech, and if Omnimaga has anything like that or any sort of posting API, I'd be glad to integrate that as well.
Title: Re: Tokens
Post by: ztrumpet on October 19, 2010, 10:07:57 pm
Wow, that looks wonderful!  Nice job! ;D
Title: Re: Tokens
Post by: DJ Omnimaga on October 20, 2010, 02:39:32 am
Nice :O

As for posting API what do you mean? ???
Title: Re: Tokens
Post by: merthsoft on October 20, 2010, 04:14:07 pm
Nice :O

As for posting API what do you mean? ???
A posting API would be something that would allow my program to interface with the omnimaga board, and post the code of a program directly to a topic.
Title: Re: Tokens
Post by: DJ Omnimaga on October 21, 2010, 02:58:29 am
Oh ok like SourceCoder. Kerm tried to implement that for Omni but it didn't work :(. Maybe it was Opera, though.
Title: Re: Tokens
Post by: Deep Toaster on October 21, 2010, 05:23:52 pm
Wow, sounds nice. So it could be used as an offline SourceCoder eventually?

EDIT: That reminds me: Feature request: For convenience: Conversion to BBCode.
Title: Re: Tokens
Post by: merthsoft on October 21, 2010, 05:34:52 pm
It basically already does what SourceCoder does, except for making actual posts. As for conversion to BBCode, how do you mean?
Title: Re: Tokens
Post by: Deep Toaster on October 21, 2010, 07:33:48 pm
When a program is uploaded to SourceCoder (from an 8xp file, not from its editor), it comes with a section that you can copy-and-paste as the BBCode to use in a forum. Could this be added?
Title: Re: Tokens
Post by: merthsoft on October 21, 2010, 09:28:30 pm
So, just wrap [code ] tags around it? Sure.
Title: Re: Tokens
Post by: DJ Omnimaga on October 21, 2010, 10:34:01 pm
Yeah that would be cool. Keep in mind SMF BBCode format for [SIZE ] tags is different, though.
Title: Re: Tokens
Post by: merthsoft on October 22, 2010, 12:33:51 am
To all my Linux friends, I have gotten it to work under Mono:
(http://myserverathome.com/TokensMono.jpg)
Title: Re: Tokens
Post by: DJ Omnimaga on October 22, 2010, 01:32:30 am
Nice to hear! There is an increasing amount of Linux users around here so they might like this.

I wonder about Mac...
Title: Re: Tokens
Post by: merthsoft on October 22, 2010, 09:03:27 am
It ought to work on Mac as well, so long as they use Mono. I don't have one to test on, tough, so if anyone wants to give it a try and post their findings, that would be awesome.
Title: Re: Tokens
Post by: DJ Omnimaga on October 22, 2010, 09:40:16 am
Ah right, mono is for Mac as well.
Title: Re: Tokens
Post by: Deep Toaster on October 22, 2010, 02:21:37 pm
So, just wrap [code ] tags around it? Sure.

Well, SourceCoder does it differently (in quote tags, IIRC) so that stuff like E and r show up correctly.

This is looking awesome, btw. Can I run it off a thumb drive? I'm not allowed to install stuff :P
Title: Re: Tokens
Post by: merthsoft on October 22, 2010, 04:48:09 pm
Ah, right. Mine actually doesn't use the special characters; the idea is that you should be able to type everything directly in a standard US keyboard. So if you want it in the special SourceCoder style, you'll have to use it. As for running off a thumb drive, yes, as long as the computer you're running it on has the .NET framework installed, or you have Mono no your thumbdrive (I don't know if Mono works like that).
Title: Re: Tokens
Post by: merthsoft on October 23, 2010, 12:50:20 am
Alright, I've added some more DCS GUI elements, here is a sampling:
(http://i52.tinypic.com/ff7fdj.png)
Which produces this:
Code: [Select]
OpenGUIStack(
PushGUIStack(1,"0000000000","LARGE
PushGUIStack(2,3,13,"0000000000","SMALL
PushGUIStack(11,6,3,0,"","
PushGUIStack(11,12,3,0,"","
PushGUIStack(11,18,3,0,"","
PushGUIStack(11,6,8,0,"","
PushGUIStack(11,12,8,0,"","
PushGUIStack(11,18,8,0,"","
PushGUIStack(22,6,15,29,17,254
PushGUIStack(21,30,4,23,9,1
PushGUIStack(14,40,16,22,14
RenderGUI(
Which renders like this:
(http://i55.tinypic.com/33dhyit.png)
So you can see it's not totally perfect, but it's a start! The only GUI items you can use right now are:

Here's the download link:
http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip)
This version has been tested under Fedora with Mono 2.8
Title: Re: Tokens
Post by: DJ Omnimaga on October 23, 2010, 01:16:50 pm
Wow, I assume this is to change the design of the DCS gui completly? Cuz that seems really awesome.
Title: Re: Tokens
Post by: merthsoft on October 25, 2010, 10:48:52 am
It's to simplify the design of GUIs. DCS exposes GUI stuff to BASIC and Asm programmers so they can make GUIs without having to make all the rendering code and all that themselves.
Title: Re: Tokens
Post by: Eeems on October 25, 2010, 11:01:48 am
looks nice :) can't wait for more to be added :D
Title: Re: Tokens
Post by: DJ Omnimaga on October 25, 2010, 05:22:25 pm
It's to simplify the design of GUIs. DCS exposes GUI stuff to BASIC and Asm programmers so they can make GUIs without having to make all the rendering code and all that themselves.
Ah cool :D. I wonder if those GUIs can be set via BASIC for BASIC programs that want to use custom ones? Is it like with Omnicalc fonts?
Title: Re: Tokens
Post by: merthsoft on October 26, 2010, 10:29:36 am
The GUI system is exposed as a library for BASIC programmers (see http://www.ticalc.org/archives/files/fileinfo/429/42991.html (http://www.ticalc.org/archives/files/fileinfo/429/42991.html)), but the actual rendering of the GUI cannot be changed, so you can't, for example, change the check boxes to rounded rectangle or anything like that.
Title: Re: Tokens
Post by: DJ Omnimaga on October 26, 2010, 03:56:03 pm
Oh ok I see. Thanks for the info.
Title: Re: Tokens
Post by: jnesselr on October 26, 2010, 06:50:33 pm
So, is this basically a full scale Basic editor?
Title: Re: Tokens
Post by: Munchor on October 26, 2010, 06:51:51 pm
So, is this basically a full scale Basic editor?

It looks more like a Visual Designer for Texas Calculators like C#, VB and C++ Visual Studio by Microsoft. Looks good ;)
Title: Re: Tokens
Post by: merthsoft on October 26, 2010, 07:00:18 pm
So, is this basically a full scale Basic editor?
That's the goal! It should work for most simple things right now (it can open and save programs, stuff like that). Most things are buggy to some extent, but they work!
Title: Re: Tokens
Post by: DJ Omnimaga on October 27, 2010, 02:56:54 am
Keeping track of all variables that are used in a program would be nice. if you got project support, maybe even accross the entire project.
Title: Re: Tokens
Post by: Munchor on October 31, 2010, 05:20:46 pm
Link to the latest version please...

Thanks!
Title: Re: Tokens
Post by: Binder News on October 31, 2010, 05:44:58 pm
I am working on a project and I need to be able to convert text files to and from 8xp. Could I maybe please begging-of-you-please have either A) an explanation of how to read and write in 8xp, or B) the code that you wrote to do that (just that little bit). I would like A better. Either way, PM me, PLEASE (can you tell I'm desperate?).
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 05:50:18 pm
Are you doing this in BASIC or ASM/Axe? The latter is fairly easy to convert to text using any of the multiple disassemblers available such as SPASM. If it's BASIC, the task is rather more difficult unless you're willing to use an emulator to view the BASIC file on-calc.
Title: Re: Tokens
Post by: Binder News on October 31, 2010, 06:08:41 pm
I'm trying to look at BASIC, and an emulator won't work. My program needs to be able to "look" at the text.
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 06:34:09 pm
Ah...

May I ask what type of program necessitates looking at ASCII characters? Text isn't a very computer friendly medium.
Title: Re: Tokens
Post by: Michael_Lee on October 31, 2010, 06:39:58 pm
Not really...
Numbers might be more friendlier to a computer, but not really human-readable, and graphics are barely comprehensible to a program.
Title: Re: Tokens
Post by: Runer112 on October 31, 2010, 06:40:38 pm
Link to the latest version please...

Thanks!

http://myserverathome.com/Tokens.zip
Title: Re: Tokens
Post by: Munchor on October 31, 2010, 06:40:51 pm
Link to the latest version please...

Thanks!
Thanks
http://myserverathome.com/Tokens.zip
Title: Re: Tokens
Post by: Binder News on October 31, 2010, 07:07:08 pm
I'm planning to write a program that will compile Axe source to z80 assembly code. That way it could be optimized, edited, etc.
Title: Re: Tokens
Post by: Runer112 on October 31, 2010, 07:18:00 pm
Could you add some sort of system that would allow users to choose what tokens result from characters or character combinations that could represent more than one thing? I was recently using your program, which is great by the way, to edit an Axe source file that someone sent me. However, when I opened the file and then recompiled it without making any changes, the output file wasn't the same and resulted in errors when compiled with Axe parser. After writing another program in Axe to search through the two programs on-calc byte-by-byte to find any differences, I found two culprits. An apostrophe character was being converted into BBD0 instead of the expected AE, and "<<" (the signed less than or equal comparison for Axe source files) was being converted into BBEB instead of the expected 6B6B.

SourceCoder handles this problem by using characters or character combinations enclosed in curly brackets (like "{L1}") to define some special tokens, and characters not found on a keyboard (like "≠") to define others. TI's Program Editor that was bundled with older versions of TI Connect handles this problem by using a custom font that replaces unused characters (namely accented letters) with the symbols needed to form the tokens (like "▸"). Both of these require a selector to pick these special tokens, which may be the way to go.

Personally, I would use a custom font and form special characters out of the same character combinations you already defined, but let the user undo the combination. For instance, if I wanted the L1 token, I would type "L1" which would be changed to "L1" as soon as I type the one (the small 1 would replace an unused ASCII value in the custom font). However, if the user wanted the literal string "L1" they could use ctrl+z to undo the combinaton. In this manner, the whole file couldn't be updated each time the user typed a new character, or it would keep changing user overrides back into special characters. I don't know how difficult it would be to do this, but the fix would probably be to only try to form special characters out of text the user just typed or pasted.



EDIT:

I'm planning to write a program that will compile Axe source to z80 assembly code. That way it could be optimized, edited, etc.

Wouldn't you just be rewriting Axe parser? That wouldn't be very fair to Quigibo.

If you want to hand-optimize compiled Axe programs, I guess you could compile source files with Axe in an emulator like wabbitemu and then transfer the compiled programs back to your computer and disassemble them there. I personally think that would undermine the spirit of using Axe, but you may feel differently.
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 07:37:47 pm
I'm planning to write a program that will compile Axe source to z80 assembly code. That way it could be optimized, edited, etc.

I believe Quigibo is writing an on-computer version of Axe as well.
Title: Re: Tokens
Post by: Binder News on October 31, 2010, 07:41:15 pm
The reason for compiling to z80 is twofold. ONE) Optimizations. I could probably write a companion program to strip away any unneeded pieces of code. This would decrease the size a lot. Also, If you needed to do some actual code (BCAlls and the like), you could add it in with this. TWO) The other reason would be for learning. I think it would help people a lot to be able to see the code they do understand, compared to the code they don't understand, to learn assembly. Finally, if Quigibo want's me to stop, I will. It is his app and his idea. I don't want to infringe.

EDIT: Then I would love to help. I am actually a very good computer programmer, and have a lot of free time.

EDIT4: I've (mostly) solved reading. Writing in 8xp is still beyond reach.
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 08:54:58 pm
Good, glad to hear it. Can this program also compile pure TI-BASIC as well? I think that would be really useful.
Title: Re: Tokens
Post by: Deep Toaster on October 31, 2010, 09:33:45 pm
Good, glad to hear it. Can this program also compile pure TI-BASIC as well? I think that would be really useful.

You mean a TI-BASIC to TI-ASM compiler? It's a lot harder than a language like Axe. There's a version of it on ticalc but it doesn't work.
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 09:36:59 pm
Good, glad to hear it. Can this program also compile pure TI-BASIC as well? I think that would be really useful.

You mean a TI-BASIC to TI-ASM compiler? It's a lot harder than a language like Axe. There's a version of it on ticalc but it doesn't work.

I can't imagine why it would be harder to translate the syntax. The interpreter doesn't do much more than Axe with respect to that. Sure, the way the instructions are handled in RAM is different than a native ASM program, but I'd take an 8kb limit in BASIC if it brought even part of the speed of compiled ASM to it.
Title: Re: Tokens
Post by: Deep Toaster on October 31, 2010, 09:41:03 pm
Good, glad to hear it. Can this program also compile pure TI-BASIC as well? I think that would be really useful.

You mean a TI-BASIC to TI-ASM compiler? It's a lot harder than a language like Axe. There's a version of it on ticalc but it doesn't work.

I can't imagine why it would be harder to translate the syntax. The interpreter doesn't do much more than Axe with respect to that. Sure, the way the instructions are handled in RAM is different than a native ASM program, but I'd take an 8kb limit in BASIC if it brought even part of the speed of compiled ASM to it.

Axe was designed to be compiled into ASM. Not TI-BASIC. Converting that to ASM would involve a lot of messy floating-point calculations and stuff like that, unless you used TI's own routines, but if you did that, it would run at essentially the same speed, so there'd be no point to convert it in the first place. It's possible, though.
Title: Re: Tokens
Post by: Binder News on October 31, 2010, 10:27:23 pm
Hmmm... I might just work on that. My main goal is to get the Axe to z80 first. I will start a new topic for this tomorrow, as it is getting to be a discussion of its own.
Title: Re: Tokens
Post by: DJ Omnimaga on October 31, 2010, 11:12:53 pm
There was a BASIC to ASM translator on Ticalc years ago but it produced broken code. Such stuff would be really hard to achieve. However, with what came out from the TI community in the past two years, I do not think it is impossible to write a decent BASIC to ASM translator. I am fairly certain something that produces relatively good speed could be done. However, the produced code might be larger than Axe.  If you want to do it, then good luck!
Title: Re: Tokens
Post by: AngelFish on October 31, 2010, 11:20:54 pm
Axe was designed to be compiled into ASM. Not TI-BASIC. Converting that to ASM would involve a lot of messy floating-point calculations and stuff like that, unless you used TI's own routines, but if you did that, it would run at essentially the same speed, so there'd be no point to convert it in the first place. It's possible, though.
The point of converting to ASM would be to remove the overhead of the interpreter, which for a high level language like TI-BASIC could be very significant, I imagine. If you compile beforehand, you remove the interpreter and the program invariably runs faster.
Title: Re: Tokens
Post by: merthsoft on November 01, 2010, 02:31:50 pm
Whoa, ok, I clearly need to check this more often...

If there is any token you want to change, you can modify the XML file. For the "<<" you have two choices, you can either type "<\<" in to the program (which is annoying, I need to work on the loading so that it'll break stuff up in the case of ambiguities like this), or you can change the XML.
Code: [Select]
<Token byte="$EB" string="<<" /> Could be change to:
Code: [Select]
<Token byte="$EB" string="<|" /> This is actually how it is with the newer version. I'm also waiting for Quigibo to get me a simplified list of all the Axe commands and what their function name should be, so that I can get an Axe.xml file so this can work better with Axe.

As for other special characters, the goal is to be able to type everything on a standard US keyboard. I don't want to require a picker, though I will probably have one because it's pretty easy to forget what things are what. So, not equals is != and store is ->. That way you can just type, and not have to be slowed down by needing a chooser.

I hope that addressed the questions that hadn't been answered. If there was something I missed, please ask again :)
Title: Re: Tokens
Post by: Binder News on November 01, 2010, 09:09:50 pm
I spent about 3 hours yesterday on figuring out the file format of the 8xp files. I THINK I can read from them, but I won't have much more time till this weekend.
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 09:13:10 am
There's plenty of literature out there about the file formats already, there's no need to figure it out on your own. http://www.ticalc.org/archives/files/fileinfo/247/24750.html (http://www.ticalc.org/archives/files/fileinfo/247/24750.html) this is where I got all my information (and asking some other people on IRC). I host that document on my website as well, http://myserverathome.com/linkguide/ (http://myserverathome.com/linkguide/)
Title: Re: Tokens
Post by: JustCause on November 02, 2010, 10:01:04 am
This is cool. Keep up the good work!
Title: Re: Tokens
Post by: DJ Omnimaga on November 02, 2010, 02:53:26 pm
Sorry if this is off-topic, but do you happen to know Runer112, Merthsoft? I swear some screenshots he posted before were hosted on Myserverathome, which appears to be your domain name (unless your site is really linkguide :P)

Back on-topic, I wonder if you plan to add something that detects all variables that were called, used, stored to or deleted in the code including pictures, strings and the like? There's a french incarnation of SourceCoder that did that and it seemed like a cool feature.
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 03:03:11 pm
Sorry if this is off-topic, but do you happen to know Runer112, Merthsoft? I swear some screenshots he posted before were hosted on Myserverathome, which appears to be your domain name (unless your site is really linkguide :P)
I do not! Though there are some screen shots of Tokens and some other projects I'm working on hosted there.

Quote
Back on-topic, I wonder if you plan to add something that detects all variables that were called, used, stored to or deleted in the code including pictures, strings and the like? There's a french incarnation of SourceCoder that did that and it seemed like a cool feature.
That is something that's planned, though a little later. The release plan at this point is to polish up the image editor, get all the DCS GUI items done (though not polished), and that will be v.5 which will be released on ticalc and other sites officially. I will then work on project support, a string editor, a matrix editor, and a map designer. The project support will include tracking vars and such. Once that's done that will be the official v1.0 release.
Title: Re: Tokens
Post by: DJ Omnimaga on November 02, 2010, 03:05:04 pm
Ah ok, maybe I got confused with a different URL. I swear he posted some of his own stuff from there, tho.

And I see. Can't wait for release :)
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 04:32:19 pm
merthsoft, do you think you could add an option to pretty-ify text? I went through the whole xml file and added lots of token alternates using non-keyboard characters that look much nicer and more visually realistic to the original tokens. Using more realistic tokens (such as Σx² instead of [Sigmax^2]) makes programs much easier to read. If I swapped the alternate strings with the main strings, or if you added some sort of "Pretty" alternate string form to the xml, could you add some sort of feature that would convert all tokens into their "Pretty" versions? Perhaps even as you type?



Ah ok, maybe I got confused with a different URL. I swear he posted some of his own stuff from there, tho.

And I see. Can't wait for release :)

Nope, never posted there. :P
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 04:38:29 pm
You're not the first person to mention that, actually. There's obvious going to be some level of re-learning that needs to be done in order to use this correctly, and the idea is that you should be able to enter everything in on the keyboard. I can think of a few ways to do this so that it's not terribly tedious. I'll think about how to implement it some more, and see if I come with anything I like. In the meantime, if you could upload your changes of the .xml file, that would be awesome!
Title: Re: Tokens
Post by: DJ Omnimaga on November 02, 2010, 04:40:35 pm
I like Runer112 idea. On forums I often see people getting confused at TI-BASIC code containing sqrt() and they ask where to find that function because it's not in the CATALOG. Finally they discover that it's the online way to type the square root function.
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 04:43:58 pm
You're not the first person to mention that, actually. There's obvious going to be some level of re-learning that needs to be done in order to use this correctly, and the idea is that you should be able to enter everything in on the keyboard. I can think of a few ways to do this so that it's not terribly tedious. I'll think about how to implement it some more, and see if I come with anything I like. In the meantime, if you could upload your changes of the .xml file, that would be awesome!

I'm just making some slight changes to it, I added the Axe Parser token alternates but forgot to specify that that's what they actually were, so I'm adding comments to signify Axe tokens.

EDIT: And following the idea of "Pretty" tags, do you think it would be possible to make something like an "Axe" tag? That way if you're editing an Axe source file, the tokens can default to the Axe token equivalences.
Title: Re: Tokens
Post by: Deep Toaster on November 02, 2010, 04:44:27 pm
Maybe a panel on the side that displays the tokens like they're displayed on the calculator, updated every time the user edits the edit panel?
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 04:46:53 pm
Well, the comments are ignored by the program, so specifying them in there won't help it at all. You should still upload that, it would be useful for me to see what people are adding and all that.

Deep Thought: There will be a chooser so you can pick things out.
Title: Re: Tokens
Post by: Deep Toaster on November 02, 2010, 04:52:10 pm
Deep Thought: There will be a chooser so you can pick things out.

Oh, sounds good, then.
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 05:25:25 pm
Ok, here's the modified xml file. Changes include:


Primary strings can all be typed on a keyboard. Alternate strings are sorted in the following manner:

Alternates that can be typed on a keyboard are first. "Pretty" alternates appear next. "Alternates" is plural because tokens that have more than one "pretty-ifiable" characters (e.g. chi^2cdf() are broken down into all possible pretty/non-pretty combinations (e.g. χ^2cdf(, chi²cdf(, and χ²cdf(). These are sorted in increasing order of "prettiness." Following any "Pretty" alternates come Axe Parser alternates, sorted according to the preceding rules.
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 06:00:08 pm
Wow, that's awesome. Thanks so much, Runner. The Axe commands I'll end up taking out into a separate Axe.xml file, but for now this is perfect. There are some things I'll change, and I'll have to look over it a little more in depth, but this is awesome. Out of curiosity, what were the duplicate tokens, and what were the missing ones?
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 06:10:31 pm
Wow, that's awesome. Thanks so much, Runner. The Axe commands I'll end up taking out into a separate Axe.xml file, but for now this is perfect. There are some things I'll change, and I'll have to look over it a little more in depth, but this is awesome. Out of curiosity, what were the duplicate tokens, and what were the missing ones?

$6338-$633F and $7E00-$7E07 were the only duplicates I think ($7E00-$7E07 were the correct ones). You missed $6237-$623C, $BBED-$BBF5, and I think few scattered others.
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 06:47:28 pm
Ok, great. Thanks. I did make some changes. Instead of using backslash as the standard, it's now the pipe character, "|". This is because backslash already has a meaning, and I don't want that ambiguity to exists. Basically, the blackslash tells the tokenizer to tokenize the next character as a single character. That way you can differentiate from, say, the max( token and the string "max(" by doing "\max(" to make it the string of four characters. Other than that it looks great :). I'll keep looking over it.

Question: Why do some of the Axe commands end with "{", like nib?
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 07:22:43 pm
To read and write to pointers in Axe, you have to enclose the pointer value in curly brackets. I believe Quigibo uses curly brackets on some of his commands, like nib{, because they take only one argument, and that argument is a pointer.

By the way, the Celtic III section seems to be malfunctioning. I accidentally stumbled into this when loading an Axe source file, because Axe uses the command det( followed by a number to allocate data sections in the compiled program. I believe all the Celtic III commands are in the format of det(number,stuff) whereas Axe uses the command in the format of det(number). Even though there is no comma after the first digit in det( calls in the Axe source file, it seems that it is being somehow intercepted by the Celtic III commands. However, because there is no comma, it doesn't actually match a token, and all it ends up doing is deleting the det( token and the first digit following it.

EDIT: It appears that this error also applies to other shell extension commands in the xml file.
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 07:50:47 pm
Hmm. That's strange. It could be having issues when there's an alt and some more tokens under it. I removed all the axe tokens in mine, actually, and made a separate Axe.xml file (since BASIC with ASM libs and AXE are two separate things). I have a new version up, by the way:
http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip)

Some things that I remember since the last update are:
An about box, using the logo designed by Weregoose!
The preview window and all that jazz for the crop image:
(http://i53.tinypic.com/6t1e6g.png)
Any other bug fixes I've made.
New tokens files. Tokens2.xml is what it now defaults to. Tokens1.xml is kept since it's what was used before, in case something doesn't looks quite right. Axe.xml has some of the Axe commands.

I got a little lazy with committing to git, so I kind of forget what I've done :( But please, use this newer version.
Title: Re: Tokens
Post by: DJ Omnimaga on November 02, 2010, 08:03:43 pm
Nice, I like how we can create pics from cropped images. It seems to convert pretty well too :)
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 08:29:00 pm
By the way, you might want to consider messing around with the main text box properties a bit :P
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 08:31:47 pm
O_o
What did you do to make that happen?
Title: Re: Tokens
Post by: Runer112 on November 02, 2010, 08:36:21 pm
O_o
What did you do to make that happen?

Copy and paste? Lol
Title: Re: Tokens
Post by: DJ Omnimaga on November 02, 2010, 08:50:14 pm
Wait you can paste file icons in there? ???
Title: Re: Tokens
Post by: merthsoft on November 02, 2010, 09:12:49 pm
Ahahaha. weird...
Title: Re: Tokens
Post by: Deep Toaster on November 03, 2010, 08:00:43 pm
And it's a ... space? :D

I'm starting to use this for some projects. It's really nice.
Title: Re: Tokens
Post by: merthsoft on November 04, 2010, 09:02:35 am
That's what I like to hear, Deep Thought :) Thanks!
Title: Re: Tokens
Post by: DJ Omnimaga on November 04, 2010, 09:40:58 am
I assume there are no random crashes or the like every 5 minutes or so, right? Because it is really tempting to start using.
Title: Re: Tokens
Post by: merthsoft on November 04, 2010, 10:27:11 am
It's crashy in places, but nothing should make it totally wreck anything. I need to make an error box and pipe all the errors to a text file for submitting bug reports, but it should be safe enough to use :)
You should use it, Kevin, you are in the credits, after all :)
Title: Re: Tokens
Post by: DJ Omnimaga on November 04, 2010, 07:36:44 pm
Really? O.O I don't remember much helping on coding and stuff. Or is it because of features suggestions?

But yeah I like it so far so it might be very useful to maybe code certain type of games or modify/update them. That said, I still love on-calc programming a lot, though, since I'm so used to it ;D
Title: Re: Tokens
Post by: merthsoft on November 04, 2010, 07:38:51 pm
You're in the credits because of Omnimaga! You're a community leader, and so you've been a big part of my participation in the community, even if indirectly!
Title: Re: Tokens
Post by: DJ Omnimaga on November 04, 2010, 10:05:29 pm
Oh ok lol, thanks a lot :D

EDIT: Hmm is your server down? It doesn't respond at all... ???
Title: Re: Tokens
Post by: merthsoft on November 05, 2010, 11:37:35 am
Yeah, we had an outage yesterday due to weather. It should be back up now.
Title: Re: Tokens
Post by: DJ Omnimaga on November 05, 2010, 06:57:07 pm
Ah, now I know where all that wind and rain we got today comes from... :P *kills Merthsoft*

Gonna try now. :D

EDIT Still down :(
Title: Re: Tokens
Post by: merthsoft on November 05, 2010, 07:26:15 pm
Works for me?
Title: Re: Tokens
Post by: DJ Omnimaga on November 05, 2010, 07:31:44 pm
Weird, it works now ???

It didn't a few mins ago...
Title: Re: Tokens
Post by: merthsoft on November 21, 2010, 03:42:03 pm
I have made an official release of Tokens. You can get it at http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip). Once this is uploaded to ticalc, I'll post that link, since their servers are a little more dependable than mine :)
Title: Re: Tokens
Post by: DJ Omnimaga on November 21, 2010, 03:43:08 pm
Cool to hear! What does this new version includes?
Title: Re: Tokens
Post by: merthsoft on November 21, 2010, 03:48:44 pm
It doesn't really have anything new over what I had up before, except for some better error-recovery and a readme :).
Title: Re: Tokens
Post by: DJ Omnimaga on November 21, 2010, 03:51:39 pm
Ah ok I see. What is a readme? J/k I'll have to try it when I have some time. My work schedule got weird again and I'm gonna be busy next weekend. X.x
Title: Re: Tokens
Post by: Munchor on November 21, 2010, 04:01:56 pm
It doesn't really have anything new over what I had up before, except for some better error-recovery and a readme :).

:) I really like Tokens, I hope some major updates are about to come, but I see none, maybe a highlighter?

Maybe an optimizer too ;)
Title: Re: Tokens
Post by: DJ Omnimaga on November 21, 2010, 04:05:38 pm
Hmm code highlighting? If you don't have that already that might be cool. I think Cooliojazz had this in his BASIC/Axe editor.
Title: Re: Tokens
Post by: Munchor on November 21, 2010, 04:06:14 pm
Hmm code highlighting? If you don't have that already that might be cool. I think Cooliojazz had this in his BASIC/Axe editor.

It is in SourceCoder too :D
Title: Re: Tokens
Post by: merthsoft on November 21, 2010, 04:08:39 pm
It's planned, I just haven't gotten around to it yet. Work and personal issues have taken most of my energy, so I just wanted to get a release out.
Title: Re: Tokens
Post by: Munchor on November 21, 2010, 04:09:39 pm
It's planned, I just haven't gotten around to it yet. Work and personal issues have taken most of my energy, so I just wanted to get a release out.

Yeah, highlighting is not very hard*, I managed to do it in C# one day :)
Title: Re: Tokens
Post by: DJ Omnimaga on November 21, 2010, 04:18:07 pm
Ah ok I see. I hope you find some more time to work on this soon.
Title: Re: Tokens
Post by: Munchor on November 22, 2010, 06:49:36 pm
Ah ok I see. I hope you find some more time to work on this soon.

I think these things like Highlighters and Updates are cool, but what he has done so far is very good! I say this because I'm trying to do a compiler too and it's going incredibly hard :(

EDIT: 700th post ;)
Title: Re: Tokens
Post by: gangsterveggies on November 22, 2010, 07:40:05 pm
And what about the source? Will it be provided?
Title: Re: Tokens
Post by: merthsoft on November 22, 2010, 09:37:13 pm
Alright, here's the ticalc link!
http://www.ticalc.org/archives/files/fileinfo/433/43315.html (http://www.ticalc.org/archives/files/fileinfo/433/43315.html)
Source will not be provided, as this is not an open source project.
Title: Re: Tokens
Post by: DJ Omnimaga on November 22, 2010, 11:15:21 pm
Cool! I'M also glad ticalc.org got fixed (an error occured for about 2 days). This project is really cool :D
Title: Re: Tokens
Post by: Munchor on December 02, 2010, 06:30:34 pm
I saw on TiCalc that a new versiow was uploaded. It looks good. Nice to see you keep on working
Title: Re: Tokens
Post by: Deep Toaster on December 05, 2010, 12:56:34 am
I'm really starting to like this :w00t:

Some feature requests (sorry if they've been mentioned before):

Also, I found a bug with the hex display: When I type in the last line in the normal box, the hex box automatically shifts so that only the first row of pixels for the last line show up (the bug I mentioned on IRC).
Title: Re: Tokens
Post by: DJ Omnimaga on December 05, 2010, 12:57:22 am
Auto Completion of commands would be like in Microsoft Visual Basic, right? That would actually be cool.
Title: Re: Tokens
Post by: Deep Toaster on December 05, 2010, 12:59:25 am
Auto Completion of commands would be like in Microsoft Visual Basic, right? That would actually be cool.

I've never tried that...

Oh, and just thought of a couple more: Find/Replace function (or at least Find), saving with Ctrl+S instead of F5 (unless you're planning to make them into projects), and a link to the readme in the About menu (if that's possible).
Title: Re: Tokens
Post by: DJ Omnimaga on December 05, 2010, 01:02:17 am
Auto Completion of commands would be like in Microsoft Visual Basic, right? That would actually be cool.

I've never tried that...
You type the beginning of a command then a drop down appears with commands starting with that prefix.

Find/Replace function (or at least Find)
And replace all.
Title: Re: Tokens
Post by: Munchor on December 05, 2010, 10:04:52 am
I'm really starting to like this :w00t:

Some feature requests (sorry if they've been mentioned before):
  • Auto-completion or tabbing to commands like in Terminal
  • Having the project name be part of the window title
  • Having an option to compile as a protected program

Also, I found a bug with the hex display: When I type in the last line in the normal box, the hex box automatically shifts so that only the first row of pixels for the last line show up (the bug I mentioned on IRC).

Yes, I also agree with those, if this is improved seriously, it can get to be the best .8xp compiler, after SourceCoder. In fact, I think it already is!
Title: Re: Tokens
Post by: merthsoft on December 05, 2010, 12:25:46 pm
These are all things I plan on putting in, but I'm glad that other people would find them useful as well. It will be project-based, so Ctrl-S will save the file as plain text, and F5 will "build" it. I'm hesitant to call this a compiler, though, as it's not really compiling anything. It's simply tokenizing the input, which isn't quite the same.
Title: Re: Tokens
Post by: Munchor on December 05, 2010, 12:42:12 pm
These are all things I plan on putting in, but I'm glad that other people would find them useful as well. It will be project-based, so Ctrl-S will save the file as plain text, and F5 will "build" it. I'm hesitant to call this a compiler, though, as it's not really compiling anything. It's simply tokenizing the input, which isn't quite the same.

I believe that tokenizing and storying the bytes in .8xp is compiling, though.

I use tokens all the time, it's fantastic:

What I noticed:

!=  stands for inequal sign

->  stands for STO Sign

This is great!
Title: Re: Tokens
Post by: merthsoft on December 05, 2010, 12:45:36 pm
It's not, though. It can be one part of compiling, but it's not really compiling itself. Compilation is the process of turning one computer language into another, and this isn't doing that. It's taking one computer language, and turning it into its tokenized form.
Title: Re: Tokens
Post by: Munchor on December 05, 2010, 03:08:26 pm
It's not, though. It can be one part of compiling, but it's not really compiling itself. Compilation is the process of turning one computer language into another, and this isn't doing that. It's taking one computer language, and turning it into its tokenized form.

You are the professional here, so I'll look up to your ideas xD

Title: Re: Tokens
Post by: DJ Omnimaga on December 05, 2010, 08:55:21 pm
These are all things I plan on putting in, but I'm glad that other people would find them useful as well. It will be project-based, so Ctrl-S will save the file as plain text, and F5 will "build" it. I'm hesitant to call this a compiler, though, as it's not really compiling anything. It's simply tokenizing the input, which isn't quite the same.

I believe that tokenizing and storying the bytes in .8xp is compiling, though.

I use tokens all the time, it's fantastic:

What I noticed:

!=  stands for inequal sign

->  stands for STO Sign

This is great!

Does it means ! and = and - and > will compile to the not equal and sto symbols? Not that this would be a problem in code syntax, but if someone uses an ASCII tilemap containing these characters one after each others, could it lead to problems?
Title: Re: Tokens
Post by: merthsoft on December 05, 2010, 08:58:53 pm
If you have "!=" it will turn in to the not equals sign. However, if you do "!\=", it will convert it to the two separate symbols.
Title: Re: Tokens
Post by: Deep Toaster on December 05, 2010, 10:54:20 pm
If you have "!=" it will turn in to the not equals sign. However, if you do "!\=", it will convert it to the two separate symbols.

Nice, so I can put \ in between any characters and it'll parse them separately?
Title: Re: Tokens
Post by: DJ Omnimaga on December 06, 2010, 01:25:30 am
I see, sounds good then. :)
Title: Re: Tokens
Post by: merthsoft on December 06, 2010, 11:18:08 am
If you have "!=" it will turn in to the not equals sign. However, if you do "!\=", it will convert it to the two separate symbols.

Nice, so I can put \ in between any characters and it'll parse them separately?
Yup! The "\" character is basically the "parse this next character as a single character" flag (or maybe it's "stop parsing this word at this charcter? I forget, I think the latter is more useful, though, so if it's not that now it will be soon).
Title: Re: Tokens
Post by: DJ Omnimaga on December 06, 2010, 01:05:42 pm
What if someone wants to type a massive RPG NPC convo in Tokens, though? ??? That must take INSANE amount of work if you have to put backslashes between every character to prevent them from being converted to actual commands! O.O
Title: Re: Tokens
Post by: merthsoft on December 06, 2010, 01:29:15 pm
Only the ones that are converted in to commands that you don't want to be tokenized. It actually helps, in some cases, because it'll save space. For instance, if the dialog is:
"Hello, friend, what would you like to buy today?"
Those will all be separate letters, because none of those are commands. However, if you have:
"Input your name:"
The "Input " will be tokenized, which would save you 9 bytes if you wanted to keep it that way. If you didn't, you'd simply put:
"Inpu\t your name:"
Title: Re: Tokens
Post by: Deep Toaster on December 06, 2010, 02:14:49 pm
Oh, and another request, having the hex box in a monospaced font.
Title: Re: Tokens
Post by: DJ Omnimaga on December 06, 2010, 02:15:12 pm
Oh right, I guess that can work. Thanks for the tip :D
Title: Re: Tokens
Post by: Munchor on December 07, 2010, 03:51:33 pm
Only the ones that are converted in to commands that you don't want to be tokenized. It actually helps, in some cases, because it'll save space. For instance, if the dialog is:
"Hello, friend, what would you like to buy today?"
Those will all be separate letters, because none of those are commands. However, if you have:
"Input your name:"
The "Input " will be tokenized, which would save you 9 bytes if you wanted to keep it that way. If you didn't, you'd simply put:
"Inpu\t your name:"

I saw your Tokens.xml and found out about all the alternative strings to be converted. :) (like -> and the real ->)
Title: Re: Tokens
Post by: Eeems on December 11, 2010, 08:23:52 pm
MOAR GUI EDITOR NEEDED!
Quote
[18:22]   <Merth>   So how about you go to the topic and yell at me to work on the DCS GUI editor ;)
[18:22]   <Eeems>   lol ok :P
Title: Re: Tokens
Post by: merthsoft on December 11, 2010, 08:26:44 pm
So, little update tonight. I've added tabbed editing, and loading/saving txt files. In addition to the text files, I've added comments. Any line that starts with "#" will be ignored once it's tokenized into a program. To demonstrate:
(http://i.imgur.com/dB9z9.png)
(http://i.imgur.com/LK5Hj.png)
Next I'll probably switch gears back toward the DCS GUI editor, as per Eeems' request :)
Title: Re: Tokens
Post by: Eeems on December 11, 2010, 08:29:52 pm
Yayz!
Title: Re: Tokens
Post by: squidgetx on December 11, 2010, 08:30:46 pm
Hey, that looks nice :)

I've pretty much neglected to tell you that Tokens is awesome for the past few weeks now, so here you go: THIS IS AWESOME :)
Title: Re: Tokens
Post by: merthsoft on December 11, 2010, 08:37:22 pm
Thanks, squidgetx, that means a lot. I'm glad people are using it :)
Title: Re: Tokens
Post by: Deep Toaster on December 12, 2010, 12:59:44 am
To sum it up:

Awesome
Awesome!
awesome,
awesome.
AWESOME :)

And the comment feature is great too :D So does this mean we can save projects now (since building to .8xp would delete them)?
Title: Re: Tokens
Post by: DJ Omnimaga on December 13, 2010, 12:55:01 am
Wow this is becoming better and better. Probably the best tool to edit BASIC programs offline now. Nice job Merth!
Title: Re: Tokens
Post by: squidgetx on December 13, 2010, 07:15:49 am
One thing I especially like is the fact that the image editor lets you change the tolerance- this means I can take a pic in 3 levels of gray and I don't need to use photoshop to separate the colors; I can just change the tolerance and get 1 pic variable with the black part and another with the gray part :)

Also I like the fact that it supports the Axe tokens (the fact that it supports other token files is pretty cool on its own)
Title: Re: Tokens
Post by: merthsoft on December 13, 2010, 09:44:27 am
Thanks DJ!

squidgetx: I'm glad the tolerance level is helpful. Some people weren't so sure it would be, and, honestly, I wasn't 100% sure it was worth it to put in. But it only took like 5 minutes to include, so I figured I might as well.
If you're feeling up to it, you're welcome to go in to the .xml file and add the Axe tokens that I've missed. Since I don't do Axe, I don't really know what everything should be.
Title: Re: Tokens
Post by: DJ Omnimaga on December 13, 2010, 10:38:51 pm
One thing I especially like is the fact that the image editor lets you change the tolerance- this means I can take a pic in 3 levels of gray and I don't need to use photoshop to separate the colors; I can just change the tolerance and get 1 pic variable with the black part and another with the gray part :)
Oh great I didn't know that. It's even better because I like to create my sprites on the computer, although I usually just use the replace color tool in photofiltre. Nice job!
Title: Re: Tokens
Post by: merthsoft on December 17, 2010, 11:15:46 pm
Alright, so I decided I should work some on the actual text editor part while also working on the DCS stuff (the DCS stuff is tedious, so I switch gears often to keep from getting too bored). I've added a reference pane, as seen here:
(http://i55.tinypic.com/vg0dvl.png)
You can set the groups and reference data in the XML file (the "Default" group is what an item without a group goes in to). Right now I have the groups, but just documentation for the stuff for DrawTileMap. Double clicking on the node adds it into the current program. I've put up a new link so you can check it out (note, some of the tabbed-editing stuff still needs to be cleaned up, so beware ;)):
http://myserverathome.com/Tokens.zip
Title: Re: Tokens
Post by: Deep Toaster on December 17, 2010, 11:37:56 pm
Really like this update! The reference pane looks really useful. Why is PicArc translated to a sequence of letter hex codes, though?

EDIT: Downloading :D
Title: Re: Tokens
Post by: merthsoft on December 17, 2010, 11:46:32 pm
Haha, when you double click it inserts it, so it's tokenizing the string "PicArc".
Title: Re: Tokens
Post by: Deep Toaster on December 18, 2010, 12:00:22 am
Oh, got it, since it isn't an actual command :D Just tried it, and it's really nice. Are you going to get rid of that bug later?
Title: Re: Tokens
Post by: merthsoft on December 18, 2010, 12:02:22 am
Yeah, there's something in there now that was supposed to prevent the non-leaf nodes from not pasting, but it didn't quite work all the way. Double clicking on the leaves, though, is supposed to paste in.
Title: Re: Tokens
Post by: DJ Omnimaga on December 18, 2010, 12:53:54 am
I like how you have the syntax being shown at the bottom. This is definitively getting more and more complete! :D
Title: Re: Tokens
Post by: Munchor on December 18, 2010, 06:37:29 am
Merth/Shaun: That looks really good!!!! It's a great idea to let us see all commands :D
Title: Re: Tokens
Post by: merthsoft on December 19, 2010, 04:15:41 pm
Alright, progress on the DCS GUI editor:
(http://i51.tinypic.com/qwygls.png)
The only items not able to be added at this point are the buttons, the sprites, and the mouse cursor. After I've added those, the next step will be to clean up the output (it's doesn't actually output correctly right now), fix the rendering of the items (at the very least use the right font), and then add the ability to save/load GUIs as XML, that way if you make one you can easily open it back up and modify it.
You may notice that GUIRWinButtons isn't an available item. I've made that part of the containers themselves. If you look at the properties pane, you can see the WinButton* items.
Also, I seem to have forgotten to include this update:
(http://i51.tinypic.com/33dbhis.png)
You can now specify a website in the XML, and you can right click to launch your browser and go to that site. If no site is found, it'll go to the site listed in the level above it. If nothing can be found, it'll yell at you.
Title: Re: Tokens
Post by: DJ Omnimaga on December 19, 2010, 04:23:09 pm
Cool progress :D
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 04:51:41 pm
It starts to remember me of Visual Studio, actually, it's very easy to design programs :9
Title: Re: Tokens
Post by: Eeems on December 19, 2010, 06:59:28 pm
merthsoft, you sir are a GOD :P
Love how this is coming along :) can't wait to play around with it :D
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:04:53 pm
merthsoft, you sir are a GOD :P
Love how this is coming along :) can't wait to play around with it :D

I will go back to BASIC when this comes out for sure! (not go back, but give it a second try!)
Title: Re: Tokens
Post by: Eeems on December 19, 2010, 07:05:56 pm
Well, this is more aimed at Hybrid with DCS...so, not quite BASIC :P
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:09:36 pm
Well, this is more aimed at Hybrid with DCS...so, not quite BASIC :P

Celtic III and xLib are BASIC for me and so is DCS, it's all based on BASIC, so it's Basic :)
Title: Re: Tokens
Post by: DJ Omnimaga on December 19, 2010, 07:14:27 pm
It works with Axe too, though, right? I think it's designed for Axe, BASIC and Hybrid.
Title: Re: Tokens
Post by: Eeems on December 19, 2010, 07:15:28 pm
Celtic III and xLib are BASIC for me and so is DCS, it's all based on BASIC, so it's Basic :)
Not based on BASIC, but an extension of BASIC
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:23:57 pm
Celtic III and xLib are BASIC for me and so is DCS, it's all based on BASIC, so it's Basic :)
Not based on BASIC, but an extension of BASIC

Upgrade*
Title: Re: Tokens
Post by: merthsoft on December 19, 2010, 07:27:10 pm
It works with Axe too, though, right? I think it's designed for Axe, BASIC and Hybrid.
It's more designed with (Hybrid)BASIC in mind, but you can do Axe. There's even an xml file for Axe, and the Hex sprites make it nice. I plan on learning Axe so that I can make it work even better for that. When I first came up with this idea (years ago...) there was no Axe. I actually first came up with this idea when Celtic III was still being written!
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:30:20 pm
It works with Axe too, though, right? I think it's designed for Axe, BASIC and Hybrid.
It's more designed with (Hybrid)BASIC in mind, but you can do Axe. There's even an xml file for Axe, and the Hex sprites make it nice. I plan on learning Axe so that I can make it work even better for that. When I first came up with this idea (years ago...) there was no Axe. I actually first came up with this idea when Celtic III was still being written!

wow, but I use tokens for Axe :D Also, I illegally use your XML files as tokens databse since the .inc file is confusing :P
Title: Re: Tokens
Post by: merthsoft on December 19, 2010, 07:32:12 pm
Also, I illegally use your XML files as tokens databse since the .inc file is confusing :P
You probably shouldn't (admit to) steal(ing) someone's intellectual property... I don't appreciate you using the file without permission for anything outside of TokenIDE.
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:33:29 pm
Also, I illegally use your XML files as tokens databse since the .inc file is confusing :P
You probably shouldn't (admit to) steal(ing) someone's intellectual property... I don't appreciate you using the file without permission for anything outside of TokenIDE.

I use it for checking tokens only. Don't worry, no commercial activity here. Also, it's not for any special program!
Title: Re: Tokens
Post by: Happybobjr on December 19, 2010, 07:34:42 pm
Also, I illegally use your XML files as tokens databse since the .inc file is confusing :P
You probably shouldn't (admit to) steal(ing) someone's intellectual property... I don't appreciate you using the file without permission for anything outside of TokenIDE.

I use it for checking tokens only. Don't worry, no commercial activity here. Also, it's not for any special program!

scout, i recommend you tread carefully right now.
Title: Re: Tokens
Post by: merthsoft on December 19, 2010, 07:35:09 pm
Well in that case, it's ok. As long as it's not getting release elsewhere or used in something that's released I'm ok with it.
Ultimately, I don't want something I worked hard on getting used in any rival software.
Title: Re: Tokens
Post by: Munchor on December 19, 2010, 07:37:24 pm
Well in that case, it's ok. As long as it's not getting release elsewhere or used in something that's released I'm ok with it.
Ultimately, I don't want something I worked hard on getting used in any rival software.

No merth, because:
1. I'd never upload it;
2. I shall not make a rival software;
3. I see the work you had organizing it and understand you do not want it stolen.
Title: Re: Tokens
Post by: DJ Omnimaga on December 19, 2010, 10:27:01 pm
Yeah it's best to ask permission before using something and releasing it with your program.
Title: Re: Tokens
Post by: Munchor on December 20, 2010, 08:08:28 am
Yeah it's best to ask permission before using something and releasing it with your program.

Yes, shaun/merth/merthsoft got it wrong :S But it's OK :D

I wouldn't be fine if someone was using my stuff :/
Title: Re: Tokens
Post by: DJ Omnimaga on December 20, 2010, 01:52:24 pm
Yeah same here, unless I allowed them to do so, in which case I prefer having some credits (and maybe a link to the site). Else it's frustrating when you spend so much time on something.
Title: Re: Tokens
Post by: merthsoft on December 21, 2010, 07:12:41 pm
Yes, shaun/merth/merthsoft...
You need only refer to me by one moniker. My chosen one on forums being merth or merthsoft ;)

Today I have an update! I've added preprocessor directives (kind of). You can #define something, and #undefine it. The syntax is:
Code: [Select]
#define word-to-replace replacement-wordand it works similar to C, just a little less functional. Basically, it'll replace all instances of word-to-replace with replacement-word, no matter where they occur in the text. This means if you have #define Map [A], it'll replace DrawTileMap with DrawTile[A], which will not give you the expected results. Also, as per the names, it only does word-by-word replacement, so you cannot #define word some series of words you want to replace. If that seems like something it ought to be able to do, I can add it (I think I probably will, that way you can do Boolean expressions). Here's a before shot:
(http://i52.tinypic.com/166epzo.png)
As you can see, comments and directives get removed from the byte-pane, using instead a blank line. I've also changed the comment-starting-thing to be "//" instead of "#", opting to use "#" as a directive starter. These will be customizable.
And here's the "compiled" program:
(http://i52.tinypic.com/6nqk9l.png)
Title: Re: Tokens
Post by: Munchor on December 21, 2010, 07:14:52 pm
I didn't really understand the define thing :S

However, nice to see you're working on this. Is there a Ctrl+F, I mean a Find/Substitute option already? I'd love to see one!
Title: Re: Tokens
Post by: Deep Toaster on December 21, 2010, 07:57:01 pm
#define! That is really useful :o Can't wait for more updates.

EDIT: It's updated on the site, right?
Title: Re: Tokens
Post by: merthsoft on December 21, 2010, 08:02:39 pm
EDIT: It's updated on the site, right?
No.
Title: Re: Tokens
Post by: DJ Omnimaga on December 21, 2010, 10:25:40 pm
Yeah I am confused at define too ???.  Sorry. Nice to see new updates, though.
Title: Re: Tokens
Post by: merthsoft on December 21, 2010, 11:57:29 pm
Yeah I am confused at define too ???.  Sorry. Nice to see new updates, though.
It's basically just aliasing a variable. So you can make anything be any value. If you use the matrix [A] to be your map matrix, you can #define MAP [A], and everywhere you would use [A], you can now use MAP. This can help make it more readable. As in the example image, the FV variable is used as the key variable, so you can alias it. You can do it with constants, too. Let's say your map width and height are 30. You can "#define map_width 30" and "#define map_height 30", and use map_width and map_height wherever you would use the constant 30 for those variable. Now, if you want to change the map_width to 45, you can just change the #define, and it'll effectively change it everywhere else in the program.
Title: Re: Tokens
Post by: DJ Omnimaga on December 22, 2010, 03:00:11 am
Ooh I see now. Should indeed make the code more readable and people who like to code for the computer might also remain interested at TI-BASIC and Axe for longer with that, since on the computer they're used to naming their variables how they want. Nice job Merth!
Title: Re: Tokens
Post by: Munchor on December 22, 2010, 06:03:39 am
I reread the first text about #define and read the second one too. Hum... I can see a lot of use for that in Axe :)
Title: Re: Tokens
Post by: merthsoft on December 25, 2010, 01:15:59 pm
I've done some more with preprocessor stuff. There's now if(n)def/else/end:
(http://i51.tinypic.com/65bsko.png)
(http://i51.tinypic.com/24dhr89.png)
You can see that it tokenizes a different line depending on what's defined and which one you use. You can embed them, as well:
(http://i56.tinypic.com/6xw9yo.png)
(http://i53.tinypic.com/29ohl7d.png)
Title: Re: Tokens
Post by: Munchor on December 25, 2010, 01:19:59 pm
Could you make elseif? That would be pretty cool. For now, looks great :)
Title: Re: Tokens
Post by: merthsoft on December 25, 2010, 01:23:51 pm
Yes.
Code: [Select]
#if blah1
...
#else
#if blah2
...
#else
#if blah3
...
#endif
#endif
#endif
This would, of course, be better with an #elseif, I just haven't gotten around to that, yet.
Title: Re: Tokens
Post by: Munchor on December 25, 2010, 01:26:14 pm
Yes.
Code: [Select]
#if blah1
...
#else
#if blah2
...
#else
#if blah3
...
#endif
#endif
#endif
This would, of course, be better with an #elseif, I just haven't gotten around to that, yet.

Please :S Else for Elseif is confusing :S
Title: Re: Tokens
Post by: merthsoft on December 25, 2010, 02:22:23 pm
I'm not sure why else if is any more confusing that elseif, but ok:
(http://i51.tinypic.com/anbm7k.png)
(http://i51.tinypic.com/2j1k5s4.png)
(http://i54.tinypic.com/1zyy4a8.png)
Title: Re: Tokens
Post by: Eeems on December 25, 2010, 04:02:54 pm
This is looking great Merth!
Title: Re: Tokens
Post by: jnesselr on December 25, 2010, 05:05:51 pm
Very nice! Any idea when this will be completed?
Title: Re: Tokens
Post by: Munchor on December 25, 2010, 05:08:34 pm
Very nice! Any idea when this will be completed?

I can't wait for a release too.
Title: Re: Tokens
Post by: merthsoft on December 26, 2010, 09:13:28 pm
Very nice! Any idea when this will be completed?
I don't really have a time line. "When it's done", I suppose.
Title: Re: Tokens
Post by: DJ Omnimaga on December 27, 2010, 04:04:34 pm
Nice merthsoft! What is ifdef, though? Does it check if the last thing that was defined is equal to onething?
Title: Re: Tokens
Post by: jnesselr on December 27, 2010, 04:21:14 pm
Nice merthsoft! What is ifdef, though? Does it check if the last thing that was defined is equal to onething?
If Defined. It checks if something is defined or not. In higher languages like Java, it is basically "If Object!=null".
Title: Re: Tokens
Post by: DJ Omnimaga on December 27, 2010, 04:24:51 pm
Oh ok, thanks.
Title: Re: Tokens
Post by: merthsoft on December 27, 2010, 06:10:32 pm
Alright, I figured I ought to release a build with all these updates. Note, the readme is not updated.
http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip)
Title: Re: Tokens
Post by: Munchor on December 27, 2010, 06:11:34 pm
Alright, I figured I out to release a build with all these updates. Note, the readme is not updated.
http://myserverathome.com/Tokens.zip (http://myserverathome.com/Tokens.zip)

Love you man :w00t:
Title: Re: Tokens
Post by: Binder News on December 27, 2010, 06:13:31 pm
Very nice!
Title: Re: Tokens
Post by: DJ Omnimaga on December 27, 2010, 10:09:52 pm
Nice Merthsoft!
Title: Re: Tokens
Post by: DJ Omnimaga on December 30, 2010, 12:49:57 am
/me can't wait until it is possible to open files with tokens by drag'n'dropping them on the app shortcut or app itself (icon or while it's open). :P
Title: Re: Tokens
Post by: Munchor on December 31, 2010, 11:57:53 am
/me can't wait until it is possible to open files with tokens by drag'n'dropping them on the app shortcut or app itself (icon or while it's open). :P

Yeah, that's true! I sometimes do that but it doesn't work :S I am used to WabbitEmu I guess.
Title: Re: Tokens
Post by: merthsoft on March 06, 2011, 03:12:21 pm
Ok, I've cleaned some stuff up and made an official release, check it out:
(http://www.ticalc.org/archives/files/ss/816/81682.gif)
Download it at:
http://www.ticalc.org/archives/files/fileinfo/433/43315.html (http://www.ticalc.org/archives/files/fileinfo/433/43315.html)
Title: Re: Tokens
Post by: ralphdspam on March 06, 2011, 04:01:28 pm
O.O Very nice.  
I do suggest color coding.  (if you were to highlight variables in one color and commands in another)
Now I don't have to use TI's discontinued editor. :)
Title: Re: Tokens
Post by: DJ Omnimaga on March 07, 2011, 09:29:30 pm
Oooh nice, I like the update. :D

I'll need to update at one point.
Title: Re: Tokens
Post by: merthsoft on June 09, 2011, 10:53:18 pm
Ugh, I haven't updated here in forever. Shame on me.

Here's a recent screenshot:
(http://i.imgur.com/1ygaR.png)
Stuff I've added:
Here's the download!
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: DJ Omnimaga on June 10, 2011, 10:20:15 pm
That's awesome! By the way are programs unlocked by default?

I think a nice feature would be a tool to batch lock/unlock/archive programs, so it's easier to edit old calc games or make games compatible with older emulators like PindurTI.
Title: Re: Tokens
Post by: defmenge on June 11, 2011, 05:58:37 am
Great program Merthsoft, this is really useful for working on larger projects!

I'm currently using it to work on my DROD clone in Axe (which, afaik, hasn't been done before), though there is one small thing bothering me about TokenIDE: when you save with Ctrl-S, you have to confirm the overwrite everytime, which is slightly annoying to people like me, who have a habit of quicksaving often to prevent data loss. ;)

Also, the included AxeTokens.xml did not work for me, importing it caused no new category or any changes to appear. I figured out the XML format and found out that the Axe commands are not assigned to any new group. AxeTokens.xml can be fixed by adding the line
Code: [Select]
<Group name="Axe" comment="The Axe parser commands." site="http://axe.omnimaga.org" /> inside the "<Tokens [...]>" tag, and adding
Code: [Select]
group="Axe" into every "<Token [...]>" tag.
This causes a new "Axe" group to appear which contains all commands displayed as if you were using Axe.

EDIT:
It seems that changing the Token file does not change the Token parser's behaviour. It will still parse the original (Tokens.xml) Tokens while only the reference list is changed.


EDIT2:
...and suddenly, it works. :P
Title: Re: Tokens
Post by: DJ Omnimaga on June 11, 2011, 02:36:35 pm
Are you planning to announce your DROD game by the way? It seems kinda fun and people might be interested here :)
Title: Re: Tokens
Post by: merthsoft on June 11, 2011, 04:21:16 pm
That's awesome! By the way are programs unlocked by default?

I think a nice feature would be a tool to batch lock/unlock/archive programs, so it's easier to edit old calc games or make games compatible with older emulators like PindurTI.
Programs are whatever they are in the 8xp by default. If you're starting a new one or loading it in from a txt file, they're unlocked, though. As for the batch lock/unlock/archive, that'll all be included when I get the whole "project" system in place, but good idea.

Great program Merthsoft, this is really useful for working on larger projects!

I'm currently using it to work on my DROD clone in Axe (which, afaik, hasn't been done before), though there is one small thing bothering me about TokenIDE: when you save with Ctrl-S, you have to confirm the overwrite everytime, which is slightly annoying to people like me, who have a habit of quicksaving often to prevent data loss. ;)

Also, the included AxeTokens.xml did not work for me, importing it caused no new category or any changes to appear. I figured out the XML format and found out that the Axe commands are not assigned to any new group. AxeTokens.xml can be fixed by adding the line
Code: [Select]
<Group name="Axe" comment="The Axe parser commands." site="http://axe.omnimaga.org" /> inside the "<Tokens [...]>" tag, and adding
Code: [Select]
group="Axe" into every "<Token [...]>" tag.
This causes a new "Axe" group to appear which contains all commands displayed as if you were using Axe.
Thanks for the input! You should definite upload/email me your changes with the axe tokens file so that I can include it in the download. I haven't really done a whole lot of work with Axe or that file, it's mostly been other contributors.
Title: Re: Tokens
Post by: Munchor on June 11, 2011, 04:23:37 pm
Great program Merthsoft, this is really useful for working on larger projects!

I'm currently using it to work on my DROD clone in Axe (which, afaik, hasn't been done before), though there is one small thing bothering me about TokenIDE: when you save with Ctrl-S, you have to confirm the overwrite everytime, which is slightly annoying to people like me, who have a habit of quicksaving often to prevent data loss. ;)

Also, the included AxeTokens.xml did not work for me, importing it caused no new category or any changes to appear. I figured out the XML format and found out that the Axe commands are not assigned to any new group. AxeTokens.xml can be fixed by adding the line
Code: [Select]
<Group name="Axe" comment="The Axe parser commands." site="http://axe.omnimaga.org" /> inside the "<Tokens [...]>" tag, and adding
Code: [Select]
group="Axe" into every "<Token [...]>" tag.
This causes a new "Axe" group to appear which contains all commands displayed as if you were using Axe.

EDIT:
It seems that changing the Token file does not change the Token parser's behaviour. It will still parse the original (Tokens.xml) Tokens while only the reference list is changed.


EDIT2:
...and suddenly, it works. :P

Did you change the Token file to AxeTokens.xml? You have to do that in order to use Axe tokens and since it always worked fine for me, I'm wondering if you did.

In fact, I also gave Merth an updated AxeTokens.xml that he will update in newest version :)
Title: Re: Tokens
Post by: merthsoft on June 11, 2011, 04:25:55 pm
Did you change the Token file to AxeTokens.xml? You have to do that in order to use Axe tokens and since it always worked fine for me, I'm wondering if you did.

In fact, I also gave Merth an updated AxeTokens.xml that he will update in newest version :)
I'm pretty sure he got it working. Also, your version is in the current download :D
Title: Re: Tokens
Post by: defmenge on June 11, 2011, 04:47:59 pm
Are you planning to announce your DROD game by the way? It seems kinda fun and people might be interested here :)
I'm planning to announce it on Omnimaga when I have at least a few screenshots to show. ;)
Currently the game is in the stage of AI and file I/O tests, I have to experiment quite a lot since this is my first real Axe project ever.

Anyway, back on topic: Merthsoft, I can send you an updated version of AxeTokens.xml if you still need one, as I said, I added an "Axe" group which contains all commands in their Axe versions. I also replaced the "u", "v" and "w" variables with "var", "appv" and "grp" respectively. I thought about attaching it, but I guess there is some minimum post number I need to reach before being able to attach files. :P

Did you change the Token file to AxeTokens.xml? You have to do that in order to use Axe tokens and since it always worked fine for me, I'm wondering if you did.
I used the "Change Token File" menu option to change it, and, if I recall correctly, it caused no groups to show up in the reference list. The tokenization might be working, but the list of commands is still useful to newbies like me who don't remember all of the commands yet. :P
Title: Re: Tokens
Post by: merthsoft on June 11, 2011, 04:51:00 pm
Anyway, back on topic: Merthsoft, I can send you an updated version of AxeTokens.xml if you still need one, as I said, I added an "Axe" group which contains all commands in their Axe versions. I also replaced the "u", "v" and "w" variables with "var", "appv" and "grp" respectively. I thought about attaching it, but I guess there is some minimum post number I need to reach before being able to attach files. :P
Yes, please do: shaunm.mcfall at gmail. Thanks :)

Edit:
Ok, he sent me the file and I looked in to it a little bit. Instead of adding the "Axe" group to each token, you can alternatively just add the main "_default" group. So just add:
Code: [Select]
<Group name="_default" comment="The standard TI-BASIC commands." site="http://tibasicdev.wikidot.com/home" /> or
Code: [Select]
<Group site="http://axe.omnimaga.org/" comment="The Axe parser commands." name="_default"/>right after the "Tokens" tag, and it should make that the default group. That should require less editing and, I think, a cleaner file.
Title: Re: Tokens
Post by: merthsoft on June 21, 2011, 06:32:13 pm
Alright, I've made a couple updates:
Next I'm working on list/matrix manipulation.
Download at the normal spot:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: Deep Toaster on June 21, 2011, 06:59:49 pm
Awesome I love drag-n-dropping files onto open apps :D
Title: Re: Tokens
Post by: merthsoft on June 22, 2011, 09:03:41 am
Yeah, that's something I've wanted to add for a while, I just never got around to it. Ideally, I should be able to make it so you can drag images on to it, and it'll open up the image editor, too.
Title: Re: Tokens
Post by: defmenge on June 27, 2011, 08:17:24 am
Nice update! It's a lot easier to work with it now. ;)

However, I noticed it started getting very slow when my project got bigger. Could you add a way to toggle retokenization on/off, so it doesn't get so slow when typing in big projects? Right now, every change causes TokenIDE to retokenize the whole program, which takes about 4 seconds for me, rendering it unresponsive during that time. It gets kind of annoying after some time and it would be nice to have a checkbox or something similar to turn instant tokenization on and off.
Title: Re: Tokens
Post by: merthsoft on June 27, 2011, 02:57:31 pm
Sure, I can work on that. Other people have noticed the same thing, too.
Title: Re: Tokens
Post by: Deep Toaster on June 27, 2011, 03:06:29 pm
Is there some way to tokenize just the current line?
Title: Re: Tokens
Post by: merthsoft on June 27, 2011, 08:57:22 pm
You can copy the line and open a tab and paste it in there.
Title: Re: Tokens
Post by: merthsoft on June 28, 2011, 08:39:16 pm
Alright, I've made some changes. First off, I've removed the "bytes" pane, since I think it's mostly useless now that there's the Tokens pane. I've also added a "live update" switch to the Tokens pane. When this is checked, the Tokens pane will update once you've stopped typing for half a second. If that seems too long or short, let me know. I'll probably make it a setting eventually. I've also added a "line numbers preview" tool (Ctrl+L) until I'm smart enough to get a real text editor in there. Here's a screen shot:
(http://i.imgur.com/a66Mq.png)
I've been googling around trying to find a good solution, but everything either uses a billion P/Invokes, or GTK, neither of which I really feel like dealing with. I might have to craft my own, which I tried a little bit, but with no real successful attempts. I'll keep working on it as I find motivation. I guess what I want to know from you guys is what should I work on next:
- Line numbers/syntax highlighting
- Matrix/List editors
- String/Y-Vars support

Let me know. I might not take do what you suggest, but it'll be taken into consideration. As always, the download link is:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: defmenge on June 29, 2011, 05:05:08 pm
Good job on this update, now I can finally work on DROD8x again without having to wait 3 seconds to type a single character. :P
Thank you a lot for this great program, I'd certainly +1 you if I could! xP
Title: Re: Tokens
Post by: XVicarious on June 29, 2011, 05:20:12 pm
What about the token in Axe "appv" which takes place of 2nd + 8?
and:
var (2nd + 7)
grp (2nd + 9)

If I knew how the token XML file was structured and about the hex codes etc I could add them myself, but I really have no clue about that.
Title: Re: Tokens
Post by: Deep Toaster on June 29, 2011, 05:25:15 pm
Alright, I've made some changes. First off, I've removed the "bytes" pane, since I think it's mostly useless now that there's the Tokens pane. I've also added a "live update" switch to the Tokens pane. When this is checked, the Tokens pane will update once you've stopped typing for half a second. If that seems too long or short, let me know. I'll probably make it a setting eventually. I've also added a "line numbers preview" tool (Ctrl+L) until I'm smart enough to get a real text editor in there.

Awesome updates! Downloaded :D

- Line numbers/syntax highlighting

That.
Title: Re: Tokens
Post by: merthsoft on June 29, 2011, 05:30:51 pm
Thanks guys :)
jkag: I'll add those in.
Title: Re: Tokens
Post by: XVicarious on June 29, 2011, 05:34:35 pm
Thanks, and possibly could you add support for subscripted numbers on Pic and Str variables? To make it a little more cross compatible with other IDEs (namely tiDE)?
Title: Re: Tokens
Post by: Deep Toaster on June 29, 2011, 05:39:59 pm
What's subscripted in Strs and Pics?
Title: Re: Tokens
Post by: XVicarious on June 29, 2011, 05:41:22 pm
The number like this: Str₃
Title: Re: Tokens
Post by: Deep Toaster on June 29, 2011, 05:45:23 pm
Isn't that written as Str3 on the calculator?
Title: Re: Tokens
Post by: XVicarious on June 29, 2011, 05:52:56 pm
Yes, but tiDE makes it like that, and that is really another valid way of writing it.
Title: Re: Tokens
Post by: Runer112 on June 29, 2011, 06:09:26 pm
I'm not really sure why you would want to purposely misrepresent tokens. If that's what tiDE uses, I might even ask if that's a bug.
Title: Re: Tokens
Post by: XVicarious on June 29, 2011, 06:18:43 pm
Well I'd ask SirCmpwn about that, but anyhow this is getting a little off topic.
Title: Re: Tokens
Post by: merthsoft on June 29, 2011, 10:17:29 pm
Ok, so I made an update that greatly increased performance of the live updating. It should now be able to tokenize it and display the tokens without any huge performance hit (if you get a big enough file you'll notice it a little bit, but nothing too big, I think). I've also added a bytes counter, so you know how big your program is:
(http://i.imgur.com/es9jK.png)
I've also fiddled with what the tabs display, now they just display the name of the program, no extension, no matter what you open. That's because save will always save a txt, and compile will always save a calc file, no matter the extension of what you opened. I'm not terrible satisfied with any of that, but of well.
Download in the same place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)

I haven't updated the xml files at all with this update, I've been busy programming.

I'm still not totally sure what I'll be working on next. I think it'll probably be strings/y-vars, list/matrix, then maybe an "official update" to ticalc, and then work on the text editor.
Title: Re: Tokens
Post by: ralphdspam on June 30, 2011, 12:19:11 am
Sounds like a nice update (especially the byte counter).  :)
Do you have any plans for syntax coloring?
Title: Re: Tokens
Post by: Deep Toaster on June 30, 2011, 01:28:57 pm
I also like the bytes counter :) Updated mine.

EDIT: Or I would have if that link worked. Your site's down :(
Title: Re: Tokens
Post by: merthsoft on June 30, 2011, 04:36:59 pm
tifreak uploaded it for me while my site's down:
http://tifreakware.net/merthsoft/Tokens.zip (http://tifreakware.net/merthsoft/Tokens.zip)
Title: Re: Tokens
Post by: DJ Omnimaga on July 01, 2011, 02:18:33 pm
Awesome update Merth :)
Title: Re: Tokens
Post by: defmenge on July 06, 2011, 01:33:04 pm
It seems TokenIDE sometimes saves files as only their program name, without any extension. Trying to open an extensionless file throws the error "File type not supported", so you have to rename them everytime you want to reopen the program. Also it caused lots of confusion about which file was the latest version and which one may have been an older version, as it seems to sometimes save a ".txt" and sometimes not, I don't want to lose any code to accidentally overwriting a file.
Please try to fix this bug, as I can't really work on DROD8x if I can't be sure that I don't mix old and new code. It shouldn't be too hard to just append a ".txt" extension to every file name to the saving files function. :P
Title: Re: Tokens
Post by: merthsoft on July 06, 2011, 04:21:36 pm
Hmm, that is weird. I'll look into that tonight.
Title: Re: Tokens
Post by: defmenge on July 06, 2011, 04:40:25 pm
Hmm, that is weird. I'll look into that tonight.
Thank you very much! This is an awesome program, keep up the great work. ;)
Title: Re: Tokens
Post by: merthsoft on July 06, 2011, 08:48:22 pm
Alright, I fixed this bug. Sorry about that! I've also made it so it remembers where you last tokenized the file to, so it doesn't prompt you each time. You can change the directory in the file menu.
Download in the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)

Also, I've been working on list and matrix support. I don't have the editors yet, but here's some pics of my tests:
Lists:
(http://i.imgur.com/WpbOZ.png)
Matrix:
(http://i.imgur.com/x9pZt.png)

Also, there's a Tokens IRC channel, just for fun. So hop on #TokenIDE on EFNet if you want :).
Title: Re: Tokens
Post by: squidgetx on July 06, 2011, 08:54:12 pm
This looks great, and has come a long way since I last downloaded it...Ready to update as soon as my computer gets fixed haha.
Title: Re: Tokens
Post by: Munchor on July 07, 2011, 06:43:58 am
Merth, was the slider updating bug fixed (because of Mono)? Thanks
Title: Re: Tokens
Post by: merthsoft on July 07, 2011, 12:13:50 pm
Merth, was the slider updating bug fixed (because of Mono)? Thanks
Not sure what you mean.
Title: Re: Tokens
Post by: Munchor on July 07, 2011, 12:14:49 pm
Merth, was the slider updating bug fixed (because of Mono)? Thanks
Not sure what you mean.

TokenIDE ran very slow on Linux using Mono and you said it's because of the sliders updating and then someone at Cemetech suggested a way of fixing that. Did you?
Title: Re: Tokens
Post by: merthsoft on July 07, 2011, 12:19:27 pm
It was running slowly because of the Tokens pane, not because of any sliders, and that has been fixed:
Ok, so I made an update that greatly increased performance of the live updating. It should now be able to tokenize it and display the tokens without any huge performance hit (if you get a big enough file you'll notice it a little bit, but nothing too big, I think).
It doesn't update until you stop typing, and does a relatively quick job of updating it. You can also turn the live updating off completely.
Title: Re: Tokens
Post by: XVicarious on July 12, 2011, 03:09:04 pm
Could you maybe eventually add the ability of hiding the tokenized window? I know you can just fold it up, but actually closing it would be nice as well.
Title: Re: Tokens
Post by: nemo on September 28, 2011, 11:38:47 pm
I'm not sure what the status of this project is, but would it be possible to have an option to use a monospaced font like Courier?

Edit: nevermind, Tokens uses the monospaced font Consolas, which i didn't have on my computer.
Title: Re: Tokens
Post by: merthsoft on May 28, 2012, 02:37:29 pm
Alright, I haven't posted here in forever. There's been progress. Syntax highlighting and token underlining and stuff. Here:
(http://img1.uploadscreenshot.com/images/orig/5/14804530749-orig.png)
Right now the only color changing that's going on is strings and comments (and preprocessor directives), but you can add it through the XML file. You can control foreground color, background color, text style, how many characters long a token has to be to underline it (-1 for no underlining), and what color the underline is. Check out the xml file if you want to see how it's laid out. The readme will be updated, um, eventually.

Keep in mind that the tokens files changed (including the schema), so this is a breaking changed with your old tokens files. Just put the "Group" tags inside a <Groups></Groups> section and add a default style and it should work. There were also some ambiguous tokens in the Tokens.XML file and the AxeTokens.xml file, and it now crashes on ambiguous tokens (sorry!). Project stuff is disabled for now.

This does work under Mono, btw:
(http://img1.uploadscreenshot.com/images/orig/5/14805253390-orig.png)

Download at the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: Adriweb on May 28, 2012, 02:39:48 pm
Wow, pretty nice :D
Congratulations!
Title: Re: Tokens
Post by: merthsoft on May 29, 2012, 11:40:00 pm
Alright, there were some bugs in how it was handling alt tokens and preprocessor directives. These have mostly been addressed:
(http://img1.uploadscreenshot.com/images/orig/5/14914204949-orig.png)
And just because I've been more rigorous about Mono support, here's it in Mono on Windows:
(http://img1.uploadscreenshot.com/images/orig/5/14914224384-orig.png)

The font can be controlled in the styles section of the XML.
I removed the tokens pane since it underlines tokens now. I've also improved performance a bit as well as made it a bit smaller.

Download in the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: DJ Omnimaga on May 29, 2012, 11:58:36 pm
Great updates Merth. Glad you're still working on this. :)

By the way for the archived checkbox, I think you should make Tokens warn with an alert box that checking that box will cause your programs to not be compatible with every emulator and linking software, in case people wonder why they get so many reports about their game not running or sending.
Title: Re: Tokens
Post by: jsj795 on May 30, 2012, 12:18:46 am
oooh nice, update :)
Title: Re: Tokens
Post by: merthsoft on June 01, 2012, 10:24:13 pm
Great updates Merth. Glad you're still working on this. :)

By the way for the archived checkbox, I think you should make Tokens warn with an alert box that checking that box will cause your programs to not be compatible with every emulator and linking software, in case people wonder why they get so many reports about their game not running or sending.
Thanks Kevin! As for the archived checkbox, I'm not sure if I'll add that or not, but it seems like a good idea.

Meanwhile, I started working on a list/matrix editor:
(http://img1.uploadscreenshot.com/images/orig/6/15213012695-orig.png)
Title: Re: Tokens
Post by: jsj795 on June 04, 2012, 05:46:36 am
Great updates Merth. Glad you're still working on this. :)

By the way for the archived checkbox, I think you should make Tokens warn with an alert box that checking that box will cause your programs to not be compatible with every emulator and linking software, in case people wonder why they get so many reports about their game not running or sending.
Thanks Kevin! As for the archived checkbox, I'm not sure if I'll add that or not, but it seems like a good idea.

Meanwhile, I started working on a list/matrix editor:
(http://img1.uploadscreenshot.com/images/orig/6/15213012695-orig.png)

The list/matrix editor is not yet in the newest version, right?

Also, with this version I can't distinguish between | and >.
Specifically, in a string, I want to put in '|' and '>=' but it gives me '|>' and '='.
Anyway I can fix this? I tried looking into XML file but i wasn't sure what I had to do.
Title: Re: Tokens
Post by: blue_bear_94 on June 04, 2012, 09:36:22 am
Great updates Merth. Glad you're still working on this. :)

By the way for the archived checkbox, I think you should make Tokens warn with an alert box that checking that box will cause your programs to not be compatible with every emulator and linking software, in case people wonder why they get so many reports about their game not running or sending.
Thanks Kevin! As for the archived checkbox, I'm not sure if I'll add that or not, but it seems like a good idea.

Meanwhile, I started working on a list/matrix editor:
(http://img1.uploadscreenshot.com/images/orig/6/15213012695-orig.png)

The list/matrix editor is not yet in the newest version, right?

Also, with this version I can't distinguish between | and >.
Specifically, in a string, I want to put in '|' and '>=' but it gives me '|>' and '='.
Anyway I can fix this? I tried looking into XML file but i wasn't sure what I had to do.
Why we should use backslashes like |\>=\...
Title: Re: Tokens
Post by: jsj795 on June 04, 2012, 09:41:42 am
but what if you want to put the backslashes in a string?
Title: Re: Tokens
Post by: blue_bear_94 on June 04, 2012, 09:42:36 am
Then you'd use two backslashes: \\
Title: Re: Tokens
Post by: jsj795 on June 04, 2012, 09:45:03 am
hm. yeah but this new version didn't allow that. the old version did tho
Title: Re: Tokens
Post by: blue_bear_94 on June 04, 2012, 09:45:38 am
At least, that's how I'd do it.
Title: Re: Tokens
Post by: DJ Omnimaga on June 04, 2012, 10:51:32 am
Will matrix editing be on a grid like on the 83+ or will it be in one single line like the TI-81? The former would be nice and something cool would be if you made some sort of tilemapping helper, where you can assign tiles from 8xi files to a specific matrix number (excluding the decimal, for those who use the fpart() trick in xLIB). It would make it easier for people using xLIB to create map. :D

Of course, this feature would be better if such matrix editor came with a way to export matrix data into BASIC code or even import it (in case you do a mistake).
Title: Re: Tokens
Post by: merthsoft on June 06, 2012, 10:00:14 am
Huh, yup, looks like two backslashes breaks it. Thanks for finding that, I'll look into it tonight and see what's going on.

DJ_O: At first I'm hoping to make it so you can copy/paste the data into the program editor and the list/matrix editor, so that if you have a matrix in your program data you can more easily edit it. As for the map editor, that'll be something for the future.

The matrix editor will be a grid, kind of like Excel.
Title: Re: Tokens
Post by: merthsoft on June 06, 2012, 10:24:28 pm
OK, I've made an update. Three changes, really:
1) File handling code now simpler, which makes my life easier.
2) Fixed bug where backslash wasn't working correctly.
3) Added some more icons, courtesy of Kerm. Here's some of them:
(http://img1.uploadscreenshot.com/images/orig/6/15713114747-orig.png)
Title: Re: Tokens
Post by: jsj795 on June 07, 2012, 12:45:19 am
backslash still doesn't work well even after the update. It still throws me unhandled exception, and also \|>= gives me \,|>,= not |, >=.
Title: Re: Tokens
Post by: blue_bear_94 on June 07, 2012, 10:02:26 am
backslash still doesn't work well even after the update. It still throws me unhandled exception, and also \|>= gives me \,|>,= not |, >=.
I think you have to do \|>\=.
Title: Re: Tokens
Post by: jsj795 on June 07, 2012, 10:05:26 am
backslash still doesn't work well even after the update. It still throws me unhandled exception, and also \|>= gives me \,|>,= not |, >=.
I think you have to do \|>\=.

when i write backslash, it automatically gives me error.
Title: Re: Tokens
Post by: merthsoft on June 07, 2012, 10:17:28 am
I'm not having any issues--are you sure you downloaded the right version and not a cached version? Your browser might not be downloading the newest one, so clear your cache on the merthsoft domain. To make sure, you can check the icons in the "Tools" menu. If the Hex Sprite Editor doesn't have an icon next to it, that's an older version.

If all that checks out, and you have the newest version but it's still crashing, tell me exactly what you're trying to type so I can try to reproduce it.
Title: Re: Tokens
Post by: jsj795 on June 07, 2012, 10:24:27 am
i do have the icons, that's why I was confused. I was typing:

Text(9,16,sub(" \|>=>|>Sigma\|<=<<|3",F,1

Whenever I type that backslash, it throws me unhandled exception, I'll post up exactly what it says tmmr, but yeah.
Even if I ignore it by pressing Continue, the groups are wrong. What I want it to be is:
|, >=, >, |>, Sigma, |, <=, <, <|, 3
but I'm getting:
\, |>, =, >, |>, Sigma, \, |, <=, <, <|, 3

Notice the backslashes in there, and also that |> instead of >= in the 2nd one.
Title: Re: Tokens
Post by: merthsoft on June 07, 2012, 12:33:45 pm
Hmm, even typing that in, I'm still not getting an exception, and the groupings I get are the right ones. Very weird. Is that the only line you're typing?

Also, try deleting the whole tokens folder and re-extracting the zip (or just deleting the exe and dlls).
Title: Re: Tokens
Post by: jsj795 on June 07, 2012, 06:07:15 pm
Yes I deleted the whole folder and extracted it. I'm heading to work now so I can provide u with more detail soon

Edit: Okay this is the message I get.

Unhandled exception has occurred in your application. If you click Continue, the application will ignore this error and attempt to continue. If you click Quit, the application will close immediately.

startIndex cannot be larger than length of string. Parameter name: startIndex.

Spoiler For detail:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.ArgumentOutOfRangeException: startIndex cannot be larger than length of string.
Parameter name: startIndex
   at System.String.InternalSubStringWithChecks(Int32 startIndex, Int32 length, Boolean fAlwaysCopy)
   at System.String.Substring(Int32 startIndex)
   at TokenIDE.Prog8xEditWindow.UpdateHighlight(List`1 tokens, Range range)
   at TokenIDE.Prog8xEditWindow.ProgramText_TextChanged(Object sender, TextChangedEventArgs e)
   at FastColoredTextBoxNS.FastColoredTextBox.timer2_Tick(Object sender, EventArgs e)
   at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/mscorlib.dll
----------------------------------------
TokenIDE
    Assembly Version: 0.10.0.0
    Win32 Version: 0.10.0.0
    CodeBase: file:///C:/Users/Student/Downloads/Calculator/TokenIDE/TokenIDE.exe
----------------------------------------
System.Windows.Forms
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Xml
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Merthsoft.Tokens
    Assembly Version: 0.9.4540.36597
    Win32 Version: 0.9.0.0
    CodeBase: file:///C:/Users/Student/Downloads/Calculator/TokenIDE/Merthsoft.Tokens.DLL
----------------------------------------
Merthsoft.Extensions
    Assembly Version: 1.0.0.0
    Win32 Version: 1.0.0.0
    CodeBase: file:///C:/Users/Student/Downloads/Calculator/TokenIDE/Merthsoft.Extensions.DLL
----------------------------------------
Merthsoft.CalcData
    Assembly Version: 1.0.0.0
    Win32 Version: 1.0.0.0
    CodeBase: file:///C:/Users/Student/Downloads/Calculator/TokenIDE/Merthsoft.CalcData.DLL
----------------------------------------
FastColoredTextBox
    Assembly Version: 2.8.1.0
    Win32 Version: 2.8.1.0
    CodeBase: file:///C:/Users/Student/Downloads/Calculator/TokenIDE/FastColoredTextBox.DLL
----------------------------------------
System.Core
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.17379 built by: FXBETAREL
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
    <system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

this happens when I type in backslash anywhere in TokenIDE (including within the string).

I don't know if you update your version number, but it's 0.10.0.0, and I have icons next to all Hex Sprite Editor, DSC Gui Designer, and Image Editor.
Title: Re: Tokens
Post by: merthsoft on June 07, 2012, 10:53:41 pm
Do you get a pop-up error before that happens?

EDIT:
I've updated it some more to add squiggly lines for untokenizable characters, as well as grouping of the backslashes so you can better tell what's a token:
(http://img1.uploadscreenshot.com/images/orig/6/15814282584-orig.png)
The version for this one is 0.10.2.0, so make sure you're using that version and let me know if you still get the errors.
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: jsj795 on June 08, 2012, 12:18:34 am
what I write was the pop up error. And I'll try the new version and let you know.

Edit: Okay, the groupings actually work like they're supposed to, but I still get an error when I write a backslash (I can ignore it, but still)

This happens when I write the backslash at line 10

(http://img.removedfromgame.com/imgs/tokenIDE%20error.png)
Title: Re: Tokens
Post by: merthsoft on June 08, 2012, 07:28:03 am
Ok! I was able to reproduce it. I'll work on fixing it. In the meantime, the workaround is to not have a single backslash and then a newline--always have something after the backslash (which is annoying, I know). I should get an update up soon.
Title: Re: Tokens
Post by: merthsoft on June 11, 2012, 12:57:55 pm
I think I've managed to fix it. Not sure how "\r" will get handled if it's actually in a token.xml file, but for now that's not a huge deal since the calcs don't use it. Here's a screen shot:
(http://img1.uploadscreenshot.com/images/orig/6/16203341571-orig.png)
The backslashes seem to be behaving.
Download at the normal spot: http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: merthsoft on June 11, 2012, 03:19:26 pm
I know this is soon for a double post, but I thought it important to let everyone know of a bug I found: If you press ctrl+s after having opened an .8xp or .8xv file, it will overwrite the file with a text-version of the program. On top of that, saving makes it have double new lines. I'm working on making this work right, but I'm at the airport right now and my battery is about to die. So, just be careful saving things :)
Title: Re: Tokens
Post by: jsj795 on June 11, 2012, 07:37:14 pm
ah okay. I never actually used the save button (only compile button, usually spamming F5 :P)

Also, I'm not sure if this was fixed in the latest version or not, but Token seems to write random 2 commands at the end of my program when I open up a program.

(http://img.removedfromgame.com/imgs/token%20end%20program%20bug.png)

(http://img.removedfromgame.com/imgs/token%20end%20program%20bug%201.png)

Will try the new version and see if this persists

Edit: tried the new version and yes this problem persists. I deleted the 2 commands and then compiled it, exited out of tokenIDE and then reopened the compiled program, and it gave me same 2 random commands (For( and nDeriv( for 1038line program.

Also, pressing F5 at the new file (that never got compiled) and then cancelling at the program name prompt gives unhandled error.
After this error, writing anything in it gives unhandled error.
Title: Re: Tokens
Post by: merthsoft on June 11, 2012, 10:24:13 pm
Ok, that was happening because if was reading in the checksum as tokens, oops! That should be fixed now. The saving issue is still there, though.
Title: Re: Tokens
Post by: jsj795 on June 12, 2012, 02:48:57 am
oh okay did u update the version that fixes the tokenizing checksum?
Title: Re: Tokens
Post by: merthsoft on June 12, 2012, 10:25:18 am
Yeah, it should be fixed in the released version now.
Title: Re: Tokens
Post by: merthsoft on June 15, 2012, 07:48:33 pm
Seems I actually didn't updated. I've uploaded a new version that should have this and the saving issue fixed. If it's still not working, please let me know.
Title: Re: Tokens
Post by: jsj795 on June 15, 2012, 08:07:09 pm
The checksum tokenizing has been fixed, but I still get unhandled error trying to save the newfile and then cancelling it at the naming prompt(although this is not a really big deal)
Title: Re: Tokens
Post by: merthsoft on June 15, 2012, 08:12:10 pm
Yeah, I didn't attempt to fix that, I was just trying to get the more important stuff fixed.
Title: Re: Tokens
Post by: jsj795 on June 15, 2012, 08:13:20 pm
yeah it's not really important :) Otherwise, it seems pretty good and stable so far!
Title: Re: Tokens
Post by: merthsoft on June 24, 2012, 08:33:01 pm
Thanks to some help from Deep Though, I've fixed (once again) how saving is handled. Note that any programs you had previously compiled with tokens may now have those extra bytes at the end, but cleaning and rebuilding should fix it. Download at the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: Hayleia on July 09, 2012, 06:56:47 am
I have to say that this is awesome. I tried it and now I wonder how I could do without it :)
However, there is one feature that would be a great addition. I'll try to explain but I am French so it might not be understandable.
This feature is the parenthese assistant. When the cursor is next to a parenthese, the parenthese is colorized and the matching parenthese is also colorized. This way, it is easy to know if there is a missing parenthese or not :)
Title: Re: Tokens
Post by: merthsoft on July 09, 2012, 05:48:20 pm
Thanks! I'm glad you find it useful! I've thought about the parentheses thing, but I think most people just leave them unbalanced, anyway, but I can look into it.
Title: Re: Tokens
Post by: Hayleia on July 17, 2012, 11:37:01 am
I have another feature request and two other things to report (not bugs, more me having problems) :)
And I excuse myself in advance for my poor sentences and explanations but I am French and I am not sure that you understand the bad English I use.
- The feature I'd like to see added is to display the number of selected characters. It is useful for example when moving sprites who don't have individual square brackets.
- I noticed that the IDE changes the indentation of a line to make it the same as the previous line. Keeping the indentation of the previous line when creating a new line is very useful, but automatically changing the indentation of an existing line when it is edited is annoying in my opinion.
- Also, the cursor seems to always blink at the same frequency, while a true text editor cursor sees its cycle changed when it is moved. I mean that when I press a key, the cursor should appear, whatever was its state right before the click. Whereas in your prog, when I press a key, it only moves the cursor but doesn't set the cursor's state as black.
(Not sure if you understood what I mean so when answering me, in the textbox, keep the down key pressed, you'll see that it never blinks until you release the key. Now, do that in TokenIDE, you see that it continues blinking, and when it disappears, it continues moving down but you don't see where it is).
Title: Re: Tokens
Post by: merthsoft on July 29, 2012, 02:59:43 pm
I have another feature request and two other things to report (not bugs, more me having problems) :)
And I excuse myself in advance for my poor sentences and explanations but I am French and I am not sure that you understand the bad English I use.
Your English is fine, don't worry!
Quote
- The feature I'd like to see added is to display the number of selected characters. It is useful for example when moving sprites who don't have individual square brackets.
I can see why that would be useful, so I added it. Now when you highlight a section, it tells you how many characters, tokens, and bytes it takes up:
(http://img1.uploadscreenshot.com/images/orig/7/21005404238-orig.png)
Quote
- I noticed that the IDE changes the indentation of a line to make it the same as the previous line. Keeping the indentation of the previous line when creating a new line is very useful, but automatically changing the indentation of an existing line when it is edited is annoying in my opinion.
That is annoying, I didn't even know it did that. I've changed auto-indent off entirely until I can invest some more time into figuring out what's going on there.
Quote
- Also, the cursor seems to always blink at the same frequency, while a true text editor cursor sees its cycle changed when it is moved. I mean that when I press a key, the cursor should appear, whatever was its state right before the click. Whereas in your prog, when I press a key, it only moves the cursor but doesn't set the cursor's state as black.
(Not sure if you understood what I mean so when answering me, in the textbox, keep the down key pressed, you'll see that it never blinks until you release the key. Now, do that in TokenIDE, you see that it continues blinking, and when it disappears, it continues moving down but you don't see where it is).
This has been fixed. Download in the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: Hayleia on July 29, 2012, 03:19:43 pm
Thanks for taking my suggestions into account :)
I'll try this tomorrow, when coding (not now, it will soon be time for me to sleep).
Title: Re: Tokens
Post by: DJ Omnimaga on July 31, 2012, 01:46:50 pm
Although I am no longer involved in calculator programming anymore, I am glad this is still progressing. :)
Title: Re: Tokens
Post by: merthsoft on July 31, 2012, 02:17:23 pm
Thanks! I'm just glad people use it and find it helpful :) I wish I had some more time to focus on it and nail out some of the harder features that I want to get done before a 1.0 release.
Title: Re: Tokens
Post by: merthsoft on July 31, 2012, 08:18:27 pm
I hope it's not too soon for a double post--I've added some much-asked-for functionality.

Due to some rather incessant nagging on tifreak's part (thank you, Daniel :)), I have added the functionality to count blocks, such as While/End etc. Here's an example:
(http://img1.uploadscreenshot.com/images/orig/7/21210535977-orig.png)
You can specify any number of groups. It's under Tools > Block Counts.
Limitations:
- It currently doesn't handle "Else" gracefully. It just treats it as another token. Ideally it would unindent the "else" line and indent the following line.
(http://img1.uploadscreenshot.com/images/orig/7/21211084668-orig.png)
- The indentation is automatic, so if you're doing this with Axe and have included your own indentation, it'll extra indent it.
- The AxeTokens.xml file doesn't have any groups defined, so check out Tokens.xml and see how it's doing it if you want to add them to it. I don't know if Axe does it the same way or what.
- Eventually this will be integrated into the main editor so that you don't have to go to a tool to use it. For now, though, it's separate.

Technical details:
There are two new tags that you'll use in the XML:
- indentGroup - String - Which group this token belongs in.
- indentGroupTerminator - Boolean - If it's a terminator.

For example, here's what the Tokens.xml looks like:
Code: [Select]
<Token byte="$CE" string="If " />
<Token byte="$CF" string="Then" indentGroup="End Group" />
<Token byte="$D0" string="Else" />
<Token byte="$D1" string="While " indentGroup="End Group" />
<Token byte="$D2" string="Repeat " indentGroup="End Group" />
<Token byte="$D3" string="For(" indentGroup="End Group" />
<Token byte="$D4" string="End" indentGroup="End Group" indentGroupTerminator="true" />

I hope that makes sense!

Download in the normal spot:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: Tokens
Post by: DJ Omnimaga on July 31, 2012, 11:59:01 pm
Wow that's a nice idea actually. You can't imagine how hectic it was to count blocks in Illusiat 13 boss programs (which are 15 KB each). One suggestion would be to add optional color support for nested blocks, like in Cooliojazz's TI editor a few years ago, although it's up to you.
Title: Re: Tokens
Post by: Juju on August 01, 2012, 12:11:09 am
Syntax highlighting, it's always a good idea.
Title: Re: Tokens
Post by: Deep Toaster on August 01, 2012, 01:15:47 am
Block counting. Now that's one enormously useful feature for TI-BASIC. There've been plenty of times while editing a TI-BASIC program (or Axe program for that matter) in the calculator editor where I wished I had some way to quickly find that one missing End screwing up the flow of the entire program :P And the feature you just added to count tokens in a selection too—they're both really useful!
Title: Re: Tokens
Post by: merthsoft on August 01, 2012, 09:25:31 am
Wow that's a nice idea actually. You can't imagine how hectic it was to count blocks in Illusiat 13 boss programs (which are 15 KB each). One suggestion would be to add optional color support for nested blocks, like in Cooliojazz's TI editor a few years ago, although it's up to you.
I think for consistency I'll just keep the syntax highlighting as it is and let people define colors for tokens.

Syntax highlighting, it's always a good idea.
Indeed, and that's why you're able to do it... Was this supposed to be a feature request or what?

Block counting. Now that's one enormously useful feature for TI-BASIC. There've been plenty of times while editing a TI-BASIC program (or Axe program for that matter) in the calculator editor where I wished I had some way to quickly find that one missing End screwing up the flow of the entire program :P And the feature you just added to count tokens in a selection too—they're both really useful!
I'm glad you think so! Tifreak has been bugging me for this feature for a while and I've been dragging my feet, but I think it definitely adds a lot to it :)
Title: Re: Tokens
Post by: blue_bear_94 on August 01, 2012, 11:09:26 am
As a sidenote, how do you insert the actual ~ token?
Title: Re: Tokens
Post by: merthsoft on August 01, 2012, 11:30:39 am
|~
To find tokens, it's useful to search the Tokens.xml file for what you need, and you might find it. For example, searching for "~" shows up:
Code: [Select]
<Token byte="$CF" string="|~" />And while those can sometimes be misleading, a quick tokenization and throwing it on an emulator can show that it's the right token. And, of course, if you want to change it to just a tilde, you certainly can, but you'll want to update the other entry as well:
Code: [Select]
<Token byte="$B0" string="~">
<Alt string="|-" />
<Alt string="⁻" />
</Token>

In this case, though, I choose ~ for negative because I assume in general there will be many more negatives that tildes.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: bilyp on September 07, 2012, 05:47:30 pm
I think I found a bug...

whenever you use the #define command, if the text that the variable equals is longer than the name of the variable, it throws an unhandled exception.

for example,
Code: [Select]
#define word k

disp "k"
works, but
Code: [Select]
#define word applesauce

disp "word"
does not, and it appears that it is because applesauce is longer than word.
The bug appears to be in the latest version.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on December 10, 2012, 12:17:58 pm
I've uploaded a new version at ticalc. Here's a screenshot:
(http://i.imgur.com/8m7Rs.png)
Download: http://www.ticalc.org/archives/files/fileinfo/433/43315.html (http://www.ticalc.org/archives/files/fileinfo/433/43315.html)
Follow the link for some nice screenshots from Linux, too :)

bilyp, thanks for the bug report, I think I know why that's going on, and the fix should be pretty easy.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: 133794m3r on December 21, 2012, 12:45:46 am
OK, quick question... am I crazy... or is the last time it was compiled was december 5th? I'm just making sure I'm running the latest version. Btw, absolutely amazing program, like seriously, I am so glad this thing exists. Being able to type out code on my calculator then be able to open it in Tilem2, and see if it works is _sooo_ much faster than writing it on my calc, and then compiling it, then running it. Effeciency has gone up by like a bajillion.

Also a feature that I'd suggest for the future is when you're clicking on the "reference" when you type a letter, jump to that letter's selection. Ala the catalog on the calculator. This'd make it much better/faster for me when I'm trying to figure out how you called a token, so I can double click it faster. That's probably the only feature that I _really_ want. That and projects working(since that feature sounds like an awesome one too). Finally, the last thing I can think of(I've not checked it yet), is that if you click on a prgrmSOMETHING, it'll take you to that program's tab(if it's in the same directory). Those are my feature requests, I don't know what your roadmap is, or if you've already had those things in the roadmap, I can't wait, if not I'd love to see them in it. I don't speak c#, so I can't help at all, but anyway keep it up, this is an awesome project.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Sorunome on December 21, 2012, 04:33:02 pm
Mhmm, how is it possible for custome variable names? What if you use then more than 27 variables?
EDIT: just noticed the pre-prozessor, lol
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on January 13, 2013, 06:05:43 pm
Congrats on the feature and interview Merth :D
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on January 19, 2013, 05:28:35 pm
Thanks Kevin :)
Alright, I've made a small update:
- Fixed saving error where it wouldn't save if it was a new file
- Fixed error where if you had a preprocesser directive that was shorter than the replace string it would crash (now it just highlights the line incorrectly :) )
- Moved some settings to INI file: Font and default tokens file.
- Grammer XML file included in download

Download at the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on January 20, 2013, 01:54:50 pm
Sorry for the double post, but I have another update:
- Fixed bug where shift+backspace did not backspace
- Added options dialog to set default tokens file when starting, and to set font:
Startup:
(http://i.imgur.com/7z78rtC.png)
Options dialog (under tools):
(http://i.imgur.com/GNjRqQX.png)
After you choose a new font:
(http://i.imgur.com/ETs1fS5.png)
Hitting save:
(http://i.imgur.com/PwjMvlh.png)
It also saves the INI file so that this persists across instances.

Download the the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on January 21, 2013, 09:03:58 pm
Another update! Runer made an awesome new AxeTokens file, this is what it looks like:
(http://img.removedfromgame.com/imgs/tokenide_axe.png)

I've included it in the download as well as addressed a couple of his suggestions:
- Removed the [expr(]
- Fixed alt-strings, I hope
- Prompts if you try to open an ASM program
- Site tab hyperlink in group pane, as well as a tooltip:
(http://i.imgur.com/UHkuMkN.png)
- Added commentString and directiveString to XML definition, defaults to "//" and "#", respectively. In AxeTokens.xml I set it to "//" and "@".

Download at the place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on January 29, 2013, 10:44:12 am
There was a bug in the Linux support and some other bugs I've since fixed. I've also added underlining back into the Axe file:
(http://i.imgur.com/vR03aLc.png)

Download at the place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: shmibs on January 29, 2013, 10:52:57 am
thanks, merth!
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on February 17, 2013, 01:32:26 pm
I found a bug with how the Alt strings were handled, and have fixed it, download at the normal place:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on February 25, 2013, 11:32:20 am
I have a question for users:
Would you prefer if instead of one .exe and six .dlls in the download folder, it's just the one .exe? This would clean up the output directory, but that's really the only thing it will change--everything else will still work the same.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Hayleia on February 25, 2013, 11:37:05 am
As long as the .exe is noticeable enough among the other files, I don't really mind the number of .dlls in the folder :)
So, for me, it is your choice ;)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: TIfanx1999 on February 25, 2013, 12:20:46 pm
I'd have to agree with Hayleia. Six Seven files is nothing really anyways.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: pimathbrainiac on February 25, 2013, 12:31:01 pm
I know this uses .NET, but is there any way to make a GNU/GTK version (still for windows)?

/me has .NET issues on one of his computers
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on February 25, 2013, 12:32:44 pm
That would take a lot of work, and isn't really the direction I'm taking the product. Have you tired installing Mono (http://www.go-mono.com/mono-downloads/download.html) on your computer that has issues with .NET? They have a Windows build.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Runer112 on February 25, 2013, 12:33:33 pm
Is there some reason why you're asking this question? It seems to me that fewer files would always be better. Is there a catch? If not, I see no reason why not to go ahead and make it one file.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on February 25, 2013, 12:37:16 pm
Is there some reason why you're asking this question?
To see if people care. There isn't a catch at all, and making it one file does make things easier for, well, everyone. I guess I just wanted to see if there was anyone who was really fond of all the dlls sitting around. It'll make the .exe slightly larger, but that's really it.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on February 25, 2013, 01:10:54 pm
I don't mind if there are multiple files, as long as I can easily find the .exe file, unlike some other software that has like 4 exe files and the readme doesn't state where the right one is. :P


Also, since Kerm has a 84 Plus C Silver Edition, is he helping you making TokenIDE recognize its extra commands and editing images? For 16 bit images we got Paint and stuff, but for 4 bit pictures it would be nice to have an editor too. :)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 02, 2013, 08:55:52 pm
Okay, preliminary 73, 82, and 83 calc support has been added. I've also cleaned up the download to just have a single .exe file. So, download it and you can delete all your dlls. Me and tifreak are working on the new xml files, but for all but the 73, the others are basically the same. I'll update when we've got more xml files.
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on March 03, 2013, 12:22:19 am
This is good to hear. Is it possible to quickly switch a program from 83+ to 82 mode by the way? That would make porting games MUCH easier (especially for the TI-73, which lacks the ALPHA key). Just have commands that are not available in a different color or something and automatically replace command names if the original is different. :D
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 03, 2013, 12:52:07 am
It is fairly simple, though it might be easier at times to, like, copy and paste. But from one format you can compile into any other, and you can select which tokens file to load up. It won't make porting an automatic process (especially with the 73, which has a weirdo token set), but it should be easier :)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on March 03, 2013, 02:33:31 am
Yeah the issue with Copy/paste is if you have to do it separately for multiple files then you might forget some parts and stuff. I remember when I tried to do it with Ti-Graph Link for Illusiat 81 and it was a major hassle :P

But yeah I guess that if there's a way to batch-copy/paste or something or if you add the ability to just switch to a different token file, then it would be easy since there's no need to check for commands that might have turned into separate characters due to renaming:

TI.82 TI.83
nMin PlotStart
nStart nMin
Un u
Vn v
UnStart u(nMin)
VnStart v(nMin)
TblMin TblStart
Sin Sin(
Cos Cos(

Of course, the user would have to solve parhenteses and order of operation issues himself (such as Sin(X-5) becoming Sin X-5), and if you allow saving to a different model you would need to be careful about limitations such as how the TI-82 has no string support, has 5 matrices and 6 pics instead of 10.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 03, 2013, 01:02:26 pm
You seem to know about all the differences--would you be willing to make the tokens files for those calcs? Or at least list all the differences. Are those the only different tokens?
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Sorunome on March 03, 2013, 08:43:10 pm
Nice update, I should try it out!
Does the .exe run under wine or how is linux compability made then?
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 03, 2013, 08:45:56 pm
It runs under Mono.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on March 03, 2013, 11:24:16 pm
You seem to know about all the differences--would you be willing to make the tokens files for those calcs? Or at least list all the differences. Are those the only different tokens?

I sadly don't know much of them. The above was pasted from the TI-83+ manual. However since the teacher had a TI-82 I got to see sin, cos and tan, for example. I also know that the 82 lacked strings and had fewer pics/matrices, along with only supporting lists 1 through 6 (99 elements max), but that's pretty much it.

And of course the syntax on the TI-85/86 is different since commands are shortened to 6 characters max or something.

By the way, do you plan to add PRIZM support?
Title: Re: TokenIDE - TI-BASIC IDE
Post by: CompSystems on March 04, 2013, 07:16:59 am
Will include TI89 editing programs?

The following editor (daisuke) has many bugs   :banghead:

http://www.ticalc.org/archives/files/fileinfo/398/39853.html
(http://sourceforge.net/dbimage.php?id=123365.jpg)

http://www.tigen.org/kevin.kofler/pcprogs/tokens89cmd.zip

This program converts TI-89/92+/V200 programs between the two formats used by the link calculator:
the tokenized binary format (89p/89f/9xp/9xf/v2p/v2f files) and the 7-bit ASCII interchange format (txt files). =)

Source
http://www.tigen.org/kevin.kofler/pcprogs.htm
Thanks
Title: Re: TokenIDE - TI-BASIC IDE
Post by: tifreak on March 04, 2013, 07:25:05 am
I'm sure he'd love to add that as a feature. All he'd need is an .xml file following the schema of the current xmls that have all the TI-89 tokens and their hex values, and it wouldn't be too hard.

Eventually he wants all calcs to be supported, but there are only so many hours in a day and all that. :)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 04, 2013, 09:11:55 am
DJ_O: Prizm support is already started--I have most of the tokens, I just need write the code to read/write the files.

CompSystems: 89 support is planned. Looking at the code from that, I should be able to create an appropriate XML file from it (it uses a giant switch statement... ugh). I can't give a good timeline for that, though. Come to think of it, though, that's probably my #1 most requested feature.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on March 04, 2013, 04:46:15 pm
Cool to hear :)

Also I bet that tifreak would love 89 support to easily port Pokémon Purple :P . I am surprised that 89 support is the most requested feature, though, considering how few people still code for it O.O
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Sorunome on March 04, 2013, 08:06:58 pm
Yeah, maybe many people code for the 89 but only small math progs and don't even care about mentioning it

But yeah, 89 support would be nice indeed :)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: blue_bear_94 on March 04, 2013, 08:15:05 pm
As a TI-89 Basic coder myself, I cannot emphasize how unsuitable the on-calc editor is for writing TI-Basic Programs. You have the long key delay, and limited find/replace functionality, not to mention that all the Local declarations should be at the beginning, so it would be useful to have a tool that consolidates them. (And automatic indentation would be nice, too!)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on March 05, 2013, 09:32:39 pm
Okay, I've updated it:
- tifreak and me worked on creating a more accurate 82 xml file.
- Sprite editor bug as been fixed
- Tokens.xml is the default token file
- Fixed problem with saving adding new lines.
As always, download:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on March 06, 2013, 03:20:07 am
Cool to hear. :) I assume that the sprite editor will eventually support color 8xi files, right? (the ones that are 265x165 and has 15 colors + transparency)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: blue_bear_94 on March 06, 2013, 04:59:23 pm
68k support is not planned, right?
Title: Re: TokenIDE - TI-BASIC IDE
Post by: TIfanx1999 on March 06, 2013, 05:15:06 pm

@bluebear: 6 posts up, same page:
DJ_O: Prizm support is already started--I have most of the tokens, I just need write the code to read/write the files.

CompSystems: 89 support is planned. Looking at the code from that, I should be able to create an appropriate XML file from it (it uses a giant switch statement... ugh). I can't give a good timeline for that, though. Come to think of it, though, that's probably my #1 most requested feature.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: blue_bear_94 on March 06, 2013, 06:25:18 pm
Derp. And yay! We'll be able to write 68k Basic more quickly.

On topic, thank you for telling me.
Title: Re: TokenIDE - TI-BASIC IDE
Post by: merthsoft on December 10, 2013, 02:53:53 pm
Wow, it's been a while since I posted here. I just wanted to make sure everyone knew that there's been lots of progress for xLIBC support! You can edit BG pics and tiles, and all the tokens are up to date. You can also edit hex sprites for the DCSE sprite support! Get the most recent download at ticalc.org:
http://www.ticalc.org/archives/files/fileinfo/433/43315.html (http://www.ticalc.org/archives/files/fileinfo/433/43315.html)
Or, of course, my server:
http://merthsoft.com/Tokens.zip (http://merthsoft.com/Tokens.zip)
Here're some screenshots!
(http://i.imgur.com/GbnXRzD.png)
(http://i.imgur.com/jXCX8L2.png)
(http://i.imgur.com/RjdNA05.png)
Title: Re: TokenIDE - TI-BASIC IDE
Post by: Sorunome on December 10, 2013, 05:35:05 pm
wow, necroupdate O.O

Anyways, nice to see that this has +C support! :D
Title: Re: TokenIDE - TI-BASIC IDE
Post by: DJ Omnimaga on April 23, 2014, 12:18:54 am
Bug report: xLIBC sprite var (8xv) exporting does not work. The appvar is created, but when listed in TI-Connect it says "Incompatible type" and cannot be sent.

I use Windows 7 and the latest version of TokenIDE. I tried with lowercase and uppercase file names of less than 8 characters, starting with an uppercase letter, to no avail.


EDIT: Nvm, I just realized appvars must start with an uppercase letter. It might be good if TokenIDE warned users about that or automatically changed it, though.