Author Topic: tok8x: a very simple on-computer tokeniser/detokeniser (Read 9935 times)

shmibs · « **Reply #15 on:** April 08, 2013, 05:01:39 pm »

coolio, i'll take a look at that, then =). the rest of your xml files have already been a big help, by the way.

as for adding in support for xml files, it sounds nice, but i'm still not convinced that it's worth the trouble. at present, all it takes for someone to add in a new token is to write a single line in the form {first_byte, second_byte, "string" }, and then run make. compilation takes less than a second, and the result is a binary that can be dragged and dropped anywhere without the need of additional files, whereas adding in xml support would increase the size considerably and either require me to include some non-standard library or write a lot more code.

EDIT: both the default BASIC and Axe token sets are now complete =D

Lionel Debroux · « **Reply #16 on:** April 09, 2013, 02:06:30 am »

I'm not really convinced either by the necessity or urgency of adding XML support, even if:
* it isn't hard to make users clone some external simple existing StAX parser, and use it in your program. Handling JSON would be easier, and more efficient, than handling XML - but Tokens IDE uses XML;
* with a bit more (easy) code, it would enable detecting mismatches between the internal, compiled-in data and the data extracted from XML files.

shmibs · « **Reply #17 on:** April 10, 2013, 01:25:21 pm »

apart from the preprocessor and another token set option for pretty printing (in case you want to display "λ" rather than "lambda", etc), i'm fairly certain this is done!

new stuff:
* full pipeline support, so things like

Code: [Select]

cat <some.8xp> | tok8x | grep -n <search term>are perfectly valid =D

* an option to strip excess whitespace (outside of strings) and comments (the format of which is specific to the language) from a generated program, so a file containing:

Code: [Select]

.ABCD
	"hey there! i'm a string"->Str1

.one line comment

...
multiple
line
comment
...

For(A,0,23)
	Text(0,A*6,{Str1+A})
End

run through:

Code: [Select]

tok8x -s -t axe -i <infile> | tok8x -t axewould generate:

Code: [Select]

.ABCD
"hey there! i'm a string"->Str1
For(A,0,23)
Text(0,A*6,{Str1+A})
End

* searching for individual tokens matching strings on the command line (so something like

Code: [Select]

tok8x Get azvd DiagnosticOffwould return:

Code: [Select]

"Get":Axe:E8
"DiagnosticOff":BASIC:BB67

* expanding leading spaces on a line to tabs for easier reading when converting from an 8xp

DJ Omnimaga · « **Reply #18 on:** April 10, 2013, 01:34:19 pm »

Will this editor support image/picture importing/exporting by the way? Even though the calc has been out for months, no existing software other than the Mac version of TI-Connect supports them and it will be annoying for BASIC programmers who use pics to store data and title screens or whatever...

shmibs · « **Reply #19 on:** April 11, 2013, 11:06:59 pm »

this isn't an editor; it's just a tool for converting between file formats. someone could build an editor that uses it as a backend, though.

i added an include preprocessor directive! the format is:
##include path/to/file
and the contents of file will be inserted verbatim into the file at that point.

Streetwalrus · « **Reply #20 on:** July 25, 2013, 08:40:38 am »

This looks pretty awesome Shmibs, just the tool I've been looking for.

shmibs · « **Reply #21 on:** July 25, 2013, 02:18:34 pm »

well, apparently i exploded something or other at some point, because it isn't working now. this version was more of a proof of concept than anything else, really, though, so i guess i'll try to redo it properly over this weekend. the long, tedious bit of writing out all the tokens is done, though, so there's nothing to worry about there.

shmibs · « **Reply #22 on:** July 28, 2013, 02:28:53 pm »

/\ignore that entirely. it's a problem of me having no memory. right now, programs generated without a specified name are always named "A", but i had completely forgotten that and thus was looking for something that wasn't there.

* shmibs goes to change it so that it defaults to the first 8 letters of the filename instead.

Streetwalrus · « **Reply #23 on:** August 04, 2013, 02:56:47 am »

I made a little pull request for you. Not much since I just fixed a missing character but...

This is the first time I make a pull request so I hope I did it right.

shmibs · « **Reply #24 on:** August 04, 2013, 04:50:08 pm »

yup, that's the way it works =)
thanks!

Streetwalrus · « **Reply #25 on:** August 05, 2013, 09:27:17 am »

You welcome, I'm glad I could help.

Hayleia · « **Reply #26 on:** July 25, 2014, 08:36:24 am »

I am having problems with this (sorry for necroposting btw).

I can't compile what I get from GitHub:
─┐ ╔[ asuka @ Luna : ~/Bureau/tok8x-master ] ╘══╩═[ make ghc --make tok8x target `tok8x' is not a module name or a source file make: *** [all] Erreur 1

And when I download the old version, it is a bit better since I can compile it and it works in one way, but not the other way (unfortunately, it's the less interesting way that works):
─┐ ╔[ asuka @ Luna : ~/CALC/SBO ] ╘══╩═[ vim TEST.8xp.txt && ./tok8x -t axe -i TEST.8xp.txt -o OUTPUT.8xp -n SBOS -f && tilem2 --help tok8x: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. Abandon (core dumped)

Am I doing something wrong ? Maybe that the old version needs a bit more than just the "tok8x" in the folder to convert (edit indeed, I can get this one to work when I don't move only the executable) ? But what about the "new" version ?

Streetwalrus · « **Reply #27 on:** July 25, 2014, 09:31:49 am »

Hmmm ghc eh ? Seems like Shmibs moved to Haskell. I don't like Haskell that much due to dependencies so I can't really help.

shmibs · « **Reply #28 on:** December 29, 2015, 10:51:36 pm »

heys, sorry about that. the previous version is in a branch labelled "old". was messing around a while ago seeing if i could make a haskell version work, but gave up and am rewriting the whole thing from scratch in C, because this was sort of the first C project thing i ever made and the old code is really gross and crufty and bug-filled. new version should be done soon, if anybody actually cares about this

TIfanx1999 · « **Reply #29 on:** December 30, 2015, 10:21:23 pm »

Oh hey, activity! Pretty nice! ^^

Author Topic: tok8x: a very simple on-computer tokeniser/detokeniser (Read 9935 times)

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Lionel Debroux

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

DJ Omnimaga

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Streetwalrus

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Streetwalrus

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Streetwalrus

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Hayleia

Re: tok8x: a very simple on-computer tokeniser/detokeniser

Streetwalrus

Re: tok8x: a very simple on-computer tokeniser/detokeniser

shmibs

Re: tok8x: a very simple on-computer tokeniser/detokeniser

TIfanx1999

Re: tok8x: a very simple on-computer tokeniser/detokeniser