The first 68k C question in this subforum! (the flash apps one I posted was in the Other Calculator Help and Support forum)

Anyway, I'm having a little trouble understanding some of the string manipulation functions in GCC4TI (string.h). It's probably because of the way strings are treated in C (as arrays) which isn't really what I'm used to, being a programmer in higher level languages (Python and VB). Also because I haven't really used pointers before in programming.

Specifically, I want to read a text file (*.89t) and split it line by line, then do operations on each line. I've gotten my program to a point where it uses a dialog box to get the name of the text file, opens and reads it, and then prints the text to the screen (this last bit is just for debugging). It's the "splitting line-by-line" that's causing me problems.

I've realized that the text files use "\r" as newline characters (at least, I think they do) and so I thought to use strscn() to get the length of the string up until the "line", then use strcpy() to copy the line into another string and strpbrk() to create a substring without the first line.

The first problem is that if you type "\", GCC4TI ignores the closing quotation mark, assuming another character is coming after it. I've had to use "\r"- but the problem of this is that if the line has "r" in it, strscn() would stop early (because it's characters from the second string, not the second string as a whole). Is there any way to solve this? Or am I wrong and, in fact, it would only stop when it found both "\" and "r"?

Then, the second problem is that I'm unsure what exactly strpbrk() does. The docs say it "returns a pointer to the first occurrence of any of the characters in s2"- is this just a pointer to that one character, or a pointer to everything from that character to the end of the string?

Well... I've figured this out- \ isn't picked up because a single backslash plus the next character (like \n or \r) is a seperate character. The actual string would need to have a double backslash for it to work. Fortunately, if \r is a single character then I can look for it using strncpy() and strcspn() without problems.

Don't know if I should even bother posting updates since there aren't really any 68k C programmers around... but I will anyway if I have more issues, just in case someone actually looks at and replies to this.

Yeah, things like '\r', '\\', '\"', '\t', and '\'' are all single characters, and are called "escape sequences." Without them you couldn't easily embed newlines, quotes, tabs, etc. in strings.

Nice to hear it's working! Yeah, just ask if you have more questions, we'll gladly answer

Hm... First of all, you probably don't need to make the method recursive.
I'd suggest an approach of looping through the string with pointers rather than making it recursive and copying the string every time.
Secondly, you probably shouldn't be using strpbrk() to move to the next token, rather, using a looping method, you should just advance the pointer manually to the next token.
So, some C: (I'm assuming the existence of some external functions and some pre-existing variables and memory locations)
tokenInfo findToken(*char string); /*Given a string, this should find the longest token and return its length and hex*/
typedef struct {
    size_t chars; /* How many chars are in the string making up the token */
    BOOL twoByteToken; /*Whether or not the token is two bytes*/
    byte hex[2]; /*the two bytes making up the token, or only hex[0] if it's a one-byte token*/
} tokenInfo;
/*Working from these two...*/
void tokenize(*char string, *byte buffer) { /*Takes a string and a buffer to store generated hex. Make the buffer dynamic on your own time. (Or ask me ;D) */
    tokenInfo currentToken;
    while(*string != '\0') {
        currentToken = findToken(string);
        *(buffer++) = currentToken.hex[0];
        if(currentToken.twoByteToken)
            *(buffer++) = currentToken.hex[1];
        string += currentToken.chars;
    }
}Would something like this work?

The tokenInfo struct was intended to simply hold enough data for the tokenization routine.
If you wanted to store information in a text file, the structure would be more like an alpha-sorted sequence of strings with the corresponding information (one- or two-byte, the token hex).
Actually, if you chose the format appropriately, you could make it user-editable, like this: (this is only a suggestion, you can probably come up with a better format that does the same) (also, note that I don't know the hex for any tokens, so it's made up)
Disp
*45
Output(
*84
GarbageCollect
BB68
sin(
*97
OpenLib(
82d2Note that you'd have to alpha-sort it at the beginning of the program (probably temp-only), but it would make it user-editable

"unsigned char" is 8 bits, "unsigned short" is 16 bits. "enum Bool", "BYTE" and "BOOL" are seldom used.

The trouble is if the "next character" in the string is the same as one of the characters that's part of the token, it won't work. So if the line is: Disp "HELLO", then the program will hang when it's truncated "HELLO" down to LLO", because two Ls are next to each other, and so strpbrk("LLO"", "L") will return "LLO"".

In many parsers, characters are divided into classes of lexical elements (string, comma, etc.), and a lexical analyzer is built to recognize them (e.g. a string starts with a '"' and it lasts until the next '"' or maybe end of line). On top of the lexical analyzer, there's a syntactical analyzer, which tries to parse a stream of lexical elements (get an element, then act upon its type - maybe there are other expected elements of various types after it). And on top of the syntactical analyzer, if the language is complicated enough, there's a semantics analyzer (e.g. {, number 234, comma, number 567, comma, number 8 is a list).

But maybe I'm replying out of scope because I'm dense ?

Unsigned char is a fine replacement for byte.
And as for the asterisks in the code, they should be between the variable type and variable name, my bad. (I haven't done C in a while, and I have never done C for the 68k calcs)
The reason I didn't have tokenInfo have the string was because it was unnecessary for the example. If you need it though, it can be added. Make sure you choose a long enough buffer for the chars, though, or that you can use dynamic memory allocation.