Hmm... what programming language is this in? I think I read somewhere that you could program on the 68k series with C. I could be misremembering though. In any case though clipping can be done by grabbing the pixels and rotating them in. I don't know how the TI-86k screen is setup but maybe this could be of some help. Just thought I would show some interest. Is there an emulator for this calculator?

I think what I would suggest is what you are already trying to do. For irregular collision boxes, I set a "collision buffer" the size of the screen (1bpp), and in that buffer I set the pixel on if it's a collision point, else off. Then when I am drawing the tile, so draw it's collision sprite to the collision buffer.

I have no idea how to do this on the 68k calcs

If you can reduce every shape down to a series of lines (or can get away with approximating them as such) then you could just take the lines that outline the shape you want to test and check it against every line that makes up the environment. There are pretty simple formulas that you can use to check if 2 line segments intersect. If the calc you are using has a built in multiplication instruction it would likely be faster than drawing lines and use less memory depending on how it is implemented. But if you use really complex shapes or need precision with curved surfaces drawing to a buffer is your best bet. Drawing to a buffer also lets you save the buffer if the area doesn't change which makes it really fast.

If you are having trouble drawing to the larger area, you could use the built in sprite routines to draw the whole thing to the 240x128 area, copy that much to the 256x256 buffer, shift everything over, draw again, copy that much to the larger buffer and repeat until all the area is filled. If you are redrawing that every frame, it could get pretty slow.

Additionally, if you only have one object you are testing for collision, you can simply shift the environment around that one object. Since the things it will collide with will be inside the size of the default sprite drawing box, you don't need to worry about using a larger buffer. This does assume that you redraw the clipping buffer each frame/update.

Since you said you are making an rpg, I assume the environment is pretty static and using my second suggestion of a collision buffer, drawing everything to the screen just copying it chunk by chunk to the buffer is probably the best since it is the simplest.