What do you mean by dithering in this case? If you are talking about the scan lines, those are only present in wabbitEmu. I tried it on my real calculator and they are hardly noticeable.

You are also definitely correct about the compression artifacts. I traced the steps through and it looks like SUPER caused them when it converted the .flv to .jpg's. But as of now, I don't know any other way to do that step.

For decompression on the calculator, there is definitely enough time. 16,000,000 hz * 50 % (grayscale) /15 = 533,000 clock cycles per frame. That's 347 clocks per byte, clearly enough time to do something amazing. The only thing I'm doing right now is converting strings of 00's and FF's to 00 00 (length) and FF FF (length).

Perhaps the jpegs themselves are the source of the problem?  Maybe you would be better off with some sort of lossless image file instead of a lossy one.

As for dithering, note that it would also raise the filesize significantly

Right, by dithering you are adding more complexity, and so RLE will not work as well.  Btw how are you doing your RLE?

the thing is though, that this video is not going to be byte based, but pixel based.  you wouldn't have 16 00's, you would have runs of either binary, or runs of 0-3

Here is my current method of compression. 00's and FF's are the target since black or white sections are all one color.
Data: 00 00 00 00 00 00 00 00 00 06 58 00 36 87 ff ff ff ff ff ff ff ff 59 ff ff    86 32
Comp: 00 00 08                   06 58 00 36 87 ff ff 07                59 ff ff 01 86 32
Expl: 00flag#-1                        one ok   FFflag#-1                  only downside

As you can see, any long string gets shortened to three bytes with this. The only downside is that since the flag is a repeated FF or 00, 00 00 and ff ff get expanded one. But those are somewhat rare. The reason that the number of bytes is -1 is just to make life easier in z80.

As for dithering I just figured out that that means artificial grayscale by alternating colors between frames. But I could more easily add 8 level grayscale, it's just the file size that would suffer.

Edit:
    Does anyone have a good idea for compression techniques? I would love to hear them.

I'm thinking of doing a variation of DEFLATE. Here is the specification if anyone is interested. (if you're going to look at it, the important part is the top of page 11) It is essentially LZ77 paired with huffman tables for the distances and lengths. It's what .png uses so I figure it's good for pictures/video. What I think I'll do is just use the uncompressed and Fixed huffman table parts of it because I don't think my file sizes are big enough to be able to use the dynamic huffman table properly.

Since LZ77 using a sliding window, that creates a sort of problem when doing it on the calculator due to memory requirements. I am thinking of using a sliding window of 768*2*2=3,072 because that is two frames. To accomplish this quickly and efficiently, I'm going to move data around by switching pages rapidly. In what might be the most hacked memory configuration yet, I will have the flash page in the $4000 slot. I will have $3A00 bytes of page $81 swapped into the $8000 range through port ($28). Then, I will alternate having pages 0 and 3 swapped into the $8000 range and the $C000 range so that the two frames are always in the same place: $BA00 and $FA00. Hopefully, this will allow me to get the speed and address space required to make this work. (I'm backing up page 0, so don't worry.)

Sounds epic!  Good luck!

It doesn't seem to work on a real calc, what do you have to do with the files?