Hey,
This game is a great idea !
At first I didn't realize I had to enter either 4/5/6 so I thought it didn't work
So, for people who like screenshots :
One thing I would like to tell you especially is that your code should be cleaner (it's better for us readers and especially for you).
For example, indentation (maybe that's just because the forum didn't get it right, though).
Also, you're making a lot of tests (if ... ~= ... etc.) that might be completely rewritten in a function to compare directly all the numbers/variables you're using. Thus, saving cpu, battery, time, game speed etc.
(the function judge() especially can be very optimized, I think)
In conclusion, you're saying it's your first Lua game, so I must say it's a very good start. You understood how the nspire lua API worked and your document can be used for educational purposes, so that's also a good point
Now, what about making the game more difficult with other calculations than basic additions ?
NB : I just found a bug (idk if that's because I tested it on the computer software ant not a calculator...) : the text is not centered at the game over screen :