GLib, as odelore has already pointed too is a good one.
There's also
JH3D, but I haven't used this one personally, so I don't know what's possible with it.
But, before you start, you should know that 3D on a calculator is still pretty much experimental. You'll only be able to draw simple shapes with a few lines with most engines, and some basic features like Z-buffering are often missing due to hardware limitations. And
while rendering environements in 3D is possible, it's deffinately not as straightforward as simply making a level and calling a function to render it. You constantly have to find workarounds for almost everything, but off course, that's part of the fun
.
I'm not trying to scare you, and I'd love to see more finished 3D games for the z80 calcs, but a 3D calculator project nothing that you should start unprepared. If you're still interested, then my advice would be to first do some test with a computer 3D engine (if you have no experience with them) to get familiar with the 3D math.
If, after all this, you're still interested in taking on this project, then I wish you good luck and lots of fun
And if you run into troubles, don't hesitate to ask. There are multiple people here that have also given calculator 3D a try.