I actually think it's quite possible.  Anyone good with circuit boards?  Let's start with the screen, the memory, and the processor, along with simple usb support to upload programs, and a simple bootloader.  So far, so good?  We can do a simple 2 buttons for an A and B thing, an ON button, and four buttons for arrows.

So, who's good at PCBs and such?

Screen is easily done by a company called LiquidWare.
CPU, memory, and USB hosts are already implemented in BeagleBoard.
Linux already has a bootloader, so we don't need to worry about that.

Mr Kerm is a pro at that stuff, and has access to equipment that can do advanced soldering.
Plus, I also have a soldering kit as well.
(Or I can take that as a sarcastic reply, and say that we'll never make a calc )

touchscreen anyone?

Snake_X: Love to, but someone said AP tests hate touchscreen calcs.
Otherwise, existing hardware (BeagleTouch) will easily be installed and used.

Ya, I thought that was who it was but I wasn't 100% for sure.

I know it's not exactly the same, but Ben Reyves over at maxcoderz built himself a Z80 computer. You can check out the thread here:
http://www.junemann.nl/maxcoderz/viewtopic.php?f=2&t=2691&hilit=z80+computer

Pretty interesting.... but I still think we should stick to ARM. As the OP himself said, it's pretty weak to do anything really powerful. Also, SirCmpwn is already making a wonderful Z80 OS, and there's already a platform to build on - the TI calculator.

What I'd like to do is to design a new OS and calculator. The OS is based off Linux, which is a pretty fun OS to work with.

Actually, I have an idea - there could be 2 branches of this project: good ol' Z80, which if successfully made, would sell for around $40-80, and gain quite a bit from the cheapness; then the ARM based calculator, which is like a Nspire, but with a colorful GUI, better processor, and of course, NOT LOCKED.

The Z80 calc in particular could run KnightOS, so no issues with DMCA or any licensing. If this happened, that would be awesome. The ARM calculator is a whole new thing, and is more or less an experiment.

We should just run TI OSes for the heck of it.

There's plenty of emulators for that.
And besides, you'll get lawsuits for doing that. This is going to be sold - that's even more of a pain to TI, since they currently have a monopoly. Selling another equally, if not better calculator is a smack to TI. (KnightOS and Z80 based calc, that is.)

If you do make a calc, be sure to include a sensor pad, just like a laptop 
Also, use 48-bit deep color for the screen with 1024x768 resolution  (j/k)

http://en.wikipedia.org/wiki/Truecolor#Beyond_truecolor:_Deep_Color

I'm game.
But, we will need a knowledge base.
I dunno how to hard mod at all.

how much are we going to spend on each calc to make?
max price for:
screen-
buttons-
shell of calculator-
battery-
back up battery-
hardware and chips-
advertisement-
shipping-
location of selling-

total cost?-
selling price?-
profit?-