Author Topic: FiXos, a UNIX-like kernel for fx9860  (Read 19073 times)

0 Members and 3 Guests are viewing this topic.

Offline Kristaba

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 16
  • Rating: +5/-0
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #15 on: June 21, 2014, 02:32:14 pm »
might be out of topic, but can anyone tell me where I can find the said GCC toolchain to biuld .g1a executables?

It's late to reply, but if anybody would like to have a sh3 GCC toolchain, as the SuperH architecture is not so common, the better way is to compile yourself GCC from sources. To avoid having to understand every details on GCC configuration, you can compile it through crosstool-NG (http://crosstool-ng.org/).
The only other tool than GCC binaries themself you need is a G1A wrapper tool, and you will be able to create G1A from a GCC toolchain ;)


About the FiXos, as usual with me, I stopped a time, and I'm working again on it from one or two months.
The kernel like more and more to a usable UNIX kernel, the main changes from the last time I posted here are :
  • Time management is now really better, kernel soft timers allow to defer kernel function execution, the 'real' time is maintained as time ellapsed since Epoch (read init time from RTC), and process kernel/user mode time ellapsed is counted. All this allow implementation of times() and gettimeofday() syscalls for user process.
  • Functional UNIX Signals (only a small subset are implemented, but the more difficult part is done and fully working).
    kill(), sigaction(), sigprocmask() syscall are usable, so it is possible to send signal to a user process from another process (or from kernel itself of course), to set signal handlers...
    Some interesting unhandleable signals are also usable (SIGSTOP and SIGCONT, SIGKILL... ^^)
  • UNIX Pipes support, this is the first pseudo-IPC in FiXos ;)
    It is now possible to create a pipe with pipe2() system call, and to use file descriptors to read/write data into the pipe.
  • Virtual Terminals! You may have several TTYs multiplexed on your screen (for now switching is done with F1~F6 keys, and only two TTYs are created with default settings, to minimize the need of memory).
    For now, sessions, process groups and TTY owning are not implemented, but user process can use any TTY by opening /dev/tty<n> device  ;) .
  • A proof-of-concept of direct screen rendering for userland is implemented as the "/dev/display device". This device implement only some ioctls :
    • one to map dedicated VRAM into user space (mmap() is not implemented for now in FiXos  :-[ )
    • an other to get display information such as size in pixel, VRAM size, color format used for the VRAM (this provides a platform-independant way to manage a screen)
    • a third for activating/deactivating the direct display rendering
    • a last to display the content of the VRAM on the real screen
    Using these ioctls allow to have graphical applications in userland, which is a first step to enable user friendly interface (but keyboard handling is only implemented in TTY, so its only the first step ^^).
  • Support for boot-time parameters (command line arguments for the kernel), as well in the bootloader and in the kernel itself. The first usable parameter is 'console={tty<virtual_terminal>|ttyUSB0}', to set the device to use for kernel console (a virtual terminal or the USB emulating a serial port).
    The USB serial output allow to use minicom-like (gtkterm for example), running on a computer host, as the kernel debug console, which is really more powerful than the tiny calc screen ;D ).
  • Better kernel critical sections, with possibility to disable kernel preemption, or all interruptions (hard critical section).
    A basic mutexes implementation was added, but not used for now.
  • Some important changes on the projet structure :
    • Now, all definitions which are used by both kernel and userland are separated from the source tree (that includes definition of types and constants  used in system calls, ioctl related stuff...)
    • Some work done to avoid useless recursive makefiles without making a unique unmaintenable one (if you are interested about why, you can find some interesting article like this one http://aegis.sourceforge.net/auug97.pdf on the internet).
    • There is now a way to set configuration options both for compile-time and build-time (inside makefiles), using a "config.h" file, in which you can enable/disable/customize feature by changing some macros. This file is then transformed into a Makefile include file, so build system and compilation share the same configuration values.
      I think I will try to make configuration easier with tools like Linux kernel's kconfig in the future (ncurse or GUI based tool to change configuration).
  • A "generic pool allocation" implementation, to make easy to have some dynamic allocation inside the kernel (this is done for fixed-size objects, by allocating physical memory pages as needed, and using them like memory pools http://en.wikipedia.org/wiki/Memory_pool ).

I think it's more or less all the important changes. If you were patient enough to read all of them, you can figure the kernel is now close to be mature to support a consequent user space, but some crutial things still missing.
The memory management is not enough powerful and will need big improvements (to support Copy-On-Write, shared area of memory, memory mapped files with mmap() syscall...).
The other really missing feature remains the shared library support. I know on some platforms it is done in user space (using mmap() and oter stuff), but I still plan to implement it in kernel space, maybe with a couple of non-POSIX syscalls.


Finaly, userland need a working libc, even before dynamic libraries implementation, in order to be really usable. Playing with syscalls is fun, but a libc is more efficient to make real programs...  [-.-]~
I'm considering to port Newlib for FiXos (Newlib is already working fine on SuperH bare-metal target, and is relatively easy to port on a new platform). I tried some month ago, but there was a problem with Newlib usage of automake, which is fixed now I hope...


I don't know if I will be active on this project during the summer, but I'll let you know if any progress is done.
And of course, if some experimented programmers are interested for working on FiXos, there is probably tons of things to do or to improve. I'm still interested to work on a port of FiXos for any other platform (Prizm or other SH4-based casio calculators, maybe m68k TI calc like TI-89, or even a x86/PC port could be fun ^^), because it will force a good kernel design to be able to separate arch/platform specific code from generic kernel code, and in the same time a portage often highlights hidden bug in a software.  ;)

Offline Eiyeron

  • Urist McEiyolobster
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1430
  • Rating: +130/-10
  • (-_(//));
    • View Profile
    • Rétro-Actif : Rétro/Prog/Blog
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #16 on: June 21, 2014, 03:03:37 pm »
I'd be glad to help this summer, but it'll need for me to read a lot of docs, I never touched a *nix kernel, so...
But, if you can give me pointers, I'd help to port to Prizm! :p

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #17 on: June 21, 2014, 03:16:07 pm »
Sorry I tl;dr'd the thread but I guess this is like KnightOS for the (more powerful) Casio calcs ?

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #18 on: June 21, 2014, 09:55:24 pm »
Just make sure that the OS plays nice with the calculator hardware, because with certain PRIZM add-ins, if you run OS 2.0 and do certain things with your calc, you can permanently brick it. (there's a topic about this on Cemetech with more details)

Offline Kristaba

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 16
  • Rating: +5/-0
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #19 on: June 22, 2014, 01:49:50 pm »
Sorry I tl;dr'd the thread but I guess this is like KnightOS for the (more powerful) Casio calcs ?

In fact, for now, the name of the project is a lie (FiXos is only a kernel currently, but it should become more or less a "full" operating system someday). So it should look like KnightOS, but more focused on a "powerful" kernel, because the targeted CPU is far more complex (MMU, USB controler, SD Card, and dozen of other on-chip peripherals) and hardware is enough powerful to support a bit heavier kernel.

Just make sure that the OS plays nice with the calculator hardware, because with certain PRIZM add-ins, if you run OS 2.0 and do certain things with your calc, you can permanently brick it. (there's a topic about this on Cemetech with more details)

I read this topic, and unfortunately with the Casio boot design, with everything on the same EEPROM, I imagine there is small chance to brick the calculator when rebooting at the (very) worst time of an EEPROM write/erase operation.
For now I try to avoid playing too much with EEPROM, it will be necessary, but I prefer to have a stable kernel before to do that.
Of course, I'll do the best to avoid any problems, especially on other calcs than my ones (in fact I'm surprised to never broke my calc as I ran thousand times my kernel to debug it, with really not stables things sometimes), but I can't be 100% sure that will never happen (as Casio themselves apparently failed...).
Only thing I can say is that a buggy free software have a chance to be fixed if a such a thing ever happen, and a protected-mode kernel which run user applications in user-mode limits the possibles location of a such issue ;)

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #20 on: June 23, 2014, 03:56:40 am »
Yeah I figured these calculators stand closer to the 68k line than to the z80 line in comparison. But the goal is still the same : a UNIX-ish OS for calculators.

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #21 on: June 23, 2014, 02:42:20 pm »
It's a shame that the PRIZM and 9860G don't use ARM processors, because it would have made it much easier to port TI-Nspire stuff to it. The HP Prime uses ARM too (although there isn't any real doc on how to create a third-party firmware for it nor a mod of the existing ones to allow ARM assembly and C).

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #22 on: June 23, 2014, 05:48:46 pm »
Well on the bright side they have a good c compiler so it makes the non asm stuff portable. All the HW is incompatible across the HP, Nspire and Casio calcs though.

Offline Kristaba

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 16
  • Rating: +5/-0
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #23 on: July 03, 2014, 08:41:31 am »
I spend several days to write many proof-of-concepts around FiXos, and I finally added some intersting things to the kernel  ;)

First of all, a lot of work was done on dynamic libraries!
After some reflexion and studying existing implementations on several platforms and OS, I decided to implement shared object stuff in kernel. In most modern OS (linux for example), dynamic linker is a userland tool, and the kernel is agnostic about any dynamic linking.
As FiXos kernel already have some ELF facilities, it seemed easier and lighter to have a kernel-part implementation (so userland is simpler to understand and to maintain). I also implemented it in a way it's still possible to write a full userland dynamic linker, without breaking the kernel implementation (and it's even possible to disable any kernel dynamic linker related stuff at build time with a config option ^^).

The main problem I faced was than dynamic linking is deeply dependant of the toolchain target arch/OS, and the one I used is configured to target sh3 arch without operating system (and it makes no sense to support dynamic linking for unexisting OS).
So, the two choices I had were to do some changes in GCC's sources to target my OS, or to cheat a bit in order to make a part of the normal linker job "by hand". As changes in GCC is not a good idea for now (a lot of time waste to maintain and distribute it...), I did the second choice.
Mainly, to build a shared library for FiXos, the following is done (and automatized by some shell scripts and Makefile itself) :
  • Mark each "public API" function with special GCC's visibility attribute "default".
  • Compile each file with -fpic or -fPIC option (which make Position Independant Code, allowing to load the library at any arbitrary virtual address), and -fvisibility=hidden (which "hide" every symbol not explicitely marked as part of the public API in C code).
  • Link all resulting object files, the -shared option is used but seems to have no effect for sh*-elf toolchains (so the resulting file is an ELF executable file, but this make no real difference with an ELF shared object).
  • Extract every symbol marked as "GLOBAL" with visibility set to "DEFAULT" from the linked file. This should be the list of public API symbols, and it is used to achieve to distinct things :
    • strip every non-public symbol information to make the dynamic library lighter!
    • generate a "stub" assembly file for each public function
  • The stubs are designed to be staticaly linked to client executable (its a set of some instructions for dynamic symbol resolution, and the symbol name as a C string), and go to the .plt section of the executable.
    They are compiled and placed together in a single static library named "lib<name>-shared.a".
When a program should use a dynamic library, you just have to link it with the corresponding stubs library, and to call the functions exactly as there were normal, local ones.
The first time a dynamic-linked function is called (which, in fact, call the corresponding stub code), some complex job is done, including a specific system call to find the corresponding symbol by its name.
Then, all subsequent calls will be faster by using the cached value of this symbol (in fact the overhead is only 4 instructions, so it should not be a problem for performances).

The kernel ELF loader will only care about a special segment of the ELF (the one of type PT_SHLIB, defined by ELF specification but with no predefined semantics). It expects to find a list of zero-terminated strings, each one corresponding to the shared object name (the library name, with no path). For each name, it will try to find and load the content of the shared object. Once loaded to an arbitrary not used address range, the kernel try to "relocate" it (a vector of absolute-based addresses is located, and each element is incremented by the base address where the code was loaded).
Please note this segment is populated automaticaly (when at least one of the stubs for a given library is used, the library name is added to this section).

I'll have to write some documentation about all that in the project source tree, because it involves many complex things, and it's possible that my implementation fail for complex patterns.
In addition, it's only an early implementation (the shared memory is not implemented, only one library may be linked to a process...), but the more difficult part is done and seems to work fine for now!


An other cool thing added is a more powerful post-mortem debug information display.
A new function, kdebug_oops(), may be used when the kernel reaches an abnormal state (to not say "crash"  [-.-]~ ), which should display every potentially useful information (some CPU registers, current process related stuff like a memory map, and a call trace of the kernel stack).
   Unfortunately, SH3 calling convention does not allow a very good call trace without a lot of additional information (more or less all the "unwind" data needed for C++ exception handling...), so the current code only display the "guessed" calls, and may repport some false-positive :/

Even if it's usefull to have addresses of caller functions, having their names allow to don't spend much time to find it by looking at the disassembly dump of the kernel. So a bit tricky mechanism may be triggered by the config option CONFIG_DEBUG_SYMBOL_NAMES to extract all symbols located in ".text" section, and to include a table for "begin address" <-> "symbol name" conversion (it is possible to keep symbol names in ELF, but it takes more memory and its more difficult to read...).
Finally, this configuration option takes many space (currently around 30% of the kernel size...), but leads to a more understandable call traces, and so should be used for kernel development.

To make this more visual, here is an example of displayed message when an unaligned data is accessed in kernel mode :
Code: [Select]
> Address : 0x3FFFFFF3
> SPC Value = 0x880025F6
Fatal kernel oops!
(CPU Access Violation (R/W))
Following informations may help :
Running pid 1 (asid=0)
Memory Map :
DIR(0x10000000~0x10008000):
  vvuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
DIR(0x3FFF8000~0x40000000):
  uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuv
(End of Memory Map)
Stack=0x880126F0
Call trace:
@0x880126F4: <_exception_handler + 272> (0x88008AC0)
@0x880126F8: <_printk + 0> (0x88006E78)
@0x88012708: <infloop + 0> (0x8800896A)
@0x88012710: <_sys_ioctl + 0> (0x88005DD8)
@0x88012714: <_interrupt_inhibit_all + 0> (0x88008680)
@0x8801271C: <_exception_handler + 0> (0x880089B0)
(End of call trace)

---- Previous Context ----
       PC: <_display_ioctl + 58> (0x880025F6)
       PR: <_exception_handler + 192> (0x88008A70)
Stack=0x8801277C
Call trace:
@0x88012780: <_exception_handler + 192> (0x88008A70)
@0x8801278C: <infloop + 0> (0x8800896A)
@0x880127A0: <_exception_handler + 0> (0x880089B0)
(End of call trace)

Of course, without knowing the kernel internals it's difficult to interpret, but in this case, the error was located in _display_ioctl(), itself called by _exception_handler(), which is displayed in the second context info.


Now, I try to add the needed syscalls to allow a fully functionnal port of Newlib.
Some work is done, and it is still possible to build Newlib for sh3-unknown-fixos target, but without a working sbrk() syscall, malloc() can't work and a large part of Newlib depends on it.
The last syscalls really needed are sbrk() and stat(), the others are either not essential for the most part of the library (symlink(), unlink(), chown()...) or implemented by using other syscalls (isatty() may be replaced by some ioctl() calls or fstat()).

Offline ProgrammerNerd

  • LV3 Member (Next: 100)
  • ***
  • Posts: 50
  • Rating: +9/-2
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #24 on: July 03, 2014, 10:19:03 am »
Wow the feature lists sound very existing. Very nice work.

Offline bb010g

  • LV6 Super Member (Next: 500)
  • ******
  • Posts: 428
  • Rating: +22/-1
  • I do stuff
    • View Profile
    • elsewhere on the net
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #25 on: July 03, 2014, 04:02:42 pm »
Awesome. :D
Arch Linux user
Haskell newbie | Warming up to Lua | Being dragged into C++
Calculators: HP 50g, HP 35s, Casio Prizm, TI-Nspire CX CAS, HP 28s, HP Prime, Mathematica 9 (if that counts)
π: 3.14...; l: 108; i: 105; e: 101; l+i+e: 314
THE CAKE IS A LIE IS A PIE

Offline aeTIos

  • Nonbinary computing specialist
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3915
  • Rating: +184/-32
    • View Profile
    • wank.party
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #26 on: July 04, 2014, 01:42:17 am »
So many TL;DR posts so I skipped the thread :trollface:
Anyway this looks prettty interesting, definitely keeping an eye on this thread even though I don't have a fx9860.
I'm not a nerd but I pretend:

Offline Eiyeron

  • Urist McEiyolobster
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1430
  • Rating: +130/-10
  • (-_(//));
    • View Profile
    • Rétro-Actif : Rétro/Prog/Blog
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #27 on: July 08, 2014, 08:32:22 am »
Is there any functions done to start a shell? I could do one if you want!


EDIT : FXShell (consider that name already taken, guys)

Offline Kristaba

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 16
  • Rating: +5/-0
    • View Profile
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #28 on: July 27, 2014, 05:50:41 pm »
Once again, some news of the kernel advancement ;)
Of course, the summer is not the better time to code everyday, but since my laptop has a great backlight, I can even enjoy the sun and write code at the same time!  [-.-]~

Main new features are :
  • Dynamic memory allocation, through sbrk() system call. This is not as powerfull as mmap() using anonymous area, but it's enough for malloc() implementation!
  • New device, /dev/keyboard, for low-level keyboard handling. It's an event-based device, which deal with raw key codes, allowing efficient way to implement IsKeyDown()/GetKey() in userspace. Of course, its more designed for graphical interface and games, console applications should deal with terminal input!
    It is possible to open this device with O_NONBLOCK flag to avoid waiting if no new keyboard event are present.
  • BSD-like sysctl() syscalls, to deal with kernel information from userspace.
    For now, available sysctls are mainly kernel/architecture information (kern.ostype, kern.osrelease, kern.osbuilddate, hw.machine, hw.model), and, more interesting, process information (kern.proc.pid.<pid> and kern.proc.all).
    Process related sysctls are the BSD way to look for process state from userspace (for ps/top and other utilities), instead of the /proc special filesystem present in Linux. For now, some data are still missing (process name and arguments...) but it is still possible to have a minimal top-like application ;)
  • Average process CPU usage is now calculated on a short period (around 1 second), in addition to total kernel ticks spent in kernel and user mode. It is never used internally, only designed to be accessed from userspace (using sysctl()).
  • A special kernel process (idle process) is now scheduled when no other process are running (it is using SuperH 'sleep' assembly instruction, so the power consumption is reduced when CPU is not needed 100% of the time).
  • Wait queues are now implemented, to allow a blocking syscall to put the corresponding process in a sleep state (only uninteruptible sleep is supported for now) until a specific event. For example, a process reading keyboard input will sleep until some characters are ready for it, instead of actively waiting for a character.
  • nanosleep() syscall implementation, to force a process to sleep some time. For now, this sleep is not interruptible, so a signal can't be handled until timeout.
  • stat() and fstat() syscalls!
  • getdents() syscall, used for listing directory entries (files and sub-directories). I think this will be the only way to list a directory content, and, as Linux and every BSD, the POSIX dirent.h functions will be part of the libc.
    The getdents() syscall is a bit tricky, and not really pleasant to use in a relatively high-level application. In the same time, the latest C standard do not define anything related to directory, so I can't rely on Newlib to have a nice interface.

Newlib should now be more or less functionnal now, I did some tests recently (mainly for malloc() and date/time related functions) and its working fine. I will try to make a new repository for the Newlib port, but in the same time I think Newlib is a good temporary solution (probably the fastest way to have a working libc/libm on any new system/architecture!), but it should be more interesting in the medium/long run to use Newlib sources but with a lot of custom changes in a lot of files.
As Newlib is designed to separate platform-specific code, and to never deal directly with the generic implementation, its probably better to don't keep the Newlib build system if changes are too much importants...


Some week ago, I tried to port a game from PierrotLL on FiXos (gravity duck for fx9860), mainly to see how difficult it is and to discover potential issues in the case of a "real-time" interactive application.
So I ported MonochromeLib in a first time (using the /dev/display device, for direct rendering), which work as well as a static library and as a dynamic library with acceptable performances (around 500 FPS for overwriting the whole screen and display it).
In a second time, some changes were needed in GravityDuck code itself for replacing Casio OS functions used (GetKey(), IsKeyDown() and some functions related to ticks ellapsed to limitate the frame rate). An other function (PrintMini()) was simply removed (a similar function should be added to MonochromeLib), but it is not a big deal as Gravity Duck do not use a lot of text... As the game makes use of malloc(), I also linked the Newlib libc, but I think it is the only libc function used.

The final result is really encouraging, the game run at normal speed, and everything seems to work as the original version!  :P


I didn't released the ported Gravity Duck for now because I did it for testing purpose, and everything is not really clean, but I will try to do that in order to show than it is still possible and relatively easy to write real applications for FiXos (less than a day for porting MonochromeLib and Gravity Duck, and less issue than I feared).


Is there any functions done to start a shell? I could do one if you want!
I think more or less everything needed is working, if you want to do that, but you will really need a libc if you want to have some helper functions, else you will have to deal only with syscalls.
Two things missing for a such project are the current/working directory (for now processes are not associated to a working directory, and every path need to be absolute for open(), stat() and similar syscalls), and maybe some ioctls and VT100-like escape codes for the TTYs. I will also need to implement process groups, sessions and controling terminal for a better UNIX concept support.

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: FiXos, a UNIX-like kernel for fx9860
« Reply #29 on: July 27, 2014, 10:59:33 pm »
Good to hear that you managed to run a Casio OS game into FiXos. It's a good step in the right direction for people interested in this OS who might still want to play older games like this. :)