Author Topic: [Library] Compress  (Read 7440 times)

0 Members and 1 Guest are viewing this topic.

Offline Morhaus

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 5
  • Rating: +3/-0
    • View Profile
[Library] Compress
« on: June 01, 2013, 04:02:18 pm »
Hey guys.

The default image format TI uses doesn't have any compression, which results in very slow loadings and big file size. I've written a small library which allows to compress images or data on your computer and decompress it in Lua on the calculator.

Decompressing in Lua might seem a bit overkill but it actually works out quite nicely, and I've experienced substantial gains in loading speed and file size.

Currently, only two algorithms are supported : RLE (Run-length encoding) and LZW (Lempel-Ziv-Welch). RLE tends to produce lighter files, but they also tend to load slower than those produced by LZW.

The library is located in src/compress.lua. You can use packages lzw.py, rle.py, lua.py and image.py to compress your data or your images. The convert.py utility allows for fast conversion between any image format to lua or tns files. Examples of use can be found below.
The image.py package requires ImageMagick to be installed on your computer and the 'convert' and 'identify' utilities to be in your PATH.
In order to produce tns files, the convert.py package requires luna to be installed on your computer and the 'luna' executable to be in your PATH.

Version 0.0.4 :
https://mega.co.nz/#!owxVXbAQ!MPb4JRfuZbVcEriqC6IqsUbJRjdcQw9CMYDzvEuwahc
Changelog: http://ourl.ca/19010/351502

Version 0.0.3 :
https://mega.co.nz/#!UxYBAZQR!BfvEC079-PEB0j98wG58RGjTjJRrt4GDA5xsJMV1KYY
Changelog: http://ourl.ca/19010/350707

Version 0.0.2 :
https://mega.co.nz/#!w8hTnIJB!SDoEbCshxMgHZbV3bNXFcxCMy-0d9H3iK7fUiFTpoZ8

Changelog :
- image.py now uses ImageMagick instead of PIL. It's a bit slower, but should produce a better output and it allows more control on the output. As a consequence, it now requires ImageMagick to be installed on your system and the 'convert' and 'identify' utilities to be in your PATH.
- example.py renamed to convert.py, now takes command line arguments to allow automatic generation of lua or tns files. TNS file generation requires luna to be installed on your system and to be in your PATH.
- lzw.decompress and rle.decompress (lua) now take a table as an optional third argument. It allows to prepend a header directly onto the output instead of concatenating strings.

Examples :
Display help message
Code: [Select]
python convert.py -h
Converts every image in img/ to lua files in lua/
Code: [Select]
python convert.py img/* -o lua/
Converts every png and gif image in img/ to tns files in the same directory, restricted to a palette of 128 colors (default is 256)
Code: [Select]
python convert.py --tns -c 128 img/*.png img/*.gif
If your path selector targets non image file, an error will be thrown.
Example :
Code: [Select]
python convert.py passwords.txt
Traceback (most recent call last):
  File "convert.py", line 23, in <module>
    image = Image(path, colors=args.colors)
  File "/Users/alexandre/Projets/compress/image.py", line 17, in __init__
    raise Exception(stderr_output)
Exception: identify: no decode delegate for this image format `passwords.txt' @ error/constitute.c/ReadImage/550.

Version 0.0.1 :
https://mega.co.nz/#!UwhnDRAJ!Tuad9XC01kTMPOZRa7DztkY7okJrhJ7NgMElQwSm8GQ

The library is located in src/compress.lua. You can use packages lzw.py, rle.py, lua.py and image.py to compress your data or your images. Examples can be found in example.py and src/example.lua.
The image.py package requires the PIL to be installed on your computer. If you already have Python 2.7.3/3.X, you can install it via "pip install pillow".
« Last Edit: June 18, 2013, 03:50:32 pm by Morhaus »

Offline Jim Bauwens

  • Lua! Nspire! Linux!
  • Editor
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1881
  • Rating: +206/-7
  • Linux!
    • View Profile
    • nothing...
Re: [Library] Compress
« Reply #1 on: June 01, 2013, 04:40:45 pm »
Very interesting!

I've creating a little RLE algorithm for TI.Image some time ago, but I never really used it because it was too slow to my likings. I'll definitely check this out (and look if I can help with optimizations) when I get some time after my exams ^^

Many thanks :)
« Last Edit: June 01, 2013, 04:43:21 pm by Jim Bauwens »

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: [Library] Compress
« Reply #2 on: June 02, 2013, 03:18:08 pm »
That's looking interesting, it would indeed be pretty useful to compress some data.
* Sorunome wonders if it would be possible to do something similar for the z80 calcs

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline Hayleia

  • Programming Absol
  • Coder Of Tomorrow
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3367
  • Rating: +393/-7
    • View Profile
Re: [Library] Compress
« Reply #3 on: June 02, 2013, 03:20:43 pm »
Great work, you solved the dilemma that coders were facing: great graphics or fast launch. Now they can have both.

* Sorunome wonders if it would be possible to do something similar for the z80 calcs
http://ourl.ca/9960/190945
I own: 83+ ; 84+SE ; 76.fr ; CX CAS ; Prizm ; 84+CSE
Sorry if I answer with something that seems unrelated, English is not my primary language and I might not have understood well. Sorry if I make English mistakes too.

click here to know where you got your last +1s

Offline ElementCoder

  • LV7 Elite (Next: 700)
  • *******
  • Posts: 611
  • Rating: +42/-2
    • View Profile
Re: [Library] Compress
« Reply #4 on: June 02, 2013, 03:21:44 pm »
This is pretty interesting :D Now I definitely need to get back into Lua :P I'm looking forward to see more of this (and other things).

Some people need a high five in the face... with a chair.
~EC

Offline Morhaus

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 5
  • Rating: +3/-0
    • View Profile
Re: [Library] Compress
« Reply #5 on: June 06, 2013, 08:07:38 am »
Updated to the last version.

New download link : https://mega.co.nz/#!w8hTnIJB!SDoEbCshxMgHZbV3bNXFcxCMy-0d9H3iK7fUiFTpoZ8

Changelog :
- image.py now uses ImageMagick instead of PIL. It's a bit slower, but should produce a better output and it allows more control on the output. As a consequence, it now requires ImageMagick to be installed on your system and the 'convert' and 'identify' utilities to be in your PATH.
- example.py renamed to convert.py, now takes command line arguments to allow automatic generation of lua or tns files. TNS file generation requires luna to be installed on your system and to be in your PATH.
- lzw.decompress and rle.decompress (lua) now take a table as an optional third argument. It allows to prepend a header directly onto the output instead of concatenating strings.

Examples :
Display help message
Code: [Select]
python convert.py -h
Converts every image in img/ to lua files in lua/
Code: [Select]
python convert.py img/* -o lua/
Converts every png and gif image in img/ to tns files in the same directory, restricted to a palette of 128 colors (default is 256)
Code: [Select]
python convert.py --tns -c 128 img/*.png img/*.gif
If your path selector targets non image file, an error will be thrown.
Example :
Code: [Select]
python convert.py passwords.txt
« Last Edit: June 06, 2013, 08:35:24 am by Morhaus »

Offline Morhaus

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 5
  • Rating: +3/-0
    • View Profile
Re: [Library] Compress
« Reply #6 on: June 06, 2013, 08:36:27 am »
Minor fixes, if you downloaded the last version before this post, you should download it again now.

Offline Morhaus

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 5
  • Rating: +3/-0
    • View Profile
Re: [Library] Compress
« Reply #7 on: June 12, 2013, 02:47:20 pm »
New version !
https://mega.co.nz/#!UxYBAZQR!BfvEC079-PEB0j98wG58RGjTjJRrt4GDA5xsJMV1KYY

Changelog :
  • image.py won't try to correct colors anymore.
  • lzw.compress is much faster.
  • convert.py throws an error when no input files were found.
  • convert.py can build lua or tns files with more than one image with the paramter -z/--zip.
  • convert.py now accepts the optional parameter -l [N], which limits the maximum size of the code table used by LZW (useful for big files).
  • convert.py now accepts the optional parameter --rle, which forces the use of the RLE encoding.
  • zoom keys :
  • +: zoom in
  • -: zoom out
  • *: the image uses the entire screen width.
  • /:  the image uses the entire screen height.
  • 0: zoom 1:1-
  • keys 1, 2, 3, 4, 6, 7, 8, 9 allow for fast travel between the different corners of the image.
  • keys A, B, C, ..., Z allow to navigate between images.
« Last Edit: June 12, 2013, 02:47:36 pm by Morhaus »

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: [Library] Compress
« Reply #8 on: June 13, 2013, 11:40:14 pm »
Heya and welcome here! :)

That's a great tool you have here.

Offline Morhaus

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 5
  • Rating: +3/-0
    • View Profile
Re: [Library] Compress
« Reply #9 on: June 18, 2013, 03:49:25 pm »
Thanks :)

New version ! 0.0.4
https://mega.co.nz/#!owxVXbAQ!MPb4JRfuZbVcEriqC6IqsUbJRjdcQw9CMYDzvEuwahc

  • RLE encoding/decoding fixed (was completely broken).
  • LZW encoding heavily optimised.
  • the "convert" executable path is now manually resolved, to avoid confusion with Windows' own "convert" executable
  • the images obtained via -z/--zip are no longer loaded at start, but rather dynamically loaded to avoir memory errors. As a side effect, scripts that contain multiple images now load faster.
« Last Edit: June 18, 2013, 03:52:02 pm by Morhaus »

Offline imath

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 18
  • Rating: +0/-0
    • View Profile
Re: [Library] Compress
« Reply #10 on: June 30, 2013, 12:40:37 am »
seems good!