Author Topic: Map Data Storage  (Read 9747 times)

0 Members and 1 Guest are viewing this topic.

Offline BlackCode

  • LV2 Member (Next: 40)
  • **
  • Posts: 28
  • Rating: +0/-0
    • View Profile
Map Data Storage
« on: August 24, 2013, 01:20:49 am »
Hello everyone, first post and all!  I just have some questions about the "best" way to store map data.  I've been playing around with Axe, and am starting to work on a map, but I'm not sure how to store data in a way that fits my needs.  I need to be able to change the map while the program is running.  Ideally, I also need a large space to store my data and/or a way of efficient compression (consider that maps will likely be highly detailed).  Any and all help is appreciated, even if it just consists of linking me to a resource I've missed.
-I post from my iPod, so please forgive any funky formatting.

Offline aeTIos

  • Nonbinary computing specialist
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3915
  • Rating: +184/-32
    • View Profile
    • wank.party
Re: Map Data Storage
« Reply #1 on: August 24, 2013, 03:29:23 am »
You need an appvar. I will explain how to create one later since I am on my phone at the moment and typing is a pita.
Welcome to omnimaga by the way!
I'm not a nerd but I pretend:

Offline Hayleia

  • Programming Absol
  • Coder Of Tomorrow
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3367
  • Rating: +393/-7
    • View Profile
Re: Map Data Storage
« Reply #2 on: August 24, 2013, 06:04:01 am »
Yeah, appvars are what you are looking for. Here is the tutorial that taught me everything about them. Besides, it doesn't only teach about appvars. Be sure to give the author a +1.

And welcome to Omnimaga, you can introduce yourself here.
I own: 83+ ; 84+SE ; 76.fr ; CX CAS ; Prizm ; 84+CSE
Sorry if I answer with something that seems unrelated, English is not my primary language and I might not have understood well. Sorry if I make English mistakes too.

click here to know where you got your last +1s

Offline Keoni29

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2466
  • Rating: +291/-16
    • View Profile
    • My electronics projects at 8times8
Re: Map Data Storage
« Reply #3 on: August 24, 2013, 06:53:24 am »
And as for world maps: it really depends on the application. If your world consists of screens of tiles you should organize your data in a different manner than if your world consists of a large tile map. As for compression: If you only use 16 different tiles you can store two tiles in every byte instead of just one per byte. If you want to go even further you can replace all zero tiles with a zero followed by the number of zero tiles until a new kind of tile occurs. This technique is only suited if there is just a single tilemap in your appvar or if you indexed where all tilemaps start.
Example for storing 2 tiles in one byte:
Code: [Select]
.Not actual sourcecode:
.tile1 = 7
.tile2 = 3
.tile1 << 4 | tile2 = $73
.Axe syntax:
[73]
Example for compressing zero tiles:
Code: [Select]
[15 00 26 00 00 00 00 00 00 00 00 24 A5]
.becomes
[15 00 01 26 00 08 24 A5]
.Downside to this technique is that single zero's will take up more space.
.Now to decompress:
.if the byte = 0
.load counter with next byte that defines the amount of zero's that should be inserted
.do
.  insert 0 tile
.  counter decrease
.while counter is not 0
.when the counter runs out go to the next tile and repeat the process until you aquired enough tiles.
If you like my work: why not give me an internet?








Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: Map Data Storage
« Reply #4 on: August 24, 2013, 09:54:42 am »
do the changes you want to make to the map have to be written back and saved for the next time the program is run? if so, that's going to make compression stuff difficult. if they don't, you can just break the map up into compressed chunks and only decompress them when that section of the map needs to be loaded. if they do, though, you might be better off storing the changes applied to the map separately so that they can be changed dynamically without having to recompress the entire chunk. having to both decompress map sections and apply a diff can slow things down quite a bit, though, so it becomes a bit of a balancing act between saving space and saving time.

Offline Hayleia

  • Programming Absol
  • Coder Of Tomorrow
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3367
  • Rating: +393/-7
    • View Profile
Re: Map Data Storage
« Reply #5 on: August 24, 2013, 10:11:28 am »
Another efficient compression method is the one deeph and chickendude showed me. Basically, it is RLE, but with a special treatment for isolated tiles. The only downside is that it only works if you have less than 128 tiles.
So, when you have 8 instances of a 4, you just put a 8 followed by a 4 in your data, and you just used 2 bytes for 8 bytes of data, so you saved space.
But when you have 1 instance of a 4, putting a 1 followed by a for would use 2 bytes for 1 byte of data. So instead of that, you'll store 4+128 (basically a 4 but with an extra bit at the "beginning").
And now, to uncompress, you just have to look at the first bit:
-if it's 0, treat the two bytes like RLE compressed data
-if it's 1, then put one instance of the byte (and don't forget to retrieve 128 from it).
I own: 83+ ; 84+SE ; 76.fr ; CX CAS ; Prizm ; 84+CSE
Sorry if I answer with something that seems unrelated, English is not my primary language and I might not have understood well. Sorry if I make English mistakes too.

click here to know where you got your last +1s

Offline BlackCode

  • LV2 Member (Next: 40)
  • **
  • Posts: 28
  • Rating: +0/-0
    • View Profile
Re: Map Data Storage
« Reply #6 on: August 24, 2013, 06:36:33 pm »
Thanks for all the responses!  I've checked out those tutorials, and appvars do sound like what I need, however I still have a few concerns.  Because I would need to be able to write to the appvar at anytime, it would have to be unarchived throughout the program, so how much space would I actually have in an appvar and/or is there any way around this that would allow me to write to an archived appvar?
-I post from my iPod, so please forgive any funky formatting.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: Map Data Storage
« Reply #7 on: August 24, 2013, 07:52:31 pm »
writing directly to archive is both tricky and a terrible idea, as it risks bricking your calculator. also, like i said above trying to write back a huge appvar every time your program is exited will be annoying for the user (taking a long time, causing lots of garbage collects, and, eventually, wearing out the flash). to get around both this issue and the issue of having too little ram during runtime, you basically have one of two options:

if you expect your user to only modify a few tiles here and there, the best method is to keep separate diff files for sections of your map, like i described above.

if you expect basically every tile on the entire map to be overwritten, you might want to break your map up into smaller chunks and write back only those that have been modified, decreasing the amount of writing necessary a bit.
« Last Edit: August 24, 2013, 07:53:41 pm by shmibs »

Offline BlackCode

  • LV2 Member (Next: 40)
  • **
  • Posts: 28
  • Rating: +0/-0
    • View Profile
Re: Map Data Storage
« Reply #8 on: August 24, 2013, 09:57:56 pm »
Shmibs, could you perhaps elaborate on the differences between those two methods?  I'm not seeing how they would vary.  Additionally, wouldn't chunking the map mean consistent slow downs throughout the game as certain chunks were unarchived and archived.  And it would compound the issue of wearing out flash (which I've seen as only happening after over 100,000 writes, something I don't forsee happening if it only writes to flash once, at the end of each program).
-I post from my iPod, so please forgive any funky formatting.

Offline Xeda112358

  • they/them
  • Moderator
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 4704
  • Rating: +719/-6
  • Calc-u-lator, do doo doo do do do.
    • View Profile
Re: Map Data Storage
« Reply #9 on: August 24, 2013, 10:07:18 pm »
With the first method he is describing, I believe he means that you basically keep a list of changes in something like a save file. This way your tilemap data can remain archived, but when you copy (I mean copy, not unarchive) it to RAM, you check the list of changes to see if any of the tiles have to be modified. So for example, if Map00 has the (3,3) tile modified to a 6, you might include in the list a sequence of bytes: 00030306. When the player enters Map00, you copy the appvar data to RAM, then you read the list of changes to see that (3,3) needs to be changed to a 6.

Offline AngelFish

  • Is this my custom title?
  • Administrator
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3242
  • Rating: +270/-27
  • I'm a Fishbot
    • View Profile
Re: Map Data Storage
« Reply #10 on: August 24, 2013, 10:10:40 pm »
And it would compound the issue of wearing out flash (which I've seen as only happening after over 100,000 writes, something I don't forsee happening if it only writes to flash once, at the end of each program).

Don't forget that other programs have to use that same flash. Every write you do is one less write for other people. That said, writing once per run is perfectly reasonable. Using flash as if it were RAM is not, which is the point I think shmibs was trying to make.
∂²Ψ    -(2m(V(x)-E)Ψ
---  = -------------
∂x²        ℏ²Ψ

Offline BlackCode

  • LV2 Member (Next: 40)
  • **
  • Posts: 28
  • Rating: +0/-0
    • View Profile
Re: Map Data Storage
« Reply #11 on: August 24, 2013, 10:47:39 pm »
Ah, thank you Xeda, that clears it up a bit.  If I were to do that though, I would still have to either archive the list of changes, or change the base map and archive that, which doesn't really seem efficient to me.  So, it seems, to me, that my best bet is using a single appvar to hold all the data, unarchiving it at the beginning, and archiving it at the end of the program.

And AngelFish, I am well aware of that, which is why I was suprised when shmibs suggested using multiple appvars as a better method.
-I post from my iPod, so please forgive any funky formatting.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: Map Data Storage
« Reply #12 on: August 24, 2013, 10:55:20 pm »
i suggested multiple because breaking up your map into smaller pieces allows you to only write back those which have had changes made to them.

Offline BlackCode

  • LV2 Member (Next: 40)
  • **
  • Posts: 28
  • Rating: +0/-0
    • View Profile
Re: Map Data Storage
« Reply #13 on: August 24, 2013, 11:02:05 pm »
But that would vastly increase the number of archive writes, would it not?  A fairly severe problem (after a while).
-I post from my iPod, so please forgive any funky formatting.

Offline Runer112

  • Project Author
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2289
  • Rating: +639/-31
    • View Profile
Re: Map Data Storage
« Reply #14 on: August 24, 2013, 11:12:15 pm »
The life of flash memory decreases with archive sector erases, not archive writes. Each sector is 64KB large. If you consistently unarchived and archived variables, each 64KB written to archive would result in one erase of the swap sector (probably the first sector likely to fail) at the next garbage collect. It is very roughly estimated that sectors may be likely to fail starting around 100,000 erases, so a total of about 6GB (yes, gigabytes) would need to be archived in the calculator's lifetime to start worrying about the flash memory failing.