0 Members and 5 Guests are viewing this topic.
OK, it _has_ to be a bug then. I'm using Axe 1.2.1 and changing the values to "1.0, and 10.0" makes the number the _same_ 250, which means that, according to you, there's a rounding error(for some reason).
Also the thing that got me into this whole crazy thing in the first place was some "big"(at least I think it is) numbers not working right in Axe.
It was (2/10)*65535. I was expecting a whole number 13,107. But, it never turns into that value... ever. I doubt Axe would be able to do it right now since you're saying it's all 256ths and 65535 256ths wouldn't be the 65535 value(a maximum for the total range in the range encoder I was working), just isn't working at all. I tried the fixed point math, it doesn't work either, nothing seems to work.The value I get is 65485(which is almost 1.0*65535), and it well should've been 0.2(51/256ths clearly close enough), but somehow that gets converted into nearly 1.0 which makes no sense to me(at the moment) but I imagine it's something with the parser.
Edit: Basically, I wanted to do a simple rouguelike with psuedo-procedural generated content. But I wanted a way to compress the premade "rooms" and other data, in range encoding(seemed like a perfect first program... at least has always worked just fine for all of my other language excursions). Well to be honest, I normally do LZW, as the first one, but axe it looked to be requiring way too much work. But now I don't even know if range encoding will even work, because that's the _only_ spot I have to use non integers, and of course the one spot where I need some sort of non-whole number math, it doesn't work... Basically even if I don't do "real numers" it says it's 0. Even though (2/10)*65535 should be that 13,107. This is _only_ spot I needed non integer math, and of course it's not going to work.Well... I guess there's two spots, the part where I calculate the share of the range, and then once again when I'm running over the data... but either way I had always used 65535(or some similar "big") number as I don't need super-duper precision. I just needed a way to do fixed point during that part. Then a single operation during the second part too. I was planning on just storing the counts(x bits) one after another and then recalculating the whole range later on. But anyway yeah, 3 decimal points _should_ be way more than enough.(with the whole range I'm using I just need for the values at the end to add up to the whole thing), and then of course make sure that the ranges work right(for the final two byte range value).
009DB900B3AB3FB4B3ABB3AB55D48AA71F000000400184788B8068CB