Jr Cartridge blank for eprom

[Shaos]

That way you can shift-and-branch and the value is already in the correct format once you get to the code that handles it.

But you still need to add 4 to get the value, right? So instead of adding 4 you may add 84H, as I said before, and highest 1 will not be involved

P.S. Thank you for your great input! I'll use your code samples as a base for public domain SHAFF0-decoder for 8086/8088, hope you are Ok with that

[Shaos]

EDIT: I just thought of another optimization: Don't use FF for the code; instead, use the value that shows up the least. Scan the entire file before compressing to determine which value shows up the least, then use that as the code. Otherwise, data that has a lot of FFs in it (like most 8-bit programming!) won't compress very well...

I don't think I'll get a lot by switching to rarest byte - usually a lot of FFs will be successfully eaten by LZ77, but I can at least estimate using outputs from my current design to see possible impact of that...

P.S. So I took results of compression of JRCARTS7.IMG and count number of literals FF that was coded in:

Code: Select all

> grep "\[255\]" JRCARTS7.out
[255] 0xFF ? = 218
[255] 0xFF ? = 175
[255] 0xFF ? = 140
[255] 0xFF ? = 39
[255] 0xFF ? = 62
[255] 0xFF ? = 98
[255] 0xFF ? = 159
[255] 0xFF ? = 240
[255] 0xFF ? = 140
> python
Python 2.5.2 (r252:60911, Jan 24 2010, 18:51:01) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 218+175+140+39+62+98+159+240+140
1271
>>> 112715-1271
111444
>>> 100.0*1271/112715    
1.127622765381715

so by switching to rarest byte I'll get extra 1.2KB (-1.13%) so 112,715 will turn into 111,444 - from one hand it's better, from other it's not too much yet to convince me to get rid of FF at the name of my compression utility (it's SHAFF because of FF : ) - it's still a little far from ZX7 and RNC method 2...[/b]

Actually I found some sample of data that gave me 4% improvement by using rarest byte instead of FF, so I think I can support it as an option...

[Trixter]

But you still need to add 4 to get the value, right? So instead of adding 4 you may add 84H, as I said before, and highest 1 will not be involved :)

Oh, right, of course :-) Yes, if you can add the 4 and adjust at the same time (and don't need to preserve carry bit), then yes, that's better :)

P.S. Thank you for your great input! I'll use your code samples as a base for public domain SHAFF0-decoder for 8086/8088, hope you are Ok with that ;)

I don't mind at all, since I was a few steps away from implementing it myself.

[Trixter]

Actually I found some sample of data that gave me 4% improvement by using rarest byte instead of FF, so I think I can support it as an option...

It's the best kind of 4% -- it's for free. I wish all my schemes could have better compression for free.

[Shaos]

Actually I found some sample of data that gave me 4% improvement by using rarest byte instead of FF, so I think I can support it as an option...

It's the best kind of 4% -- it's for free. I wish all my schemes could have better compression for free.

OK, for JRCARTS7.IMG I've got these results (with 0FFH and with 92H):

Code: Select all

147,456 JRCARTS7.IMG - original
130,281 JRCARTS8.IMG - my 00-00-00 compression
112,745 JRCARTS7.IMGFF - my byte-oriented LZ77-like compression (SHAFF0) <<<<<<<<<<<<<<<<<<<<<<
111,526 JRCARTS7.IMGFF - my byte-oriented LZ77-like compression (SHAFF0 with option -x92) <<<<<
103,307 JRCARTS7.PP2 - RNC Pro Pack (method 2)
101,009 JRCARTS7.ZX7
100,180 JRCARTS7.IMG.hrm - Russian Hrum
 99,974 JRCARTS7.IMG.mlz - Russian MegaLZ
 98,045 JRCARTS7.IMGFF - my bit-oriented LZ77-like compression (SHAFF1) <<<<<<<<<<<<<<<<<<<<<<
 98,006 JRCARTS7.LZH - created by LHA v2.13
 80,016 JRCARTS7.IMG.bz2
 79,098 JRCARTS7.LZ4
 72,959 JRCARTS7.PP1 - RNC Pro Pack (method 1)
 71,950 JRCARTS7.zip - modern ZIP from Debian
 71,925 JRCARTS7.ARJ - created by ARJ v2.41a
 71,865 JRCARTS7.AIN - another Russian archiver from 90s
 71,855 JRCARTS7.ZIP - old PKZIP v2.06
 71,808 JRCARTS7.IMG.gz
 70,041 JRCARTS7.HA - HA 0.98 (Harri Hirvola)
 67,437 JRCARTS7.IMG.hst - Russian Hrust
 66,970 JRCARTS7.RAR - old RAR v1
 65,765 JRCARTS7.RAR - modern RAR v5
 62,543 JRCARTS7.7z - modern 7-zip
 62,508 JRCARTS7.IMG.xz - modern LZMA2

So now I have 19.5KB for menu

P.S. Latest SHAFF0 encoder/decoder is located on GitHub https://github.com/shaos/shaff (tested on WinXP, Linux x86_64 and Linux PowerPC G4)

P.P.S. SHAFF1 moved me to 98KB (confirmed on Feb 8), but because it's bit-oriented, decoder will be a little bigger (and slower)

P.P.P.S. SHAFF2 (SHAFF1 + Huffman for literals) potentially may give us about 87K in this case, but I don't think I finish it in 2017

[Trixter]

Very cool. But looking at LZ4 size, I don't know why you just don't use that. My LZ4 decoder only decodes up to a single segment, so you'd have to compress and decompress each ROM by itself if you wanted to use my code, but even still it seems like a win...

[Shaos]

Very cool. But looking at LZ4 size, I don't know why you just don't use that. My LZ4 decoder only decodes up to a single segment, so you'd have to compress and decompress each ROM by itself if you wanted to use my code, but even still it seems like a win...

Yea, may be I should try that...

P.S. Interesting fact - ZX7 shows much better results than LZ4 on compression of Z80 code, for example:

Code: Select all

 49179 thor.sna
 32557 thor.SNAFF <<<<< SHAFF0
 31211 thor.SNAFF <<<<< SHAFF0 (-xE9)
 29770 thor.LZ4
 27293 thor.sna.bz2
 26970 thor.PP2 - RNC Pro Pack (method 2)
 26643 thor.SNAFF <<<<< SHAFF1
 26291 thor.ZX7
 26164 thor.sna.hrm
 26041 thor.sna.mlz
 25761 thor.PP1 - RNC Pro Pack (method 1)
 25235 thor.zip
 25114 thor.sna.gz
 25031 thor.sna.hst
 24132 thor.rar
 22100 thor.sna.xz
 21788 thor.EX2

[Trixter]

P.S. Interesting fact - ZX7 shows much better results than LZ4 on compression of Z80 code, for example:

ZX7 always compresses smaller than LZ4 because ZX7 has an entropy encoder (range coding) on the back-end, whereas LZ4 does not. Where you pay for it is on the decompression side: LZ4 is extremely fast, ranging from 0.8x to 3.2x memcpy() speed, whereas ZX7 hovers around 9x memcpy() speed. Of course, decompression speed isn't a concern for a ROM cartridge -- nobody will care if it takes 0.5 seconds vs. 1 second to start the game...

[Shaos]

P.S. Interesting fact - ZX7 shows much better results than LZ4 on compression of Z80 code, for example:

ZX7 always compresses smaller than LZ4 because ZX7 has an entropy encoder (range coding) on the back-end, whereas LZ4 does not...

But it's not the case for JRCARTS7.IMG:

Code: Select all

101,009 JRCARTS7.ZX7
..........
 79,098 JRCARTS7.LZ4

[Trixter]

LZ4 has an optimal parser; ZX7 claims something similar, but does not. There might be a special case in that data (like, a lot of runs) that LZ4's compressor is able to tackle better.