[4.5.9] CrunchyNES: Graphics compression + up-to-512-BG-tiles + sprite overlays + easy-split-screens in NESMaker

michel_iwaniec

New member
Lately I've worked on a little conversion utility + NES library that I think NESMakers might find useful.

github.com/michel-iwaniec/CrunchyNES

github.com/michel-iwaniec/CrunchyNES/releases/download/v1.1/CrunchyNES-v1.1-win64.zip

CrunchyNES basically aims to make it a lot easier to display title screens / cutscenes with full-screen artwork that use "intermediate-level-tricks" in your Mapper30 NES game.

Namely, the aforementioned intermediate-level-tricks are:
  • Sprite overlays in either 8x8 or 8x16 mode - for more colorful full-screen pictures
  • Up to 512 8x8 background tiles using mid-frame CHR bank-switching - for bigger / less repetitive pixel art
  • Easily controlled scrolling and split-screen - for those big enemies or parallax horizons
  • Codemasters / Tokumaru tile compression - so those extra background tiles come for free*

Future things I'm likely to add, in semi-priority order (shout if you actually care / disagree on the priorities)
  • Multiple banks (CHR / nametable decompression reading bytes from other banks should be trivial to add)
  • Make CrunchyBuild conversion settings be per-picture instead of global for more flexibility
  • DPCM IRQs to avoid wasting lots of frame time for mid-frame CHR bankswitching / split screen effects
  • DPCM-sample playback raster timing compensation (though you can't use it at the same time as DPCM IRQs obviously)
  • Scrolling Pictures spanning multiple nametables (512x240, and maybe even 256x480)
  • PAL support (currently mid-frame CHR bank-switching / split screen is NTSC-only)
  • Palette fades
  • OAM write optimisations
  • Less noticeable sprite#0 pixel generation
  • Top/bottom borders. And maybe a vertical "clamp" mode to bypass the wrap-around effect with vertical mirroring.
CrunchyNES relies on two separate components: CrunchyBuild and CrunchyLib.

CrunchyBuild is a command-line tool. It takes a list of indexed color images and builds a dedicated bank with these images converted and compressed. This bank also includes the CrunchyLib NES code.

CrunchyLib is the actual NES code library that contains subroutines for loading the picture / updating sprites, and an NMI handler function to perform the timed scrolling updates.

CrunchyNES isn't NESMaker-specific... but I have made a patch for NESMaker that integrates it in a as-simple-as-I-could way, using a post-screen-loading subroutine. I've attached this integration to this thread (details to follow).

It repurposes the top overworld screen bank (16x2 = 32 screens) to load the CrunchyNES compressed pictures instead of the normal NESMaker level screens.

The sprite overlays will be scrolled along with the picture using the scrolling coordinates in CrunchyLib, without having to manage them as separate NESMaker objects.

Animation is not supported directly by CrunchyNES at the moment. But you can still use NESMaker objects on top of the CrunchyNES sprite overlay pictures to add some animation, if you wish to have some Ninja Gaiden-style animated cutscenes.
But keep in mind that CrunchyNES uses 8x16 sprites by default, so you'll need to switch to running CrunchyBuild with 8x8 sprite generation for your NESMaker objects to look correctly... or figure out how to use 8x16 sprites in NESMaker if you're brave. ;)

You can also use NESMaker textboxes on top of CrunchyNES. At the moment this requires the textbox area to have blank tiles, as the CrunchyNES NESMaker integration currently can't restore the original picture the textbox drew on top of.
If you do this, you also want to set the "--max_bg_slots" parameter of CrunchyBuild to 208 to leave some room for the NESMaker text characters at the end of the BG pattern table. You can use this mixing of CrunchyNES pictures with regular NESMaker text together with the mid-frame CHR-bank-switching, which still gives you up to 512 - 2*48 = 416 background tiles for your picture.

To get started with CrunchyNES, I would first recommend trying CrunchyNES on its own outside of NESMaker, as this will make it easier to test the conversion process and debug any potential issues.
CrunchyNES comes with a stand-alone NES-viewer CrunchyView which allows you to switch between a number of images, setting the split-screen parameters, and scrolling the picture - using just the joypad to drive this.

Have a read through the user guide to understand how CrunchyNES works and what parameters are supported, and try running CrunchyView in Mesen and use the "PPU viewer" to see how your pictures use the character memory / sprites. That should give you a feel for how you can either pimp or simplify your full-screen artwork before putting it into your game.

CrunchyView might also be a useful stand-alone-tool if you have a need to just quickly check how your background-with-sprite-overlays picture looks on a CRT while being either still or scrolling, before you spend time trying to to put it into your NESMaker game. Just build a few images with CrunchyBuild and put them onto your flash cart... and enjoy the slideshow :)
 

Attachments

  • CrunchyLib_zero_page_RAM.png
    CrunchyLib_zero_page_RAM.png
    34.2 KB · Views: 16
  • CrunchyNESMakerIntegration.zip
    118.4 KB · Views: 17

michel_iwaniec

New member
Using CrunchyNES with NESMaker

Once you've spent some time playing around with CrunchyBuild & CrunchyView on their own, using CrunchyBuild & CrunchyLib with the NESMaker integration is fairly straightforward.

Start by downloading the CrunchyNESMakerIntegration.zip attached to this post.

This .zip file contains some patched source files from NESMaker 4.5.9 which you can copy straight into the NESmaker_4_5_x directory. It also contains a .patch file if you prefer to use the Linux-world "patch" tool, or just want to browse through them in a single file.

Once the changes are applied you'll have a new directory Game NESmaker_4_5_x\GameEngineData\CrunchyNES. This directory contains just a single file CrunchyNM.asm, which will include the files generated by CrunchyBuild's image conversion / build process.
After you've succesfully run CrunchyBuild this directory will be filled with crunchylib.asm, the Codemasters / Tokumaru compressor decompress.asm, .chr / nametable /OAM data and another include file that puts them together.

In the original NESMaker files that have been modified, every single place has been clearly marked with "CrunchyNES: " comments to make these patches easy to spot and combine with whatever changes you may already have done to NESMaker.

To summarize these patches to the original NESMaker source code in brief:

1) In NMI.asm, we add the call to the CrunchyLib_Display subroutine.
CrunchyLib_Display sets the initial scroll, ensures a sprite#0 hit on the first two scanlines, and potentially delays a number of scanlines if CHR bank-switching / split-screen is active, before returning back control to the NESMaker NMI handler code.

We also fix a race condition bug that can occasionally cause CHR corruption on screen loading, by skipping the writes to $2005 if the "soft2001" variable is zero. This isn't strictly required for the CrunchyNES integration as it's actually a bug in the original NESMaker code. But we might as well fix this nuisance while we're messing around in the NMI. :)

2) In MainGameLoop.asm we make sure ordinary NESMaker objects start at sprite#1 instead of sprite#0 in OAM, and call the CrunchyLib_WriteOAM subroutine.
(Note this call to CrunchyLib_WriteOAM can be avoided if you only wish to display non-scrolling pictures)

3) In doDrawBox.asm we override the restore-tile value and the restore-attribute value with 0 if the screen is a CrunchyNES screen. This hack avoids the complexity of having to either re-read CrunchyNES's compressed nametable data or read from PPU memory. (which are both a bit more complex than we need for the time being)

4) In doLoadChrRam.asm we make sure to force bank-switching to always use CHR bank#0 for writing the plain-old NESMaker CHR data, even if CrunchyLib has previously set a different bank during picture display.

5) In Bank00.asm we comment out all the .include directives which include the (now redundant) uncompressed nametables / attribute tables that NESMaker would normally use.

We leave the collision tables intact however, as these contain screen info bytes at offset 120, which are needed for correct loading of objects and whatnot during screen transitions. They also allow using the CrunchyNES pictures as sort-of-maps if you wish to (although this is more by accident than design).

6) In doLoadScreen.asm we switch to the CRUNCHY_PRG_BANK and call CrunchyNM_LoadPicture to load the CrunchyNES picture if we're in the topmost overworld screen bank.

7) CrunchyNM_LoadPicture itself performs a straight mapping of the 32 first screens to CrunchyNES picture indices. Note that this mapping is just an arbitrary convenient choice, and you could change it to whatever you prefer it to be - potentially using these coordinate bits to denote new ways to load the picture if you so wish.

You also need to do one more setup step from the NESMaker GUI in order to define memory space for CrunchyLib's persistent variables.

Under "Project Settings" -> "Zero Page RAM", define a new variable "Crunchy_ZP_RAM" with at least 14 bytes, and make sure to move this variables so it is placed *right after the temp16 variable*.
This is the zeropage memory CrunchyLib will use for the various persistent variable. (see attached screenshot)

CrunchyLib_zero_page_RAM.png

For temporary variables, the CrunchyNES NESMaker integration just repurposes the NESMaker temporary zeropage variables to conserve zeropage memory. However, the Codemasters / Tokumaru CHR decompression (which needs some additional memory to function) will also use parts of this zeropage as temporary storage, with CrunchyLib re-initialising the scrolling / display variables to zero after CHR decompression.

Finally, you build the CrunchyNES 16kB bank by calling CrunchyBuild on the command-line as such:

CrunchyBuild.exe --input C:\CrunchyNES_test_pictures\Bernie-converted.png C:\CrunchyNES_test_pictures\Tutorititle2_indexed_now_with_compress.png C:\CrunchyNES_test_pictures\rocket_clipart-converted.png --sprite_size 8x8 --prgbank 0 --prefix_dir CrunchyNES\ --output C:\NESmaker_4_5_x\GameEngineData\CrunchyNES --max_bg_slots 208 -v


And that's it. You can now load compressed 512-background-tiles-with-sprite-overlay-pictures in the top overworld screen bank, and manipulate the CrunchyVar_* variables to have scrolling and / or split-scrolling effects.

For more details on how these CrunchyVar variables work, please see the user guide at github.com/michel-iwaniec/CrunchyNES/ and again try running the CrunchyView stand-alone viewer to play around with them and see how they work.

A small-n-silly demo .mst file is included in the attached .zip. This is based on the beginning of the adventure tutorial and just loads 3 different test pictures based on which warp you enter.

The pictures loaded demonstrate some of the different features you can use:

West warp: Shows a CrunchyNES picture (Bernie-converted.png) that uses mid-frame CHR bank-switching to display more than > 256 background tiles, with a textbox on top.

North warp: Shows how to use split-screen scrolling. There's some extra code in CrunchyNM.asm which manipulates the CrunchyVar_* variables to scroll the different parts of the screen. (delete that code for a real project)

East warp: Just shows some regular animated NESMaker objects together with the CrunchyNES picture.

Disclaimer: I'm still quite a n00b when it comes to NESMaker... so while CrunchyBuild and CrunchyLib themselves should be pretty sensible code, my NESMaker integration is just a first attempt at hacking this in, and there's probably much cleaner ways to do this. So suggestions from more experienced folks are quite welcome. :)

I've also only tested this with the Adventure module so far. But in theory it ought to work with any of the modules... perhaps with some minor tweaks.
 

michel_iwaniec

New member
Using OverlayPal to prepare images for CrunchyBuild

If you're finding creating indexed-color PNG background + sprite overlay images (which CrunchyNES takes as input) a bit challenging to work with, then I've also made another recent tool called OverlayPal which tries to simplify this process.


Bernie-screenshot.png

You can download the latest version of OverlayPal from github.com/michel-iwaniec/OverlayPal/releases/download/v0.9/OverlayPal-v0.9-win64-Release.zip

Documentation is available on the github page:
github.com/michel-iwaniec/OverlayPal/

OverlayPal is a graphical tool which aims to make the process of drawing fullscreen artwork that combines backgrounds + sprites more easy.

This method would usually require carefully deciding on background / sprite palettes and duplicating colors in multiple palettes and building an image from these low-level "building blocks". OverlayPal instead decomposes an existing image into these building blocks.

OverlayPal can RGB images that can use any of the 52 arbitrary colors in the NES palette, and then tries to find a valid partitioning that decompose these colors into either background or sprite overlay.
The output from OverlayPal is a new indexed image with the first 32 colors representing the NES palette, with colors duplicated in different palette groups as necessary.

A word of warning: Because finding such a palette partitioning is a non-trivial programming problem, OverlayPal's conversion process can be rather slow. And for complex images you may need to bump the default timeout value up quite a bit for a successful conversion. Your computer's speed will also determine how high a timeout value you need...

However, OverlayPal also has an auto-conversion mode where it will monitor the last file loaded for any changes and re-trigger the conversion if it changes on disk. This mode allows drawing artwork in your favorite pixel editor, with OverlayPal automatically re-running the conversion in the background whenever you save your image.
You can thus use OverlayPal as a verification tool to glance at from time to time, in order to make sure you're still staying within the NES limits whilst letting your artistic side run wild. ;)

OverlayPal still has some rough edges, so go easy on it and don't try to max out your sprites / scanline count just yet if you want to use it for automatic conversion of your graphics. But do let me know if you run into problems using it.
 

Jonny

Well-known member
Amazing thank-you so much, this will help a lot of people. The mid-frame technique was explained in the Discord server only yesterday. What a crazy coincidence!
 

Logana

Well-known member
Looks cool, don’t think I will ever use it personally tho since I don’t wanna risk and entire bank, also because I know logistically how getting the most colors out of the nes works
 

michel_iwaniec

New member
Looks cool, don’t think I will ever use it personally tho since I don’t wanna risk and entire bank, also because I know logistically how getting the most colors out of the nes works

Yeah, it's definitely not a perfect fit for every type of project :)

Anyway, to clarify one thing: There's no requirement to sacrifice the entire 16x2 screen bank if you don't need that much data for CrunchyNES pictures. You could easily change one or two instructions which do the screen -> CrunchyNES picture mapping to only consider the upper 16 screens of the top screen bank as CrunchyNES pictures. That'd be just half a screen bank, with the other 16 screens functioning as normal. Or take it further and use just 8 or less screens for CrunchyNES pictures.

But as the decompression code takes around 10% of the 16kB bank, it wouldn't make sense to use it if you're only using one or two pictures. It's once you start to have a handful of them that the compression really starts giving you space savings (given certain assumptions on content).

It's also not mandatory to use screen banks at all if screens are the most precious resource in your NESmaker game. The example integration is just that: An example of how CrunchyNES can work with NESmaker.
I chose to sacrifice 32 screens / 1 bank because that kind of data is the most plentiful in NESmaker, with most released games not using anything close to the 512 total screens available for overworld / underworld screens. And having each CrunchyNES picture "pretend" to be a NESmaker screen allows most other screen-related things in NESmaker (like warp-to and NPC messages) to keep functioning the same without changes, keeping it simple.

But CrunchyLib itself doesn't actually deal with screens on a map, just pictures with a number index. So it's perfectly possible to use some completely different system unrelated to NESmaker screens to trigger loading of specific pictures, like say a monster script. Although you'd have to figure it out on your own... and for the average user I think it'd be a lot less useful in practice than the current solution of mapping CrunchyNES pictures to NESmaker screens.
 
Sim, definitivamente não é um ajuste perfeito para todo tipo de projeto:)

De qualquer forma, para esclarecer uma coisa: não há necessidade de sacrificar todo o banco de tela 16x2 se você não precisar de tantos dados para as fotos do CrunchyNES. Você pode facilmente alterar uma ou duas instruções que fazem a tela -> mapeamento de imagem do CrunchyNES para considerar apenas as 16 telas superiores do banco de tela superior como imagens do CrunchyNES. Isso seria apenas metade de um banco de telas, com as outras 16 telas funcionando normalmente. Ou vá mais longe e use apenas 8 telas ou menos para fotos do CrunchyNES.

Mas como o código de descompressão ocupa cerca de 10% do banco de 16kB, não faria sentido usá-lo se você estiver usando apenas uma ou duas fotos. É quando você começa a ter um punhado deles que a compactação realmente começa a oferecer economia de espaço (dadas certas suposições sobre o conteúdo).

Também não é obrigatório usar bancos de tela se as telas forem o recurso mais precioso em seu jogo NESmaker. O exemplo de integração é apenas isso: Um exemplo de como o CrunchyNES pode trabalhar com o NESmaker.
Escolhi sacrificar 32 telas / 1 banco porque esse tipo de dado é o mais abundante no NESmaker, com a maioria dos jogos lançados não usando nada perto do total de 512 telas disponíveis para telas de overworld / underworld. E fazer com que cada imagem do CrunchyNES "finja" ser uma tela do NESmaker permite que a maioria das outras coisas relacionadas à tela no NESmaker (como warp-to e mensagens NPC) continuem funcionando da mesma forma sem alterações, mantendo-o simples.

Mas o próprio CrunchyLib não lida com telas em um mapa, apenas imagens com um índice numérico. Portanto, é perfeitamente possível usar algum sistema completamente diferente não relacionado às telas do NESmaker para acionar o carregamento de imagens específicas, como, digamos, um script de monstro. Embora você tenha que descobrir por conta própria... e para o usuário médio eu acho que seria muito menos útil na prática do que a solução atual de mapear imagens do CrunchyNES para telas do NESmaker.
Amazing
 
Lately I've worked on a little conversion utility + NES library that I think NESMakers might find useful.

github.com/michel-iwaniec/CrunchyNES

github.com/michel-iwaniec/CrunchyNES/releases/download/v1.1/CrunchyNES-v1.1-win64.zip

CrunchyNES basically aims to make it a lot easier to display title screens / cutscenes with full-screen artwork that use "intermediate-level-tricks" in your Mapper30 NES game.

Namely, the aforementioned intermediate-level-tricks are:
  • Sprite overlays in either 8x8 or 8x16 mode - for more colorful full-screen pictures
  • Up to 512 8x8 background tiles using mid-frame CHR bank-switching - for bigger / less repetitive pixel art
  • Easily controlled scrolling and split-screen - for those big enemies or parallax horizons
  • Codemasters / Tokumaru tile compression - so those extra background tiles come for free*

Future things I'm likely to add, in semi-priority order (shout if you actually care / disagree on the priorities)
  • Multiple banks (CHR / nametable decompression reading bytes from other banks should be trivial to add)
  • Make CrunchyBuild conversion settings be per-picture instead of global for more flexibility
  • DPCM IRQs to avoid wasting lots of frame time for mid-frame CHR bankswitching / split screen effects
  • DPCM-sample playback raster timing compensation (though you can't use it at the same time as DPCM IRQs obviously)
  • Scrolling Pictures spanning multiple nametables (512x240, and maybe even 256x480)
  • PAL support (currently mid-frame CHR bank-switching / split screen is NTSC-only)
  • Palette fades
  • OAM write optimisations
  • Less noticeable sprite#0 pixel generation
  • Top/bottom borders. And maybe a vertical "clamp" mode to bypass the wrap-around effect with vertical mirroring.
CrunchyNES relies on two separate components: CrunchyBuild and CrunchyLib.

CrunchyBuild is a command-line tool. It takes a list of indexed color images and builds a dedicated bank with these images converted and compressed. This bank also includes the CrunchyLib NES code.

CrunchyLib is the actual NES code library that contains subroutines for loading the picture / updating sprites, and an NMI handler function to perform the timed scrolling updates.

CrunchyNES isn't NESMaker-specific... but I have made a patch for NESMaker that integrates it in a as-simple-as-I-could way, using a post-screen-loading subroutine. I've attached this integration to this thread (details to follow).

It repurposes the top overworld screen bank (16x2 = 32 screens) to load the CrunchyNES compressed pictures instead of the normal NESMaker level screens.

The sprite overlays will be scrolled along with the picture using the scrolling coordinates in CrunchyLib, without having to manage them as separate NESMaker objects.

Animation is not supported directly by CrunchyNES at the moment. But you can still use NESMaker objects on top of the CrunchyNES sprite overlay pictures to add some animation, if you wish to have some Ninja Gaiden-style animated cutscenes.
But keep in mind that CrunchyNES uses 8x16 sprites by default, so you'll need to switch to running CrunchyBuild with 8x8 sprite generation for your NESMaker objects to look correctly... or figure out how to use 8x16 sprites in NESMaker if you're brave. ;)

You can also use NESMaker textboxes on top of CrunchyNES. At the moment this requires the textbox area to have blank tiles, as the CrunchyNES NESMaker integration currently can't restore the original picture the textbox drew on top of.
If you do this, you also want to set the "--max_bg_slots" parameter of CrunchyBuild to 208 to leave some room for the NESMaker text characters at the end of the BG pattern table. You can use this mixing of CrunchyNES pictures with regular NESMaker text together with the mid-frame CHR-bank-switching, which still gives you up to 512 - 2*48 = 416 background tiles for your picture.

To get started with CrunchyNES, I would first recommend trying CrunchyNES on its own outside of NESMaker, as this will make it easier to test the conversion process and debug any potential issues.
CrunchyNES comes with a stand-alone NES-viewer CrunchyView which allows you to switch between a number of images, setting the split-screen parameters, and scrolling the picture - using just the joypad to drive this.

Have a read through the user guide to understand how CrunchyNES works and what parameters are supported, and try running CrunchyView in Mesen and use the "PPU viewer" to see how your pictures use the character memory / sprites. That should give you a feel for how you can either pimp or simplify your full-screen artwork before putting it into your game.

CrunchyView might also be a useful stand-alone-tool if you have a need to just quickly check how your background-with-sprite-overlays picture looks on a CRT while being either still or scrolling, before you spend time trying to to put it into your NESMaker game. Just build a few images with CrunchyBuild and put them onto your flash cart... and enjoy the slideshow :)
I think you are the person I was looking for. I'd like to show you a NES game. How do I talk to you?
 
Top Bottom