Super Mario 3D All-Stars

From Nintendo Switch Brew
Revision as of 20:09, 17 November 2020 by Yellows8 (talk | contribs) (→‎v1.1.0)
Jump to navigation Jump to search

This page documents the Super Mario 3D All-Stars (SM3DAS) game.

The initial menu is the Stardust launcher, which has the base ProgramId 010049900F546000. Each game launched from here has a different ProgramIndex, therefore each ProgramId differs: 0x1 (010049900F546001) SuperMario64, 0x2 (010049900F546002) SuperMarioSunshine, 0x3 (010049900F546003) SuperMarioGalaxy.

The launcher launches each game by using ExecuteProgram with ProgramIndex={above values}, with 0x10-byte parameter-data: {+0x0 = LanguageCode selected in the launcher for this game, +0x8/+0xC = floats loaded from a fixed-data array specific for each game, which games can use with SetAudioDeviceOutputVolume}. Each game returns to the launcher by using ExecuteProgram with ProgramIndex=0, no parameter-data is passed to the launcher.

Each game uses different methods of emulation.

These games use Vulkan for rendering.

These games save PlayReports with prepo, the fields include "num_stars" etc.

The pause-menu when pressing the '-' button in each game uses the Offline web-applet with WebSession.

NPDM

This section documents the NPDM for the launcher and each game.

  • Launcher: Roughly the same as most Switch games' NPDM.
  • SuperMario64: Locked down since this uses JIT. Compared to the Launcher NPDM (besides ProgramId):
  • SuperMarioSunshine: Same as Launcher (besides ProgramId), except flag OptimizeMemoryAllocation is set.
  • SuperMarioGalaxy: Same as Launcher (besides ProgramId), except flags is 0x4: Is64BitInstruction=0(AArch32) and ProcessAddressSpace=0x2.

SuperMario64

This is a Nintendo 64 emulator. This uses JIT for running the emulated executable. RomFs also contains an AOT NRO. A NRO does get loaded, however it's not used other than that (no symbols from the NRO are resolved).

At the time of release, this was the very first title to use JIT on retail.

The ROMIMAGE .bin in RomFs is identical to the Shindou region N64 ROM. Patches are applied at runtime with the config/lua files.

The LanguageCode is loaded from the launch-parameter passed by the launcher, if not available it will fallback to the output from GetDesiredLanguage. This is used to select the <language> to load from RomFs "/Message/", the data from there is then used for patching the ROM. Note that the .lua file runs n64InitMessageData(); from the GameBegin() function.

The ROMIMAGE .bin in RomFs can be replaced with other data (an unofficial image for example) when the config files are adjusted, however note that in some cases a hang may occur during the loading screen.

RomFs contains the following:

  • "/Pipeline.cache", "/shader_p1_frag.spv", "/shader_p1_vert.spv", "/shader_vulkan_frag.spv", "/shader_vulkan_vert.spv"
  • "/Message/<language>/" contains:
    • "Messages.bin", "UI_Font8.bin", "UI_Font16.bin", "UI_Font16_code2.bin", "UI_FontS88.bin".
  • "/.nrr/" contains:
  • "/nro/" contains:
    • "EmuSystemJITPlugin.nro": The NRO for the JitPlugin.
    • "NROBuilder.nro": AOT NRO.
    • "05_UNSMJ3.002.nro": Same as "NROBuilder.nro".
    • "/rom/": This contains the ROM sub-directory, in this case that dir-name is "Stardust_JP" which contains the following:
      • "Textures/texture_pack.cpio": Standard (?) .cpio archive containing textures.
      • "00_UNSMJ3.002.meta": Contains json metadata for the other files in this directory, etc.
      • "01_UNSMJ3.002.bin": This is the ROMIMAGE.
      • "02_UNSMJ3.002.cfg": Configuration json file, ROMCONFIG.
      • "03_UNSMJ3.002.pcb": This is the "PRECMP" file referenced by the .meta. See #EmuSystemJITPlugin. Using this can be disabled by changing the "PRECMP" string in the .meta to an invalid string, etc.
      • "04_UNSMJ3.002.nrr": Unused NRR which is the same as "/.nrr/04_UNSMJ3.002.nrr", except the Certification and ApplicationId are all-zero.
      • "05_UNSMJ3.002.nro": Identical to "/nro/05_UNSMJ3.002.nro".
      • "06_UNSMJ3.002.lua": This is the "ROMPATCH" Lua file referenced by the .meta.
      • "07_UNSMJ3.002.ppl": This is the "PIPELINE" file referenced by the .meta.

Emulation Notes

The generated-code runs from the main-thread. The following 4 threads are created:

  • Audio thread.
  • "g_trlRenderTaskThread". Calls funcptr/object-vfunc from a list of tasks (funcptr/object only used if set), where the tasks are sent by the main-thread. These tasks can be triggered by accessing various IO regs with Write32 (see below), etc.
  • A thread which basically does nothing, just uses a Semaphore and sleeps, while checking state.
  • An unknown thread.


  • n64MemDefault{Read/Write}{bitsize}(...), where bitsize is {8, 16, 32, 64}: implements memory read/write, all read/written data is byte-swapped. First param is the u32 vaddr. For Write, the second param is the value. For Read, the returned u32 is the read value.
  • n64MemDefaultReadSpecial{bitsize}(...), where bitsize is {8, 16, 32, 64}: same as n64MemDefaultRead{bitsize}, except this runs the additional Special functionality when accessing RAM.
  • n64MemDefault{Read/Write}{bitsize}WithDestLog(...), where bitsize is {8, 16, 32, 64}: same as n64MemDefault{Read/Write}{bitsize}, except this calls an additional logging func when accessing RAM.


  • Read8/Write8 only implements access for RAM and SP_DMEM/SP_IMEM. 0 is returned for Read when the address is unrecognized.
  • Read16/Write16 only implements access for RAM. Read16WithDestLog/Write16WithDestLog will additionally call the logging func if the address is in the IO region (high byte is 0x04). 0 is returned for Read when the address is unrecognized.
  • Read32/Write32 implements RAM access and all supported non-RAM IO/memory access. 0 is returned for Read when the address is unrecognized.
  • Read64/Write64 are implemented the same as *16 above, except this is for 64bit.

EmuSystemJITPlugin

This section documents the JIT plugin.

First, Control is used with a 0xEB0-byte struct for the InBuffer, and cmd in_u64=0. The OutBuffer is unused. This does initialization, and copies the input struct to workmem+0x0.

The u64 for GenerateCode is the type: 1-4 are valid. Type1 is used first, then type2. Then type3 is used with data loaded from the "PRECMP" file (the passed array data will be empty if the PRECMP file is not available). Type4 is only used when executing an emu_pc_addr which was not previously compiled, with a matching emu_pc_addr.

The u32 for the input Struct32 size is ignored by the plugin.

  • Type1: Struct32 and InBuffer are unused. Generates the entry code which can be called for running the rest of the code in the CodeMemory. After the generated code finishes setup, it jumps to x0.
  • Type2: Struct32 and InBuffer are unused. Generates the code for returning from CodeMemory, which is used by the rest of the generated code.
  • Type3: The emulator passes the 0x8-bytes from InBuffer+0 as the input Struct32, however the plugin doesn't use the Struct32. The emulator passes InBuffer_size=0x8C00048 and OutBuffer_size=0x2900008. This generates code for the MIPS code specified in the InBuffer. The InBuffer contains array data etc. The InBuffer can contain multiple blocks of code, which are generally individual MIPS functions. The MIPS instructions are big-endian.
  • Type4: 0x20-bytes are passed for the input Struct32: u32 +0 = emu_pc_addr, u32 +4 = set by emulator but unused by the plugin, u32 +0x18 = instruction_count (which determines the InBuffer size), the rest is unused / overwritten by the plugin. This generates code for the MIPS code specified in the InBuffer. InBuffer+0 is the u32s for the big-endian MIPS instructions. The OutBuffer is 0x2000C-bytes:
    • +0x0 is an array of u32s with the count from u32 +0x20004. Each entry is the generated-code offset for each compiled MIPS instruction, relative to the CodeRange offset.
    • +0x10000 is an array of u32s with the count from u32 +0x20004. Each entry is flags for each compiled MIPS instruction.
    • The 3 u32s at +0x20000 are the same as OutBuffer+0x0 with type1/type2 (except the last two fields can be non-zero). +0x20004 is the total compiled MIPS instructions. +0x20008 is emu_pc_addr for the first compiled instruction.

0xC-bytes are copied to the OutBuffer with type1/type2, these come from workmem. These 3 u32s are left at all-zero, except for +0 which is the generated_code_size (which is then copied to the output CodeRange size).

The 0xEB0-byte controldata struct has the following structure (all addrs/data are from the emulator user-process):

Offset Size Description
0x0 0x8 Data addr: g_pGPR
0x8 0x8 Data addr: g_pFPR
0x10 0x8 Data addr: g_pCP0
0x18 0x8 Data addr: g_nRegHi
0x20 0x8 Data addr: g_nRegLo
0x28 0x8 Data addr.
0x30 0x8 Data addr: g_nFCSR
0x38 0x8 Data addr: g_bN64CpuCmpEndCheck
0x40 0x8 Data addr: g_nN64CpuPC
0x48 0x8 Data addr: g_nN64CpuJmpPC. s32, normally -1. When not -1 this is a value which can be copied into g_nN64CpuJmpPC by the generated code, with g_nN64CpuJmpPC being reset to -1 afterwards regardless of the original value.
0x50 0x8 Data addr.
0x58 0x8 Data addr: g_nN64CpuHaveExp
0x60 0x658 Array of funcptrs. Called with w0=instruction_value from generated code for handling instructions' implementation, when the plugin itself doesn't handle it.
0x6B8 0x8 Funcptr: n64CPUProcInstCOP1Check()
0x6C0 0x8 Funcptr.
0x6C8 0x8 Funcptr.
0x6D0 0x8 Funcptr: n64CPUSetRegisterCP0(int, unsigned int)
0x6D8 0x8 Funcptr: n64CPUGetRegisterCP0(int)
0x6E0 0x20 Funcptrs for reading data: n64MemDefaultRead{bitsize}(unsigned int), where bitsize is {8, 16, 32, 64} (32 uses n64MemDefaultRead32WithDestLog). Unused by the plugin.
0x700 0x20 Funcptrs for reading Special data: n64MemDefaultReadSpecial{bitsize}(unsigned int), where bitsize is {8, 16, 32, 64}.
0x720 0x20 Funcptrs for writing data: n64MemDefaultWrite{bitsize}(unsigned int, ...), where bitsize is {8, 16, 32, 64} (32 uses n64MemDefaultWrite32WithDestLog). Unused by the plugin.
0x740 0x20 Funcptrs for reading data via the WithDestLog version: n64MemDefaultRead{bitsize}WithDestLog(unsigned int), where bitsize is {8, 16, 32, 64}.
0x760 0x20 Funcptrs for writing data via the WithDestLog version: n64MemDefaultWrite{bitsize}WithDestLog(unsigned int, ...), where bitsize is {8, 16, 32, 64}.
0x780 0x8 Funcptr: n64CPU_CallLuaAfterInst(unsigned int, unsigned int)
0x788 0x8 Funcptr: n64CPU_CallLuaBeforeInst(unsigned int, unsigned int)
0x790 0x8 Funcptr.
0x798 0x4 Array count for the the following array.
0x79C 0x180 (0x20*0xC) Array data with the above count, each entry is 0xC-bytes. The emulator loads this from config array "Idle". +0 = u32 emu_pc_addr (config "JmpAddr"), +4 = u32 instruction_value (config "JmpInst"), +8 = u32 is_idle (config "IsIdle", 0x1 = enabled).
0x91C 0x4 Array count for the the following array.
0x920 0x500 (0x40*0x14) Array data with the above count, each entry is 0x14-bytes. The emulator loads this from config array "SpecialInst". +0 = u32 emu_pc_addr (config "Addr"), +4 = u32 instruction_value (config "Inst"), +8 = u32 flags. The emulator sets the flags to 0 by default, then compares config "Type" with various strings to determine the value. Flags: bit0 = Type "BreakLoop", bit5 = Type "LoadFrameBuffer", bit6 = Type "Lua_BeforeInst", bit7 = Type "Lua_AfterInst".

When the config for {any of the previously mentioned Type strings} is set to value 1, the emulator will also enable the corresponding flags bit mentioned above. Likewise with config "Return", which sets flags bit8.

When the plugin loads a funcptr from +0x740, it will load a funcptr from +0x700 instead if there's a matching entry here which has bit5 set.

0xE20 0x4 Array count for the the following array. The plugin doesn't use this.
0xE24 0x80 (0x10*0x8) Array data with the above count, each entry is 0x8-bytes. The emulator loads this from config array "FrameBuffer". +0 = u32 (config "Addr"), +4 = u32 (config "Length"). The plugin doesn't use this.
0xEA4 0x4 Set to 0 by default, then if available the emulator sets bit flags here depending on the content of the config for "Cmp", "g_nN64CpuCmpTLBCheckLevel".
0xEA8 0x4 Set to 0 by default by the emulator. With config "Cmp", "CodeMultiAddr" this is set to 1, and with config "Cmp", "BlockDynamicPC" this is set to 2. The plugin only checks whether this ==2 or !=2, so values 0-1 are basically the same.
0xEAC 0x4 Padding

SuperMarioSunshine

This is a Nintendo Gamecube emulator. The emulator (which is the main-codebin) name is "Hagi". This uses an AOT NRO for running the emulated executable.

Since this uses AOT, attempting to run a different .dol with this emulator is equivalent to attempting to run an emulator with an executable where .text (code) is the from the original executable, while everything else is from the other executable.

This emulator supports arguments with argc/argv, when launched officially zero arguments are passed for this (unless SetProgramArgument is used manually prior to launching). The emulator will also load arguments from RomFs "/default_cmd.txt" when argc==1, otherwise it loads arguments from a filepath loaded elsewhere. Hence, when not launched with arguments with an official launch, "/default_cmd.txt" is not used. Some of the supported arguments are:

This emulator uses various hashes, SHA-1 is used for this.

This supports loading Bios from a file where the path is determined using the specified LanguageCode, when config flags "Emulate" and "LoadBios" are true, however with the MarioSunshine config the latter config flag is false.

RomFs contains the following:

  • "/hagi.config": Emulator json configuration.
  • "/hagi-override.config": Emulator json configuration.
  • "/rom.json": ROM json configuration, contains the following fields:
    • "UID": Game-name.
    • "DolHash": Hash.
    • "ROM": An array of strings for the absolute file path for the ROM. In this case, it only contains the following: "rom:/MarioSunshine/Super_Mario_Sunshine_Stardust-trimmed.gcm".
  • "/.nrr/Hagi.nrr": NRR for the AOT NRO.
  • "/MarioSunshine/": This contains the following:
    • "Super_Mario_Sunshine_Stardust-trimmed.gcm": This is the .gcm image mentioned above.
    • "Super_Mario_Sunshine_Stardust-trimmed.hash": Contains json configuration for various hashes.
  • "/data/": This contains the following:
    • "database/": This contains the following:
      • "config/MarioSunshine.config": This contains game-specific emulator json configuration.
      • "hagi/MarioSunshine.json": This contains ROM json configuration. IDs, names, etc. This also contains various hashes, including a duplicate of the above .hash file.
      • "pip/MarioSunshine/pipelineDB.pip"
      • "vulkan/MarioSunshine/B1317A6632265CF763766CF6339AC8D1.cache": This is the .cache file mentioned in #Savedata.
    • "gekko/MarioSunshine/nx/MarioSunshine.nrolz": This is the AOT NRO. LZ4 compressed, first 4-bytes is the big-endian decompressed size, then the compressed data follows.
    • "lazy_texture_replace/MarioSunshine": Contains PNG textures.
    • "saves/MarioSunshine/A.sav": Savedata. Presumably used to create the initial savedata file?
    • "texture_replace/": Contains "MarioSunshine.har" and "MarioSunshine.hix".
    • "video_replace/MarioSunshine/": Contains MP4 videos.

AOT NRO

This section documents the AOT NRO. There are only two symbols from NRO which the main-codebin does lookup for and uses: "SetGekkoMemoryPointer" and "GetStatifierInfo".

During initialization, nn::os::AllocateAddressRegion is used with size 0x100200000. Then nn::os::AllocateMemoryPages is used (which uses svcMapPhysicalMemory):

  • base+0 size 0x1000 is allocated.
  • base+0x80000000 size 0x1800000 is allocated.
  • base+0xE0000000 size 0x4000 is allocated.
  • base+0xFFF00000 size 0x200000 is allocated.

Initially SetGekkoMemoryPointer is called with base+0x80000000, however at some point it's called (?) with base+0.

The AOT NRO does all main-memory accesses by accessing {global ptr set by SetGekkoMemoryPointer}+{ppc_addr}. Therefore, above base+0 == ppc_addr 0x0. Since svcMapPhysicalMemory is used above, attempting to access any PPC-addrs not in the allocated regions above will result in the emulator crashing due to accessing unmapped memory - accessing non-PPC emulator data with invalid PPC-addrs is not possible. These crashes may happen when attempting to run a different .dol with the emulator as previously mentioned.

GetStatifierInfo takes an input string param and an output u64* param. This returns a bool: 0 = error, 1 = success. The NRO compares the input string with various strings to determine an array index, returning 0 if there's no match. Otherwise, the u64 data from the located array entry is copied to the output u64.

  • "GET_FUNCTION_INFO": Ptr to a table, where each entry is 0x18-bytes. Structure: +0x0 = ppc_code_addr, +0x8 = nro_code_addr, +0x10 = string ptr (normally "").
  • "GET_FUNCTION_INFO_SIZE" Ptr to the size of the FUNCTION_INFO table, in entries.
  • ...

Each func in the NRO listed in the above table (which are the AOT funcs) is called with x0={state ptr}, where state contains:

  • +0x0: u32 array for the PPC GPRs.
  • ...

SuperMarioGalaxy

This appears to use an emulator similar to the one used by #SuperMarioSunshine. The code/data for the MarioGalaxy executable is included directly in the main-codebin.

RomFs contains the following:

  • "/hagi.config": This is mostly the same as the "/hagi.config" for #SuperMarioSunshine, except with various config fields adjusted.
  • "/data/": This contains the following:
    • "database/": This contains "pip/pipelineDB.pip" and "vulkan/FDBD9F0C3FEA5778E10FFFA74093E0B8.cache".
    • "dsp/": This contains "drom.bin" and "irom.bin".
    • "ondemand_texture_replace/MarioGalaxy/": This contains PNG textures.
    • "texture_replace/": This contains "MarioGalaxy.har", "MarioGalaxy.hix", "MarioGalaxy_<language>.har", "MarioGalaxy_<language>.hix", "MarioGalaxy_Sub.har", "MarioGalaxy_Sub.hix".
    • "video_replace/MarioGalaxy/MovieData/": This contains MP4 videos.
  • The rest is the MarioGalaxy game assets stored directly in RomFs.

Versions

This section documents the game-update changes.

v1.0.1

In ExeFs, the only NPDM change was the usual ACID pubkey/signature update. The only other ExeFs change, was that the main-codebin was updated only for SuperMarioSunshine/SuperMarioGalaxy.

RomFs changes:

  • Launcher: Not updated.
  • SuperMario64: Both NRRs were updated, pubkey/signature was updated. No other files were updated.
  • SuperMarioSunshine: Only "/.nrr/Hagi.nrr" was updated, same as above.
  • SuperMarioGalaxy:
    • "/data/texture_replace/": Various files here were updated.
    • "/<language>/LayoutData/Font.arc": Updated.
    • "/LayoutData/CameraInfo.arc", "/LayoutData/IconAButton.arc", "/StationedData/GDDR3.arc", "/StationedData/GDDR3.zst": Updated.

v1.1.0

ExeFs:

  • Updated for using SDK 10.8.1.0, except for SuperMarioSunshine/SuperMarioGalaxy.
  • NPDMs for all Programs only had the usual update (see #v1.0.1).
    • Launcher: All files in ExeFs were updated.
    • SuperMario64: All files in ExeFs were updated, except subsdk*.
    • SuperMarioSunshine/SuperMarioGalaxy: Only main-codebin and the NPDM were updated.

RomFs:

  • Launcher: The following was updated:
    • /Layout/CursorArrow.Nin_NX_NVN.blarc.zs
    • /Layout/DialogKeyButton.Nin_NX_NVN.blarc.zs
    • /Layout/DialogTwoButton.Nin_NX_NVN.blarc.zs
    • /Message/Msg_CNzh.d.sarc.zs
    • /Message/Msg_JPja.d.sarc.zs
  • SuperMario64: The following was updated:
    • /nro/05_UNSMJ3.002.nro
    • /nro/EmuSystemJITPlugin.nro
    • /nro/NROBuilder.nro
    • /.nrr/04_UNSMJ3.002.nrr
    • /.nrr/EmuAppNX64.nrr
    • /Pipeline.cache
    • "/rom/Stardust_JP/00_UNSMJ3.002.meta" Newlines now use LF instead of CRLF.
    • "/rom/Stardust_JP/02_UNSMJ3.002.cfg" Newlines now use LF instead of CRLF. The content of the "Comment" entries under "SpecialInst" had the string characters replaced with '*' (for redacting the string). The Lua hook entries were updated under "SpecialInst". The "Comment" entries under "VertexHack" were redacted the same way, nothing else changed under "VertexHack".
    • /rom/Stardust_JP/04_UNSMJ3.002.nrr
    • /rom/Stardust_JP/05_UNSMJ3.002.nro
    • "/rom/Stardust_JP/06_UNSMJ3.002.lua" Newlines now use LF instead of CRLF. Comments were redacted, see above. HookFunc_AfterInst was updated for handling the updated hooks mentioned above.
  • SuperMarioSunshine:
    • "/data/database/config/MarioSunshine.config" The config entries for "KeyPadAxisTriggerLeft"/"KeyPadAxisTriggerRight" were updated.
    • "/data/gekko/MarioSunshine/nx/MarioSunshine.nrolz" updated
    • The following files were added under "/data/lazy_texture_replace/MarioSunshine/":
      • 5323227429978722713_lod1.png
      • 5323227429978722713_lod2.png
      • 5323227429978722713_lod3.png
      • 5323227429978722713.png
    • "/hagi-override.config" Formatting was adjusted.
    • "/MarioSunshine/Super_Mario_Sunshine_Stardust-trimmed.gcm" updated
    • "/.nrr/Hagi.nrr" updated
    • "/rom.json" The DolHash was updated.
  • SuperMarioGalaxy: The only change was removing the following files from "/StationedData/":
    • DEBUG.arc
    • DEBUG.zst
    • GDDR3.arc
    • NAPA.zst

The Manual RomFs for each game was also updated.

DeltaFragment NCAs were also added.

Savedata

The savedata contains the following:

  • "/PlayReport/account.prsav"
  • "/StardustLauncher/game_data.sav"
  • "/Mario64/Mario64.sav": The raw N64 savedata.
  • "/MarioSunshine/": This contains the following data, for MarioSunshine:
    • "gekko/log/", "gekko/trace/", and "hanafuda/": Empty directories.
    • "A.sav": The memory card image. The first 4-bytes are the big-endian decompressed size, followed by the LZ4 compressed data.
    • "B.raw": Uncompressed raw version of the memory card image. All-zeros, since it doesn't seem to be used after creation? This normally doesn't exist: this only exists if the application had argv parameter "--card-uncompressed" passed.
    • "B1317A6632265CF763766CF6339AC8D1.cache": This is probably graphics pipeline related?(RomFs has an identical file)
  • "/MarioGalaxy/GameData.bin": The savedata for SuperMarioGalaxy.
Retrieved from "https://switchbrew.org/w/index.php?title=Super_Mario_3D_All-Stars&oldid=10176"