13.0.0: Difference between revisions

Line 80:

* [[HID_services#Firmware_update|ControllerFirmware]]: updated "FirmwareInfo.csv", "TouchScreenFirmwareInfo.csv", "ukyosakyo_ep2_ota.bin". Added "FTS_33000510.fts256" and "FTS_98000004.ftb".

* Various applet UI/gfx data was updated.

===BootImagePackage===

All files in RomFS were updated.

====Kernel====

* Compiler upgrade to LLVM 11.1.

** Most notably, certain code now emits "ands" rather than "and; and; tst" + "bfxil" patterns are more commonly used.

* C++ language upgrade to C++20 (or C++17, but less likely).

** Slab heaps/object containers are now constinit, no longer constructed during .init_array.

* Initialize0 changes:

** KernelLdr now sends back the initial process binary address in state.

** Initialize0 now uses a helper function for selecting random virtual regions, and now verifies KInitialPageTable->IsFree() before selecting regions.

** Initialize0 now verifies that the initial process binary address is in correct place (pool partition), and sets global=initial address.

*** This global is now used where GetInitialProcessBinaryAddress() was used previously.

* KPort limit increased to 0x180 from 0x100.

* SvcMapDeviceAddressSpace() was removed.

** Userland only ever used SvcMapDeviceAddressSpaceByForce and SvcMapDeviceAddressSpaceAligned, so this doesn't break any official software.

** Map only allowed one page table to be allocated, only partially mapping the desired range if more than one would be used.

*** Similarly, it returned an output mapped size so a caller could continue mapping partially until the whole range was done, one page table at a time.

** All parameters required to implement partial mapping have been removed, correspondingly.

*** No out_mapped_size, out_page_table_count, pt_limit parameters to any KDevicePageTable mapping functions any more.

*** UnlockForDevicePageTablePartialMap no longer takes in a partially-mapped size.

**** This code is simplified, since it is no longer possible to have a partially-mapped size other than zero.

* The two new kernel objects/four new SVCs added in 11.0.0 are finally present/instantiated.

** Prototypes for the new SVCs:

*** 0x39: Result CreateIoPool(Handle *out_handle, uint32_t which);

*** 0x3A: Result CreateIoRegion(Handle *out_handle, Handle io_pool, PhysicalAddress physical_address, size_t size, MemoryMapping mapping, MemoryPermission perm);

**** MemoryMapping is a new enum, 0 = IO, 1 = Uncached memory, 2 = Normal Memory.

*** 0x46: Result MapIoRegion(Handle region_handle, uintptr_t address, size_t size, MemoryPermission perm);

*** 0x47: Result UnmapIoRegion(Handle region_handle, uintptr_t address, size_t size);

** These SVCs conceptually allow creating an object for mapping in certain physical address ranges at user-specified virtual addresses, without having to include the ranges in npdm/kip capabilities ahead of time.

*** The only allowed id right now is 0, which corresponds to PCIE_A2 (physical address range 0x12000000-0x1FFFFFFF).

* KWritableEvent was deleted.

** Class tokens for types after KWritableEvent have been adjusted downwards to compensate (see mesosphere for class token generation algorithm).

** Handles which were previously returned to KWritableEvent are now returned directly to KEvent.

** KEvent has new boolean member to track whether the readable event has been destroyed.

** KReadableEvent::Initialize now opens reference to the parent event, rather than inline in KEvent::Initialize.

* KReadableEvent::Signal/Clear are no longer virtual functions.

* KAutoObject no longer has virtual destructor.

** This mostly means that base/deleting destructor no longer occur inside vtables.

* New memory state (0x16/0x2016) "Coverage", currently not exposed via any SVCs.

* A number of functions are now devirtualized when possible, indicating either the virtual functions (or the classes) were marked final.

** KProcess::GetId()

** KThread::GetId()

** KSessionRequest::Finalize()

** KInitialPageAllocator::Free()

* KConditionVariable::WaitForAddress/KConditionalVariable::SignalToAddress are now static

* KMemoryManager now operates on physical addresses instead of virtual addresses.

** KMemoryManager::Initialize now iterates the physical tree rather than the virtual tree, when finding pool regions.

** KMemoryManager::Allocate now returns a physical address instead of a virtual address.

** KMemoryManager::Close now takes in a physical address instead of a virtual address.

** KPageHeap now operates on physical addresses rather than virtual addresses.

*** New member stores the linear virtual address for the physical range the heap operates on.

** KBlockInfo now size 0x10 instead of 0x20.

*** Before: struct { KBlockInfo *prev; KBlockInfo *next; size_t num_pages; KVirtualAddress address; }

*** Now: struct { KBlockInfo *next; u32 phys_address_page; u32 num_pages; };

*** phys_address_page is a KPhysicalAddress / PageSize, to ensure it fits in u32.

** This halves memory requirements for KBlockInfos system-wide, effectively doubling the KPageGroup capacity.

** Memory range helper object used by page table functions also now operates on physical addresses.

* KPageTableBase no longer contains virtual memory region cache.

** This was only used in KPageTableBase::MapPageGroupImpl to do ABORT_UNLESS(IsHeapVirtualAddress(...));

** This abort is no longer present, likely because page groups are now physical blocks rather than virtual ones.

* KPageTable::Unmap now incrementally frees pages, rather than freeing them all at once.

* KHandleTableEntryInfo no longer stores object class token, KHandleTable::Add/Register no longer takes class token as argument.

** This reduces KHandleTable size (and thus KProcess size) by 0x800.

* The following types no longer have (unused) slab heaps:

** KClientSession, KLightClientSession, KLightServerSession

* A complete re-work/unification was done for the various kinds of thread waiting operations in the kernel.

** Previously, there were a number of different ways to initiate and end waits, handled manually in each location by invoking thread->SetState(...);

** Now, all waits use a common interface based around thread queue objects with virtual functions for the three kinds of supported waits.

** Supported queue virtual functions:

*** void NotifyAvailable(KThread *waiting_thread, KSynchronizationObject *signaled_object, Result wait_result);

**** This is supported only by the queue used by WaitSynchronization, and sets the signaled object/sync'd index for the thread.

*** void EndWait(KThread *waiting_thread, Result wait_result);

**** This conceptually ends a wait "normally", usually without additional cleanup. This is exclusively called by kernel-handled wait codepaths.

*** void CancelWait(KThread *waiting_thread, Result wait_result, bool cancel_timer_task);

**** This conceptually ends a wait "by force", interrupting it regardless of normal wait completion.

***** This usually involves extra cleanup.

**** This is invoked by e.g. CancelSynchronization, thread termination, thread finalize, etc.

** KThread::SetState is likely private now; it is called only by internal KThread functions.

** KLightSession was notably reworked substantially to take advantage of the new unified wait semantics.

** KSynchronizationObject::DumpWaiters() no longer exists.

* KDebug is now substantially more careful about management of its process pointer.

** New field "is_attached" explicitly tracks whether the KDebug is attached (instead of m_process != nullptr).

** m_process replaced by a new helper object containing KProcess * and reference count.

*** Code which previously used the process field now opens/closes references on m_process_holder.

*** When m_process_holder's reference count hits zero, the process is closed and m_pointer is set to 1 instead of nullptr.

** This makes the way KDebug treats the attached process's reference count much more correct/consistent.

*** In particular, "the KDebug is attached" now counts as one reference to the KProcess, no matter what the KDebug is doing with it.

* Resource management has changed substantially.

** Dynamic slabheaps no longer contain a pointer to an associated page allocator; instead, a page allocator is now passed as an argument to .Allocate().

** There are new helper objects which contain a dynamic slab heap pointer and a page allocator pointer, used for conveniently referencing a slab/page resource pair.

*** There are now helper objects differentiating between KBlockInfo and KPageTableManager allocators for system/application context.

*** KPageTable now has a helper function to allocate a page from the KPageTableManager helper object.

** Instead of allocating all unused pages to the page table page heap, all but 70 pages are allocated to the page table page heap.

** When a new flag from the secure monitor is zero (this is always the case on retail), new logic for "dynamic resource expansion" is enabled.

*** In particular, this causes the helper objects for the system memory blocks, block infos, and page table managers to be set to the dynamic page allocator.

**** Thus, the last 70 unused pages are dynamically allocated to (system memory blocks, system block infos, system page table pages) on a first-come basis as the system uses these resources.

**** NOTE: There is no "free"-ing of these resources back to the unused page heap, once they've been allocated to a specific slab they will remain in that slab until reboot.

*** In addition, when allocating a KSession, KEvent, or KLightSession from the system resource limit fails, an object of the desired type will be allocated from the otherwise unused "gaps" in the slab region.

**** When allocating a KSession dynamically from gap-space, kernel also allocates two KSessionRequests from gap-space and frees them to the KSessionRequest slabheap.

**** Allocator for this uses an intrusive red black tree on in-place nodes in the gaps.

* Minor changes to the atomic operations for slab heaps.

*** Slab heap initialization now panics if L1 cache is direct-mapped (1-way associative).

*** Slab heap allocation no longer issues a clrex on failure.

*** For types supporting the new dynamic allocation, Slab heap free now checks that the object lives within the slab region instead of panicking when object is not within slab extents.

* The interrupt task manager thread no longer exists, functionality has been rolled into KScheduler.

** KScheduler now has member pointer to the interrupt task manager for the current core.

** KScheduler::EnableScheduling now sends scheduler interrupt unconditionally, regardless of disable count.

** KScheduler::RescheduleCurrentCore now uses double-checked-locking-esque strategy for checking scheduling necessity around interrupt disables.

** KScheduler::Schedule now processes interrupt tasks directly (calls a KInterruptTaskManager member function) rather than setting the interrupt task thread to runnable.

* KInterruptEventTask no longer contains a KLightLock member.

** KSchedulerLock is used in places where interrupt events were locked previously.

* KSchedulerInterruptTask is now a KInterruptHandler, rather than a KInterruptTask (no ::DoTask() implementation, any more).

* KProcess::AddSharedMemory now sets all fields in one block/scope, rather than two.

* KWorkerTaskManager::Initialize now hardcodes id=0 instead of taking it as a parameter.

** KWorkerTaskManager no longer has "id"/"active" fields.

* Certain unnecessary conditionals were optimized in KServerSession::SendReply/ReceiveMessage.

** Code of the form `x = condition ? a : b` with condition's value fixed for the block was optimized to `x = a`.

** This may just be a relic of compiler upgrade mentioned above.

* Minor changes to GetInfo:

** Logic which previously disabled interrupt for profiling InfoTypes now instead disables dispatch (KScopedDisableDispatch instead of KScopedDisableInterrupt).

** InfoType_ThreadTickCount has had its value changed from 0xF0000002 to 0x19. This is presumably to generate better asm for the switch statement.

* SendAsyncRequestWithUserBuffer now accepts ResultThreadTerminating as a success result (it does not unlock memory when it is returned).

* KClientSession::SendSyncRequest/SendAsyncRequest no longer hold the scheduler lock while calling KServerSession::OnRequest.

** Lock is now acquired as needed by KServerSession::OnRequest itself.

* KClientSession, KLightClientSession, KLightServerSession no longer inherit from KSlabAllocated/KAutoObjectWithList, and are now correspondingly smaller.

* KResourceLimit::Allocate() now calls KAutoObject::Create().

* KPrintf has been replaced with a function which takes a non-format string, and prints the string, then <current program ID formatted as %016lx>, then a newline.

** "Break() called. %016lx\n" and "Exception occurred. %016lx\n" are now "Break() called. " and "Exception occurred. ", respectively.

* StoreDataCacheSharedForInit has slightly different (but equivalent) iteration logic.

===[[Bluetooth_Driver_services|bluetooth]]===

@@ Line 80: / Line 80: @@
 * [[HID_services#Firmware_update|ControllerFirmware]]: updated "FirmwareInfo.csv", "TouchScreenFirmwareInfo.csv", "ukyosakyo_ep2_ota.bin". Added "FTS_33000510.fts256" and "FTS_98000004.ftb".
 * Various applet UI/gfx data was updated.
+===BootImagePackage===
+All files in RomFS were updated.
+====Kernel====
+* Compiler upgrade to LLVM 11.1.
+** Most notably, certain code now emits "ands" rather than "and; and; tst" + "bfxil" patterns are more commonly used.
+* C++ language upgrade to C++20 (or C++17, but less likely).
+** Slab heaps/object containers are now constinit, no longer constructed during .init_array.
+* Initialize0 changes:
+** KernelLdr now sends back the initial process binary address in state.
+** Initialize0 now uses a helper function for selecting random virtual regions, and now verifies KInitialPageTable->IsFree() before selecting regions.
+** Initialize0 now verifies that the initial process binary address is in correct place (pool partition), and sets global=initial address.
+*** This global is now used where GetInitialProcessBinaryAddress() was used previously.
+* KPort limit increased to 0x180 from 0x100.
+* SvcMapDeviceAddressSpace() was removed.
+** Userland only ever used SvcMapDeviceAddressSpaceByForce and SvcMapDeviceAddressSpaceAligned, so this doesn't break any official software.
+** Map only allowed one page table to be allocated, only partially mapping the desired range if more than one would be used.
+*** Similarly, it returned an output mapped size so a caller could continue mapping partially until the whole range was done, one page table at a time.
+** All parameters required to implement partial mapping have been removed, correspondingly.
+*** No out_mapped_size, out_page_table_count, pt_limit parameters to any KDevicePageTable mapping functions any more.
+*** UnlockForDevicePageTablePartialMap no longer takes in a partially-mapped size.
+**** This code is simplified, since it is no longer possible to have a partially-mapped size other than zero.
+* The two new kernel objects/four new SVCs added in 11.0.0 are finally present/instantiated.
+** Prototypes for the new SVCs:
+*** 0x39: Result CreateIoPool(Handle *out_handle, uint32_t which);
+*** 0x3A: Result CreateIoRegion(Handle *out_handle, Handle io_pool, PhysicalAddress physical_address, size_t size, MemoryMapping mapping, MemoryPermission perm);
+**** MemoryMapping is a new enum, 0 = IO, 1 = Uncached memory, 2 = Normal Memory.
+*** 0x46: Result MapIoRegion(Handle region_handle, uintptr_t address, size_t size, MemoryPermission perm);
+*** 0x47: Result UnmapIoRegion(Handle region_handle, uintptr_t address, size_t size);
+** These SVCs conceptually allow creating an object for mapping in certain physical address ranges at user-specified virtual addresses, without having to include the ranges in npdm/kip capabilities ahead of time.
+*** The only allowed id right now is 0, which corresponds to PCIE_A2 (physical address range 0x12000000-0x1FFFFFFF).
+* KWritableEvent was deleted.
+** Class tokens for types after KWritableEvent have been adjusted downwards to compensate (see mesosphere for class token generation algorithm).
+** Handles which were previously returned to KWritableEvent are now returned directly to KEvent.
+** KEvent has new boolean member to track whether the readable event has been destroyed.
+** KReadableEvent::Initialize now opens reference to the parent event, rather than inline in KEvent::Initialize.
+* KReadableEvent::Signal/Clear are no longer virtual functions.
+* KAutoObject no longer has virtual destructor.
+** This mostly means that base/deleting destructor no longer occur inside vtables.
+* New memory state (0x16/0x2016) "Coverage", currently not exposed via any SVCs.
+* A number of functions are now devirtualized when possible, indicating either the virtual functions (or the classes) were marked final.
+** KProcess::GetId()
+** KThread::GetId()
+** KSessionRequest::Finalize()
+** KInitialPageAllocator::Free()
+* KConditionVariable::WaitForAddress/KConditionalVariable::SignalToAddress are now static
+* KMemoryManager now operates on physical addresses instead of virtual addresses.
+** KMemoryManager::Initialize now iterates the physical tree rather than the virtual tree, when finding pool regions.
+** KMemoryManager::Allocate now returns a physical address instead of a virtual address.
+** KMemoryManager::Close now takes in a physical address instead of a virtual address.
+** KPageHeap now operates on physical addresses rather than virtual addresses.
+*** New member stores the linear virtual address for the physical range the heap operates on.
+** KBlockInfo now size 0x10 instead of 0x20.
+*** Before: struct { KBlockInfo *prev; KBlockInfo *next; size_t num_pages; KVirtualAddress address; }
+*** Now:    struct { KBlockInfo *next; u32 phys_address_page; u32 num_pages; };
+*** phys_address_page is a KPhysicalAddress / PageSize, to ensure it fits in u32.
+** This halves memory requirements for KBlockInfos system-wide, effectively doubling the KPageGroup capacity.
+** Memory range helper object used by page table functions also now operates on physical addresses.
+* KPageTableBase no longer contains virtual memory region cache.
+** This was only used in KPageTableBase::MapPageGroupImpl to do ABORT_UNLESS(IsHeapVirtualAddress(...));
+** This abort is no longer present, likely because page groups are now physical blocks rather than virtual ones.
+* KPageTable::Unmap now incrementally frees pages, rather than freeing them all at once.
+* KHandleTableEntryInfo no longer stores object class token, KHandleTable::Add/Register no longer takes class token as argument.
+** This reduces KHandleTable size (and thus KProcess size) by 0x800.
+* The following types no longer have (unused) slab heaps:
+** KClientSession, KLightClientSession, KLightServerSession
+* A complete re-work/unification was done for the various kinds of thread waiting operations in the kernel.
+** Previously, there were a number of different ways to initiate and end waits, handled manually in each location by invoking thread->SetState(...);
+** Now, all waits use a common interface based around thread queue objects with virtual functions for the three kinds of supported waits.
+** Supported queue virtual functions:
+*** void NotifyAvailable(KThread *waiting_thread, KSynchronizationObject *signaled_object, Result wait_result);
+**** This is supported only by the queue used by WaitSynchronization, and sets the signaled object/sync'd index for the thread.
+*** void EndWait(KThread *waiting_thread, Result wait_result);
+**** This conceptually ends a wait "normally", usually without additional cleanup. This is exclusively called by kernel-handled wait codepaths.
+*** void CancelWait(KThread *waiting_thread, Result wait_result, bool cancel_timer_task);
+**** This conceptually ends a wait "by force", interrupting it regardless of normal wait completion.
+***** This usually involves extra cleanup.
+**** This is invoked by e.g. CancelSynchronization, thread termination, thread finalize, etc.
+** KThread::SetState is likely private now; it is called only by internal KThread functions.
+** KLightSession was notably reworked substantially to take advantage of the new unified wait semantics.
+** KSynchronizationObject::DumpWaiters() no longer exists.
+* KDebug is now substantially more careful about management of its process pointer.
+** New field "is_attached" explicitly tracks whether the KDebug is attached (instead of m_process != nullptr).
+** m_process replaced by a new helper object containing KProcess * and reference count.
+*** Code which previously used the process field now opens/closes references on m_process_holder.
+*** When m_process_holder's reference count hits zero, the process is closed and m_pointer is set to 1 instead of nullptr.
+** This makes the way KDebug treats the attached process's reference count much more correct/consistent.
+*** In particular, "the KDebug is attached" now counts as one reference to the KProcess, no matter what the KDebug is doing with it.
+* Resource management has changed substantially.
+** Dynamic slabheaps no longer contain a pointer to an associated page allocator; instead, a page allocator is now passed as an argument to .Allocate().
+** There are new helper objects which contain a dynamic slab heap pointer and a page allocator pointer, used for conveniently referencing a slab/page resource pair.
+*** There are now helper objects differentiating between KBlockInfo and KPageTableManager allocators for system/application context.
+*** KPageTable now has a helper function to allocate a page from the KPageTableManager helper object.
+** Instead of allocating all unused pages to the page table page heap, all but 70 pages are allocated to the page table page heap.
+** When a new flag from the secure monitor is zero (this is always the case on retail), new logic for "dynamic resource expansion" is enabled.
+*** In particular, this causes the helper objects for the system memory blocks, block infos, and page table managers to be set to the dynamic page allocator.
+**** Thus, the last 70 unused pages are dynamically allocated to (system memory blocks, system block infos, system page table pages) on a first-come basis as the system uses these resources.
+**** NOTE: There is no "free"-ing of these resources back to the unused page heap, once they've been allocated to a specific slab they will remain in that slab until reboot.
+*** In addition, when allocating a KSession, KEvent, or KLightSession from the system resource limit fails, an object of the desired type will be allocated from the otherwise unused "gaps" in the slab region.
+**** When allocating a KSession dynamically from gap-space, kernel also allocates two KSessionRequests from gap-space and frees them to the KSessionRequest slabheap.
+**** Allocator for this uses an intrusive red black tree on in-place nodes in the gaps.
+* Minor changes to the atomic operations for slab heaps.
+*** Slab heap initialization now panics if L1 cache is direct-mapped (1-way associative).
+*** Slab heap allocation no longer issues a clrex on failure.
+*** For types supporting the new dynamic allocation, Slab heap free now checks that the object lives within the slab region instead of panicking when object is not within slab extents.
+* The interrupt task manager thread no longer exists, functionality has been rolled into KScheduler.
+** KScheduler now has member pointer to the interrupt task manager for the current core.
+** KScheduler::EnableScheduling now sends scheduler interrupt unconditionally, regardless of disable count.
+** KScheduler::RescheduleCurrentCore now uses double-checked-locking-esque strategy for checking scheduling necessity around interrupt disables.
+** KScheduler::Schedule now processes interrupt tasks directly (calls a KInterruptTaskManager member function) rather than setting the interrupt task thread to runnable.
+* KInterruptEventTask no longer contains a KLightLock member.
+** KSchedulerLock is used in places where interrupt events were locked previously.
+* KSchedulerInterruptTask is now a KInterruptHandler, rather than a KInterruptTask (no ::DoTask() implementation, any more).
+* KProcess::AddSharedMemory now sets all fields in one block/scope, rather than two.
+* KWorkerTaskManager::Initialize now hardcodes id=0 instead of taking it as a parameter.
+** KWorkerTaskManager no longer has "id"/"active" fields.
+* Certain unnecessary conditionals were optimized in KServerSession::SendReply/ReceiveMessage.
+** Code of the form `x = condition ? a : b` with condition's value fixed for the block was optimized to `x = a`.
+** This may just be a relic of compiler upgrade mentioned above.
+* Minor changes to GetInfo:
+** Logic which previously disabled interrupt for profiling InfoTypes now instead disables dispatch (KScopedDisableDispatch instead of KScopedDisableInterrupt).
+** InfoType_ThreadTickCount has had its value changed from 0xF0000002 to 0x19. This is presumably to generate better asm for the switch statement.
+* SendAsyncRequestWithUserBuffer now accepts ResultThreadTerminating as a success result (it does not unlock memory when it is returned).
+* KClientSession::SendSyncRequest/SendAsyncRequest no longer hold the scheduler lock while calling KServerSession::OnRequest.
+** Lock is now acquired as needed by KServerSession::OnRequest itself.
+* KClientSession, KLightClientSession, KLightServerSession no longer inherit from KSlabAllocated/KAutoObjectWithList, and are now correspondingly smaller.
+* KResourceLimit::Allocate() now calls KAutoObject::Create().
+* KPrintf has been replaced with a function which takes a non-format string, and prints the string, then <current program ID formatted as %016lx>, then a newline.
+** "Break() called. %016lx\n" and "Exception occurred. %016lx\n" are now "Break() called. " and "Exception occurred. ", respectively.
+* StoreDataCacheSharedForInit has slightly different (but equivalent) iteration logic.
 ===[[Bluetooth_Driver_services|bluetooth]]===