AMDMTLBronzeDriver.bundle, the Metal driver. It took a week of reverse engineering the AMD drivers and reading the open source AMD implementation in Linux and Mesa3D to figure the issue out.
src/amdand for Metal, it is in
AMDMTLBronzeDriver.bundle(which is a userland driver). We can do a search for "polaris 22" and "vega m" in the codebase and find a few references in Mesa.
AMDMTLBronzeDriver.bundle! But it turns out to be false hope because even though addrlib is included in Apple's drivers (multiple times actually; it's found in the OpenGL drivers and the graphics accelerator kext as well), it seems to be vestigial code that's never called. It's likely that addrlib is used previously at some point but was later parts were replaced by a closed-source implementation.
amdBronzeMtlAddrLibGetBaseArrayModetakes as parameters: some blob of pre-computed data, a texture description, and a pointer of hardware configuration (which turns out to be unused). It returns a
AddrTileModethat is passed to some addrlib functions. For the sample code, the function returns
ADDR_TM_2D_TILED_THIN1but if we change it to
ADDR_TM_1D_TILED_THIN1then the textured square renders correctly! This likely is somehow related to what we found before about the pipe configuration being different in Vega M. Somehow, the pre-computed blob doesn't take account of the Vega M's configuration and
amdBronzeMtlAddrLibGetBaseArrayModeproduces the wrong result. There is likely a deeper reason, but a patch to always return
ADDR_TM_1D_TILED_THIN1when it would have returned
ADDR_TM_2D_TILED_THIN1seems to fix all observable issues.