Improve "Add raw copy dependencies for incompatible textures"

Initial PR was made by Avan for Ryubing.

This allows for Trails in the Sky 1st Chapter (and by extension Trails through Daybreak, Trails through Daybreak II, Trails Beyond the Horizon, and the future Trails in the Sky 2nd Chapter – tested via demo – to be fully playable.

Initial PR caused issues on macOS devices, specifically crashing in certain games (such as Mario Kart 8 Deluxe and the aforementioned Trails games). It was reverted on the original Ryubing project (at the time of writing this description) for causing small rendering issues on The Legend of Zelda: Breath of the Wild.

The rendering issues do not appear on macOS.

This commit improves the original PR by eliminating said macOS crash.

Trails in the Sky 1st writes its exposure/brightness value through an R32G32Float 1x1 texture and later reads the raw bits back through an R32Uint 2x1 texture mapped to the same guest memory. The two texture formats/dimensions are fully incompatible as texture views, so they end
up as separate host textures, and nothing kept their contents coherent - the reader saw stale zeroed data, making the exposure calculation (and the rendered image) too dark. (This is the initial fix for the game)

TextureGroup already had a raw-copy dependency mechanism intended to
handle this class of alias (TextureDependency, TextureGroupHandle
raw-copy plumbing), gated by CanCreateRawCopyDependency's strict same-guest-memory checks. This commit fixes three bugs that kept that mechanism from working correctly (and thus crashing Trails and other games, such as Mario Kart 8 Deluxe):

(1) TextureGroup.InitializeOverlaps() and TextureGroup.RegisterIncompatibleOverlap() only forwarded overlaps to CreateCopyDependency() when compatibility was LayoutIncompatible or better, silently excluding the fully Incompatible case the raw-copy path exists for. Widened both guards to let Incompatible overlaps through so CanCreateRawCopyDependency actually gets a chance to run.

(2) Once Incompatible overlaps were allowed through, CreateCopyDependency(TextureGroup, ...) could still fall back to a regular, non-raw textureCopy for such pairs, since ViewLayoutCompatible/CopySizeMatches only check byte size and were never meant to reason about fully incompatible pairs (e.g. a depth format aliasing a color format with the same byte size). On Vulkan running through MoltenVK on macOS, that non-raw copy path requires a pixel-format-reinterpreting texture view, and MoltenVK/Metal refuses to create any view onto a depth-format texture ("not castable"), aborting emulation. textureCopy is now forced false for Incompatible pairs, leaving raw copy (which already excludes depth/stencil formats) as the only route for that severity level.

(3) TextureGroupHandle.Inherit() copied a handle's pending DeferredCopy to the new handle when a view was recreated, but did not copy DeferredCopyRaw alongside it. A handle that inherited a pending raw copy would then execute it through the regular (non-raw) CopyTo path once acknowledged, reinterpreting the source bytes with the wrong row layout and corrupting the image - visible as vertical flickering stripes whenever a view happened to be recreated with a raw copy still pending. DeferredCopyRaw is now carried over together with DeferredCopy.

(As mentioned earlier, vertical stripes aren't present on macOS, so (3) doesn't negatively have an effect on macOS in any way).

src/Ryujinx.Graphics.Gpu/Image/TextureGroup.cs

@@ -182,7 +182,11 @@ namespace Ryujinx.Graphics.Gpu.Image
         {
             foreach (TextureIncompatibleOverlap overlap in _incompatibleOverlaps)
             {
-                if (overlap.Compatibility <= TextureViewCompatibility.LayoutIncompatible)
+                // LayoutIncompatible and better may still use a regular texture copy dependency.
+                // Fully Incompatible pairs are not copy compatible in general, but may still qualify for an
+                // exact raw byte copy dependency (checked internally by CreateCopyDependency) when they map
+                // to exactly the same guest memory, such as differently typed/sized aliases of the same data.
+                if (overlap.Compatibility <= TextureViewCompatibility.Incompatible)
                 {
                     CreateCopyDependency(overlap.Group, false, overlap.Compatibility);
                 }
@@ -226,7 +230,6 @@ namespace Ryujinx.Graphics.Gpu.Image
             }
         }
 
-
         /// <summary>
         /// Flushes incompatible overlaps if the storage format requires it, and they have been modified.
         /// This allows unsupported host formats to accept data written to format aliased textures.
@@ -659,7 +662,7 @@ namespace Ryujinx.Graphics.Gpu.Image
 
                 bool canImport = Storage.Info.IsLinear && Storage.Info.Stride >= Storage.Info.Width * Storage.Info.FormatInfo.BytesPerPixel;
 
-                IntPtr hostPointer = canImport ? _physicalMemory.GetHostPointer(Storage.Range) : 0;
+                nint hostPointer = canImport ? _physicalMemory.GetHostPointer(Storage.Range) : 0;
 
                 if (hostPointer != 0 && _context.Renderer.PrepareHostMapping(hostPointer, Storage.Size))
                 {
@@ -1049,7 +1052,7 @@ namespace Ryujinx.Graphics.Gpu.Image
             int endOffset = _allOffsets[viewEnd] + _sliceSizes[lastLevel];
             int size = endOffset - offset;
 
-            List<RegionHandle> result = new();
+            List<RegionHandle> result = [];
 
             for (int i = 0; i < TextureRange.Count; i++)
             {
@@ -1163,7 +1166,6 @@ namespace Ryujinx.Graphics.Gpu.Image
             SignalAllDirty();
         }
 
-
         /// <summary>
         /// Removes a view from the group, removing it from all overlap lists.
         /// </summary>
@@ -1385,7 +1387,7 @@ namespace Ryujinx.Graphics.Gpu.Image
 
                 if (_is3D)
                 {
-                    List<TextureGroupHandle> handlesList = new();
+                    List<TextureGroupHandle> handlesList = [];
 
                     for (int i = 0; i < levelHandles; i++)
                     {
@@ -1468,8 +1470,8 @@ namespace Ryujinx.Graphics.Gpu.Image
             // Get the location of each texture within its storage, so we can find the handles to apply the dependency to.
             // This can consist of multiple disjoint regions, for example if this is a mip slice of an array texture.
 
-            List<(int BaseHandle, int RegionCount)> targetRange = new();
-            List<(int BaseHandle, int RegionCount)> otherRange = new();
+            List<(int BaseHandle, int RegionCount)> targetRange = [];
+            List<(int BaseHandle, int RegionCount)> otherRange = [];
 
             EvaluateRelevantHandles(firstLayer, firstLevel, other.Info.GetSlices(), other.Info.Levels, (baseHandle, regionCount, _, _) =>
             {
@@ -1601,7 +1603,15 @@ namespace Ryujinx.Graphics.Gpu.Image
                         TextureInfo info = Storage.Info;
                         TextureInfo otherInfo = other.Storage.Info;
 
-                        bool textureCopy = TextureCompatibility.ViewLayoutCompatible(info, otherInfo, level, otherLevel) &&
+                        // ViewLayoutCompatible/CopySizeMatches only reason about textures with some genuine
+                        // format relationship (LayoutIncompatible or better) - they are not aware of, and must
+                        // never be used to justify, a plain texture-to-texture copy (which some backends
+                        // implement via a reinterpreting view) between fully Incompatible aliases such as a
+                        // depth format and an unrelated color format. For Incompatible pairs, only the strict
+                        // raw byte copy dependency below (which explicitly excludes depth/stencil formats) may
+                        // be used.
+                        bool textureCopy = compatibility != TextureViewCompatibility.Incompatible &&
+                            TextureCompatibility.ViewLayoutCompatible(info, otherInfo, level, otherLevel) &&
                             TextureCompatibility.CopySizeMatches(info, otherInfo, level, otherLevel);
 
                         if (textureCopy || rawCopy)
@@ -1659,9 +1669,12 @@ namespace Ryujinx.Graphics.Gpu.Image
         {
             if (!_incompatibleOverlaps.Any(overlap => overlap.Group == other.Group))
             {
-                if (copy && other.Compatibility <= TextureViewCompatibility.LayoutIncompatible)
+                if (copy && other.Compatibility <= TextureViewCompatibility.Incompatible)
                 {
                     // Any of the group's views may share compatibility, even if the parents do not fully.
+                    // Fully Incompatible groups are also let through here, since CreateCopyDependency will
+                    // fall back to an exact raw byte copy dependency for them when the strict requirements
+                    // for one are met (see CanCreateRawCopyDependency).
                     CreateCopyDependency(other.Group, false, other.Compatibility);
                 }
 
@@ -1803,4 +1816,3 @@ namespace Ryujinx.Graphics.Gpu.Image
         }
     }
 }
-

src/Ryujinx.Graphics.Gpu/Image/TextureGroupHandle.cs

@@ -717,6 +717,7 @@ namespace Ryujinx.Graphics.Gpu.Image
                 }
 
                 DeferredCopy = old.DeferredCopy;
+                DeferredCopyRaw = old.DeferredCopyRaw;
             }
         }