Rewrite chunk system (#8177)

Patch documentation to come

Issues with the old system that are fixed now:
- World generation does not scale with cpu cores effectively.
- Relies on the main thread for scheduling and maintaining chunk state, dropping chunk load/generate rates at lower tps.
- Unreliable prioritisation of chunk gen/load calls that block the main thread.
- Shutdown logic is utterly unreliable, as it has to wait for all chunks to unload - is it guaranteed that the chunk system is in a state on shutdown that it can reliably do this? Watchdog shutdown also typically failed due to thread checks, which is now resolved.
- Saving of data is not unified (i.e can save chunk data without saving entity data, poses problems for desync if shutdown is really abnormal.
- Entities are not loaded with chunks. This caused quite a bit of headache for Chunk#getEntities API, but now the new chunk system loads entities with chunks so that they are ready whenever the chunk loads in. Effectively brings the behavior back to 1.16 era, but still storing entities in their own separate regionfiles.

The above list is not complete. The patch documentation will complete it.

New chunk system hard relies on starlight and dataconverter, and most importantly the new concurrent utilities in ConcurrentUtil.

Some of the old async chunk i/o interface (i.e the old file io thread reroutes _some_ calls to the new file io thread) is kept for plugin compat reasons. It will be removed in the next major version of minecraft.

The old legacy chunk system patches have been moved to the removed folder in case we need them again.

1 files changed, 73 insertions, 0 deletions

diff --git a/patches/removed/1.19.2-legacy-chunksystem/0397-Fix-Chunk-Post-Processing-deadlock-risk.patch b/patches/removed/1.19.2-legacy-chunksystem/0397-Fix-Chunk-Post-Processing-deadlock-risk.patch
new file mode 100644
index 0000000000..13219b5e56
--- /dev/null
+++ b/patches/removed/1.19.2-legacy-chunksystem/0397-Fix-Chunk-Post-Processing-deadlock-risk.patch
@@ -0,0 +1,73 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Aikar <[email protected]>
+Date: Sat, 18 Apr 2020 04:36:11 -0400
+Subject: [PATCH] Fix Chunk Post Processing deadlock risk
+
+See: https://gist.github.com/aikar/dd22bbd2a3d78a2fd3d92e95e9f28dc6
+
+as part of post processing a chunk, we can call ChunkConverter.
+
+ChunkConverter then kicks off major physics updates, and when blocks
+that have connections across chunk boundaries occur, a recursive risk
+can occur where A updates a block that triggers a physics request.
+
+That physics request may trigger a chunk request, that then enqueues
+a task into the Mailbox ChunkTaskQueueSorter.
+
+If anything requests that same chunk that is in the middle of conversion,
+it's mailbox queue is going to be held up, so the subsequent chunk request
+will be unable to proceed.
+
+We delay post processing of Chunk.A() 1 "pass" by re stuffing it back into
+the executor so that the mailbox ChunkQueue is now considered empty.
+
+This successfully fixed a reoccurring and highly reproducible crash
+for heightmaps.
+
+diff --git a/src/main/java/net/minecraft/server/level/ChunkMap.java b/src/main/java/net/minecraft/server/level/ChunkMap.java
+index fcc9dd6e1c54e4ca16102150aa4c12ecc7de06df..aed3da6ef2d498d3f1c9c64177bf1ba6b8157493 100644
+--- a/src/main/java/net/minecraft/server/level/ChunkMap.java
++++ b/src/main/java/net/minecraft/server/level/ChunkMap.java
+@@ -193,6 +193,7 @@ public class ChunkMap extends ChunkStorage implements ChunkHolder.PlayerProvider
+     };
+     // CraftBukkit end
+ 
++    final CallbackExecutor chunkLoadConversionCallbackExecutor = new CallbackExecutor(); // Paper
+     // Paper start - distance maps
+     private final com.destroystokyo.paper.util.misc.PooledLinkedHashSets<ServerPlayer> pooledLinkedPlayerHashSets = new com.destroystokyo.paper.util.misc.PooledLinkedHashSets<>();
+ 
+@@ -1132,16 +1133,15 @@ public class ChunkMap extends ChunkStorage implements ChunkHolder.PlayerProvider
+         });
+         CompletableFuture<Either<LevelChunk, ChunkHolder.ChunkLoadingFailure>> completablefuture1 = completablefuture.thenApplyAsync((either) -> {
+             return either.mapLeft((list) -> {
+-                return (LevelChunk) list.get(list.size() / 2);
+-            });
+-        }, (runnable) -> {
+-            this.mainThreadMailbox.tell(ChunkTaskPriorityQueueSorter.message(holder, runnable));
+-        }).thenApplyAsync((either) -> {
+-            return either.ifLeft((chunk) -> {
++                // Paper start - revert 1.18.2 diff
++                final LevelChunk chunk = (LevelChunk) list.get(list.size() / 2);
+                 chunk.postProcessGeneration();
+                 this.level.startTickingChunk(chunk);
++                return chunk;
+             });
+-        }, this.mainThreadExecutor);
++        }, (runnable) -> {
++            this.mainThreadMailbox.tell(ChunkTaskPriorityQueueSorter.message(holder, () -> ChunkMap.this.chunkLoadConversionCallbackExecutor.execute(runnable))); // Paper - delay running Chunk post processing until outside of the sorter to prevent a deadlock scenario when post processing causes another chunk request.
++        }); // Paper end - revert 1.18.2 diff
+ 
+         completablefuture1.thenAcceptAsync((either) -> {
+             either.ifLeft((chunk) -> {
+diff --git a/src/main/java/net/minecraft/server/level/ServerChunkCache.java b/src/main/java/net/minecraft/server/level/ServerChunkCache.java
+index 471cc00c677b6581ba84c8cac25d2246c2a14bc9..497827822a64eeff2a4901f0e7c62f0f2c359b48 100644
+--- a/src/main/java/net/minecraft/server/level/ServerChunkCache.java
++++ b/src/main/java/net/minecraft/server/level/ServerChunkCache.java
+@@ -994,6 +994,7 @@ public class ServerChunkCache extends ChunkSource {
+                 return super.pollTask() || execChunkTask; // Paper
+             }
+         } finally {
++            chunkMap.chunkLoadConversionCallbackExecutor.run(); // Paper - Add chunk load conversion callback executor to prevent deadlock due to recursion in the chunk task queue sorter
+             chunkMap.callbackExecutor.run();
+         }
+         // CraftBukkit end