Gate .sentinel removal on graceful shutdown behind env flag

Default behavior unchanged: .sentinel is removed on graceful shutdown
(preserves existing semantics for the 99% of deployments that don't
need the kubectl-delete-pod recovery path).

With LIBSQL_PRESERVE_SENTINEL_ON_SHUTDOWN=1 set, the sentinel survives
graceful shutdown. This re-enables the documented operator recovery
procedure:

  1. kubectl exec <pod> -- touch /data/dbs/<ns>/.sentinel
  2. kubectl delete pod <pod>      # SIGTERM → graceful shutdown
  3. Kubernetes recreates pod
  4. Next namespace access triggers dirty-recovery on the preserved
     .sentinel, rebuilding wallog/snapshots from the live data file

Without this flag, step 2's graceful shutdown removes the sentinel
BEFORE the pod stops, so step 4 doesn't find a sentinel and skips
the dirty-recovery path.

Now that POST /v1/namespaces/:ns/reset-replication is the primary
recovery primitive, this flag is a low-priority belt-and-suspenders
for emergency ops workflows (e.g. when the admin API is unavailable).

Verified end-to-end with /tmp/run_sentinel_preserve_simple.sh: sentinel
preserved with flag, dirty-recovery fires on next access, data
preserved through the cycle.

Author: sle-c

libsql-server/src/namespace/mod.rs

@@ -139,8 +139,31 @@ impl Namespace {
             self.checkpoint().await?;
         }
         self.db.shutdown().await?;
-        if let Err(e) = tokio::fs::remove_file(self.path.join(".sentinel")).await {
-            tracing::error!("unable to remove .sentinel file: {}", e);
+        // Historically `.sentinel` was removed unconditionally on graceful
+        // shutdown. This makes the documented `touch .sentinel + kubectl
+        // delete pod` operator recovery path silently ineffective, because
+        // kubectl sends SIGTERM first which invokes this graceful shutdown
+        // and removes the sentinel before the pod actually stops.
+        //
+        // Guard the removal behind `LIBSQL_PRESERVE_SENTINEL_ON_SHUTDOWN`.
+        // When set, the sentinel survives graceful shutdown, so the next
+        // namespace init will correctly trigger dirty-recovery from the
+        // live `data` file.
+        //
+        // Default remains: remove (preserves existing behavior for the
+        // 99% of deployments that don't need this recovery path, now that
+        // `POST /v1/namespaces/:ns/reset-replication` is the primary
+        // recovery primitive).
+        let preserve_sentinel =
+            std::env::var("LIBSQL_PRESERVE_SENTINEL_ON_SHUTDOWN").is_ok();
+        if !preserve_sentinel {
+            if let Err(e) = tokio::fs::remove_file(self.path.join(".sentinel")).await {
+                tracing::error!("unable to remove .sentinel file: {}", e);
+            }
+        } else {
+            tracing::info!(
+                "LIBSQL_PRESERVE_SENTINEL_ON_SHUTDOWN set; keeping .sentinel for recovery"
+            );
         }
         Ok(())
     }