Files
archer/docs/checkpoint-and-resume.md
voidandClaude Sonnet 4.6 366d61ca25 Add archer import and allow date-mode bulk delete
archer import replays a sealed archive into an existing table: shard
selection (`--select`), SHA-256 pre-verification, idempotent modes
(`ignore`/`insert`/`replace`), and both LOAD DATA and batched-INSERT
engines. It reads manifest.jsonl strictly so corruption is reported
instead of skipped, and supports --dry-run, --allow-partial, and
per-shard workers.

--bulk-delete no longer rejects --shard-by date. Range deletion now
derives its bounds from the chunk's actual keys (date scanners rewrite
LastKey into a resumption cursor) and keeps safe-auto-increment
semantics by excluding the current max key. A startup warning documents
the monotonic-shard-column assumption.

Also splits CLI/format/sharding/progress/safety reference docs out of
the README, and adds the corresponding OpenSpec change artifacts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-11-08 21:14:23 +08:00

1.4 KiB
Raw Permalink Blame History

清单、断点与一致性

清单与断点

每个分片关闭后向 manifest.jsonl 追加一行并 fsync:

{"seq":1,"path":"000001.1-1000000.dump","pk_lo":"1","pk_hi":"1000000","rows":1000000,"bytes":12345678,"sha256":"...","closed_at":"2026-11-06T12:00:00Z"}

state.json 在每个 chunk 提交后原子更新(tmp + fsync + rename),记录:

  • config_hash 与完整的配置指纹;
  • 扫描游标(主键、日期键或当前日期);
  • 当前日期桶与分片序号;
  • 已处理行数。

续跑时使用相同输出目录与配置即可。指纹不一致会拒绝,除非显式 --force-resume;不一致的字段会列在错误信息里。密码不参与指纹,允许凭据轮换。

已封分片的文件如果缺少 manifest 记录(例如在 rename 与 manifest 追加之间崩溃), 续跑会自动识别并补录,不会被覆盖。带 --purge 时还会保留活动分片的 .partial 与 sidecar 元数据,续跑时截断到一致偏移并追加,避免已删除数据丢失。

一致性

at-least-once。文件写入与数据库提交不在同一个原子域内,崩溃后重跑可能重复写入 已处理的行,但不会丢失已提交的数据。导入端可用 INSERT IGNORE / 去重逻辑处理。

扫描期间不提供快照一致性:只保证读取到的行,并发写入可能改变行集合。