Files
archer/docs/checkpoint-and-resume.md
T
voidandClaude Sonnet 4.6 366d61ca25 Add archer import and allow date-mode bulk delete
archer import replays a sealed archive into an existing table: shard
selection (`--select`), SHA-256 pre-verification, idempotent modes
(`ignore`/`insert`/`replace`), and both LOAD DATA and batched-INSERT
engines. It reads manifest.jsonl strictly so corruption is reported
instead of skipped, and supports --dry-run, --allow-partial, and
per-shard workers.

--bulk-delete no longer rejects --shard-by date. Range deletion now
derives its bounds from the chunk's actual keys (date scanners rewrite
LastKey into a resumption cursor) and keeps safe-auto-increment
semantics by excluding the current max key. A startup warning documents
the monotonic-shard-column assumption.

Also splits CLI/format/sharding/progress/safety reference docs out of
the README, and adds the corresponding OpenSpec change artifacts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-11-08 21:14:23 +08:00

31 lines
1.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 清单、断点与一致性
## 清单与断点
每个分片关闭后向 `manifest.jsonl` 追加一行并 fsync:
```json
{"seq":1,"path":"000001.1-1000000.dump","pk_lo":"1","pk_hi":"1000000","rows":1000000,"bytes":12345678,"sha256":"...","closed_at":"2026-11-06T12:00:00Z"}
```
`state.json` 在每个 chunk 提交后原子更新(tmp + fsync + rename),记录:
- `config_hash` 与完整的配置指纹;
- 扫描游标(主键、日期键或当前日期);
- 当前日期桶与分片序号;
- 已处理行数。
续跑时使用相同输出目录与配置即可。指纹不一致会拒绝,除非显式
`--force-resume`;不一致的字段会列在错误信息里。密码不参与指纹,允许凭据轮换。
已封分片的文件如果缺少 manifest 记录(例如在 rename 与 manifest 追加之间崩溃),
续跑会自动识别并补录,不会被覆盖。带 `--purge` 时还会保留活动分片的
`.partial` 与 sidecar 元数据,续跑时截断到一致偏移并追加,避免已删除数据丢失。
## 一致性
**at-least-once**。文件写入与数据库提交不在同一个原子域内,崩溃后重跑可能重复写入
已处理的行,但不会丢失已提交的数据。导入端可用 `INSERT IGNORE` / 去重逻辑处理。
扫描期间不提供快照一致性:只保证读取到的行,并发写入可能改变行集合。