Add archer import and allow date-mode bulk delete
archer import replays a sealed archive into an existing table: shard selection (`--select`), SHA-256 pre-verification, idempotent modes (`ignore`/`insert`/`replace`), and both LOAD DATA and batched-INSERT engines. It reads manifest.jsonl strictly so corruption is reported instead of skipped, and supports --dry-run, --allow-partial, and per-shard workers. --bulk-delete no longer rejects --shard-by date. Range deletion now derives its bounds from the chunk's actual keys (date scanners rewrite LastKey into a resumption cursor) and keeps safe-auto-increment semantics by excluding the current max key. A startup warning documents the monotonic-shard-column assumption. Also splits CLI/format/sharding/progress/safety reference docs out of the README, and adds the corresponding OpenSpec change artifacts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc4a6b504f
commit
366d61ca25
49 files changed
+5099
-342
No files matched your search
@@ -0,0 +1,30 @@
|
||||
# 清单、断点与一致性
|
||||
|
||||
## 清单与断点
|
||||
|
||||
每个分片关闭后向 `manifest.jsonl` 追加一行并 fsync:
|
||||
|
||||
```json
|
||||
{"seq":1,"path":"000001.1-1000000.dump","pk_lo":"1","pk_hi":"1000000","rows":1000000,"bytes":12345678,"sha256":"...","closed_at":"2026-11-06T12:00:00Z"}
|
||||
```
|
||||
|
||||
`state.json` 在每个 chunk 提交后原子更新(tmp + fsync + rename),记录:
|
||||
|
||||
- `config_hash` 与完整的配置指纹;
|
||||
- 扫描游标(主键、日期键或当前日期);
|
||||
- 当前日期桶与分片序号;
|
||||
- 已处理行数。
|
||||
|
||||
续跑时使用相同输出目录与配置即可。指纹不一致会拒绝,除非显式
|
||||
`--force-resume`;不一致的字段会列在错误信息里。密码不参与指纹,允许凭据轮换。
|
||||
|
||||
已封分片的文件如果缺少 manifest 记录(例如在 rename 与 manifest 追加之间崩溃),
|
||||
续跑会自动识别并补录,不会被覆盖。带 `--purge` 时还会保留活动分片的
|
||||
`.partial` 与 sidecar 元数据,续跑时截断到一致偏移并追加,避免已删除数据丢失。
|
||||
|
||||
## 一致性
|
||||
|
||||
**at-least-once**。文件写入与数据库提交不在同一个原子域内,崩溃后重跑可能重复写入
|
||||
已处理的行,但不会丢失已提交的数据。导入端可用 `INSERT IGNORE` / 去重逻辑处理。
|
||||
|
||||
扫描期间不提供快照一致性:只保证读取到的行,并发写入可能改变行集合。
|
||||
Reference in new issue
Block a user