Add archer import and allow date-mode bulk delete

archer import replays a sealed archive into an existing table: shard
selection (`--select`), SHA-256 pre-verification, idempotent modes
(`ignore`/`insert`/`replace`), and both LOAD DATA and batched-INSERT
engines. It reads manifest.jsonl strictly so corruption is reported
instead of skipped, and supports --dry-run, --allow-partial, and
per-shard workers.

--bulk-delete no longer rejects --shard-by date. Range deletion now
derives its bounds from the chunk's actual keys (date scanners rewrite
LastKey into a resumption cursor) and keeps safe-auto-increment
semantics by excluding the current max key. A startup warning documents
the monotonic-shard-column assumption.

Also splits CLI/format/sharding/progress/safety reference docs out of
the README, and adds the corresponding OpenSpec change artifacts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
voidandClaude Sonnet 4.6 committed 2026-11-08 21:14:23 +08:00
1 parent bc4a6b504f
commit 366d61ca25
49 files changed
+5099 -342

No files matched your search

+50
View File
@@ -0,0 +1,50 @@
# 输出格式与导入
## 输出格式
| `--format` | 说明 |
|---|---|
| `dump`(默认) | MySQL `LOAD DATA` 兼容的 TSV:TAB 分隔、`\N` 表示 NULL、MySQL 转义表、无表头 |
| `csv` | RFC4180:逗号分隔、表头、双引号转义、CRLF;`--csv-null` 覆盖 NULL 标记 |
| `jsonl` | 每行一个 JSON 对象;DECIMAL 为字符串、BLOB 为 base64、时间列为 RFC3339、NULL 为 `null` |
## 压缩
`--compress none|gzip|zstd|xz`,默认 `none`。每个分片独立压缩,扩展名分别为 `.gz` / `.zst` / `.xz`。压缩文件可单独解压后导入。
`--compress-level 1..9`,默认 `3`。1–9 是跨压缩器的统一档位:数值越大压缩率越高、吞吐越低,`9` 表示该压缩器的最强档位。各格式的映射:
| 档位 | gzip | zstd | xz 字典 |
|---|---|---|---|
| 1 | 1 | fastest | 1 MiB |
| 2 | 2 | fastest | 2 MiB |
| 3(默认) | 3 | default | 4 MiB |
| 4 | 4 | default | 4 MiB |
| 5 | 5 | default | 8 MiB |
| 6 | 6 | better | 8 MiB |
| 7 | 7 | better | 16 MiB |
| 8 | 8 | better | 32 MiB |
| 9 | 9 | best | 64 MiB |
默认档位与引入该参数前的版本保持一致,存量任务可直接续跑;非默认档位会计入配置指纹,续跑时不一致会被拒绝。`--compress none` 不能搭配非默认档位。
xz 由纯 Go 库实现,产物是标准 xz 流,`xz -dc` 可直接解码。其压缩吞吐明显低于 gzip/zstd,适合长期归档、写入速度不敏感的场景;zstd 则在吞吐与压缩率之间更均衡。
## 导入
```bash
# dump -> MySQL(列顺序一致时)
zcat 000001.1-1000000.dump.gz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
# xz -> MySQL
xz -dc 000001.1-1000000.dump.xz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
# csv -> PostgreSQL
gunzip -c 000001.1-1000.csv.gz | psql -c "\copy orders FROM STDIN WITH (FORMAT csv, HEADER true)"
# csv -> ClickHouse
clickhouse-client --query "INSERT INTO orders FORMAT CSVWithNames" < 000001.csv
# jsonl -> ClickHouse
clickhouse-client --query "INSERT INTO orders FORMAT JSONEachRow" < 000001.jsonl
```