Add archer import and allow date-mode bulk delete
archer import replays a sealed archive into an existing table: shard selection (`--select`), SHA-256 pre-verification, idempotent modes (`ignore`/`insert`/`replace`), and both LOAD DATA and batched-INSERT engines. It reads manifest.jsonl strictly so corruption is reported instead of skipped, and supports --dry-run, --allow-partial, and per-shard workers. --bulk-delete no longer rejects --shard-by date. Range deletion now derives its bounds from the chunk's actual keys (date scanners rewrite LastKey into a resumption cursor) and keeps safe-auto-increment semantics by excluding the current max key. A startup warning documents the monotonic-shard-column assumption. Also splits CLI/format/sharding/progress/safety reference docs out of the README, and adds the corresponding OpenSpec change artifacts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc4a6b504f
commit
366d61ca25
49 files changed
+5099
-342
No files matched your search
@@ -0,0 +1,50 @@
|
||||
# 输出格式与导入
|
||||
|
||||
## 输出格式
|
||||
|
||||
| `--format` | 说明 |
|
||||
|---|---|
|
||||
| `dump`(默认) | MySQL `LOAD DATA` 兼容的 TSV:TAB 分隔、`\N` 表示 NULL、MySQL 转义表、无表头 |
|
||||
| `csv` | RFC4180:逗号分隔、表头、双引号转义、CRLF;`--csv-null` 覆盖 NULL 标记 |
|
||||
| `jsonl` | 每行一个 JSON 对象;DECIMAL 为字符串、BLOB 为 base64、时间列为 RFC3339、NULL 为 `null` |
|
||||
|
||||
## 压缩
|
||||
|
||||
`--compress none|gzip|zstd|xz`,默认 `none`。每个分片独立压缩,扩展名分别为 `.gz` / `.zst` / `.xz`。压缩文件可单独解压后导入。
|
||||
|
||||
`--compress-level 1..9`,默认 `3`。1–9 是跨压缩器的统一档位:数值越大压缩率越高、吞吐越低,`9` 表示该压缩器的最强档位。各格式的映射:
|
||||
|
||||
| 档位 | gzip | zstd | xz 字典 |
|
||||
|---|---|---|---|
|
||||
| 1 | 1 | fastest | 1 MiB |
|
||||
| 2 | 2 | fastest | 2 MiB |
|
||||
| 3(默认) | 3 | default | 4 MiB |
|
||||
| 4 | 4 | default | 4 MiB |
|
||||
| 5 | 5 | default | 8 MiB |
|
||||
| 6 | 6 | better | 8 MiB |
|
||||
| 7 | 7 | better | 16 MiB |
|
||||
| 8 | 8 | better | 32 MiB |
|
||||
| 9 | 9 | best | 64 MiB |
|
||||
|
||||
默认档位与引入该参数前的版本保持一致,存量任务可直接续跑;非默认档位会计入配置指纹,续跑时不一致会被拒绝。`--compress none` 不能搭配非默认档位。
|
||||
|
||||
xz 由纯 Go 库实现,产物是标准 xz 流,`xz -dc` 可直接解码。其压缩吞吐明显低于 gzip/zstd,适合长期归档、写入速度不敏感的场景;zstd 则在吞吐与压缩率之间更均衡。
|
||||
|
||||
## 导入
|
||||
|
||||
```bash
|
||||
# dump -> MySQL(列顺序一致时)
|
||||
zcat 000001.1-1000000.dump.gz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
|
||||
|
||||
# xz -> MySQL
|
||||
xz -dc 000001.1-1000000.dump.xz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
|
||||
|
||||
# csv -> PostgreSQL
|
||||
gunzip -c 000001.1-1000.csv.gz | psql -c "\copy orders FROM STDIN WITH (FORMAT csv, HEADER true)"
|
||||
|
||||
# csv -> ClickHouse
|
||||
clickhouse-client --query "INSERT INTO orders FORMAT CSVWithNames" < 000001.csv
|
||||
|
||||
# jsonl -> ClickHouse
|
||||
clickhouse-client --query "INSERT INTO orders FORMAT JSONEachRow" < 000001.jsonl
|
||||
```
|
||||
Reference in new issue
Block a user