Initial commit: archer, a Go-native MySQL/MariaDB table archiver
Renamed from arcdb: module path, cmd/archer, internal/archererr, docs, build scripts and ARCHER_* environment variables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
commit
bc4a6b504f
99 files changed
+12752
No files matched your search
@@ -0,0 +1,185 @@
|
||||
---
|
||||
name: openspec-apply-change
|
||||
description: Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Implement tasks from an OpenSpec change.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/openspec-apply-change <other>`).
|
||||
|
||||
2. **Check status to understand the schema**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used (e.g., "spec-driven")
|
||||
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
|
||||
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
|
||||
|
||||
3. **Get apply instructions**
|
||||
|
||||
```bash
|
||||
openspec instructions apply --change "<name>" --json
|
||||
```
|
||||
|
||||
This returns:
|
||||
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
|
||||
- Progress (total, complete, remaining)
|
||||
- Task list with status
|
||||
- Dynamic instruction based on current state
|
||||
- Optional `context`: current required project instruction input from the selected root
|
||||
- Optional `operationGuidance`: current advisory guidance for apply
|
||||
|
||||
**Handle states:**
|
||||
- If `state: "blocked"` (missing artifacts): show message, suggest using openspec-continue-change (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
|
||||
- If `state: "all_done"`: congratulate, suggest archive
|
||||
- Otherwise: proceed to implementation
|
||||
|
||||
Treat `context` as a required prompt-level input. Read and consider it, and
|
||||
apply relevant project facts, conventions, and constraints while implementing.
|
||||
Treat `operationGuidance` as optional additive advice. Read and consider every
|
||||
entry, and follow entries that are applicable and compatible with the built-in
|
||||
workflow.
|
||||
|
||||
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
|
||||
progress, `contextFiles`, and the built-in `instruction`. They are not
|
||||
evidence of task completion, do not replace the built-in instruction, and do
|
||||
not permit bypassing a blocked state. If context conflicts with the built-in
|
||||
instruction, an explicit user choice, or a CLI-controlled value, report the
|
||||
conflict and preserve the controlling value. If guidance is inapplicable or
|
||||
conflicts with those controlling inputs, do not follow it and explain why.
|
||||
These are prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
4. **Read context files**
|
||||
|
||||
Read every file path listed under `contextFiles` from the apply instructions output.
|
||||
The files depend on the schema being used:
|
||||
- **spec-driven**: proposal, specs, design, tasks
|
||||
- Other schemas: follow the contextFiles from CLI output
|
||||
|
||||
Do not copy `context` or `operationGuidance` verbatim into implementation
|
||||
files or planning artifacts unless the user separately asks for that content.
|
||||
|
||||
5. **Show current progress**
|
||||
|
||||
Display:
|
||||
- Schema being used
|
||||
- Progress: "N/M tasks complete"
|
||||
- Remaining tasks overview
|
||||
- Dynamic instruction from CLI
|
||||
|
||||
6. **Implement tasks (loop until done or blocked)**
|
||||
|
||||
For each pending task:
|
||||
- Show which task is being worked on
|
||||
- Make the code changes required
|
||||
- Keep changes minimal and focused
|
||||
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
|
||||
- Continue to next task
|
||||
|
||||
**Pause if:**
|
||||
- Task is unclear → ask for clarification
|
||||
- Implementation reveals a design issue → suggest updating artifacts
|
||||
- Error or blocker encountered → report and wait for guidance
|
||||
- User interrupts
|
||||
|
||||
7. **On completion or pause, show status**
|
||||
|
||||
Display:
|
||||
- Tasks completed this session
|
||||
- Overall progress: "N/M tasks complete"
|
||||
- If all done: suggest archive
|
||||
- If paused: explain why and wait for guidance
|
||||
|
||||
**Output During Implementation**
|
||||
|
||||
```
|
||||
## Implementing: <change-name> (schema: <schema-name>)
|
||||
|
||||
Working on task 3/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
|
||||
Working on task 4/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
```
|
||||
|
||||
**Output On Completion**
|
||||
|
||||
```
|
||||
## Implementation Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 7/7 tasks complete ✓
|
||||
|
||||
### Completed This Session
|
||||
- [x] Task 1
|
||||
- [x] Task 2
|
||||
...
|
||||
|
||||
All tasks complete! Ready to archive this change.
|
||||
```
|
||||
|
||||
**Output On Pause (Issue Encountered)**
|
||||
|
||||
```
|
||||
## Implementation Paused
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 4/7 tasks complete
|
||||
|
||||
### Issue Encountered
|
||||
<description of the issue>
|
||||
|
||||
**Options:**
|
||||
1. <option 1>
|
||||
2. <option 2>
|
||||
3. Other approach
|
||||
|
||||
What would you like to do?
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Keep going through tasks until done or blocked
|
||||
- Always read context files before starting (from the apply instructions output)
|
||||
- If task is ambiguous, pause and ask before implementing
|
||||
- If implementation reveals issues, pause and suggest artifact updates
|
||||
- Keep code changes minimal and scoped to each task
|
||||
- Update task checkbox immediately after completing each task
|
||||
- Pause on errors, blockers, or unclear requirements - don't guess
|
||||
- Use contextFiles from CLI output, don't assume specific file names
|
||||
- Do not use context or operation guidance as proof that a task is complete
|
||||
- Apply relevant project context; report conflicts with controlling workflow inputs
|
||||
- Consider every guidance entry; explain any inapplicable or conflicting advice
|
||||
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
|
||||
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
|
||||
|
||||
**Fluid Workflow Integration**
|
||||
|
||||
This skill supports the "actions on a change" model:
|
||||
|
||||
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
|
||||
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
name: openspec-archive-change
|
||||
description: Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Archive a completed change in the experimental workflow.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show only active changes (not already archived).
|
||||
Include the schema used for each change if available.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/openspec-archive-change <other>`).
|
||||
|
||||
**Load current archive inputs before the existing archive checks:**
|
||||
|
||||
After resolving the selected change and planning root, run:
|
||||
```bash
|
||||
openspec instructions archive --change "<name>" --json
|
||||
```
|
||||
Keep the same selected-root flags on this command. This lookup is advisory and
|
||||
optional: it only supplies extra prompt inputs, so it must never block archiving.
|
||||
If it exits non-zero or returns invalid JSON — for example on an older CLI that
|
||||
does not support this command yet — continue the archive workflow with no
|
||||
context and no operation guidance. Do not report an error and do not stop.
|
||||
|
||||
A successful response may omit both optional fields. Treat `context` as a
|
||||
required prompt-level input: read and consider it, and apply relevant project
|
||||
facts, conventions, and constraints. Treat `operationGuidance` as optional
|
||||
additive advice: read and consider every entry, and follow entries that are
|
||||
applicable and compatible with the built-in archive workflow.
|
||||
|
||||
Keep both fields separate from built-in steps, explicit user choices, resolved
|
||||
paths, CLI checks, and command contracts. If context conflicts with one of those
|
||||
controlling inputs, report the conflict and preserve the controlling value. If
|
||||
guidance is inapplicable or conflicts with a controlling input, do not follow it
|
||||
and explain why. Do not infer replacement paths, skipped prompts, or flags from
|
||||
either field, and do not copy their text verbatim into specs, change artifacts,
|
||||
or archive summaries unless the user separately asks for it. These are
|
||||
prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
2. **Check artifact completion status**
|
||||
|
||||
Run `openspec status --change "<name>" --json` to check artifact completion.
|
||||
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
|
||||
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
|
||||
|
||||
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
|
||||
- Display warning listing incomplete artifacts
|
||||
- Ask the user to confirm they want to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
3. **Check task completion status**
|
||||
|
||||
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
|
||||
|
||||
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
|
||||
|
||||
**If incomplete tasks found:**
|
||||
- Display warning showing count of incomplete tasks
|
||||
- Ask the user to confirm they want to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
**If no tasks file exists:** Proceed without task-related warning.
|
||||
|
||||
4. **Assess delta spec sync state**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
|
||||
delta-spec source. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
|
||||
delta specs from other artifacts.
|
||||
|
||||
**If delta specs exist:**
|
||||
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
|
||||
- Determine what changes would be applied (adds, modifications, removals, renames)
|
||||
- Show a combined summary before prompting
|
||||
|
||||
**Prompt options:**
|
||||
- If changes needed: "Sync now (recommended)", "Archive without syncing"
|
||||
- If already synced: "Archive now", "Sync anyway", "Cancel"
|
||||
|
||||
Route on the answer:
|
||||
- "Cancel" — stop, do not archive
|
||||
- "Archive without syncing" or "Archive now" — proceed to archive
|
||||
- "Sync now" or "Sync anyway" — sync, then verify (below)
|
||||
- Anything else — ask again rather than archiving
|
||||
|
||||
Before a selected sync writes any main spec, run
|
||||
`openspec instructions specs --change "<name>" --json` once with the same
|
||||
selected-root flags. Require a zero exit status and valid artifact-instruction
|
||||
JSON. If the lookup fails or returns invalid JSON, report the error and stop
|
||||
before writing any main spec or moving the change. A valid response with omitted
|
||||
`rules` is the no-rules case. Apply returned `rules` only to the content and
|
||||
form of main specs produced by this merge; do not use them as archive guidance,
|
||||
change CLI behavior, or copy the rule text into any output file.
|
||||
|
||||
Then run the `openspec-sync-specs` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
|
||||
|
||||
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
|
||||
- ADDED requirements present
|
||||
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
|
||||
- REMOVED requirements gone
|
||||
- RENAMED requirements present under the new name and absent under the old one
|
||||
|
||||
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
|
||||
|
||||
5. **Perform the archive**
|
||||
|
||||
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
|
||||
```bash
|
||||
mkdir -p "<planningHome.changesDir>/archive"
|
||||
```
|
||||
|
||||
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
|
||||
|
||||
**Check if target already exists:**
|
||||
- If yes: Fail with error, suggest renaming existing archive or using different date
|
||||
- If no: Move `changeRoot` to the archive directory
|
||||
|
||||
```bash
|
||||
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
|
||||
```
|
||||
|
||||
6. **Display summary**
|
||||
|
||||
Show archive completion summary including:
|
||||
- Change name
|
||||
- Schema that was used
|
||||
- Archive location
|
||||
- Whether specs were synced (if applicable)
|
||||
- Note about any warnings (incomplete artifacts/tasks)
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Archive Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
**Specs:** <"✓ Synced to main specs" only if the step 4 verification passed; otherwise "No delta specs" or "Sync skipped">
|
||||
|
||||
<"All artifacts complete. All tasks complete." — or, if archived with warnings, list them instead (e.g. "Archived with 2 incomplete tasks")>
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Announce the selected change; prompt for selection when it is ambiguous
|
||||
- Use artifact graph (openspec status --json) for completion checking
|
||||
- Don't block archive on warnings - just inform and confirm
|
||||
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
|
||||
- Show clear summary of what happened
|
||||
- If sync is requested, run the `openspec-sync-specs` workflow inline (agent-driven)
|
||||
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
|
||||
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
|
||||
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
|
||||
- Consider every guidance entry and explain any inapplicable or conflicting advice
|
||||
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
|
||||
- Artifact rules constrain only the specs being written and are never operation guidance
|
||||
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
|
||||
@@ -0,0 +1,296 @@
|
||||
---
|
||||
name: openspec-explore
|
||||
description: Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
|
||||
|
||||
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing.
|
||||
|
||||
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
---
|
||||
|
||||
## The Stance
|
||||
|
||||
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
|
||||
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
|
||||
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
|
||||
- **Adaptive** - Follow interesting threads, pivot when new information emerges
|
||||
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
|
||||
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
|
||||
|
||||
---
|
||||
|
||||
## What You Might Do
|
||||
|
||||
Depending on what the user brings, you might:
|
||||
|
||||
**Explore the problem space**
|
||||
- Ask clarifying questions that emerge from what they said
|
||||
- Challenge assumptions
|
||||
- Reframe the problem
|
||||
- Find analogies
|
||||
|
||||
**Investigate the codebase**
|
||||
- Map existing architecture relevant to the discussion
|
||||
- Find integration points
|
||||
- Identify patterns already in use
|
||||
- Surface hidden complexity
|
||||
|
||||
**Compare options**
|
||||
- Brainstorm multiple approaches
|
||||
- Build comparison tables
|
||||
- Sketch tradeoffs
|
||||
- Recommend a path (if asked)
|
||||
|
||||
**Visualize**
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Use ASCII diagrams liberally │
|
||||
├─────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────┐ ┌────────┐ │
|
||||
│ │ State │────────▶│ State │ │
|
||||
│ │ A │ │ B │ │
|
||||
│ └────────┘ └────────┘ │
|
||||
│ │
|
||||
│ System diagrams, state machines, │
|
||||
│ data flows, architecture sketches, │
|
||||
│ dependency graphs, comparison tables │
|
||||
│ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Surface risks and unknowns**
|
||||
- Identify what could go wrong
|
||||
- Find gaps in understanding
|
||||
- Suggest spikes or investigations
|
||||
|
||||
---
|
||||
|
||||
## OpenSpec Awareness
|
||||
|
||||
You have full context of the OpenSpec system. Use it naturally, don't force it.
|
||||
|
||||
### Check for context
|
||||
|
||||
At the start, quickly check what exists:
|
||||
```bash
|
||||
openspec list --json
|
||||
```
|
||||
|
||||
This tells you:
|
||||
- If there are active changes
|
||||
- Their names, schemas, and status
|
||||
- What the user might be working on
|
||||
|
||||
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
|
||||
- `context`: project background - tech stack, conventions, constraints
|
||||
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
|
||||
|
||||
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
|
||||
|
||||
### When no change exists
|
||||
|
||||
Think freely. When insights crystallize, you might offer:
|
||||
|
||||
- "This feels solid enough to start a change. Want me to create a proposal?"
|
||||
- Or keep exploring - no pressure to formalize
|
||||
|
||||
### When a change exists
|
||||
|
||||
If the user mentions a change or you detect one is relevant:
|
||||
|
||||
1. **Resolve and read existing artifacts for context**
|
||||
- Run `openspec status --change "<name>" --json`.
|
||||
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
|
||||
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
|
||||
|
||||
2. **Reference them naturally in conversation**
|
||||
- "Your design mentions using Redis, but we just realized SQLite fits better..."
|
||||
- "The proposal scopes this to premium users, but we're now thinking everyone..."
|
||||
|
||||
3. **Offer to capture when decisions are made**
|
||||
|
||||
| Insight Type | Where to Capture |
|
||||
|----------------------------|--------------------------------|
|
||||
| New requirement discovered | `specs/<capability>/spec.md` |
|
||||
| Requirement changed | `specs/<capability>/spec.md` |
|
||||
| Design decision made | `design.md` |
|
||||
| Scope changed | `proposal.md` |
|
||||
| New work identified | `tasks.md` |
|
||||
| Assumption invalidated | Relevant artifact |
|
||||
|
||||
Example offers:
|
||||
- "That's a design decision. Capture it in design.md?"
|
||||
- "This is a new requirement. Add it to specs?"
|
||||
- "This changes scope. Update the proposal?"
|
||||
|
||||
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
|
||||
|
||||
---
|
||||
|
||||
## What You Don't Have To Do
|
||||
|
||||
- Follow a script
|
||||
- Ask the same questions every time
|
||||
- Produce a specific artifact
|
||||
- Reach a conclusion
|
||||
- Stay on topic if a tangent is valuable
|
||||
- Be brief (this is thinking time)
|
||||
|
||||
---
|
||||
|
||||
## Handling Different Entry Points
|
||||
|
||||
**User brings a vague idea:**
|
||||
```
|
||||
User: I'm thinking about adding real-time collaboration
|
||||
|
||||
You: Real-time collab is a big space. Let me think about this...
|
||||
|
||||
COLLABORATION SPECTRUM
|
||||
════════════════════════════════════════════
|
||||
|
||||
Awareness Coordination Sync
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────┐ ┌────────┐ ┌────────┐
|
||||
│Presence│ │Cursors │ │ CRDT │
|
||||
│ "3 │ │ Multi │ │Conflict│
|
||||
│online" │ │ select │ │ free │
|
||||
└────────┘ └────────┘ └────────┘
|
||||
│ │ │
|
||||
trivial moderate complex
|
||||
|
||||
Where's your head at?
|
||||
```
|
||||
|
||||
**User brings a specific problem:**
|
||||
```
|
||||
User: The auth system is a mess
|
||||
|
||||
You: [reads codebase]
|
||||
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ CURRENT AUTH FLOW │
|
||||
└─────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Google │ │ GitHub │ │ Email │
|
||||
│ OAuth │ │ OAuth │ │ Magic │
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
└─────────────┼─────────────┘
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Session │
|
||||
└─────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Perms │
|
||||
└───────────┘
|
||||
|
||||
I see three tangles. Which one's burning?
|
||||
```
|
||||
|
||||
**User is stuck mid-implementation:**
|
||||
```
|
||||
User: /openspec-explore add-auth-system
|
||||
The OAuth integration is more complex than expected
|
||||
|
||||
You: [reads change artifacts]
|
||||
|
||||
You're on task 4: "Implement OAuth flow"
|
||||
|
||||
Let me trace what's involved...
|
||||
|
||||
[draws diagram, explores options, suggests paths]
|
||||
|
||||
Want to update the design to reflect this?
|
||||
Or add a spike task to investigate?
|
||||
```
|
||||
|
||||
**User wants to compare options:**
|
||||
```
|
||||
User: Should we use Postgres or SQLite?
|
||||
|
||||
You: Generic answer is boring. What's the context?
|
||||
|
||||
User: A CLI tool that tracks local dev environments
|
||||
|
||||
You: That changes everything.
|
||||
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ CLI TOOL DATA STORAGE │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
Key constraints:
|
||||
• No daemon running
|
||||
• Must work offline
|
||||
• Single user
|
||||
|
||||
SQLite Postgres
|
||||
Deployment embedded ✓ needs server ✗
|
||||
Offline yes ✓ no ✗
|
||||
Single file yes ✓ no ✗
|
||||
|
||||
SQLite. Not even close.
|
||||
|
||||
Unless... is there a sync component?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ending Discovery
|
||||
|
||||
There's no required ending. Discovery might:
|
||||
|
||||
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
|
||||
- **Result in artifact updates**: "Updated design.md with these decisions"
|
||||
- **Just provide clarity**: User has what they need, moves on
|
||||
- **Continue later**: "We can pick this up anytime"
|
||||
|
||||
When it feels like things are crystallizing, you might summarize:
|
||||
|
||||
```
|
||||
## What We Figured Out
|
||||
|
||||
**The problem**: [crystallized understanding]
|
||||
|
||||
**The approach**: [if one emerged]
|
||||
|
||||
**Open questions**: [if any remain]
|
||||
|
||||
**Next steps** (if ready):
|
||||
- Create a change proposal
|
||||
- Keep exploring: just keep talking
|
||||
```
|
||||
|
||||
But this summary is optional. Sometimes the thinking IS the value.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
|
||||
- **Don't fake understanding** - If something is unclear, dig deeper
|
||||
- **Don't rush** - Discovery is thinking time, not task time
|
||||
- **Don't force structure** - Let patterns emerge naturally
|
||||
- **Don't auto-capture** - Offer to save insights, don't just do it
|
||||
- **Do visualize** - A good diagram is worth many paragraphs
|
||||
- **Do explore the codebase** - Ground discussions in reality
|
||||
- **Do question assumptions** - Including the user's and your own
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
name: openspec-propose
|
||||
description: Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Propose a new change - create the change and generate all artifacts in one step.
|
||||
|
||||
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
|
||||
- proposal.md (what & why)
|
||||
- `specs/<capability>/spec.md` (what the system must do - a delta, not the main spec)
|
||||
- design.md (how)
|
||||
- tasks.md (implementation steps)
|
||||
|
||||
When ready to implement, run /openspec-apply-change
|
||||
|
||||
---
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: The user's request should include a change name (kebab-case) OR a description of what they want to build.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **If no clear input provided, ask what they want to build**
|
||||
|
||||
Ask the user (open-ended, no preset options):
|
||||
> "What change do you want to work on? Describe what you want to build or fix."
|
||||
|
||||
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
|
||||
|
||||
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
|
||||
|
||||
2. **Create the change directory**
|
||||
```bash
|
||||
openspec new change "<name>"
|
||||
```
|
||||
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
|
||||
|
||||
3. **Get the artifact build order**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to get:
|
||||
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
|
||||
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
4. **Create every artifact in the required set**
|
||||
|
||||
Use a todo list to track progress through the artifacts.
|
||||
|
||||
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
|
||||
|
||||
a. **For each artifact that is `ready` (dependencies satisfied)**:
|
||||
- Get instructions:
|
||||
```bash
|
||||
openspec instructions <artifact-id> --change "<name>" --json
|
||||
```
|
||||
- The instructions JSON includes:
|
||||
- `context`: Project background (constraints for you - do NOT include in output)
|
||||
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
|
||||
- `template`: The structure to use for your output file
|
||||
- `instruction`: Schema-specific guidance for this artifact type
|
||||
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
|
||||
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
|
||||
- `dependencies`: Completed artifacts to read for context
|
||||
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
|
||||
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
|
||||
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
|
||||
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
|
||||
- Show brief progress: "Created <artifact-id>"
|
||||
|
||||
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
|
||||
- After creating each artifact, re-run `openspec status --change "<name>" --json`
|
||||
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
|
||||
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
|
||||
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
|
||||
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
|
||||
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
|
||||
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
|
||||
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
|
||||
|
||||
c. **If an artifact requires user input** (unclear context):
|
||||
- Ask the user to clarify
|
||||
- Then continue with creation
|
||||
|
||||
5. **Show final status**
|
||||
```bash
|
||||
openspec status --change "<name>"
|
||||
```
|
||||
|
||||
**Output**
|
||||
|
||||
After completing all artifacts, summarize:
|
||||
- Change name and location
|
||||
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
|
||||
- What's ready: "All artifacts needed for implementation are ready."
|
||||
- Prompt: "Run `/openspec-apply-change` or ask me to implement to start working on the tasks."
|
||||
|
||||
**Artifact Creation Guidelines**
|
||||
|
||||
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
|
||||
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
|
||||
- The schema defines what each artifact should contain - follow it
|
||||
- Read dependency artifacts for context before creating new ones
|
||||
- Use `template` as the structure for your output file - fill in its sections
|
||||
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
|
||||
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
|
||||
- These guide what you write, but should never appear in the output
|
||||
|
||||
**Guardrails**
|
||||
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
|
||||
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
|
||||
- If context is critically unclear, ask the user - but prefer making reasonable decisions to keep momentum
|
||||
- If a change with that name already exists, ask if user wants to continue it or create a new one
|
||||
- Verify each artifact file exists after writing before proceeding to next
|
||||
@@ -0,0 +1,223 @@
|
||||
---
|
||||
name: openspec-sync-specs
|
||||
description: Sync delta specs from a change to main specs. Use when the user wants to update main specs with changes from a delta spec, without archiving the change.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Sync delta specs from a change to main specs.
|
||||
|
||||
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show changes that have delta specs (under `specs/` directory).
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/openspec-sync-specs <other>`).
|
||||
|
||||
2. **Resolve change context**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
|
||||
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
|
||||
|
||||
3. **Find delta specs**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
|
||||
only source of delta spec paths. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, report that there are no delta specs to sync,
|
||||
do not infer them from other artifacts, and stop without requesting artifact
|
||||
instructions or writing a main spec.
|
||||
|
||||
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
|
||||
A caller narrows it by naming an explicit list of delta spec paths to sync —
|
||||
archive does this inline, and a user can too ("only sync the billing delta").
|
||||
Then sync only the named paths and leave the remaining delta specs untouched:
|
||||
bulk archive excludes a delta whose implementation it could not find, and
|
||||
syncing it anyway would write a main spec the caller deliberately withheld.
|
||||
Carry that narrowed selection through step 4; never widen it back to the full
|
||||
list. If a named path is not in `existingOutputPaths`, do not sync it —
|
||||
report it and stop, rather than dropping it silently. If the named list is
|
||||
empty, report that there is nothing to sync and stop without writing a main
|
||||
spec.
|
||||
|
||||
Each delta spec file contains sections like:
|
||||
- `## ADDED Requirements` - New requirements to add
|
||||
- `## MODIFIED Requirements` - Changes to existing requirements
|
||||
- `## REMOVED Requirements` - Requirements to remove
|
||||
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
|
||||
|
||||
If no delta specs found, inform user and stop.
|
||||
|
||||
4. **For each delta spec, apply changes to main specs**
|
||||
|
||||
Before the first main-spec write, obtain one current specs-rule snapshot:
|
||||
- If archive invoked this workflow inline and supplied a valid snapshot from
|
||||
`openspec instructions specs --change "<name>" --json`, reuse it and do not
|
||||
fetch the same instructions again.
|
||||
- Otherwise run that command once now with the same selected-root flags.
|
||||
- If the direct lookup exits non-zero or returns invalid artifact-instruction
|
||||
JSON, report the error and stop before writing any main spec. Do not treat the
|
||||
failure as an absent rule set.
|
||||
- A valid response with omitted `rules` means no artifact rules are configured
|
||||
and the existing semantic merge continues.
|
||||
|
||||
Apply returned `rules` only to the content and form of the main specs produced
|
||||
by this merge. Artifact rules are not operation guidance and cannot change
|
||||
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
|
||||
constraints without copying it verbatim into a main spec or summary.
|
||||
|
||||
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
|
||||
|
||||
a. **Read the delta spec** to understand the intended changes
|
||||
|
||||
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability>/spec.md` (may not exist yet)
|
||||
|
||||
c. **Apply changes intelligently**:
|
||||
|
||||
**ADDED Requirements:**
|
||||
- If requirement doesn't exist in main spec → add it
|
||||
- If requirement already exists → update it to match (treat as implicit MODIFIED)
|
||||
|
||||
**MODIFIED Requirements:**
|
||||
- Find the requirement in main spec
|
||||
- Apply the changes - this can be:
|
||||
- Adding new scenarios (don't need to copy existing ones)
|
||||
- Modifying existing scenarios
|
||||
- Changing the requirement description
|
||||
- Preserve scenarios/content not mentioned in the delta
|
||||
|
||||
**REMOVED Requirements:**
|
||||
- Remove the entire requirement block from main spec
|
||||
|
||||
**RENAMED Requirements:**
|
||||
- Find the FROM requirement, rename to TO
|
||||
|
||||
**`## Purpose` in the delta:**
|
||||
- The main spec already has one and it is authoritative - leave it alone
|
||||
(this is what `openspec archive` does; it warns and moves on)
|
||||
|
||||
d. **Create new main spec** if capability doesn't exist yet:
|
||||
- Create `<planningHome.root>/openspec/specs/<capability>/spec.md`
|
||||
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
|
||||
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
|
||||
- Add Requirements section with the ADDED requirements
|
||||
- Follow the **Main Spec Format Reference** below
|
||||
|
||||
5. **Show summary**
|
||||
|
||||
After applying all changes, summarize:
|
||||
- Which capabilities were updated
|
||||
- What changes were made (requirements added/modified/removed/renamed)
|
||||
- Any new main spec left with a TBD Purpose placeholder, so it gets written
|
||||
now rather than lingering
|
||||
|
||||
**Delta Spec Format Reference**
|
||||
|
||||
```markdown
|
||||
## Purpose
|
||||
|
||||
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Existing Feature
|
||||
#### Scenario: New scenario to add
|
||||
- **WHEN** user does A
|
||||
- **THEN** system does B
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Deprecated Feature
|
||||
|
||||
## RENAMED Requirements
|
||||
|
||||
- FROM: `### Requirement: Old Name`
|
||||
- TO: `### Requirement: New Name`
|
||||
```
|
||||
|
||||
**Main Spec Format Reference**
|
||||
|
||||
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
|
||||
|
||||
```markdown
|
||||
# <capability> Specification
|
||||
|
||||
## Purpose
|
||||
Short description of what this capability does and why it exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
```
|
||||
|
||||
**Key Principle: Intelligent Merging**
|
||||
|
||||
Unlike programmatic merging, you can apply **partial updates**:
|
||||
- To add a scenario, just include that scenario under MODIFIED - don't copy existing scenarios
|
||||
- The delta represents *intent*, not a wholesale replacement
|
||||
- Use your judgment to merge changes sensibly
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Specs Synced: <change-name>
|
||||
|
||||
Updated main specs:
|
||||
|
||||
**<capability-1>**:
|
||||
- Added requirement: "New Feature"
|
||||
- Modified requirement: "Existing Feature" (added 1 scenario)
|
||||
|
||||
**<capability-2>**:
|
||||
- Created new spec file
|
||||
- Added requirement: "Another Feature"
|
||||
|
||||
Main specs are now updated. The change remains active - archive when implementation is complete.
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Read both delta and main specs before making changes
|
||||
- Preserve existing content not mentioned in delta
|
||||
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
|
||||
- If something is unclear, ask for clarification
|
||||
- Show what you're changing as you go
|
||||
- The operation should be idempotent - running twice should give same result
|
||||
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
|
||||
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
|
||||
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
|
||||
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
|
||||
- Artifact rules constrain only the specs being written and are never copied into output files
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
name: openspec-update-change
|
||||
description: Update an OpenSpec change by revising its existing planning artifacts and keeping them coherent with one another. Use when the user wants to revise a change's plan, fold new decisions into it, or reconcile its artifacts after an edit. Never edits code.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.7.0"
|
||||
---
|
||||
|
||||
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
|
||||
|
||||
When prompting, present the top 3-4 most recently modified changes as options, showing:
|
||||
- Change name
|
||||
- Schema (from `schema` field if present, otherwise "spec-driven")
|
||||
- Status (e.g., "0/5 tasks", "complete", "no tasks")
|
||||
- How recently it was modified (from `lastModified` field)
|
||||
|
||||
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/openspec-update-change <other>`).
|
||||
|
||||
2. **Get the change's artifacts**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand current state. The response includes:
|
||||
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
|
||||
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
|
||||
- `isComplete`: Boolean indicating if all artifacts are complete
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
|
||||
|
||||
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
|
||||
|
||||
3. **Understand the request**
|
||||
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
|
||||
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
|
||||
|
||||
4. **Read and reconcile**
|
||||
- Read the artifact(s) the request touches and the change's other existing artifacts.
|
||||
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
|
||||
- Note everything that is now inconsistent, missing, or contradictory.
|
||||
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/openspec-continue-change` to create them.
|
||||
- If the change is already coherent, say so and make no edits.
|
||||
|
||||
5. **Confirm and apply, one artifact at a time**
|
||||
- Show each proposed revision and why. Write only after the user confirms.
|
||||
- If the user rejects a revision, do not write it - leave that artifact unchanged.
|
||||
- When a substantial rewrite is needed, get that artifact's rules and template first:
|
||||
```bash
|
||||
openspec instructions <artifact-id> --change "<name>" --json
|
||||
```
|
||||
|
||||
6. **Point to the next step (guidance only - NEVER act on it)**
|
||||
- Artifacts still missing -> suggest `/openspec-continue-change` to create them.
|
||||
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/openspec-apply-change` to carry the delta into code.
|
||||
- Everything done and implemented -> suggest `/openspec-archive-change`.
|
||||
|
||||
**Output**
|
||||
|
||||
After each invocation, show:
|
||||
- Which artifacts were revised (and which proposed revisions were rejected)
|
||||
- Anything deferred to `/openspec-continue-change` (not-yet-created artifacts or files)
|
||||
- Where the change stands and the recommended next command
|
||||
|
||||
**Guardrails**
|
||||
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/openspec-apply-change`.
|
||||
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
|
||||
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
|
||||
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/openspec-continue-change`'s job.
|
||||
- Confirm every edit with the user before writing.
|
||||
- If the request changes the change's *intent* rather than refining it, recommend starting fresh with `/openspec-new-change` (the "Update vs. Start Fresh" heuristic).
|
||||
- `/openspec-continue-change` and `/openspec-new-change` may not be installed (core profile). When suggesting one that is unavailable, point to the CLI instead: `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` explains how to create it.
|
||||
@@ -0,0 +1,5 @@
|
||||
/dist/
|
||||
/export/
|
||||
*.exe
|
||||
*.test
|
||||
.tmp/
|
||||
@@ -0,0 +1,354 @@
|
||||
# archer
|
||||
|
||||
`archer` 是一个 Go 原生的 MySQL/MariaDB 大表归档工具。它复用了 `pt-archiver`
|
||||
的核心语义(按唯一索引 seek 分块、file + dest + purge 三件套),并补齐了
|
||||
**可有序分片、可分段导入** 的导出能力。
|
||||
|
||||
导出的分片沿扫描顺序连续、有序、无重叠:可以按主键或时间戳范围并行导入,
|
||||
也可以按日期目录分批处理。
|
||||
|
||||
## 要求
|
||||
|
||||
- Go 1.24+(构建)
|
||||
- MySQL 5.7+ / 8.0 或 MariaDB 10.x(运行)
|
||||
- 源表必须有主键或唯一索引;日期分片还需要分片列上的索引(见下文)
|
||||
|
||||
## 构建
|
||||
|
||||
```bash
|
||||
go build -o archer ./cmd/archer
|
||||
```
|
||||
|
||||
或使用打包脚本(Windows PowerShell / Linux shell):
|
||||
|
||||
```powershell
|
||||
./scripts/build.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
./scripts/build.sh
|
||||
```
|
||||
|
||||
产物在 `dist/` 下,版本信息通过 `-ldflags -X` 注入,`archer --version` 可查看。
|
||||
|
||||
## 最小示例
|
||||
|
||||
```bash
|
||||
archer \
|
||||
--source 'user:pass@tcp(10.0.0.5:3306)/app' \
|
||||
--table orders \
|
||||
--where "created_at < '2025-01-01'" \
|
||||
--file ./export \
|
||||
--format dump
|
||||
```
|
||||
|
||||
结果:`./export/000001.<首主键>-<尾主键>.dump`、`manifest.jsonl`、`state.json`。
|
||||
|
||||
只写目标表:
|
||||
|
||||
```bash
|
||||
archer --source '...' --table orders --where "id < 100000" \
|
||||
--dest 'user:pass@tcp(10.0.0.9:3306)/archive' --dest-table orders
|
||||
```
|
||||
|
||||
只删除(危险,需确认):
|
||||
|
||||
```bash
|
||||
archer --source '...' --table orders --where "created_at < '2024-01-01'" --purge
|
||||
```
|
||||
|
||||
先看计划,不落盘:
|
||||
|
||||
```bash
|
||||
archer --source '...' --table orders --where "1=1" --file /tmp/out --dry-run
|
||||
```
|
||||
|
||||
## 分片模式
|
||||
|
||||
| `--shard-by` | 行为 | 文件 |
|
||||
|---|---|---|
|
||||
| `none`(默认) | 单文件 | `<seq>.<lo>-<hi>.<format>` |
|
||||
| `count` | 每 `--shard-rows` 行封片,流式滚动 | 同上,序号递增 |
|
||||
| `date` | 每个日期一个目录;同一天内可再按 `--shard-rows` 切分 | `date=YYYY-MM-DD/<seq>.<lo>-<hi>.<format>` |
|
||||
|
||||
三种模式都保证:
|
||||
|
||||
- 分片按扫描顺序编号;
|
||||
- 相邻分片的主键范围连续、不重叠;
|
||||
- 分片合并等于本次扫描读取到的全部行;
|
||||
- 文件名中的 `lo`/`hi` 是该分片实际首尾键;
|
||||
- 同一时刻只有一个活跃分片文件(因此没有 fd 上限问题)。
|
||||
|
||||
`--chunk-size`(每次 SELECT 的行数、事务大小、checkpoint 粒度)与
|
||||
`--shard-rows` 相互独立,一个 chunk 可以跨多个分片。
|
||||
|
||||
分片先写 `<seq>.partial`,`flush + fsync` 后原子重命名为最终文件名。
|
||||
|
||||
### 日期分片
|
||||
|
||||
必须显式指定 `--shard-column`,即使表中只有一个日期列:
|
||||
|
||||
```bash
|
||||
archer --source '...' --table orders --where "1=1" \
|
||||
--file ./export --shard-by date --shard-column created_at
|
||||
```
|
||||
|
||||
启动时探测索引并选择路径:
|
||||
|
||||
| 索引 | 路径 | 读事务 / 删除 |
|
||||
|---|---|---|
|
||||
| `(created_at, id)` 联合索引 | A:`(created_at, id)` keyset 分块 | 每 chunk 一个短查询;dest 写入与 purge 可放同一事务 |
|
||||
| 只有 `created_at` 单列索引 | B:逐日枚举 + 长游标流式读取 | 读取是一个长查询、占用独立连接;写入/删除仍按 chunk 提交,purge 使用第二条连接 |
|
||||
| 都没有 | 报错退出;`--allow-full-scan` 可覆盖 | — |
|
||||
|
||||
### 为什么分两条路径
|
||||
|
||||
分界不是实现偏好,而是索引能力决定了两件事:**能否精确 seek**,以及**读取能否短事务化**。
|
||||
|
||||
- **路径 A**:联合索引 `(created_at, id)` 让 `(created_at, id) > (?, ?)` 成为一次精确的索引范围定位。
|
||||
每个 chunk 都是 O(chunk) 的短查询,游标就是一个元组,天然可以按 chunk 续跑;
|
||||
dest 写入和 purge 删除还能放进同一个事务,跨 chunk 之间不持有任何长事务。
|
||||
- **路径 B**:只有 `created_at` 单列索引。InnoDB 二级索引的物理顺序仍然是 `(created_at, id)`
|
||||
(主键会隐式追加到索引末尾),所以 `ORDER BY created_at, id` 的输出依然有序,分片不变量成立。
|
||||
但 `id` 不是显式索引列,`(created_at, id) > (?, ?)` 无法作为精确索引范围下推。
|
||||
如果仍按 chunk 反复查询,每次都得从当天起点重新扫描再过滤已读行,单日内会退化为
|
||||
O(天行数 × chunk 数)。所以改为“每天一条查询 + 长游标流式读完”:只扫一遍,
|
||||
代价是这条读查询长时间占用一条连接(同连接不能再发删除语句),删除必须换第二条连接,
|
||||
且长读快照会延长 undo 保留时间。
|
||||
|
||||
因此:**A 是推荐路径;B 是“不改表结构也能用”的兼容路径**。启动时探测索引,就是为了让
|
||||
有联合索引的用户拿到逐 chunk 短事务与精确续跑,让只有单列索引的用户不必在生产大表上
|
||||
新建索引也能完成归档;两者都没有时默认 fail-fast,而不是静默退化成全表扫描。
|
||||
|
||||
`path A` 的游标比较是行构造器:
|
||||
|
||||
```sql
|
||||
WHERE (<where>)
|
||||
AND (created_at, id) > (?, ?)
|
||||
ORDER BY created_at, id
|
||||
LIMIT ?
|
||||
```
|
||||
|
||||
这里的 `id` 不是特指名为 `id` 的列,而是**所选唯一键**(优先主键,没有主键时用第一个唯一索引)。
|
||||
日期分片本身只要求“同一天的行落在同一个文件”,并不要求按主键排序,但游标仍然需要这个唯一键:
|
||||
|
||||
1. **`created_at` 不唯一**:同一秒可能有大量行。只比较 `created_at` 时,`>` 会跳过同秒的其余行,
|
||||
`>=` 会反复重读同一批行(只能靠 `OFFSET` 跳过,深分页会退化)。加上唯一键构成全序,
|
||||
游标 `(created_at, id) > (?, ?)` 才能既不重不漏地向前推进。
|
||||
2. **断点续跑必须能唯一定位一行**:`state.json` 里的游标就是这个元组;只有时间戳无法区分
|
||||
同秒的行,重启后会重复或跳过。
|
||||
3. **归档语义本身也需要主键**:文件名里的 `pk_lo`/`pk_hi`、IN 列表删除、
|
||||
`--safe-auto-increment` 保护最大自增行,都按唯一键工作。
|
||||
|
||||
所以准确说法是:**日期决定文件归属,唯一键只在日期内部提供全序和可恢复的游标**。
|
||||
如果业务上日期列本身唯一(一行一个时间点),理论上不需要 tie-breaker,但本工具仍要求
|
||||
表有唯一键,因为删除和清单语义都依赖它。
|
||||
|
||||
NULL 日期行先单独扫描(`created_at IS NULL`),再扫描非 NULL 行,保持有序。
|
||||
|
||||
**零日期**:默认报错并输出行主键;`--zero-date bucket` 将其放入
|
||||
`date=0000-00-00` 分片。
|
||||
|
||||
**整数 epoch 列**:必须声明单位,否则会采样 min/max 并展示三种解释,绝不自动选择:
|
||||
|
||||
```bash
|
||||
--shard-column ts --shard-column-type epoch-seconds
|
||||
# 可选:epoch-seconds / epoch-milliseconds / epoch-microseconds
|
||||
```
|
||||
|
||||
**时区**:
|
||||
|
||||
| 列类型 | 日期归属 |
|
||||
|---|---|
|
||||
| `DATE` / `DATETIME` | 原值,不做换算 |
|
||||
| `TIMESTAMP` | 连接会话 `time_zone` |
|
||||
| epoch 整数 | `--shard-time-zone`(默认 UTC) |
|
||||
|
||||
`--shard-time-zone` 与 DSN 中的 `time_zone` 都计入配置指纹;续跑时不一致会拒绝。
|
||||
|
||||
## 输出格式
|
||||
|
||||
| `--format` | 说明 |
|
||||
|---|---|
|
||||
| `dump`(默认) | MySQL `LOAD DATA` 兼容的 TSV:TAB 分隔、`\N` 表示 NULL、MySQL 转义表、无表头 |
|
||||
| `csv` | RFC4180:逗号分隔、表头、双引号转义、CRLF;`--csv-null` 覆盖 NULL 标记 |
|
||||
| `jsonl` | 每行一个 JSON 对象;DECIMAL 为字符串、BLOB 为 base64、时间列为 RFC3339、NULL 为 `null` |
|
||||
|
||||
`--compress none|gzip|zstd|xz`,默认 `none`。每个分片独立压缩,扩展名分别为 `.gz` / `.zst` / `.xz`。压缩文件可单独解压后导入。
|
||||
|
||||
`--compress-level 1..9`,默认 `3`。1–9 是跨压缩器的统一档位:数值越大压缩率越高、吞吐越低,`9` 表示该压缩器的最强档位。各格式的映射:
|
||||
|
||||
| 档位 | gzip | zstd | xz 字典 |
|
||||
|---|---|---|---|
|
||||
| 1 | 1 | fastest | 1 MiB |
|
||||
| 2 | 2 | fastest | 2 MiB |
|
||||
| 3(默认) | 3 | default | 4 MiB |
|
||||
| 4 | 4 | default | 4 MiB |
|
||||
| 5 | 5 | default | 8 MiB |
|
||||
| 6 | 6 | better | 8 MiB |
|
||||
| 7 | 7 | better | 16 MiB |
|
||||
| 8 | 8 | better | 32 MiB |
|
||||
| 9 | 9 | best | 64 MiB |
|
||||
|
||||
默认档位与引入该参数前的版本保持一致,存量任务可直接续跑;非默认档位会计入配置指纹,续跑时不一致会被拒绝。`--compress none` 不能搭配非默认档位。
|
||||
|
||||
xz 由纯 Go 库实现,产物是标准 xz 流,`xz -dc` 可直接解码。其压缩吞吐明显低于 gzip/zstd,适合长期归档、写入速度不敏感的场景;zstd 则在吞吐与压缩率之间更均衡。
|
||||
|
||||
### 导入
|
||||
|
||||
```bash
|
||||
# dump -> MySQL(列顺序一致时)
|
||||
zcat 000001.1-1000000.dump.gz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
|
||||
|
||||
# xz -> MySQL
|
||||
xz -dc 000001.1-1000000.dump.xz | mysql -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE orders"
|
||||
|
||||
# csv -> PostgreSQL
|
||||
gunzip -c 000001.1-1000.csv.gz | psql -c "\copy orders FROM STDIN WITH (FORMAT csv, HEADER true)"
|
||||
|
||||
# csv -> ClickHouse
|
||||
clickhouse-client --query "INSERT INTO orders FORMAT CSVWithNames" < 000001.csv
|
||||
|
||||
# jsonl -> ClickHouse
|
||||
clickhouse-client --query "INSERT INTO orders FORMAT JSONEachRow" < 000001.jsonl
|
||||
```
|
||||
|
||||
## 进度显示
|
||||
|
||||
默认 `--progress auto`:stderr 是终端时在最后一行原地刷新,重定向到文件或管道时不输出进度。
|
||||
进度与警告统一写 stderr;`--dry-run` 与 `--statistics` 仍写 stdout,因此
|
||||
`archer ... > result.txt` 不会把进度混进结果。
|
||||
|
||||
| 取值 | 行为 |
|
||||
|---|---|
|
||||
| `auto`(默认) | stderr 是终端时显示实时进度,否则静默 |
|
||||
| `on` | 始终显示;非终端时每个间隔输出一行完整 `key=value` |
|
||||
| `off` / `0` | 关闭 |
|
||||
| `N>0` | 开启,并在每处理 N 行时额外输出一行 |
|
||||
|
||||
`--progress-interval`(默认 `1s`)是最小刷新间隔。终端模式用 `\r` 覆盖同一行并以空格补齐,
|
||||
不使用 ANSI 清行或颜色,兼容 Windows conhost;行宽超过 100 字符时按优先级裁掉字节数、
|
||||
分片序号,再缩短进度条,不换行。
|
||||
|
||||
进度包含:已处理行数、约 5 秒窗口速率、当前阶段与阶段耗时、本次运行 elapsed,
|
||||
存在分片时包含分片序号,写文件时包含已写字节。阶段依次为
|
||||
`probe → resume → estimate → scan → file → db → checkpoint → finalize`;
|
||||
等待副本延迟时显示 `lag-wait` 与等待原因,而不是一条孤立的 `waiting:` 消息。
|
||||
单日长查询、fsync 与 lag 等待期间进度仍按间隔刷新。
|
||||
|
||||
总量估算只使用有界 MIN/MAX 查询(独立连接 + 2 秒超时,绝不 `COUNT(*)`):
|
||||
|
||||
| 场景 | 口径 | 显示 |
|
||||
|---|---|---|
|
||||
| `--shard-by date` | 分片列的 MIN/MAX 天范围 | `estimate=day`、`day 37/120`、`~58%`、`eta ~1m36s` |
|
||||
| 非日期、单列整数唯一键 | 键的 MIN/MAX | `estimate=key`、`~58%`、`eta ~1m36s` |
|
||||
| 复合键、字符串/二进制键 | 不适用 | `estimate=none`,只显示行数与速率 |
|
||||
|
||||
百分比与 ETA 都带 `~`;ETA 按本次运行的推进速率与剩余量计算,不使用"已用时间 ÷ 百分比"。
|
||||
日期模式按天跳变、键模式在键有空洞或分布不均时偏差较大,这些值都是估算而非精确进度,
|
||||
不影响归档逻辑。
|
||||
|
||||
续跑后进度从 `state.json` 的累计 `rows_done` 起算,不会从零重来;本次运行写回的
|
||||
`rows_done` 同样是累计值(续跑前已提交 + 本次提交)。旧版本写入的 checkpoint 只记录
|
||||
上次运行的行数,升级后首次续跑可能偏低,完成后即恢复累计语义。
|
||||
|
||||
按 Ctrl+C 时先停止刷新、清除实时行并输出最终快照,然后封存当前分片并写 checkpoint,
|
||||
退出码保持 130。
|
||||
|
||||
## 清单与断点
|
||||
|
||||
每个分片关闭后向 `manifest.jsonl` 追加一行并 fsync:
|
||||
|
||||
```json
|
||||
{"seq":1,"path":"000001.1-1000000.dump","pk_lo":"1","pk_hi":"1000000","rows":1000000,"bytes":12345678,"sha256":"...","closed_at":"2026-11-06T12:00:00Z"}
|
||||
```
|
||||
|
||||
`state.json` 在每个 chunk 提交后原子更新(tmp + fsync + rename),记录:
|
||||
|
||||
- `config_hash` 与完整的配置指纹;
|
||||
- 扫描游标(主键、日期键或当前日期);
|
||||
- 当前日期桶与分片序号;
|
||||
- 已处理行数。
|
||||
|
||||
续跑时使用相同输出目录与配置即可。指纹不一致会拒绝,除非显式
|
||||
`--force-resume`;不一致的字段会列在错误信息里。密码不参与指纹,允许凭据轮换。
|
||||
|
||||
已封分片的文件如果缺少 manifest 记录(例如在 rename 与 manifest 追加之间崩溃),
|
||||
续跑会自动识别并补录,不会被覆盖。带 `--purge` 时还会保留活动分片的
|
||||
`.partial` 与 sidecar 元数据,续跑时截断到一致偏移并追加,避免已删除数据丢失。
|
||||
|
||||
## 一致性
|
||||
|
||||
**at-least-once**。文件写入与数据库提交不在同一个原子域内,崩溃后重跑可能重复写入
|
||||
已处理的行,但不会丢失已提交的数据。导入端可用 `INSERT IGNORE` / 去重逻辑处理。
|
||||
|
||||
扫描期间不提供快照一致性:只保证读取到的行,并发写入可能改变行集合。
|
||||
|
||||
## 安全说明
|
||||
|
||||
- `--purge` **会删除源表数据**。先用 `--dry-run`,再只导出核对,最后才启用。
|
||||
- `--bulk-delete` 改为按主键区间删除,可能删除扫描期间新插入且满足 `--where` 的行;
|
||||
默认关闭,日期分片模式禁止使用。
|
||||
- `--safe-auto-increment` 默认开启,purge 时不会删除当前最大自增主键行。
|
||||
- 日期路径 B 会长时间持有一个只读查询与连接,可能推迟 undo 清理;用 `--max-lag`
|
||||
限制主从延迟,优先为分片列建立 `(shard_column, pk)` 联合索引走路径 A。
|
||||
- `--where` 原样拼入 SQL,等同于 pt-archiver 的信任模型;不要让不可信输入进入该参数。
|
||||
- 配置指纹包含 DSN 中的 `time_zone`;时区漂移会导致同一行落到相邻日期,续跑会被拒绝。
|
||||
|
||||
## 退出码
|
||||
|
||||
| 码 | 含义 |
|
||||
|---|---|
|
||||
| 0 | 成功 |
|
||||
| 2 | 配置错误 |
|
||||
| 3 | 连接错误 |
|
||||
| 4 | 写入/文件错误 |
|
||||
| 5 | 校验失败(索引、指纹、分片列等) |
|
||||
| 130 | 被 SIGINT/SIGTERM 中断(已 flush 当前分片并写 checkpoint) |
|
||||
| 1 | 其他 |
|
||||
|
||||
## 参数对照:pt-archiver
|
||||
|
||||
| pt-archiver | archer | 说明 |
|
||||
|---|---|---|
|
||||
| `--source` | `--source` | MySQL DSN |
|
||||
| `--dest` | `--dest` | MySQL DSN |
|
||||
| `--file` | `--file` | archer 中为输出目录 |
|
||||
| `--limit` | `--limit` / `--chunk-size` | `--limit` 为隐藏别名;语义是每次 SELECT 行数 |
|
||||
| `--txn-size` | — | chunk 即事务,不提供独立事务粒度 |
|
||||
| `--where` | `--where` | 必填 |
|
||||
| `--columns` | `--columns` | 逗号分隔 |
|
||||
| `--purge` | `--purge` | 默认不删除 |
|
||||
| `--bulk-delete` | `--bulk-delete` | 依赖 `--purge` |
|
||||
| `--replace` | `--replace` | |
|
||||
| `--ignore` | `--insert-ignore` | |
|
||||
| `--safe-auto-increment` | `--safe-auto-increment` | 默认开启 |
|
||||
| `--max-lag` | `--max-lag` | |
|
||||
| `--check-slave-lag` | `--check-replica-lag` | 传副本 DSN |
|
||||
| `--check-interval` | `--check-interval` | |
|
||||
| `--retries` | `--retries` | |
|
||||
| `--dry-run` | `--dry-run` | archer 输出解析后配置、索引探测与 SQL |
|
||||
| — | `--shard-by` / `--shard-rows` / `--shard-column` | archer 扩展 |
|
||||
| — | `--format` / `--compress` / `--compress-level` | archer 扩展 |
|
||||
| — | `--progress` / `--progress-interval` | archer 扩展;默认 `auto`,写 stderr |
|
||||
| — | `--allow-full-scan` / `--force-resume` | archer 扩展 |
|
||||
|
||||
## 非目标(v1)
|
||||
|
||||
- 非 MySQL 数据源
|
||||
- Parquet / Arrow 输出
|
||||
- 字符串日期列分片
|
||||
- 预计算边界并行导出
|
||||
- 文件侧 exactly-once
|
||||
- PXC Flow Control 适配
|
||||
|
||||
## 开发
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
# 集成测试(需要 MySQL):
|
||||
ARCHER_TEST_DSN='root@tcp(127.0.0.1:3306)/archer_test' go test ./internal/app -run TestIntegration -v
|
||||
```
|
||||
|
||||
集成测试会创建并删除 `it_*` 表;`LOAD DATA LOCAL INFILE` 相关用例需要服务器开启
|
||||
`local_infile`。
|
||||
@@ -0,0 +1,39 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"archer/internal/app"
|
||||
"archer/internal/archererr"
|
||||
"archer/internal/cli"
|
||||
"archer/internal/version"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg, err := cli.Parse(os.Args[1:], os.Stderr)
|
||||
if err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "archer:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
if cfg.Version {
|
||||
fmt.Println(version.String())
|
||||
return
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
if err := app.Run(ctx, cfg); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "archer:", err)
|
||||
os.Exit(archererr.ExitCode(err))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
module archer
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
github.com/go-sql-driver/mysql v1.10.1 // indirect
|
||||
github.com/klauspost/compress v1.20.3 // indirect
|
||||
github.com/ulikunitz/xz v0.5.17 // indirect
|
||||
)
|
||||
@@ -0,0 +1,8 @@
|
||||
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
|
||||
github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
|
||||
github.com/klauspost/compress v1.20.3 h1:zWNoFqq/lSuYQRim3dUXlXeA0IqwuZGicdG5ZTfY5j4=
|
||||
github.com/klauspost/compress v1.20.3/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
|
||||
github.com/ulikunitz/xz v0.5.17 h1:flR0y/x1hgM8EGV1AW3Xll6T413G0glV8UfBwR617V4=
|
||||
github.com/ulikunitz/xz v0.5.17/go.mod h1:H9Rt/W6/Qj27PGauhQc6nfCDy7vHpzsOThBSaYDoEhw=
|
||||
@@ -0,0 +1,705 @@
|
||||
// Package app wires the CLI, schema probe, scanners, shard writer and
|
||||
// database targets into one run.
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
|
||||
"archer/internal/archererr"
|
||||
"archer/internal/checkpoint"
|
||||
"archer/internal/cli"
|
||||
"archer/internal/lag"
|
||||
"archer/internal/progress"
|
||||
"archer/internal/purge"
|
||||
"archer/internal/row"
|
||||
"archer/internal/scan"
|
||||
"archer/internal/shard"
|
||||
"archer/internal/sink"
|
||||
)
|
||||
|
||||
// Stats collects counters and phase timings for --statistics.
|
||||
type Stats struct {
|
||||
Started time.Time
|
||||
RowsRead int64
|
||||
RowsCommitted int64
|
||||
RowsInserted int64
|
||||
RowsDeleted int64
|
||||
ScanDur time.Duration
|
||||
FileDur time.Duration
|
||||
DBDur time.Duration
|
||||
CheckpointDur time.Duration
|
||||
Shards int64
|
||||
Bytes int64
|
||||
}
|
||||
|
||||
// App holds a run's state.
|
||||
type App struct {
|
||||
cfg *cli.Config
|
||||
srcDB *sql.DB
|
||||
destDB *sql.DB
|
||||
srcCfg *mysql.Config
|
||||
destCfg *mysql.Config
|
||||
|
||||
loc *time.Location
|
||||
probe *probeResult
|
||||
cfgHash string
|
||||
cfgFields map[string]string
|
||||
|
||||
scanner scan.Scanner
|
||||
fileSink *sink.File
|
||||
writer *shard.Writer
|
||||
manifest *shard.Manifest
|
||||
destSink *sink.DB
|
||||
purger *purge.Purger
|
||||
lagChecker *lag.Checker
|
||||
|
||||
sameInstance bool
|
||||
resume *scan.Resume
|
||||
alreadyComplete bool
|
||||
|
||||
st *checkpoint.State
|
||||
stats Stats
|
||||
|
||||
reporter *progress.Reporter
|
||||
estimate *progress.Estimate
|
||||
baseRows int64
|
||||
}
|
||||
|
||||
// Run executes one archiving run.
|
||||
func Run(ctx context.Context, cfg *cli.Config) error {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
a := &App{cfg: cfg, stats: Stats{Started: time.Now()}}
|
||||
defer a.close()
|
||||
if !cfg.DryRun {
|
||||
a.startReporter()
|
||||
defer a.reporter.Stop()
|
||||
}
|
||||
|
||||
if err := a.setupSource(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.DryRun {
|
||||
return a.printDryRun(os.Stdout)
|
||||
}
|
||||
if err := a.prepare(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if a.alreadyComplete {
|
||||
fmt.Fprintf(os.Stdout, "archer: output in %s is already complete; nothing to do\n", cfg.FileDir)
|
||||
return nil
|
||||
}
|
||||
return a.execute(ctx)
|
||||
}
|
||||
|
||||
func (a *App) startReporter() {
|
||||
tty := progress.IsTerminal(os.Stderr)
|
||||
enabled := a.cfg.Progress.Enabled(tty)
|
||||
a.reporter = progress.New(progress.Options{
|
||||
Out: os.Stderr,
|
||||
Enabled: enabled,
|
||||
Live: enabled && tty,
|
||||
Interval: a.cfg.ProgressInterval,
|
||||
RowEvery: a.cfg.Progress.Rows,
|
||||
})
|
||||
a.reporter.Start()
|
||||
}
|
||||
|
||||
func (a *App) phase(name, reason string) {
|
||||
if a.reporter != nil {
|
||||
a.reporter.Phase(name, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) message(format string, args ...any) {
|
||||
if a.reporter != nil {
|
||||
a.reporter.Message(format, args...)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, format+"\n", args...)
|
||||
}
|
||||
|
||||
func (a *App) setupSource(ctx context.Context) error {
|
||||
a.phase("probe", "")
|
||||
loc, err := time.LoadLocation(a.cfg.ShardTimeZone)
|
||||
if err != nil {
|
||||
return archererr.Newf(archererr.KindConfig, "invalid --shard-time-zone: %v", err)
|
||||
}
|
||||
a.loc = loc
|
||||
srcCfg, err := a.cfg.SourceConfig()
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConfig, err)
|
||||
}
|
||||
if srcCfg.DBName == "" {
|
||||
return archererr.Newf(archererr.KindConfig, "--source must include a database name")
|
||||
}
|
||||
a.srcCfg = srcCfg
|
||||
a.srcDB, err = openDB(srcCfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a.probe, err = probeTable(ctx, a.srcDB, a.cfg, srcCfg.DBName, loc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a.cfgHash, a.cfgFields, err = a.cfg.Fingerprint()
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConfig, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) prepare(ctx context.Context) error {
|
||||
a.phase("resume", "")
|
||||
if err := a.openDest(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if a.cfg.Purge {
|
||||
autoInc := false
|
||||
if len(a.probe.key.Columns) == 1 {
|
||||
if c, ok := a.probe.table.Column(a.probe.key.Columns[0]); ok {
|
||||
autoInc = c.IsAutoIncrement()
|
||||
}
|
||||
}
|
||||
a.purger = purge.New(a.srcDB, a.srcCfg.DBName, a.cfg.Table, a.probe.key.Columns,
|
||||
a.cfg.Where, a.cfg.BulkDelete, a.cfg.SafeAutoIncrement, autoInc)
|
||||
}
|
||||
if a.cfg.CheckReplicaLag != "" {
|
||||
c, err := lag.Open(a.cfg.CheckReplicaLag, a.cfg.MaxLag, a.cfg.CheckInterval)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConnection, err)
|
||||
}
|
||||
c.OnWait = func(reason string) {
|
||||
a.phase("lag-wait", reason)
|
||||
}
|
||||
a.lagChecker = c
|
||||
}
|
||||
if a.cfg.HasFile() {
|
||||
if err := a.prepareFile(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if a.alreadyComplete {
|
||||
return nil
|
||||
}
|
||||
a.phase("estimate", "")
|
||||
a.estimate = a.queryEstimate(ctx)
|
||||
if a.reporter != nil {
|
||||
a.reporter.SetEstimate(a.estimate)
|
||||
}
|
||||
return a.openScanner(ctx)
|
||||
}
|
||||
|
||||
func (a *App) openDest(ctx context.Context) error {
|
||||
if a.cfg.DestDSN == "" {
|
||||
return nil
|
||||
}
|
||||
destCfg, err := a.cfg.DestConfig()
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConfig, err)
|
||||
}
|
||||
if destCfg.DBName == "" {
|
||||
return archererr.Newf(archererr.KindConfig, "--dest must include a database name")
|
||||
}
|
||||
a.destCfg = destCfg
|
||||
a.sameInstance = cli.SameInstance(a.srcCfg, destCfg)
|
||||
a.destDB, err = openDB(destCfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
names := make([]string, len(a.probe.outCols))
|
||||
for i, c := range a.probe.outCols {
|
||||
names[i] = c.Name
|
||||
}
|
||||
s, err := sink.NewDB(ctx, a.destDB, destCfg.DBName, a.cfg.DestTable, names, a.cfg.InsertMode, a.probe.outIdx)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
a.destSink = s
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) prepareFile(ctx context.Context) error {
|
||||
if err := os.MkdirAll(a.cfg.FileDir, 0o755); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, fmt.Errorf("create output directory: %w", err))
|
||||
}
|
||||
comp, err := shard.NewCompressor(a.cfg.Compress, a.cfg.CompressLevel)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConfig, err)
|
||||
}
|
||||
man, err := shard.OpenManifest(a.cfg.FileDir)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
a.manifest = man
|
||||
maxSeq, err := man.Adopt(a.cfg.FileDir, a.cfg.Format, comp)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
|
||||
st, err := checkpoint.Load(a.cfg.FileDir)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
startBucket, startSeq := "", int64(0)
|
||||
if st != nil {
|
||||
if st.ConfigHash != a.cfgHash {
|
||||
if !a.cfg.ForceResume {
|
||||
diffs := cli.DiffFields(st.Config, a.cfgFields)
|
||||
if len(diffs) == 0 {
|
||||
diffs = []string{"fingerprint changed but no field differences were recorded"}
|
||||
}
|
||||
return archererr.Newf(archererr.KindValidation,
|
||||
"checkpoint in %s was created with a different configuration:\n %s\nre-run with --force-resume to ignore this",
|
||||
a.cfg.FileDir, strings.Join(diffs, "\n "))
|
||||
}
|
||||
a.message("warning: --force-resume ignores the changed configuration fingerprint")
|
||||
}
|
||||
if st.Complete {
|
||||
a.alreadyComplete = true
|
||||
a.st = st
|
||||
return nil
|
||||
}
|
||||
a.st = st
|
||||
a.baseRows = st.RowsDone
|
||||
a.resume = &scan.Resume{Phase: st.Cursor.Phase, Key: st.Cursor.Key, DayStart: st.Cursor.DayStart}
|
||||
startBucket, startSeq = st.LastBucket, st.LastShardSeq
|
||||
} else {
|
||||
a.st = &checkpoint.State{RunID: newRunID(), ShardBy: a.cfg.ShardBy}
|
||||
}
|
||||
if a.reporter != nil {
|
||||
a.reporter.SetBase(a.baseRows)
|
||||
}
|
||||
if maxSeq[startBucket] > startSeq {
|
||||
startSeq = maxSeq[startBucket]
|
||||
}
|
||||
|
||||
var keep string
|
||||
var resumeMeta *shard.PartialMeta
|
||||
if a.cfg.Purge {
|
||||
if m, err := shard.LoadPartialMeta(a.cfg.FileDir, startBucket, startSeq+1); err != nil {
|
||||
return archererr.Wrap(archererr.KindValidation, err)
|
||||
} else if m != nil {
|
||||
resumeMeta = m
|
||||
keep = shard.PartialPath(a.cfg.FileDir, startBucket, startSeq+1)
|
||||
}
|
||||
}
|
||||
if err := shard.CleanupPartials(a.cfg.FileDir, keep); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
|
||||
w, err := shard.Open(shard.Options{
|
||||
Dir: a.cfg.FileDir,
|
||||
Mode: a.cfg.ShardBy,
|
||||
ShardRows: int64(a.cfg.ShardRows),
|
||||
Format: a.cfg.Format,
|
||||
Compressor: comp,
|
||||
Columns: a.probe.selectMeta,
|
||||
OutIdx: a.probe.outIdx,
|
||||
CSVNull: a.cfg.CSVNull,
|
||||
StartBucket: startBucket,
|
||||
StartSeq: startSeq,
|
||||
}, man)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
if resumeMeta != nil {
|
||||
if err := w.Resume(resumeMeta); err != nil {
|
||||
return archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
}
|
||||
a.writer = w
|
||||
a.fileSink = &sink.File{W: w}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) openScanner(ctx context.Context) error {
|
||||
opts := scan.Options{
|
||||
DB: a.srcDB,
|
||||
DBName: a.srcCfg.DBName,
|
||||
Table: a.cfg.Table,
|
||||
Where: a.cfg.Where,
|
||||
SelectCols: a.probe.selectCols,
|
||||
KeyCols: a.probe.key.Columns,
|
||||
KeyIdx: a.probe.keyIdx,
|
||||
ChunkSize: a.cfg.ChunkSize,
|
||||
Shard: a.probe.shard,
|
||||
ShardIdx: a.probe.shardIdx,
|
||||
ShardLoc: a.loc,
|
||||
ZeroDate: a.cfg.ZeroDate,
|
||||
Resume: a.resume,
|
||||
}
|
||||
switch a.probe.path {
|
||||
case "A":
|
||||
a.scanner = scan.NewDateA(opts)
|
||||
case "B":
|
||||
s, err := scan.NewDateB(ctx, opts)
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindConnection, err)
|
||||
}
|
||||
a.scanner = s
|
||||
default:
|
||||
a.scanner = scan.NewPK(opts)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) execute(ctx context.Context) error {
|
||||
if a.cfg.HasFile() {
|
||||
a.message("note: file output is at-least-once; a crash and rerun may duplicate rows in shard files")
|
||||
}
|
||||
if a.cfg.Purge {
|
||||
a.message("warning: --purge deletes archived rows from the source table")
|
||||
}
|
||||
if a.cfg.BulkDelete {
|
||||
a.message("warning: --bulk-delete removes all rows in the scanned key range that match --where, including concurrent inserts")
|
||||
}
|
||||
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return a.interrupt(err)
|
||||
}
|
||||
if a.lagChecker != nil {
|
||||
if err := a.lagChecker.Wait(ctx); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return a.interrupt(err)
|
||||
}
|
||||
return archererr.Wrap(archererr.KindConnection, err)
|
||||
}
|
||||
}
|
||||
a.phase("scan", "")
|
||||
start := time.Now()
|
||||
chunk, err := a.scanner.Next(ctx)
|
||||
a.stats.ScanDur += time.Since(start)
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return a.interrupt(err)
|
||||
}
|
||||
return archererr.Wrap(archererr.KindOther, fmt.Errorf("scan: %w", err))
|
||||
}
|
||||
if err := a.processChunk(ctx, chunk); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return a.interrupt(err)
|
||||
}
|
||||
return err
|
||||
}
|
||||
a.stats.RowsRead += int64(chunk.Len())
|
||||
a.publishProgress(chunk)
|
||||
}
|
||||
return a.finish()
|
||||
}
|
||||
|
||||
func (a *App) processChunk(ctx context.Context, chunk *row.Chunk) error {
|
||||
fileStart := time.Now()
|
||||
if a.fileSink != nil {
|
||||
a.phase("file", "")
|
||||
if err := a.fileSink.WriteChunk(ctx, chunk); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
if a.cfg.Purge {
|
||||
if err := a.fileSink.Checkpoint(); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
a.stats.FileDur += time.Since(fileStart)
|
||||
|
||||
dbStart := time.Now()
|
||||
if a.destSink != nil || a.purger != nil {
|
||||
a.phase("db", "")
|
||||
}
|
||||
if err := a.runDB(ctx, chunk); err != nil {
|
||||
return err
|
||||
}
|
||||
a.stats.DBDur += time.Since(dbStart)
|
||||
|
||||
if a.fileSink != nil && !a.cfg.Purge {
|
||||
if err := a.fileSink.Flush(ctx); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
}
|
||||
|
||||
a.stats.RowsCommitted += int64(chunk.Len())
|
||||
if a.cfg.HasFile() {
|
||||
a.phase("checkpoint", "")
|
||||
}
|
||||
cpStart := time.Now()
|
||||
if err := a.saveState(false); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, fmt.Errorf("save checkpoint: %w", err))
|
||||
}
|
||||
a.stats.CheckpointDur += time.Since(cpStart)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) runDB(ctx context.Context, chunk *row.Chunk) error {
|
||||
attempts := a.cfg.Retries + 1
|
||||
if a.destSink != nil && a.purger != nil && a.sameInstance {
|
||||
err := retry(ctx, attempts, func() error {
|
||||
tx, err := a.destDB.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := a.destSink.InsertChunk(ctx, tx, chunk)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
return err
|
||||
}
|
||||
d, err := a.purger.DeleteChunk(ctx, tx, chunk)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
a.stats.RowsInserted += n
|
||||
a.stats.RowsDeleted += d
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if a.destSink != nil {
|
||||
err := retry(ctx, attempts, func() error {
|
||||
tx, err := a.destDB.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := a.destSink.InsertChunk(ctx, tx, chunk)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
a.stats.RowsInserted += n
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
}
|
||||
if a.purger != nil {
|
||||
err := retry(ctx, attempts, func() error {
|
||||
tx, err := a.srcDB.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d, err := a.purger.DeleteChunk(ctx, tx, chunk)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
a.stats.RowsDeleted += d
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func retry(ctx context.Context, attempts int, fn func() error) error {
|
||||
var err error
|
||||
for i := 0; i < attempts; i++ {
|
||||
if i > 0 {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Duration(i) * time.Second):
|
||||
}
|
||||
}
|
||||
if err = fn(); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (a *App) saveState(complete bool) error {
|
||||
if a.cfg.FileDir == "" || a.st == nil || a.alreadyComplete {
|
||||
return nil
|
||||
}
|
||||
st := a.st
|
||||
st.ConfigHash = a.cfgHash
|
||||
st.Config = a.cfgFields
|
||||
st.ShardBy = a.cfg.ShardBy
|
||||
st.RowsDone = a.baseRows + a.stats.RowsCommitted
|
||||
if a.scanner != nil {
|
||||
cs := a.scanner.ResumeState()
|
||||
st.Cursor = checkpoint.Cursor{Phase: cs.Phase, Key: cs.Key, DayStart: cs.DayStart}
|
||||
}
|
||||
if a.writer != nil {
|
||||
st.LastBucket, st.LastShardSeq = a.writer.StatePosition()
|
||||
}
|
||||
st.Complete = complete
|
||||
return checkpoint.Save(a.cfg.FileDir, st)
|
||||
}
|
||||
|
||||
func (a *App) finish() error {
|
||||
a.phase("finalize", "")
|
||||
if a.writer != nil {
|
||||
if err := a.writer.Close(); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, err)
|
||||
}
|
||||
var shards, bytes int64
|
||||
_, shards, bytes = a.writer.Stats()
|
||||
a.stats.Shards, a.stats.Bytes = shards, bytes
|
||||
}
|
||||
if err := a.saveState(true); err != nil {
|
||||
return archererr.Wrap(archererr.KindWrite, fmt.Errorf("save checkpoint: %w", err))
|
||||
}
|
||||
a.printStats()
|
||||
if a.reporter != nil {
|
||||
if a.writer != nil {
|
||||
a.reporter.SetShards(a.stats.Shards)
|
||||
a.reporter.SetBytes(a.stats.Bytes)
|
||||
}
|
||||
a.reporter.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) interrupt(cause error) error {
|
||||
if a.reporter != nil {
|
||||
a.reporter.Interrupted()
|
||||
}
|
||||
if a.writer != nil {
|
||||
if err := a.writer.SealCurrent(); err != nil {
|
||||
a.message("warning: could not seal current shard: %v", err)
|
||||
}
|
||||
}
|
||||
if err := a.saveState(false); err != nil {
|
||||
a.message("warning: could not save checkpoint: %v", err)
|
||||
}
|
||||
return archererr.Wrap(archererr.KindInterrupted, cause)
|
||||
}
|
||||
|
||||
func (a *App) publishProgress(chunk *row.Chunk) {
|
||||
if a.reporter == nil {
|
||||
return
|
||||
}
|
||||
if a.estimate != nil {
|
||||
switch a.estimate.Kind {
|
||||
case "day":
|
||||
if len(chunk.Buckets) > 0 {
|
||||
b := chunk.Buckets[len(chunk.Buckets)-1]
|
||||
if b != "" && b != "_null" && !strings.HasPrefix(b, "0000-") {
|
||||
if t, err := time.ParseInLocation("2006-01-02", b, a.loc); err == nil {
|
||||
a.reporter.SetPosition(float64(t.Unix()))
|
||||
}
|
||||
}
|
||||
}
|
||||
case "key":
|
||||
if len(chunk.LastKey) == 1 && !chunk.LastKey[0].Null {
|
||||
if n, err := strconv.ParseInt(chunk.LastKey[0].Val, 10, 64); err == nil {
|
||||
a.reporter.SetPosition(float64(n))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if a.writer != nil {
|
||||
a.reporter.SetShard(a.writer.CurrentShard())
|
||||
a.reporter.SetBytes(a.writer.WrittenBytes())
|
||||
}
|
||||
a.reporter.AddRows(int64(chunk.Len()))
|
||||
}
|
||||
|
||||
func (a *App) printStats() {
|
||||
if !a.cfg.Statistics {
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(os.Stdout, "this run:")
|
||||
fmt.Fprintf(os.Stdout, " rows read: %d\n", a.stats.RowsRead)
|
||||
fmt.Fprintf(os.Stdout, " rows committed: %d\n", a.stats.RowsCommitted)
|
||||
fmt.Fprintf(os.Stdout, " rows inserted: %d\n", a.stats.RowsInserted)
|
||||
fmt.Fprintf(os.Stdout, " rows deleted: %d\n", a.stats.RowsDeleted)
|
||||
if a.writer != nil {
|
||||
fmt.Fprintf(os.Stdout, " shards: %d\n", a.stats.Shards)
|
||||
fmt.Fprintf(os.Stdout, " bytes written: %d\n", a.stats.Bytes)
|
||||
}
|
||||
fmt.Fprintf(os.Stdout, " elapsed: %s (scan %s, file %s, db %s, checkpoint %s)\n",
|
||||
time.Since(a.stats.Started).Round(time.Millisecond),
|
||||
a.stats.ScanDur.Round(time.Millisecond), a.stats.FileDur.Round(time.Millisecond),
|
||||
a.stats.DBDur.Round(time.Millisecond), a.stats.CheckpointDur.Round(time.Millisecond))
|
||||
|
||||
fmt.Fprintln(os.Stdout, "cumulative (including prior runs):")
|
||||
fmt.Fprintf(os.Stdout, " rows read: %d\n", a.baseRows+a.stats.RowsRead)
|
||||
fmt.Fprintf(os.Stdout, " rows committed: %d\n", a.baseRows+a.stats.RowsCommitted)
|
||||
if a.writer != nil && a.manifest != nil {
|
||||
var rows, bytes int64
|
||||
entries := a.manifest.Entries()
|
||||
for _, e := range entries {
|
||||
rows += e.Rows
|
||||
bytes += e.Bytes
|
||||
}
|
||||
fmt.Fprintf(os.Stdout, " shards: %d\n", len(entries))
|
||||
fmt.Fprintf(os.Stdout, " rows written: %d\n", rows)
|
||||
fmt.Fprintf(os.Stdout, " bytes written: %d\n", bytes)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) close() {
|
||||
if a.scanner != nil {
|
||||
_ = a.scanner.Close()
|
||||
}
|
||||
if a.writer != nil {
|
||||
_ = a.writer.Abandon()
|
||||
} else if a.manifest != nil {
|
||||
_ = a.manifest.Close()
|
||||
}
|
||||
if a.lagChecker != nil {
|
||||
_ = a.lagChecker.Close()
|
||||
}
|
||||
if a.destDB != nil {
|
||||
_ = a.destDB.Close()
|
||||
}
|
||||
if a.srcDB != nil {
|
||||
_ = a.srcDB.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func openDB(cfg *mysql.Config) (*sql.DB, error) {
|
||||
db, err := sql.Open("mysql", cfg.FormatDSN())
|
||||
if err != nil {
|
||||
return nil, archererr.Wrap(archererr.KindConnection, err)
|
||||
}
|
||||
db.SetMaxOpenConns(8)
|
||||
db.SetMaxIdleConns(4)
|
||||
db.SetConnMaxLifetime(time.Hour)
|
||||
if err := db.Ping(); err != nil {
|
||||
db.Close()
|
||||
return nil, archererr.Wrap(archererr.KindConnection, fmt.Errorf("connect %s: %w", cfg.Addr, err))
|
||||
}
|
||||
return db, nil
|
||||
}
|
||||
|
||||
func newRunID() string {
|
||||
b := make([]byte, 8)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return time.Now().UTC().Format("20060102T150405Z")
|
||||
}
|
||||
return time.Now().UTC().Format("20060102T150405Z") + "-" + hex.EncodeToString(b)
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
|
||||
"archer/internal/purge"
|
||||
"archer/internal/shard"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
func (a *App) printDryRun(w io.Writer) error {
|
||||
cfg := a.cfg
|
||||
comp, err := shard.NewCompressor(cfg.Compress, cfg.CompressLevel)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(w, "== resolved configuration ==")
|
||||
fmt.Fprintf(w, "source: %s\n", redactDSN(cfg.SourceDSN))
|
||||
if cfg.DestDSN != "" {
|
||||
fmt.Fprintf(w, "dest: %s\n", redactDSN(cfg.DestDSN))
|
||||
}
|
||||
fmt.Fprintf(w, "table: %s\n", cfg.Table)
|
||||
if cfg.DestDSN != "" {
|
||||
fmt.Fprintf(w, "dest table: %s\n", cfg.DestTable)
|
||||
}
|
||||
fmt.Fprintf(w, "where: %s\n", cfg.Where)
|
||||
if len(cfg.Columns) > 0 {
|
||||
fmt.Fprintf(w, "columns: %s\n", strings.Join(cfg.Columns, ", "))
|
||||
} else {
|
||||
fmt.Fprintf(w, "columns: (all)\n")
|
||||
}
|
||||
targets := []string{}
|
||||
if cfg.HasFile() {
|
||||
targets = append(targets, "file="+cfg.FileDir)
|
||||
}
|
||||
if cfg.DestDSN != "" {
|
||||
targets = append(targets, "dest")
|
||||
}
|
||||
if cfg.Purge {
|
||||
targets = append(targets, "purge")
|
||||
}
|
||||
fmt.Fprintf(w, "targets: %s\n", strings.Join(targets, ", "))
|
||||
fmt.Fprintf(w, "chunk size: %d\n", cfg.ChunkSize)
|
||||
fmt.Fprintf(w, "shard by: %s\n", cfg.ShardBy)
|
||||
if cfg.ShardBy != "none" {
|
||||
fmt.Fprintf(w, "shard rows: %d\n", cfg.ShardRows)
|
||||
}
|
||||
if a.probe.shard != nil {
|
||||
fmt.Fprintf(w, "shard column: %s (%s)\n", a.probe.shard.Column, a.probe.shard.Type)
|
||||
fmt.Fprintf(w, "shard time zone: %s\n", cfg.ShardTimeZone)
|
||||
}
|
||||
fmt.Fprintf(w, "format: %s\n", cfg.Format)
|
||||
if comp.Name == "none" {
|
||||
fmt.Fprintf(w, "compress: none\n")
|
||||
} else {
|
||||
fmt.Fprintf(w, "compress: %s (level %d)\n", comp.Name, comp.Level)
|
||||
}
|
||||
if cfg.InsertMode != "" {
|
||||
fmt.Fprintf(w, "insert mode: %s\n", cfg.InsertMode)
|
||||
}
|
||||
fmt.Fprintf(w, "max lag: %s (check interval %s)\n", cfg.MaxLag, cfg.CheckInterval)
|
||||
fmt.Fprintf(w, "retries: %d\n", cfg.Retries)
|
||||
fmt.Fprintf(w, "safe auto inc: %t\n", cfg.SafeAutoIncrement)
|
||||
fmt.Fprintf(w, "config hash: %s\n", a.cfgHash)
|
||||
|
||||
fmt.Fprintln(w, "\n== index probe ==")
|
||||
fmt.Fprintf(w, "scan key: %s (%s)\n", a.probe.key.Name, strings.Join(a.probe.key.Columns, ", "))
|
||||
if a.probe.path == "A" {
|
||||
fmt.Fprintf(w, "scan path: A (keyset chunks using index %s)\n", a.probe.joint.Name)
|
||||
} else if a.probe.path == "B" {
|
||||
if a.probe.single != nil {
|
||||
fmt.Fprintf(w, "scan path: B (day-at-a-time using index %s)\n", a.probe.single.Name)
|
||||
} else {
|
||||
fmt.Fprintln(w, "scan path: B (day-at-a-time, full table scan allowed)")
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintln(w, "scan path: primary key seek")
|
||||
}
|
||||
|
||||
fmt.Fprintln(w, "\n== first SELECT ==")
|
||||
for _, q := range a.dryRunSelects() {
|
||||
fmt.Fprintf(w, " %s\n", q)
|
||||
}
|
||||
|
||||
fmt.Fprintln(w, "\n== write statements ==")
|
||||
if cfg.DestDSN != "" {
|
||||
fmt.Fprintf(w, " %s\n", a.dryRunInsert())
|
||||
}
|
||||
if cfg.Purge {
|
||||
p := purge.New(nil, a.srcCfg.DBName, cfg.Table, a.probe.key.Columns, cfg.Where, cfg.BulkDelete, cfg.SafeAutoIncrement, false)
|
||||
for _, q := range p.DryRunSQL() {
|
||||
fmt.Fprintf(w, " %s\n", q)
|
||||
}
|
||||
}
|
||||
if cfg.HasFile() {
|
||||
fmt.Fprintf(w, " file: %s/[...]<seq>.<pk_lo>-<pk_hi>.%s%s (written as .partial until sealed)\n",
|
||||
cfg.FileDir, cfg.Format, comp.Ext)
|
||||
}
|
||||
fmt.Fprintln(w, "\nnote: file output is at-least-once; a crash and rerun may duplicate rows in shard files")
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *App) dryRunSelects() []string {
|
||||
cols := sqlutil.Columns(a.probe.selectCols)
|
||||
table := sqlutil.Qualified(a.srcCfg.DBName, a.cfg.Table)
|
||||
where := a.cfg.Where
|
||||
key := sqlutil.Columns(a.probe.key.Columns)
|
||||
limit := fmt.Sprintf(" LIMIT %d", a.cfg.ChunkSize)
|
||||
switch a.probe.path {
|
||||
case "A":
|
||||
shard := sqlutil.Ident(a.probe.shard.Column)
|
||||
colsWithShard := sqlutil.Columns(append([]string{a.probe.shard.Column}, a.probe.key.Columns...))
|
||||
return []string{
|
||||
fmt.Sprintf("SELECT %s FROM %s WHERE (%s) AND %s IS NULL ORDER BY %s%s", cols, table, where, shard, key, limit),
|
||||
fmt.Sprintf("SELECT %s FROM %s WHERE (%s) AND %s IS NOT NULL AND (%s) > (?, ...) ORDER BY %s%s", cols, table, where, shard, colsWithShard, colsWithShard, limit),
|
||||
}
|
||||
case "B":
|
||||
shard := sqlutil.Ident(a.probe.shard.Column)
|
||||
colsWithShard := sqlutil.Columns(append([]string{a.probe.shard.Column}, a.probe.key.Columns...))
|
||||
return []string{
|
||||
fmt.Sprintf("SELECT %s FROM %s WHERE (%s) AND %s IS NULL ORDER BY %s", cols, table, where, shard, key),
|
||||
fmt.Sprintf("SELECT %s FROM %s WHERE (%s) AND %s >= ? AND %s < ? ORDER BY %s", cols, table, where, shard, shard, colsWithShard),
|
||||
}
|
||||
default:
|
||||
return []string{fmt.Sprintf("SELECT %s FROM %s WHERE (%s) ORDER BY %s%s", cols, table, where, key, limit)}
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) dryRunInsert() string {
|
||||
destCfg, err := a.cfg.DestConfig()
|
||||
dbName := ""
|
||||
if err == nil && destCfg != nil {
|
||||
dbName = destCfg.DBName
|
||||
}
|
||||
verb := "INSERT INTO"
|
||||
switch a.cfg.InsertMode {
|
||||
case "ignore":
|
||||
verb = "INSERT IGNORE INTO"
|
||||
case "replace":
|
||||
verb = "REPLACE INTO"
|
||||
}
|
||||
names := make([]string, len(a.probe.outCols))
|
||||
for i, c := range a.probe.outCols {
|
||||
names[i] = c.Name
|
||||
}
|
||||
tuple := sqlutil.ParamTuple(len(names))
|
||||
return fmt.Sprintf("%s %s (%s) VALUES %s, %s, ...",
|
||||
verb, sqlutil.Qualified(dbName, a.cfg.DestTable), sqlutil.Columns(names), tuple, tuple)
|
||||
}
|
||||
|
||||
func redactDSN(dsn string) string {
|
||||
c, err := mysql.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
return "<invalid dsn>"
|
||||
}
|
||||
c.Passwd = ""
|
||||
return c.FormatDSN()
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"archer/internal/estimate"
|
||||
"archer/internal/progress"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// estimateTimeout bounds the one-off MIN/MAX query at startup.
|
||||
const estimateTimeout = 2 * time.Second
|
||||
|
||||
// estimateSpec decides which cheap position metric this run can expose.
|
||||
func (a *App) estimateSpec() estimate.Spec {
|
||||
if a.cfg.IsDateShard() && a.probe.shard != nil {
|
||||
return estimate.Spec{
|
||||
Kind: "day",
|
||||
Column: a.probe.shard.Column,
|
||||
ShardKind: a.probe.shard.Kind,
|
||||
Unit: a.probe.shard.Unit,
|
||||
Loc: a.loc,
|
||||
}
|
||||
}
|
||||
types := make([]string, len(a.probe.key.Columns))
|
||||
for i, name := range a.probe.key.Columns {
|
||||
c, _ := a.probe.table.Column(name)
|
||||
types[i] = c.DataType
|
||||
}
|
||||
col, ok := estimate.KeyColumn(a.probe.key.Columns, types)
|
||||
if !ok {
|
||||
return estimate.Spec{}
|
||||
}
|
||||
return estimate.Spec{Kind: "key", Column: col}
|
||||
}
|
||||
|
||||
// queryEstimate runs the bounded range query on a dedicated connection and
|
||||
// silently returns nil when the metric does not apply or the query fails.
|
||||
func (a *App) queryEstimate(ctx context.Context) *progress.Estimate {
|
||||
spec := a.estimateSpec()
|
||||
if spec.Kind == "" {
|
||||
return nil
|
||||
}
|
||||
conn, err := a.srcDB.Conn(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer conn.Close()
|
||||
query := func(qctx context.Context, column string) (sql.NullString, sql.NullString, error) {
|
||||
col := sqlutil.Ident(column)
|
||||
q := fmt.Sprintf("SELECT MIN(%s), MAX(%s) FROM %s WHERE (%s)",
|
||||
col, col, sqlutil.Qualified(a.srcCfg.DBName, a.cfg.Table), a.cfg.Where)
|
||||
if spec.Kind == "day" {
|
||||
q += " AND " + col + " IS NOT NULL"
|
||||
}
|
||||
var minRaw, maxRaw sql.NullString
|
||||
err := conn.QueryRowContext(qctx, q).Scan(&minRaw, &maxRaw)
|
||||
return minRaw, maxRaw, err
|
||||
}
|
||||
return estimate.Bounded(ctx, query, spec, estimateTimeout)
|
||||
}
|
||||
@@ -0,0 +1,918 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
"github.com/klauspost/compress/zstd"
|
||||
"github.com/ulikunitz/xz"
|
||||
|
||||
"archer/internal/cli"
|
||||
)
|
||||
|
||||
func testDSN(t *testing.T) string {
|
||||
d := os.Getenv("ARCHER_TEST_DSN")
|
||||
if d == "" {
|
||||
t.Skip("ARCHER_TEST_DSN is not set; skipping MySQL integration test")
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
if dsn := os.Getenv("ARCHER_TEST_DSN"); dsn != "" {
|
||||
cfg, err := mysql.ParseDSN(dsn)
|
||||
if err == nil && cfg.DBName != "" {
|
||||
dbName := cfg.DBName
|
||||
cfg.DBName = ""
|
||||
db, err := sql.Open("mysql", cfg.FormatDSN())
|
||||
if err == nil {
|
||||
_, _ = db.Exec("CREATE DATABASE IF NOT EXISTS `" + dbName + "`")
|
||||
_, _ = db.Exec("SET GLOBAL local_infile=ON")
|
||||
_, _ = db.Exec("SET GLOBAL sql_mode=''")
|
||||
db.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func openRoot(t *testing.T, dsn string) *sql.DB {
|
||||
t.Helper()
|
||||
cfg, err := mysql.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := sql.Open("mysql", cfg.FormatDSN())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Ping(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return db
|
||||
}
|
||||
|
||||
func execSQL(t *testing.T, db *sql.DB, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := db.Exec(q, args...); err != nil {
|
||||
t.Fatalf("exec %q: %v", q, err)
|
||||
}
|
||||
}
|
||||
|
||||
var stdoutMu sync.Mutex
|
||||
var stderrMu sync.Mutex
|
||||
|
||||
func captureStdout(fn func() error) (string, error) {
|
||||
stdoutMu.Lock()
|
||||
defer stdoutMu.Unlock()
|
||||
old := os.Stdout
|
||||
r, w, err := os.Pipe()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
os.Stdout = w
|
||||
var buf bytes.Buffer
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
_, _ = io.Copy(&buf, r)
|
||||
close(done)
|
||||
}()
|
||||
runErr := fn()
|
||||
_ = w.Close()
|
||||
os.Stdout = old
|
||||
<-done
|
||||
return buf.String(), runErr
|
||||
}
|
||||
|
||||
func captureStderr(fn func() error) (string, error) {
|
||||
stderrMu.Lock()
|
||||
defer stderrMu.Unlock()
|
||||
old := os.Stderr
|
||||
r, w, err := os.Pipe()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
os.Stderr = w
|
||||
var buf bytes.Buffer
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
_, _ = io.Copy(&buf, r)
|
||||
close(done)
|
||||
}()
|
||||
runErr := fn()
|
||||
_ = w.Close()
|
||||
os.Stderr = old
|
||||
<-done
|
||||
return buf.String(), runErr
|
||||
}
|
||||
|
||||
func runCLI(t *testing.T, ctx context.Context, args ...string) (string, error) {
|
||||
t.Helper()
|
||||
cfg, err := cli.Parse(args, io.Discard)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return captureStdout(func() error { return Run(ctx, cfg) })
|
||||
}
|
||||
|
||||
func runCLIStderr(t *testing.T, ctx context.Context, args ...string) (string, error) {
|
||||
t.Helper()
|
||||
cfg, err := cli.Parse(args, io.Discard)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return captureStderr(func() error { return Run(ctx, cfg) })
|
||||
}
|
||||
|
||||
func baseCLI(dsn, table, out string, extra ...string) []string {
|
||||
args := []string{
|
||||
"--source", dsn,
|
||||
"--table", table,
|
||||
"--where", "1=1",
|
||||
"--file", out,
|
||||
}
|
||||
return append(args, extra...)
|
||||
}
|
||||
|
||||
func listFiles(t *testing.T, dir, pattern string) []string {
|
||||
t.Helper()
|
||||
matches, err := filepath.Glob(filepath.Join(dir, pattern))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sort.Strings(matches)
|
||||
return matches
|
||||
}
|
||||
|
||||
func dumpIDs(t *testing.T, path string) []int {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var ids []int
|
||||
for _, line := range strings.Split(strings.TrimRight(string(b), "\n"), "\n") {
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
id, err := strconv.Atoi(strings.SplitN(line, "\t", 2)[0])
|
||||
if err != nil {
|
||||
t.Fatalf("%s: bad line %q", path, line)
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func TestIntegrationCountShards(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_count")
|
||||
execSQL(t, db, "CREATE TABLE it_count (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
var b strings.Builder
|
||||
b.WriteString("INSERT INTO it_count (id, v) VALUES ")
|
||||
for i := 1; i <= 25; i++ {
|
||||
if i > 1 {
|
||||
b.WriteString(",")
|
||||
}
|
||||
fmt.Fprintf(&b, "(%d,'v%d')", i, i)
|
||||
}
|
||||
execSQL(t, db, b.String())
|
||||
|
||||
out := t.TempDir()
|
||||
_, err := runCLI(t, context.Background(), baseCLI(dsn, "it_count", out,
|
||||
"--shard-by", "count", "--shard-rows", "10", "--chunk-size", "7", "--format", "dump")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := listFiles(t, out, "*.dump")
|
||||
if len(files) != 3 {
|
||||
t.Fatalf("got %d shard files: %v", len(files), files)
|
||||
}
|
||||
wantNames := []string{"000001.1-10.dump", "000002.11-20.dump", "000003.21-25.dump"}
|
||||
var got []int
|
||||
for i, f := range files {
|
||||
if filepath.Base(f) != wantNames[i] {
|
||||
t.Fatalf("file %d = %s want %s", i, filepath.Base(f), wantNames[i])
|
||||
}
|
||||
got = append(got, dumpIDs(t, f)...)
|
||||
}
|
||||
if len(got) != 25 {
|
||||
t.Fatalf("got %d rows", len(got))
|
||||
}
|
||||
for i, id := range got {
|
||||
if id != i+1 {
|
||||
t.Fatalf("row %d = %d, shards are not contiguous", i, id)
|
||||
}
|
||||
}
|
||||
|
||||
entries := readManifest(t, out)
|
||||
if len(entries) != 3 {
|
||||
t.Fatalf("manifest entries %d", len(entries))
|
||||
}
|
||||
for i, e := range entries {
|
||||
if e.Seq != int64(i+1) || e.Rows != int64(len(dumpIDs(t, filepath.Join(out, e.Path)))) {
|
||||
t.Fatalf("entry %d %+v", i, e)
|
||||
}
|
||||
if e.Bytes == 0 || e.SHA == "" {
|
||||
t.Fatalf("entry %d missing bytes/hash", i)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(out, "state.json")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var complete bool
|
||||
st, err := os.ReadFile(filepath.Join(out, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
complete = bytes.Contains(st, []byte(`"complete": true`))
|
||||
if !complete {
|
||||
t.Fatalf("state not complete: %s", st)
|
||||
}
|
||||
}
|
||||
|
||||
type manifestEntry struct {
|
||||
Seq int64 `json:"seq"`
|
||||
Path string `json:"path"`
|
||||
Date string `json:"date"`
|
||||
Rows int64 `json:"rows"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
SHA string `json:"sha256"`
|
||||
PKLo string `json:"pk_lo"`
|
||||
PKHi string `json:"pk_hi"`
|
||||
}
|
||||
|
||||
func readManifest(t *testing.T, dir string) []manifestEntry {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(dir, "manifest.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out []manifestEntry
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") {
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
var e manifestEntry
|
||||
if err := json.Unmarshal([]byte(line), &e); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestIntegrationDatePathsAgree(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
for _, spec := range []struct{ name, index string }{
|
||||
{"it_date_a", "KEY da (created_at, id)"},
|
||||
{"it_date_b", "KEY db (created_at)"},
|
||||
} {
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS "+spec.name)
|
||||
execSQL(t, db, "CREATE TABLE "+spec.name+` (
|
||||
id BIGINT NOT NULL,
|
||||
v VARCHAR(32),
|
||||
created_at DATETIME NULL,
|
||||
PRIMARY KEY (id),
|
||||
`+spec.index+`)`)
|
||||
execSQL(t, db, `INSERT INTO `+spec.name+` (id, v, created_at) VALUES
|
||||
(1,'a','2025-01-01 10:00:00'),
|
||||
(2,'b','2025-01-01 12:00:00'),
|
||||
(3,'c','2025-01-02 01:00:00'),
|
||||
(4,'d',NULL),
|
||||
(5,'e','2025-01-01 11:00:00'),
|
||||
(6,'f','2025-01-03 00:00:00')`)
|
||||
}
|
||||
outA := t.TempDir()
|
||||
outB := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_date_a", outA,
|
||||
"--shard-by", "date", "--shard-column", "created_at", "--format", "dump", "--chunk-size", "2")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_date_b", outB,
|
||||
"--shard-by", "date", "--shard-column", "created_at", "--format", "dump", "--chunk-size", "2")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a := readTree(t, outA)
|
||||
b := readTree(t, outB)
|
||||
if len(a) != len(b) {
|
||||
t.Fatalf("file counts differ: %d vs %d (%v / %v)", len(a), len(b), a, b)
|
||||
}
|
||||
for k, va := range a {
|
||||
if b[k] != va {
|
||||
t.Fatalf("%s:\nA=%q\nB=%q", k, va, b[k])
|
||||
}
|
||||
}
|
||||
if _, ok := a["date=_null/000001.4-4.dump"]; !ok {
|
||||
t.Fatalf("NULL row not routed to _null: %v", a)
|
||||
}
|
||||
if len(a) != 4 {
|
||||
t.Fatalf("expected 4 shards (null + 3 days), got %v", a)
|
||||
}
|
||||
}
|
||||
|
||||
func readTree(t *testing.T, dir string) map[string]string {
|
||||
t.Helper()
|
||||
out := map[string]string{}
|
||||
err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
if !strings.HasSuffix(d.Name(), ".dump") {
|
||||
return nil
|
||||
}
|
||||
rel, _ := filepath.Rel(dir, path)
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out[filepath.ToSlash(rel)] = string(b)
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestIntegrationZeroDateAndNull(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_zero")
|
||||
execSQL(t, db, `CREATE TABLE it_zero (
|
||||
id BIGINT NOT NULL,
|
||||
d DATE NULL,
|
||||
PRIMARY KEY (id),
|
||||
KEY dz (d, id))`)
|
||||
execSQL(t, db, `INSERT INTO it_zero (id, d) VALUES
|
||||
(1, NULL), (2, '0000-00-00'), (3, '2024-01-01'), (4, '2024-01-01')`)
|
||||
|
||||
out := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_zero", out,
|
||||
"--shard-by", "date", "--shard-column", "d")...); err == nil {
|
||||
t.Fatal("expected zero-date error")
|
||||
} else if !strings.Contains(err.Error(), "zero date") {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
out = t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_zero", out,
|
||||
"--shard-by", "date", "--shard-column", "d", "--zero-date", "bucket")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tree := readTree(t, out)
|
||||
for _, want := range []string{
|
||||
"date=_null/000001.1-1.dump",
|
||||
"date=0000-00-00/000001.2-2.dump",
|
||||
"date=2024-01-01/000001.3-4.dump",
|
||||
} {
|
||||
if _, ok := tree[want]; !ok {
|
||||
t.Fatalf("missing %s in %v", want, tree)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationEpochUnits(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_epoch")
|
||||
execSQL(t, db, `CREATE TABLE it_epoch (
|
||||
id BIGINT NOT NULL,
|
||||
ts BIGINT NOT NULL,
|
||||
PRIMARY KEY (id),
|
||||
KEY e (ts, id))`)
|
||||
execSQL(t, db, `INSERT INTO it_epoch (id, ts) VALUES
|
||||
(1, 1717200000), (2, 1717200001)`)
|
||||
|
||||
for _, tc := range []struct {
|
||||
unit string
|
||||
vals string
|
||||
}{
|
||||
{"epoch-seconds", "(3, 1717200000), (4, 1717200001)"},
|
||||
{"epoch-milliseconds", "(3, 1717200000000), (4, 1717200001000)"},
|
||||
{"epoch-microseconds", "(3, 1717200000000000), (4, 1717200001000000)"},
|
||||
} {
|
||||
table := "it_epoch_" + strings.Split(tc.unit, "-")[1]
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS "+table)
|
||||
execSQL(t, db, `CREATE TABLE `+table+` (
|
||||
id BIGINT NOT NULL, ts BIGINT NOT NULL,
|
||||
PRIMARY KEY (id), KEY e (ts, id))`)
|
||||
execSQL(t, db, "INSERT INTO "+table+" (id, ts) VALUES "+tc.vals)
|
||||
out := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, table, out,
|
||||
"--shard-by", "date", "--shard-column", "ts", "--shard-column-type", tc.unit,
|
||||
"--shard-time-zone", "UTC")...); err != nil {
|
||||
t.Fatalf("%s: %v", tc.unit, err)
|
||||
}
|
||||
tree := readTree(t, out)
|
||||
if len(tree) != 1 {
|
||||
t.Fatalf("%s: expected one date bucket, got %v", tc.unit, tree)
|
||||
}
|
||||
for k := range tree {
|
||||
if !strings.Contains(k, "date=2024-06-01") {
|
||||
t.Fatalf("%s: wrong bucket %s", tc.unit, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationNoIndexFailFast(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_noindex")
|
||||
execSQL(t, db, `CREATE TABLE it_noindex (
|
||||
id BIGINT NOT NULL, d DATETIME NULL, PRIMARY KEY (id))`)
|
||||
execSQL(t, db, `INSERT INTO it_noindex (id, d) VALUES (1,'2024-01-01 00:00:00')`)
|
||||
|
||||
out := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_noindex", out,
|
||||
"--shard-by", "date", "--shard-column", "d")...); err == nil {
|
||||
t.Fatal("expected no-index error")
|
||||
} else if !strings.Contains(err.Error(), "no usable index") {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
out = t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_noindex", out,
|
||||
"--shard-by", "date", "--shard-column", "d", "--allow-full-scan")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(out, "date=2024-01-01")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationDryRunCreatesNothing(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_dry")
|
||||
execSQL(t, db, "CREATE TABLE it_dry (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
execSQL(t, db, "INSERT INTO it_dry (id, v) VALUES (1,'a')")
|
||||
|
||||
out := filepath.Join(t.TempDir(), "never-created")
|
||||
stdout, err := runCLI(t, context.Background(), baseCLI(dsn, "it_dry", out, "--dry-run")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, statErr := os.Stat(out); !os.IsNotExist(statErr) {
|
||||
t.Fatal("dry-run created the output directory")
|
||||
}
|
||||
if !strings.Contains(stdout, "== first SELECT ==") || !strings.Contains(stdout, "config hash") {
|
||||
t.Fatalf("dry-run output incomplete:\n%s", stdout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationResumeAfterInterrupt(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_resume")
|
||||
execSQL(t, db, "CREATE TABLE it_resume (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
var b strings.Builder
|
||||
b.WriteString("INSERT INTO it_resume (id, v) VALUES ")
|
||||
const total = 20000
|
||||
for i := 1; i <= total; i++ {
|
||||
if i > 1 {
|
||||
b.WriteString(",")
|
||||
}
|
||||
fmt.Fprintf(&b, "(%d,'v')", i)
|
||||
}
|
||||
execSQL(t, db, b.String())
|
||||
|
||||
out := t.TempDir()
|
||||
args := baseCLI(dsn, "it_resume", out, "--shard-by", "count", "--shard-rows", "137",
|
||||
"--chunk-size", "50", "--progress", "100")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
stderrMu.Lock()
|
||||
old := os.Stderr
|
||||
r, w, _ := os.Pipe()
|
||||
os.Stderr = w
|
||||
cfg, err := cli.Parse(args, io.Discard)
|
||||
if err != nil {
|
||||
os.Stderr = old
|
||||
stderrMu.Unlock()
|
||||
t.Fatal(err)
|
||||
}
|
||||
firstErr := make(chan error, 1)
|
||||
go func() { firstErr <- Run(ctx, cfg) }()
|
||||
go func() {
|
||||
buf := make([]byte, 4096)
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
if n > 0 && strings.Contains(string(buf[:n]), "progress:") {
|
||||
cancel()
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
err = <-firstErr
|
||||
_ = w.Close()
|
||||
os.Stderr = old
|
||||
stderrMu.Unlock()
|
||||
if err == nil {
|
||||
t.Fatal("first run should have been interrupted")
|
||||
}
|
||||
|
||||
if _, err := runCLI(t, context.Background(), args...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
seen := map[int]bool{}
|
||||
for _, f := range listFiles(t, out, "*.dump") {
|
||||
for _, id := range dumpIDs(t, f) {
|
||||
seen[id] = true
|
||||
}
|
||||
}
|
||||
for i := 1; i <= total; i++ {
|
||||
if !seen[i] {
|
||||
t.Fatalf("row %d missing after resume", i)
|
||||
}
|
||||
}
|
||||
st, err := os.ReadFile(filepath.Join(out, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.Contains(st, []byte(`"complete": true`)) {
|
||||
t.Fatalf("state not complete: %s", st)
|
||||
}
|
||||
var saved struct {
|
||||
RowsDone int64 `json:"rows_done"`
|
||||
}
|
||||
if err := json.Unmarshal(st, &saved); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if saved.RowsDone != total {
|
||||
t.Fatalf("rows_done = %d after resume, want cumulative %d", saved.RowsDone, total)
|
||||
}
|
||||
if _, err := filepath.Glob(filepath.Join(out, "*.partial")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationProgressModes(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_progress")
|
||||
execSQL(t, db, "CREATE TABLE it_progress (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
execSQL(t, db, "INSERT INTO it_progress (id, v) VALUES (1,'a'),(2,'b'),(3,'c'),(4,'d'),(5,'e')")
|
||||
|
||||
for _, extra := range [][]string{
|
||||
nil,
|
||||
{"--progress", "off"},
|
||||
{"--progress", "0"},
|
||||
} {
|
||||
out := t.TempDir()
|
||||
args := baseCLI(dsn, "it_progress", out, extra...)
|
||||
errOut, err := runCLIStderr(t, context.Background(), args...)
|
||||
if err != nil {
|
||||
t.Fatalf("%v: %v", extra, err)
|
||||
}
|
||||
if strings.Contains(errOut, "progress:") || strings.Contains(errOut, "done rows=") {
|
||||
t.Fatalf("%v: unexpected progress output %q", extra, errOut)
|
||||
}
|
||||
}
|
||||
|
||||
out := t.TempDir()
|
||||
errOut, err := runCLIStderr(t, context.Background(), baseCLI(dsn, "it_progress", out, "--progress", "on")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(errOut, "done rows=5") {
|
||||
t.Fatalf("forced progress missing done summary: %q", errOut)
|
||||
}
|
||||
if strings.Contains(errOut, "\r") {
|
||||
t.Fatalf("redirected progress must not use carriage returns: %q", errOut)
|
||||
}
|
||||
|
||||
out = t.TempDir()
|
||||
errOut, err = runCLIStderr(t, context.Background(), baseCLI(dsn, "it_progress", out, "--progress", "1")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(errOut, "progress: rows=5") {
|
||||
t.Fatalf("row-threshold progress line missing: %q", errOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationEstimateMetrics(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_estimate")
|
||||
execSQL(t, db, `CREATE TABLE it_estimate (
|
||||
id BIGINT NOT NULL,
|
||||
v VARCHAR(32),
|
||||
created_at DATETIME NULL,
|
||||
PRIMARY KEY (id),
|
||||
KEY de (created_at, id))`)
|
||||
execSQL(t, db, `INSERT INTO it_estimate (id, v, created_at) VALUES
|
||||
(1,'a','2025-01-01 01:00:00'),
|
||||
(2,'b','2025-01-01 02:00:00'),
|
||||
(3,'c','2025-01-02 01:00:00'),
|
||||
(4,'d','2025-01-03 01:00:00'),
|
||||
(5,'e','2025-01-03 02:00:00')`)
|
||||
|
||||
out := t.TempDir()
|
||||
errOut, err := runCLIStderr(t, context.Background(), baseCLI(dsn, "it_estimate", out, "--progress", "1")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(errOut, "estimate=key") {
|
||||
t.Fatalf("primary key estimate missing: %q", errOut)
|
||||
}
|
||||
|
||||
out = t.TempDir()
|
||||
errOut, err = runCLIStderr(t, context.Background(),
|
||||
baseCLI(dsn, "it_estimate", out, "--progress", "1", "--shard-by", "date", "--shard-column", "created_at")...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(errOut, "estimate=day") {
|
||||
t.Fatalf("day estimate missing: %q", errOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationImportDump(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_import_src")
|
||||
execSQL(t, db, "CREATE TABLE it_import_src (id BIGINT NOT NULL, v VARCHAR(64), PRIMARY KEY (id))")
|
||||
execSQL(t, db, `INSERT INTO it_import_src (id, v) VALUES
|
||||
(1,'plain'), (2,'tab\there'), (3,'line\nbreak'), (4,'back\\slash')`)
|
||||
|
||||
for _, comp := range []string{"none", "gzip", "zstd", "xz"} {
|
||||
out := t.TempDir()
|
||||
args := baseCLI(dsn, "it_import_src", out, "--format", "dump", "--compress", comp)
|
||||
if _, err := runCLI(t, context.Background(), args...); err != nil {
|
||||
t.Fatalf("%s: %v", comp, err)
|
||||
}
|
||||
files := listFiles(t, out, "*.dump*")
|
||||
if len(files) != 1 {
|
||||
t.Fatalf("%s: files %v", comp, files)
|
||||
}
|
||||
loadPath := files[0]
|
||||
if comp != "none" {
|
||||
loadPath = filepath.Join(t.TempDir(), "decompressed.dump")
|
||||
if err := decompressFile(files[0], loadPath, comp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_import_dst")
|
||||
execSQL(t, db, "CREATE TABLE it_import_dst (id BIGINT NOT NULL, v VARCHAR(64), PRIMARY KEY (id))")
|
||||
loadLocal(t, dsn, loadPath, "it_import_dst")
|
||||
var n int
|
||||
if err := db.QueryRow("SELECT COUNT(*) FROM it_import_dst").Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n != 4 {
|
||||
t.Fatalf("%s: loaded %d rows", comp, n)
|
||||
}
|
||||
var v string
|
||||
if err := db.QueryRow("SELECT v FROM it_import_dst WHERE id=3").Scan(&v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v != "line\nbreak" {
|
||||
t.Fatalf("%s: round trip mismatch %q", comp, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func decompressFile(src, dst, comp string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
var r io.Reader
|
||||
switch comp {
|
||||
case "gzip":
|
||||
gr, err := gzip.NewReader(in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer gr.Close()
|
||||
r = gr
|
||||
case "zstd":
|
||||
zr, err := zstd.NewReader(in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer zr.Close()
|
||||
r = zr
|
||||
case "xz":
|
||||
xr, err := xz.NewReader(in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r = xr
|
||||
default:
|
||||
r = in
|
||||
}
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer out.Close()
|
||||
_, err = io.Copy(out, r)
|
||||
return err
|
||||
}
|
||||
|
||||
func loadLocal(t *testing.T, dsn, path, table string) {
|
||||
t.Helper()
|
||||
cfg, err := mysql.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.AllowAllFiles = true
|
||||
db, err := sql.Open("mysql", cfg.FormatDSN())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
p := filepath.ToSlash(path)
|
||||
p = strings.ReplaceAll(p, `'`, `''`)
|
||||
q := fmt.Sprintf("LOAD DATA LOCAL INFILE '%s' INTO TABLE %s", p, table)
|
||||
if _, err := db.Exec(q); err != nil {
|
||||
t.Fatalf("load data: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationCSVAndJSONL(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_fmt")
|
||||
execSQL(t, db, `CREATE TABLE it_fmt (
|
||||
id BIGINT NOT NULL, name VARCHAR(64), price DECIMAL(10,2), created DATETIME,
|
||||
data BLOB, PRIMARY KEY (id))`)
|
||||
execSQL(t, db, `INSERT INTO it_fmt (id, name, price, created, data) VALUES
|
||||
(1,'a,\"b\"',12.30,'2024-01-02 03:04:05',X'6869'), (2,'',NULL,NULL,NULL)`)
|
||||
|
||||
// CSV
|
||||
out := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_fmt", out, "--format", "csv")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := listFiles(t, out, "*.csv")
|
||||
if len(files) != 1 {
|
||||
t.Fatalf("csv files %v", files)
|
||||
}
|
||||
f, err := os.Open(files[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
recs, err := csv.NewReader(f).ReadAll()
|
||||
f.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(recs) != 3 {
|
||||
t.Fatalf("csv records %d", len(recs))
|
||||
}
|
||||
if recs[0][0] != "id" || recs[0][1] != "name" {
|
||||
t.Fatalf("header %v", recs[0])
|
||||
}
|
||||
if recs[1][1] != `a,"b"` || recs[1][2] != "12.30" {
|
||||
t.Fatalf("row %v", recs[1])
|
||||
}
|
||||
if recs[2][2] != `\N` {
|
||||
t.Fatalf("NULL decimal %q", recs[2][2])
|
||||
}
|
||||
|
||||
// JSONL
|
||||
out = t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_fmt", out, "--format", "jsonl")...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files = listFiles(t, out, "*.jsonl")
|
||||
var shardFiles []string
|
||||
for _, f := range files {
|
||||
if filepath.Base(f) != "manifest.jsonl" {
|
||||
shardFiles = append(shardFiles, f)
|
||||
}
|
||||
}
|
||||
files = shardFiles
|
||||
if len(files) != 1 {
|
||||
t.Fatalf("jsonl files %v", files)
|
||||
}
|
||||
b, err := os.ReadFile(files[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
if len(lines) != 2 {
|
||||
t.Fatalf("jsonl lines %d", len(lines))
|
||||
}
|
||||
var obj map[string]any
|
||||
if err := json.Unmarshal([]byte(lines[0]), &obj); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if obj["price"] != "12.30" {
|
||||
t.Fatalf("price %v", obj["price"])
|
||||
}
|
||||
if obj["created"] != "2024-01-02T03:04:05Z" {
|
||||
t.Fatalf("created %v", obj["created"])
|
||||
}
|
||||
if obj["data"] != "aGk=" {
|
||||
t.Fatalf("data %v", obj["data"])
|
||||
}
|
||||
var second map[string]any
|
||||
if err := json.Unmarshal([]byte(lines[1]), &second); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if second["price"] != nil || second["created"] != nil {
|
||||
t.Fatalf("null mapping %v", second)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationPurgeAndDest(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_purge_src")
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_purge_dst")
|
||||
execSQL(t, db, "CREATE TABLE it_purge_src (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
execSQL(t, db, "CREATE TABLE it_purge_dst (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
execSQL(t, db, "INSERT INTO it_purge_src (id, v) VALUES (1,'a'),(2,'b'),(3,'c')")
|
||||
|
||||
out := t.TempDir()
|
||||
args := []string{
|
||||
"--source", dsn, "--table", "it_purge_src", "--where", "1=1",
|
||||
"--file", out, "--dest", dsn, "--dest-table", "it_purge_dst",
|
||||
"--purge", "--chunk-size", "2",
|
||||
}
|
||||
if _, err := runCLI(t, context.Background(), args...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var srcCount, dstCount int
|
||||
db.QueryRow("SELECT COUNT(*) FROM it_purge_src").Scan(&srcCount)
|
||||
db.QueryRow("SELECT COUNT(*) FROM it_purge_dst").Scan(&dstCount)
|
||||
if srcCount != 0 || dstCount != 3 {
|
||||
t.Fatalf("src=%d dst=%d", srcCount, dstCount)
|
||||
}
|
||||
if len(listFiles(t, out, "*.dump")) != 1 {
|
||||
t.Fatal("expected one shard")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationCheckpointConfigMismatch(t *testing.T) {
|
||||
dsn := testDSN(t)
|
||||
db := openRoot(t, dsn)
|
||||
defer db.Close()
|
||||
execSQL(t, db, "DROP TABLE IF EXISTS it_mismatch")
|
||||
execSQL(t, db, "CREATE TABLE it_mismatch (id BIGINT NOT NULL, v VARCHAR(32), PRIMARY KEY (id))")
|
||||
execSQL(t, db, "INSERT INTO it_mismatch (id, v) VALUES (1,'a'),(2,'b')")
|
||||
out := t.TempDir()
|
||||
if _, err := runCLI(t, context.Background(), baseCLI(dsn, "it_mismatch", out)...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Remove the complete flag so resume logic runs.
|
||||
stPath := filepath.Join(out, "state.json")
|
||||
b, _ := os.ReadFile(stPath)
|
||||
b = bytes.Replace(b, []byte(`"complete": true`), []byte(`"complete": false`), 1)
|
||||
if err := os.WriteFile(stPath, b, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := baseCLI(dsn, "it_mismatch", out)
|
||||
for i := range args {
|
||||
if args[i] == "1=1" {
|
||||
args[i] = "id > 0"
|
||||
}
|
||||
}
|
||||
if _, err := runCLI(t, context.Background(), args...); err == nil {
|
||||
t.Fatal("expected config mismatch error")
|
||||
} else if !strings.Contains(err.Error(), "different configuration") {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
args = append(args, "--force-resume")
|
||||
if _, err := runCLI(t, context.Background(), args...); err != nil {
|
||||
t.Fatalf("force resume: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"archer/internal/archererr"
|
||||
"archer/internal/cli"
|
||||
"archer/internal/row"
|
||||
"archer/internal/schema"
|
||||
)
|
||||
|
||||
// probeResult is everything learned about the source table needed to plan the
|
||||
// scan and the encoders.
|
||||
type probeResult struct {
|
||||
table *schema.Table
|
||||
key schema.KeyInfo
|
||||
shard *schema.ShardSpec
|
||||
path string // pk | A | B
|
||||
joint *schema.Index
|
||||
single *schema.Index
|
||||
selectCols []string
|
||||
selectMeta []row.Column
|
||||
keyIdx []int
|
||||
shardIdx int
|
||||
outCols []row.Column
|
||||
outIdx []int
|
||||
}
|
||||
|
||||
func probeTable(ctx context.Context, db *sql.DB, cfg *cli.Config, dbName string, loc *time.Location) (*probeResult, error) {
|
||||
table, err := schema.Load(ctx, db, dbName, cfg.Table)
|
||||
if err != nil {
|
||||
return nil, archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
key, err := table.ChooseKey(cfg.Index)
|
||||
if err != nil {
|
||||
return nil, archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
outCols, err := table.OutputColumns(cfg.Columns)
|
||||
if err != nil {
|
||||
return nil, archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
p := &probeResult{table: table, key: key, path: "pk", shardIdx: -1, outCols: outCols}
|
||||
|
||||
if cfg.IsDateShard() {
|
||||
spec, err := schema.ResolveShardColumn(ctx, db, table, cfg.ShardColumn, cfg.ShardColumnType, cfg.Where, loc)
|
||||
if err != nil {
|
||||
return nil, archererr.Wrap(archererr.KindValidation, err)
|
||||
}
|
||||
p.shard = spec
|
||||
prefix := append([]string{spec.Column}, key.Columns...)
|
||||
p.joint = table.FindIndexPrefix(prefix)
|
||||
p.single = table.FindIndexPrefix([]string{spec.Column})
|
||||
switch {
|
||||
case p.joint != nil:
|
||||
p.path = "A"
|
||||
case p.single != nil:
|
||||
p.path = "B"
|
||||
case cfg.AllowFullScan:
|
||||
p.path = "B"
|
||||
default:
|
||||
return nil, archererr.Newf(archererr.KindValidation,
|
||||
"no usable index on %s for date sharding\n joint: create an index on (%s) for keyset chunks\n or: create an index on (%s) for day-at-a-time streaming\n or: pass --allow-full-scan to accept a full table scan\n%s",
|
||||
table.Name, strings.Join(prefix, ", "), spec.Column, describeIndexes(table))
|
||||
}
|
||||
}
|
||||
|
||||
needed := map[string]bool{}
|
||||
for _, c := range outCols {
|
||||
needed[strings.ToLower(c.Name)] = true
|
||||
}
|
||||
for _, c := range key.Columns {
|
||||
needed[strings.ToLower(c)] = true
|
||||
}
|
||||
if p.shard != nil {
|
||||
needed[strings.ToLower(p.shard.Column)] = true
|
||||
}
|
||||
for _, c := range table.Columns {
|
||||
if needed[strings.ToLower(c.Name)] {
|
||||
p.selectCols = append(p.selectCols, c.Name)
|
||||
p.selectMeta = append(p.selectMeta, row.Column{Name: c.Name, Type: c.DataType})
|
||||
}
|
||||
}
|
||||
pos := map[string]int{}
|
||||
for i, name := range p.selectCols {
|
||||
pos[name] = i
|
||||
}
|
||||
p.keyIdx = make([]int, len(key.Columns))
|
||||
for i, c := range key.Columns {
|
||||
idx, ok := pos[c]
|
||||
if !ok {
|
||||
return nil, archererr.Newf(archererr.KindValidation, "internal: key column %s missing from select list", c)
|
||||
}
|
||||
p.keyIdx[i] = idx
|
||||
}
|
||||
if p.shard != nil {
|
||||
idx, ok := pos[p.shard.Column]
|
||||
if !ok {
|
||||
return nil, archererr.Newf(archererr.KindValidation, "internal: shard column %s missing from select list", p.shard.Column)
|
||||
}
|
||||
p.shardIdx = idx
|
||||
}
|
||||
p.outIdx = make([]int, len(outCols))
|
||||
for i, c := range outCols {
|
||||
p.outIdx[i] = pos[c.Name]
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func describeIndexes(t *schema.Table) string {
|
||||
if len(t.Indexes) == 0 {
|
||||
return " indexes: (none)"
|
||||
}
|
||||
var b strings.Builder
|
||||
b.WriteString(" indexes:\n")
|
||||
for _, ix := range t.Indexes {
|
||||
kind := "index"
|
||||
if ix.Name == "PRIMARY" {
|
||||
kind = "primary key"
|
||||
} else if ix.Unique {
|
||||
kind = "unique index"
|
||||
}
|
||||
fmt.Fprintf(&b, " - %s (%s) %s\n", ix.Name, strings.Join(ix.Columns, ", "), kind)
|
||||
}
|
||||
return strings.TrimRight(b.String(), "\n")
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"archer/internal/checkpoint"
|
||||
"archer/internal/cli"
|
||||
)
|
||||
|
||||
func TestStateConfigStoresRedactedDSN(t *testing.T) {
|
||||
const source = "u:s3cr3t@tcp(h:3306)/db?charset=utf8mb4"
|
||||
cfg, err := cli.Parse([]string{"--source", source, "--table", "t", "--where", "id > 0", "--file", "out"}, &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hash, fields, err := cfg.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dir := t.TempDir()
|
||||
if err := checkpoint.Save(dir, &checkpoint.State{ConfigHash: hash, Config: fields}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := checkpoint.Load(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := redactDSN(source)
|
||||
if st.Config["source"] != want {
|
||||
t.Fatalf("config.source = %q, want %q", st.Config["source"], want)
|
||||
}
|
||||
raw, err := os.ReadFile(checkpoint.Path(dir))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if bytes.Contains(raw, []byte("s3cr3t")) {
|
||||
t.Fatal("state.json must not contain the password")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
// Package archererr classifies runtime errors so the process can pick an exit
|
||||
// code and so callers can distinguish interrupted runs.
|
||||
package archererr
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
type Kind int
|
||||
|
||||
const (
|
||||
KindOther Kind = iota
|
||||
KindConfig
|
||||
KindConnection
|
||||
KindWrite
|
||||
KindValidation
|
||||
KindInterrupted
|
||||
)
|
||||
|
||||
// Error wraps an error with a classification.
|
||||
type Error struct {
|
||||
Kind Kind
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *Error) Error() string { return e.Err.Error() }
|
||||
func (e *Error) Unwrap() error { return e.Err }
|
||||
|
||||
func Wrap(kind Kind, err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
var existing *Error
|
||||
if errors.As(err, &existing) {
|
||||
return err
|
||||
}
|
||||
return &Error{Kind: kind, Err: err}
|
||||
}
|
||||
|
||||
func Newf(kind Kind, format string, args ...any) error {
|
||||
return &Error{Kind: kind, Err: fmt.Errorf(format, args...)}
|
||||
}
|
||||
|
||||
func KindOf(err error) Kind {
|
||||
var e *Error
|
||||
if errors.As(err, &e) {
|
||||
return e.Kind
|
||||
}
|
||||
return KindOther
|
||||
}
|
||||
|
||||
// ExitCode maps a classification to a process exit code.
|
||||
func ExitCode(err error) int {
|
||||
if err == nil {
|
||||
return 0
|
||||
}
|
||||
switch KindOf(err) {
|
||||
case KindConfig:
|
||||
return 2
|
||||
case KindConnection:
|
||||
return 3
|
||||
case KindWrite:
|
||||
return 4
|
||||
case KindValidation:
|
||||
return 5
|
||||
case KindInterrupted:
|
||||
return 130
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
// Package atomicfile 提供 Windows 上更稳健的原子文件替换。
|
||||
//
|
||||
// 直接 os.Rename 在 Windows 上走 MoveFileEx(MOVEFILE_REPLACE_EXISTING),
|
||||
// 只要目标文件此刻被任何进程以「不共享删除」方式打开(杀毒软件实时扫描、
|
||||
// Windows Search 索引器、编辑器预览等都会这样),就会返回
|
||||
// ERROR_ACCESS_DENIED,而且这种占用通常是毫秒级的瞬时状态。
|
||||
// 因此这里对 rename 做有限次指数退避重试;非 Windows 平台无此问题,
|
||||
// 但同样行为无害(第一次就成功)。
|
||||
package atomicfile
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"runtime"
|
||||
"time"
|
||||
)
|
||||
|
||||
// renameRetries 是 rename 的最大尝试次数(含首次)。
|
||||
const renameRetries = 12
|
||||
|
||||
// Replace 原子地用 tmp 覆盖 final。tmp 与 final 必须在同一文件系统内。
|
||||
//
|
||||
// 失败时保留 tmp 供调用者排查,并返回最后一次 rename 的原始错误。
|
||||
func Replace(tmp, final string) error {
|
||||
var err error
|
||||
delay := time.Millisecond
|
||||
for i := 0; i < renameRetries; i++ {
|
||||
if err = os.Rename(tmp, final); err == nil {
|
||||
return nil
|
||||
}
|
||||
// 非 Windows 上的失败基本都是真错误(跨设备、权限、目录不存在),
|
||||
// 重试无意义,立即返回。
|
||||
if runtime.GOOS != "windows" || !isTransient(err) {
|
||||
return err
|
||||
}
|
||||
time.Sleep(delay)
|
||||
if delay < 20*time.Millisecond {
|
||||
delay *= 2
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%w (after %d attempts; the file is likely held open by "+
|
||||
"antivirus or the search indexer)", err, renameRetries)
|
||||
}
|
||||
|
||||
// isTransient 判断 rename 失败是否属于「目标被短暂占用」。
|
||||
//
|
||||
// 只识别 Windows 上真实的瞬时错误码。os.Rename 已经把 Win32 错误包成
|
||||
// *os.LinkError,其 Err 是 syscall.Errno,可直接与 syscall 常量比较。
|
||||
func isTransient(err error) bool {
|
||||
var le *os.LinkError
|
||||
if !errors.As(err, &le) {
|
||||
return false
|
||||
}
|
||||
return isTransientErrno(le.Err)
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
//go:build !windows
|
||||
|
||||
package atomicfile
|
||||
|
||||
// 非 Windows 平台没有「目标被短暂占用导致 rename 失败」这一语义,
|
||||
// 因此不做重试判断。
|
||||
func isTransientErrno(err error) bool { return false }
|
||||
@@ -0,0 +1,29 @@
|
||||
//go:build windows
|
||||
|
||||
package atomicfile
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// Windows 上值得重试的错误码。
|
||||
const (
|
||||
errAccessDenied syscall.Errno = 5 // ERROR_ACCESS_DENIED
|
||||
errSharingViol syscall.Errno = 32 // ERROR_SHARING_VIOLATION
|
||||
errLockViolation syscall.Errno = 33 // ERROR_LOCK_VIOLATION
|
||||
errBusy syscall.Errno = 170 // ERROR_BUSY
|
||||
errUserMapped syscall.Errno = 1224 // ERROR_USER_MAPPED_FILE
|
||||
)
|
||||
|
||||
func isTransientErrno(err error) bool {
|
||||
var errno syscall.Errno
|
||||
if !errors.As(err, &errno) {
|
||||
return false
|
||||
}
|
||||
switch errno {
|
||||
case errAccessDenied, errSharingViol, errLockViolation, errBusy, errUserMapped:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
// Package checkpoint persists and restores scan progress in state.json.
|
||||
package checkpoint
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"archer/internal/atomicfile"
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
const Version = 1
|
||||
|
||||
// Cursor identifies where the scanner should resume. Phase is one of
|
||||
// "pk", "null", "value" or "day"; the meaning of Key depends on it.
|
||||
type Cursor struct {
|
||||
Phase string `json:"phase,omitempty"`
|
||||
Key row.Key `json:"key,omitempty"`
|
||||
DayStart string `json:"day_start,omitempty"`
|
||||
}
|
||||
|
||||
// State is the entire persisted checkpoint.
|
||||
type State struct {
|
||||
Version int `json:"version"`
|
||||
RunID string `json:"run_id"`
|
||||
ConfigHash string `json:"config_hash"`
|
||||
Config map[string]string `json:"config,omitempty"`
|
||||
ShardBy string `json:"shard_by"`
|
||||
Cursor Cursor `json:"cursor"`
|
||||
LastBucket string `json:"last_bucket,omitempty"`
|
||||
LastShardSeq int64 `json:"last_shard_seq"`
|
||||
RowsDone int64 `json:"rows_done"`
|
||||
Complete bool `json:"complete"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
func Path(dir string) string { return filepath.Join(dir, "state.json") }
|
||||
|
||||
// Load reads state.json. A missing file returns (nil, nil).
|
||||
func Load(dir string) (*State, error) {
|
||||
b, err := os.ReadFile(Path(dir))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read state: %w", err)
|
||||
}
|
||||
var st State
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return nil, fmt.Errorf("parse state.json: %w", err)
|
||||
}
|
||||
if st.Version != Version {
|
||||
return nil, fmt.Errorf("unsupported state.json version %d", st.Version)
|
||||
}
|
||||
return &st, nil
|
||||
}
|
||||
|
||||
// Save atomically replaces state.json using a temporary file, fsync and
|
||||
// rename.
|
||||
func Save(dir string, st *State) error {
|
||||
st.Version = Version
|
||||
st.UpdatedAt = time.Now().UTC().Format(time.RFC3339)
|
||||
b, err := json.MarshalIndent(st, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode state: %w", err)
|
||||
}
|
||||
b = append(b, '\n')
|
||||
|
||||
final := Path(dir)
|
||||
tmp := final + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create state temp: %w", err)
|
||||
}
|
||||
if _, err := f.Write(b); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("write state temp: %w", err)
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("sync state temp: %w", err)
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("close state temp: %w", err)
|
||||
}
|
||||
if err := atomicfile.Replace(tmp, final); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("replace state.json: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package checkpoint
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
func TestSaveLoadRoundTrip(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
st := &State{
|
||||
RunID: "run-1",
|
||||
ConfigHash: "abc",
|
||||
ShardBy: "date",
|
||||
Cursor: Cursor{Phase: "value", Key: row.KeyOf("2025-01-01 00:00:00", "42")},
|
||||
LastBucket: "2025-01-01",
|
||||
RowsDone: 99,
|
||||
}
|
||||
if err := Save(dir, st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(Path(dir) + ".tmp"); !os.IsNotExist(err) {
|
||||
t.Fatal("temp state file should not remain")
|
||||
}
|
||||
got, err := Load(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got == nil || got.RunID != "run-1" || got.RowsDone != 99 || got.LastBucket != "2025-01-01" {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
if got.Cursor.Phase != "value" || len(got.Cursor.Key) != 2 || got.Cursor.Key[1].Val != "42" {
|
||||
t.Fatalf("cursor %+v", got.Cursor)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadMissing(t *testing.T) {
|
||||
st, err := Load(t.TempDir())
|
||||
if err != nil || st != nil {
|
||||
t.Fatalf("got %v %v", st, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,352 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
func baseArgs() []string {
|
||||
return []string{
|
||||
"--source", "u:p@tcp(127.0.0.1:3306)/db",
|
||||
"--table", "t",
|
||||
"--where", "id > 0",
|
||||
"--file", "out",
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaults(t *testing.T) {
|
||||
c, err := Parse(baseArgs(), &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c.ChunkSize != 2000 || c.ShardBy != "none" || c.Format != "dump" || c.Compress != "none" {
|
||||
t.Fatalf("bad defaults: %+v", c)
|
||||
}
|
||||
if c.MaxLag.String() != "1s" || c.CheckInterval.String() != "1s" || c.Retries != 1 {
|
||||
t.Fatalf("bad defaults: %+v", c)
|
||||
}
|
||||
if !c.SafeAutoIncrement {
|
||||
t.Fatal("safe auto increment must default true")
|
||||
}
|
||||
if err := c.Validate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLimitAlias(t *testing.T) {
|
||||
args := append(baseArgs(), "--limit", "500")
|
||||
c, err := Parse(args, &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c.ChunkSize != 500 {
|
||||
t.Fatalf("got %d", c.ChunkSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseInsertModesMutuallyExclusive(t *testing.T) {
|
||||
args := append(baseArgs(), "--insert-ignore", "--replace")
|
||||
if _, err := Parse(args, &bytes.Buffer{}); err == nil {
|
||||
t.Fatal("expected error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRequired(t *testing.T) {
|
||||
c, _ := Parse([]string{"--source", "u:p@tcp(h:3306)/db", "--table", "t", "--file", "out"}, &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "--where") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateNeedsTarget(t *testing.T) {
|
||||
c, _ := Parse([]string{"--source", "u:p@tcp(h:3306)/db", "--table", "t", "--where", "1=1"}, &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "output target") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateShardRules(t *testing.T) {
|
||||
args := append(baseArgs(), "--shard-by", "count", "--shard-column", "created_at", "--shard-rows", "10")
|
||||
c, err := Parse(args, &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "--shard-column") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
|
||||
args = append(baseArgs(), "--shard-by", "count")
|
||||
c, _ = Parse(args, &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "--shard-rows") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
|
||||
args = append(baseArgs(), "--shard-by", "date", "--shard-column", "c", "--bulk-delete", "--purge")
|
||||
c, _ = Parse(args, &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "bulk-delete") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateBulkDeleteNeedsPurge(t *testing.T) {
|
||||
args := append(baseArgs(), "--bulk-delete")
|
||||
c, _ := Parse(args, &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "requires --purge") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFingerprintExcludesPassword(t *testing.T) {
|
||||
a, _ := Parse([]string{"--source", "u:one@tcp(h:3306)/db", "--table", "t", "--where", "1=1", "--file", "out"}, &bytes.Buffer{})
|
||||
b, _ := Parse([]string{"--source", "u:two@tcp(h:3306)/db", "--table", "t", "--where", "1=1", "--file", "out"}, &bytes.Buffer{})
|
||||
ha, fa, err := a.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hb, fb, err := b.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ha != hb {
|
||||
t.Fatal("password must not affect the fingerprint")
|
||||
}
|
||||
if fa["source"] != fb["source"] {
|
||||
t.Fatalf("password must not affect the source field: %q vs %q", fa["source"], fb["source"])
|
||||
}
|
||||
}
|
||||
|
||||
func fingerprintFor(t *testing.T, dsn string) (string, map[string]string) {
|
||||
t.Helper()
|
||||
c, err := Parse([]string{"--source", dsn, "--table", "t", "--where", "id > 0", "--file", "out"}, &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h, f, err := c.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return h, f
|
||||
}
|
||||
|
||||
func TestFingerprintTracksConnectionCharset(t *testing.T) {
|
||||
hBase, _ := fingerprintFor(t, "u:p@tcp(h:3306)/db")
|
||||
hCharset, fCharset := fingerprintFor(t, "u:p@tcp(h:3306)/db?charset=utf8mb4")
|
||||
if hBase == hCharset {
|
||||
t.Fatal("charset must affect the fingerprint")
|
||||
}
|
||||
if !strings.Contains(fCharset["source"], "charset=utf8mb4") {
|
||||
t.Fatalf("source field missing charset: %q", fCharset["source"])
|
||||
}
|
||||
|
||||
hCollation, _ := fingerprintFor(t, "u:p@tcp(h:3306)/db?charset=utf8mb4&collation=utf8mb4_bin")
|
||||
if hCharset == hCollation {
|
||||
t.Fatal("collation must affect the fingerprint")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDsnFingerprintRedactsPassword(t *testing.T) {
|
||||
c, err := mysql.ParseDSN("u:s3cr3t@tcp(h:3306)/db")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := dsnFingerprint(c)
|
||||
if got != "u@tcp(h:3306)/db" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if strings.Contains(got, "s3cr3t") {
|
||||
t.Fatal("password leaked into the fingerprint")
|
||||
}
|
||||
if c.Passwd != "s3cr3t" {
|
||||
t.Fatal("dsnFingerprint must not modify the input config")
|
||||
}
|
||||
if dsnFingerprint(nil) != "" {
|
||||
t.Fatal("nil config must yield an empty fragment")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFingerprintDestEmptyAndNoPassword(t *testing.T) {
|
||||
h, f := fingerprintFor(t, "u:s3cr3t@tcp(h:3306)/db")
|
||||
if f["dest"] != "" {
|
||||
t.Fatalf("dest must be empty without --dest: %q", f["dest"])
|
||||
}
|
||||
if f["source"] != "u@tcp(h:3306)/db" {
|
||||
t.Fatalf("source = %q", f["source"])
|
||||
}
|
||||
if strings.Contains(h, "s3cr3t") || strings.Contains(f["source"], "s3cr3t") {
|
||||
t.Fatal("password leaked into the fingerprint")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFingerprintDetectsTimezoneChange(t *testing.T) {
|
||||
a, _ := Parse(append(baseArgs(), "--shard-by", "date", "--shard-column", "c", "--shard-time-zone", "UTC"), &bytes.Buffer{})
|
||||
b, _ := Parse(append(baseArgs(), "--shard-by", "date", "--shard-column", "c", "--shard-time-zone", "Asia/Shanghai"), &bytes.Buffer{})
|
||||
ha, fa, _ := a.Fingerprint()
|
||||
hb, _, _ := b.Fingerprint()
|
||||
if ha == hb {
|
||||
t.Fatal("time zone must affect the fingerprint")
|
||||
}
|
||||
_ = fa
|
||||
}
|
||||
|
||||
func TestDiffFields(t *testing.T) {
|
||||
old := map[string]string{"a": "1", "b": "2"}
|
||||
cur := map[string]string{"a": "1", "b": "3", "c": "4"}
|
||||
d := DiffFields(old, cur)
|
||||
if len(d) != 2 {
|
||||
t.Fatalf("got %v", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCompressLevel(t *testing.T) {
|
||||
for _, level := range []string{"0", "10"} {
|
||||
c, err := Parse(append(baseArgs(), "--compress", "gzip", "--compress-level", level), &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "between 1 and 9") {
|
||||
t.Fatalf("level %s: got %v", level, err)
|
||||
}
|
||||
}
|
||||
|
||||
c, _ := Parse(append(baseArgs(), "--compress", "none", "--compress-level", "5"), &bytes.Buffer{})
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "only valid when compression is enabled") {
|
||||
t.Fatalf("none + level 5: got %v", err)
|
||||
}
|
||||
|
||||
c, _ = Parse(append(baseArgs(), "--compress", "none", "--compress-level", "3"), &bytes.Buffer{})
|
||||
if err := c.Validate(); err != nil {
|
||||
t.Fatalf("none + default level must be accepted: %v", err)
|
||||
}
|
||||
|
||||
for _, name := range []string{"gzip", "zstd", "xz"} {
|
||||
c, _ = Parse(append(baseArgs(), "--compress", name, "--compress-level", "9"), &bytes.Buffer{})
|
||||
if err := c.Validate(); err != nil {
|
||||
t.Fatalf("%s level 9: %v", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProgressModes(t *testing.T) {
|
||||
cases := []struct {
|
||||
arg string
|
||||
mode ProgressMode
|
||||
rows int64
|
||||
valid bool
|
||||
}{
|
||||
{"", ProgressAuto, 0, true},
|
||||
{"auto", ProgressAuto, 0, true},
|
||||
{"on", ProgressOn, 0, true},
|
||||
{"off", ProgressOff, 0, true},
|
||||
{"0", ProgressOff, 0, true},
|
||||
{"100000", ProgressRows, 100000, true},
|
||||
{"1", ProgressRows, 1, true},
|
||||
{"abc", 0, 0, false},
|
||||
{"-1", 0, 0, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
args := baseArgs()
|
||||
if tc.arg != "" {
|
||||
args = append(args, "--progress", tc.arg)
|
||||
}
|
||||
c, err := Parse(args, &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatalf("%q: parse: %v", tc.arg, err)
|
||||
}
|
||||
err = c.Validate()
|
||||
if !tc.valid {
|
||||
if err == nil {
|
||||
t.Fatalf("%q: expected a configuration error", tc.arg)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("%q: validate: %v", tc.arg, err)
|
||||
}
|
||||
if c.Progress.Mode != tc.mode || c.Progress.Rows != tc.rows {
|
||||
t.Fatalf("%q: got mode=%d rows=%d", tc.arg, c.Progress.Mode, c.Progress.Rows)
|
||||
}
|
||||
}
|
||||
|
||||
c, _ := Parse(baseArgs(), &bytes.Buffer{})
|
||||
if c.Progress.Mode != ProgressAuto {
|
||||
t.Fatalf("default progress mode = %d, want auto", c.Progress.Mode)
|
||||
}
|
||||
if c.ProgressInterval != time.Second {
|
||||
t.Fatalf("default progress interval = %s", c.ProgressInterval)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProgressEnabledDependsOnTTY(t *testing.T) {
|
||||
on, _ := Parse(append(baseArgs(), "--progress", "on"), &bytes.Buffer{})
|
||||
off, _ := Parse(append(baseArgs(), "--progress", "off"), &bytes.Buffer{})
|
||||
auto, _ := Parse(baseArgs(), &bytes.Buffer{})
|
||||
for _, c := range []*Config{on, off, auto} {
|
||||
if err := c.Validate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if !on.Progress.Enabled(false) || !on.Progress.Enabled(true) {
|
||||
t.Fatal("on must always be enabled")
|
||||
}
|
||||
if off.Progress.Enabled(true) || off.Progress.Enabled(false) {
|
||||
t.Fatal("off must never be enabled")
|
||||
}
|
||||
if !auto.Progress.Enabled(true) {
|
||||
t.Fatal("auto must be enabled on a terminal")
|
||||
}
|
||||
if auto.Progress.Enabled(false) {
|
||||
t.Fatal("auto must be silent without a terminal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProgressIntervalValidation(t *testing.T) {
|
||||
c, err := Parse(append(baseArgs(), "--progress-interval", "0s"), &bytes.Buffer{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "--progress-interval") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFingerprintCompressLevel(t *testing.T) {
|
||||
implicit, _ := Parse(append(baseArgs(), "--compress", "gzip"), &bytes.Buffer{})
|
||||
explicit, _ := Parse(append(baseArgs(), "--compress", "gzip", "--compress-level", "3"), &bytes.Buffer{})
|
||||
hImplicit, fImplicit, err := implicit.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hExplicit, _, err := explicit.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if hImplicit != hExplicit {
|
||||
t.Fatal("the default level must not change the fingerprint")
|
||||
}
|
||||
if _, ok := fImplicit["compress_level"]; ok {
|
||||
t.Fatal("the default level must not be fingerprinted")
|
||||
}
|
||||
|
||||
five, _ := Parse(append(baseArgs(), "--compress", "gzip", "--compress-level", "5"), &bytes.Buffer{})
|
||||
hFive, fFive, err := five.Fingerprint()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if hFive == hImplicit {
|
||||
t.Fatal("a non-default level must change the fingerprint")
|
||||
}
|
||||
if fFive["compress_level"] != "5" {
|
||||
t.Fatalf("compress_level = %q", fFive["compress_level"])
|
||||
}
|
||||
|
||||
if diffs := DiffFields(fImplicit, fFive); len(diffs) != 1 || !strings.Contains(diffs[0], "compress_level") {
|
||||
t.Fatalf("raising the level: %v", diffs)
|
||||
}
|
||||
if diffs := DiffFields(fFive, fImplicit); len(diffs) != 1 || !strings.Contains(diffs[0], "compress_level") {
|
||||
t.Fatalf("lowering the level back to default: %v", diffs)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
// Package cli parses and validates archer command-line configuration.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
// Config is the fully resolved runtime configuration.
|
||||
type Config struct {
|
||||
SourceDSN string
|
||||
DestDSN string
|
||||
Table string
|
||||
DestTable string
|
||||
Where string
|
||||
Columns []string
|
||||
FileDir string
|
||||
Purge bool
|
||||
BulkDelete bool
|
||||
InsertMode string // "", "ignore", "replace"
|
||||
|
||||
ChunkSize int
|
||||
|
||||
ShardBy string // none|count|date
|
||||
ShardRows int
|
||||
ShardColumn string
|
||||
ShardColumnType string
|
||||
ShardTimeZone string
|
||||
ZeroDate string
|
||||
|
||||
Format string
|
||||
Compress string
|
||||
CompressLevel int
|
||||
CSVNull string
|
||||
|
||||
MaxLag time.Duration
|
||||
CheckInterval time.Duration
|
||||
CheckReplicaLag string
|
||||
Retries int
|
||||
|
||||
SafeAutoIncrement bool
|
||||
Progress ProgressValue
|
||||
ProgressInterval time.Duration
|
||||
Statistics bool
|
||||
DryRun bool
|
||||
AllowFullScan bool
|
||||
ForceResume bool
|
||||
Index string
|
||||
Version bool
|
||||
}
|
||||
|
||||
// Defaults returns a Config with documented default values applied.
|
||||
func Defaults() *Config {
|
||||
return &Config{
|
||||
ChunkSize: 2000,
|
||||
ShardBy: "none",
|
||||
ShardRows: 0,
|
||||
ShardTimeZone: "UTC",
|
||||
Format: "dump",
|
||||
Compress: "none",
|
||||
CompressLevel: 3,
|
||||
CSVNull: `\N`,
|
||||
MaxLag: time.Second,
|
||||
CheckInterval: time.Second,
|
||||
Retries: 1,
|
||||
SafeAutoIncrement: true,
|
||||
Progress: ProgressValue{Mode: ProgressAuto},
|
||||
ProgressInterval: time.Second,
|
||||
}
|
||||
}
|
||||
|
||||
// SourceConfig parses and normalizes the source DSN, forcing parseTime off so
|
||||
// values arrive as raw bytes.
|
||||
func (c *Config) SourceConfig() (*mysql.Config, error) {
|
||||
return normalize(c.SourceDSN)
|
||||
}
|
||||
|
||||
// DestConfig parses and normalizes the dest DSN.
|
||||
func (c *Config) DestConfig() (*mysql.Config, error) {
|
||||
if c.DestDSN == "" {
|
||||
return nil, nil
|
||||
}
|
||||
return normalize(c.DestDSN)
|
||||
}
|
||||
|
||||
func normalize(dsn string) (*mysql.Config, error) {
|
||||
cfg, err := mysql.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid DSN: %w", err)
|
||||
}
|
||||
cfg.ParseTime = false
|
||||
if cfg.Loc == nil {
|
||||
cfg.Loc = time.UTC
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// SameInstance reports whether two normalized DSNs point at the same MySQL
|
||||
// server, which determines whether dest writes and purges share a transaction.
|
||||
func SameInstance(src, dst *mysql.Config) bool {
|
||||
if src == nil || dst == nil {
|
||||
return false
|
||||
}
|
||||
return src.Net == dst.Net && src.Addr == dst.Addr
|
||||
}
|
||||
|
||||
// ShardMode helpers.
|
||||
func (c *Config) IsDateShard() bool { return c.ShardBy == "date" }
|
||||
func (c *Config) IsCountShard() bool { return c.ShardBy == "count" }
|
||||
func (c *Config) HasFile() bool { return c.FileDir != "" }
|
||||
@@ -0,0 +1,102 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
// Fingerprint returns a stable hash of every configuration input that can
|
||||
// change scan or shard semantics, plus the normalized fields used to build it.
|
||||
// The source and dest fields are standard MySQL DSNs that include every
|
||||
// connection parameter the driver parsed (charset, collation, timeout, ...),
|
||||
// so a resume notices any of them changing. The password is the one exclusion:
|
||||
// rotating it must not break resume.
|
||||
func (c *Config) Fingerprint() (string, map[string]string, error) {
|
||||
src, err := c.SourceConfig()
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("source DSN: %w", err)
|
||||
}
|
||||
dst, err := c.DestConfig()
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("dest DSN: %w", err)
|
||||
}
|
||||
fields := map[string]string{
|
||||
"source": dsnFingerprint(src),
|
||||
"dest": dsnFingerprint(dst),
|
||||
"table": c.Table,
|
||||
"dest_table": c.DestTable,
|
||||
"where": c.Where,
|
||||
"columns": strings.Join(c.Columns, ","),
|
||||
"purge": strconv.FormatBool(c.Purge),
|
||||
"bulk_delete": strconv.FormatBool(c.BulkDelete),
|
||||
"insert_mode": c.InsertMode,
|
||||
"shard_by": c.ShardBy,
|
||||
"shard_rows": strconv.Itoa(c.ShardRows),
|
||||
"shard_column": c.ShardColumn,
|
||||
"shard_column_type": c.ShardColumnType,
|
||||
"shard_time_zone": c.ShardTimeZone,
|
||||
"zero_date": c.ZeroDate,
|
||||
"format": c.Format,
|
||||
"compress": c.Compress,
|
||||
"csv_null": c.CSVNull,
|
||||
"safe_auto_increment": strconv.FormatBool(c.SafeAutoIncrement),
|
||||
}
|
||||
// The level only changes the output when compression is on, and omitting
|
||||
// the default keeps fingerprints of tasks started before it existed
|
||||
// byte-identical so they can resume.
|
||||
if c.Compress != "" && c.Compress != "none" && c.CompressLevel != 3 {
|
||||
fields["compress_level"] = strconv.Itoa(c.CompressLevel)
|
||||
}
|
||||
keys := make([]string, 0, len(fields))
|
||||
for k := range fields {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var b strings.Builder
|
||||
for _, k := range keys {
|
||||
b.WriteString(k)
|
||||
b.WriteByte('=')
|
||||
b.WriteString(fields[k])
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
sum := sha256.Sum256([]byte(b.String()))
|
||||
return hex.EncodeToString(sum[:]), fields, nil
|
||||
}
|
||||
|
||||
// dsnFingerprint renders c as a standard MySQL DSN covering every parameter
|
||||
// the driver parsed, except the password. Copying by value keeps the driver's
|
||||
// unexported fields (charsets, timeTruncate, ...) intact, which is the only way
|
||||
// FormatDSN can serialize them.
|
||||
func dsnFingerprint(c *mysql.Config) string {
|
||||
if c == nil {
|
||||
return ""
|
||||
}
|
||||
redacted := *c
|
||||
redacted.Passwd = ""
|
||||
return redacted.FormatDSN()
|
||||
}
|
||||
|
||||
// DiffFields lists keys whose values differ between two fingerprints.
|
||||
func DiffFields(old, cur map[string]string) []string {
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for k, v := range cur {
|
||||
seen[k] = true
|
||||
if old[k] != v {
|
||||
out = append(out, fmt.Sprintf("%s: %q -> %q", k, old[k], v))
|
||||
}
|
||||
}
|
||||
for k, v := range old {
|
||||
if !seen[k] {
|
||||
out = append(out, fmt.Sprintf("%s: %q -> (absent)", k, v))
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Parse parses args (without the program name) into a Config. It does not
|
||||
// validate combinations; call Validate for that.
|
||||
func Parse(args []string, errOut io.Writer) (*Config, error) {
|
||||
c := Defaults()
|
||||
var (
|
||||
columnsArg string
|
||||
limitArg = -1
|
||||
insertIgnoreArg bool
|
||||
replaceArg bool
|
||||
)
|
||||
fs := flag.NewFlagSet("archer", flag.ContinueOnError)
|
||||
fs.SetOutput(errOut)
|
||||
|
||||
fs.StringVar(&c.SourceDSN, "source", "", "source MySQL DSN (required)")
|
||||
fs.StringVar(&c.DestDSN, "dest", "", "destination MySQL DSN")
|
||||
fs.StringVar(&c.Table, "table", "", "source table (required)")
|
||||
fs.StringVar(&c.DestTable, "dest-table", "", "destination table (defaults to --table)")
|
||||
fs.StringVar(&c.Where, "where", "", "row filter for the source table (required)")
|
||||
fs.StringVar(&columnsArg, "columns", "", "comma separated columns to export (default: all)")
|
||||
fs.StringVar(&c.FileDir, "file", "", "output directory for shard files")
|
||||
fs.BoolVar(&c.Purge, "purge", false, "delete archived rows from the source")
|
||||
fs.BoolVar(&c.BulkDelete, "bulk-delete", false, "delete by key range instead of IN list (requires --purge)")
|
||||
fs.BoolVar(&insertIgnoreArg, "insert-ignore", false, "use INSERT IGNORE for --dest")
|
||||
fs.BoolVar(&replaceArg, "replace", false, "use REPLACE for --dest")
|
||||
|
||||
fs.IntVar(&c.ChunkSize, "chunk-size", c.ChunkSize, "rows per SELECT / transaction / checkpoint")
|
||||
fs.IntVar(&limitArg, "limit", -1, "deprecated alias for --chunk-size")
|
||||
|
||||
fs.StringVar(&c.ShardBy, "shard-by", c.ShardBy, "shard mode: none|count|date")
|
||||
fs.IntVar(&c.ShardRows, "shard-rows", c.ShardRows, "rows per shard file (count/date modes)")
|
||||
fs.StringVar(&c.ShardColumn, "shard-column", "", "date or epoch column used for date sharding")
|
||||
fs.StringVar(&c.ShardColumnType, "shard-column-type", "", "epoch unit: epoch-seconds|epoch-milliseconds|epoch-microseconds")
|
||||
fs.StringVar(&c.ShardTimeZone, "shard-time-zone", c.ShardTimeZone, "time zone for epoch shard values")
|
||||
fs.StringVar(&c.ZeroDate, "zero-date", "", "zero date handling: bucket (default: error)")
|
||||
|
||||
fs.StringVar(&c.Format, "format", c.Format, "output format: dump|csv|jsonl")
|
||||
fs.StringVar(&c.Compress, "compress", c.Compress, "compression: none|gzip|zstd|xz")
|
||||
fs.IntVar(&c.CompressLevel, "compress-level", c.CompressLevel, "compression effort 1-9 (higher is smaller and slower)")
|
||||
fs.StringVar(&c.CSVNull, "csv-null", c.CSVNull, "string used for NULL in csv output")
|
||||
|
||||
fs.DurationVar(&c.MaxLag, "max-lag", c.MaxLag, "maximum tolerated replica lag")
|
||||
fs.DurationVar(&c.CheckInterval, "check-interval", c.CheckInterval, "wait between replica lag checks")
|
||||
fs.StringVar(&c.CheckReplicaLag, "check-replica-lag", "", "DSN of a replica whose lag is checked before each fetch")
|
||||
fs.IntVar(&c.Retries, "retries", c.Retries, "retries for cross-instance commits")
|
||||
|
||||
fs.BoolVar(&c.SafeAutoIncrement, "safe-auto-increment", c.SafeAutoIncrement, "never purge the maximum auto-increment row")
|
||||
fs.Var(&c.Progress, "progress", "progress output: auto|on|off|N (default auto; N: also every N rows)")
|
||||
fs.DurationVar(&c.ProgressInterval, "progress-interval", c.ProgressInterval, "minimum interval between progress updates")
|
||||
fs.BoolVar(&c.Statistics, "statistics", false, "print a summary when finished")
|
||||
fs.BoolVar(&c.DryRun, "dry-run", false, "print resolved config and SQL without writing")
|
||||
fs.BoolVar(&c.AllowFullScan, "allow-full-scan", false, "allow date sharding without a usable index")
|
||||
fs.BoolVar(&c.ForceResume, "force-resume", false, "resume even when the config fingerprint changed")
|
||||
fs.StringVar(&c.Index, "index", "", "force a unique index for the scan key")
|
||||
fs.BoolVar(&c.Version, "version", false, "print version and exit")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if c.Version {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
set := map[string]bool{}
|
||||
fs.Visit(func(f *flag.Flag) { set[f.Name] = true })
|
||||
|
||||
if set["limit"] {
|
||||
if set["chunk-size"] {
|
||||
return nil, fmt.Errorf("--limit is a deprecated alias of --chunk-size; pass only one")
|
||||
}
|
||||
c.ChunkSize = limitArg
|
||||
}
|
||||
if columnsArg != "" {
|
||||
for _, part := range strings.Split(columnsArg, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
if part != "" {
|
||||
c.Columns = append(c.Columns, part)
|
||||
}
|
||||
}
|
||||
}
|
||||
if insertIgnoreArg && replaceArg {
|
||||
return nil, fmt.Errorf("--insert-ignore and --replace are mutually exclusive")
|
||||
}
|
||||
switch {
|
||||
case insertIgnoreArg:
|
||||
c.InsertMode = "ignore"
|
||||
case replaceArg:
|
||||
c.InsertMode = "replace"
|
||||
}
|
||||
if c.DestTable == "" {
|
||||
c.DestTable = c.Table
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ProgressMode is the resolved meaning of --progress.
|
||||
type ProgressMode int
|
||||
|
||||
const (
|
||||
ProgressAuto ProgressMode = iota // live on a terminal, silent otherwise
|
||||
ProgressOff
|
||||
ProgressOn
|
||||
ProgressRows // enabled, plus a row-count threshold
|
||||
)
|
||||
|
||||
// ProgressValue is the flag.Value behind --progress. It keeps the raw
|
||||
// argument so invalid values are reported as configuration errors by Validate
|
||||
// rather than as bare flag parse errors.
|
||||
type ProgressValue struct {
|
||||
Mode ProgressMode
|
||||
Rows int64
|
||||
raw string
|
||||
}
|
||||
|
||||
// Set records the raw argument; resolve happens in Validate.
|
||||
func (p *ProgressValue) Set(s string) error {
|
||||
p.raw = s
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *ProgressValue) String() string {
|
||||
if p.raw != "" {
|
||||
return p.raw
|
||||
}
|
||||
switch p.Mode {
|
||||
case ProgressOn:
|
||||
return "on"
|
||||
case ProgressOff:
|
||||
return "off"
|
||||
case ProgressRows:
|
||||
return strconv.FormatInt(p.Rows, 10)
|
||||
default:
|
||||
return "auto"
|
||||
}
|
||||
}
|
||||
|
||||
// resolve parses the recorded argument into Mode and Rows.
|
||||
func (p *ProgressValue) resolve() error {
|
||||
switch strings.ToLower(strings.TrimSpace(p.raw)) {
|
||||
case "", "auto":
|
||||
p.Mode, p.Rows = ProgressAuto, 0
|
||||
case "on":
|
||||
p.Mode, p.Rows = ProgressOn, 0
|
||||
case "off":
|
||||
p.Mode, p.Rows = ProgressOff, 0
|
||||
default:
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(p.raw), 10, 64)
|
||||
if err != nil || n < 0 {
|
||||
return fmt.Errorf("--progress must be auto, on, off or a non-negative integer, got %q", p.raw)
|
||||
}
|
||||
if n == 0 {
|
||||
p.Mode, p.Rows = ProgressOff, 0
|
||||
} else {
|
||||
p.Mode, p.Rows = ProgressRows, n
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Enabled reports whether progress output should be produced. tty describes
|
||||
// whether stderr is a terminal.
|
||||
func (p ProgressValue) Enabled(tty bool) bool {
|
||||
switch p.Mode {
|
||||
case ProgressOn, ProgressRows:
|
||||
return true
|
||||
case ProgressAuto:
|
||||
return tty
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"archer/internal/archererr"
|
||||
)
|
||||
|
||||
// Validate checks required flags and the inter-parameter dependency rules.
|
||||
// The shard column's existence and type are checked later, after metadata is
|
||||
// loaded, so the error can list candidate columns.
|
||||
func (c *Config) Validate() error {
|
||||
if c.SourceDSN == "" {
|
||||
return cfgErr("--source is required")
|
||||
}
|
||||
if c.Table == "" {
|
||||
return cfgErr("--table is required")
|
||||
}
|
||||
if strings.TrimSpace(c.Where) == "" {
|
||||
return cfgErr("--where is required")
|
||||
}
|
||||
if !c.HasFile() && c.DestDSN == "" && !c.Purge {
|
||||
return cfgErr("at least one output target is required: --file, --dest or --purge")
|
||||
}
|
||||
if c.ChunkSize <= 0 {
|
||||
return cfgErr("--chunk-size must be greater than 0")
|
||||
}
|
||||
if c.Retries < 0 {
|
||||
return cfgErr("--retries must be 0 or greater")
|
||||
}
|
||||
if err := c.Progress.resolve(); err != nil {
|
||||
return cfgErr("%v", err)
|
||||
}
|
||||
if c.ProgressInterval <= 0 {
|
||||
return cfgErr("--progress-interval must be greater than 0")
|
||||
}
|
||||
|
||||
switch c.ShardBy {
|
||||
case "none", "count", "date":
|
||||
default:
|
||||
return cfgErr("--shard-by must be one of none, count, date")
|
||||
}
|
||||
|
||||
switch c.ShardBy {
|
||||
case "date":
|
||||
if c.ShardRows < 0 {
|
||||
return cfgErr("--shard-rows must be 0 or greater")
|
||||
}
|
||||
default:
|
||||
if c.ShardColumn != "" {
|
||||
return cfgErr("--shard-column is only valid with --shard-by date")
|
||||
}
|
||||
if c.ShardColumnType != "" {
|
||||
return cfgErr("--shard-column-type is only valid with --shard-by date")
|
||||
}
|
||||
if c.ZeroDate != "" {
|
||||
return cfgErr("--zero-date is only valid with --shard-by date")
|
||||
}
|
||||
}
|
||||
if c.ShardBy == "none" && c.ShardRows != 0 {
|
||||
return cfgErr("--shard-rows is only valid with --shard-by count or date")
|
||||
}
|
||||
if c.ShardBy == "count" && c.ShardRows == 0 {
|
||||
return cfgErr("--shard-rows must be greater than 0 for --shard-by count")
|
||||
}
|
||||
if c.BulkDelete && !c.Purge {
|
||||
return cfgErr("--bulk-delete requires --purge")
|
||||
}
|
||||
if c.BulkDelete && c.ShardBy == "date" {
|
||||
return cfgErr("--bulk-delete cannot be used with --shard-by date; delete by IN list instead")
|
||||
}
|
||||
|
||||
switch c.Format {
|
||||
case "dump", "csv", "jsonl":
|
||||
default:
|
||||
return cfgErr("--format must be one of dump, csv, jsonl")
|
||||
}
|
||||
switch c.Compress {
|
||||
case "none", "gzip", "zstd", "xz":
|
||||
default:
|
||||
return cfgErr("--compress must be one of none, gzip, zstd, xz")
|
||||
}
|
||||
if c.CompressLevel < 1 || c.CompressLevel > 9 {
|
||||
return cfgErr("--compress-level must be between 1 and 9")
|
||||
}
|
||||
if c.Compress == "none" && c.CompressLevel != 3 {
|
||||
return cfgErr("--compress-level is only valid when compression is enabled")
|
||||
}
|
||||
switch c.ZeroDate {
|
||||
case "", "bucket":
|
||||
default:
|
||||
return cfgErr("--zero-date must be empty or bucket")
|
||||
}
|
||||
if _, err := time.LoadLocation(c.ShardTimeZone); err != nil {
|
||||
return cfgErr("--shard-time-zone %q is not a valid location: %v", c.ShardTimeZone, err)
|
||||
}
|
||||
if _, err := c.SourceConfig(); err != nil {
|
||||
return cfgErr("--source: %v", err)
|
||||
}
|
||||
if _, err := c.DestConfig(); err != nil {
|
||||
return cfgErr("--dest: %v", err)
|
||||
}
|
||||
if c.CheckReplicaLag != "" {
|
||||
if err := validateDSN(c.CheckReplicaLag); err != nil {
|
||||
return cfgErr("--check-replica-lag: %v", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDSN(dsn string) error {
|
||||
_, err := normalize(dsn)
|
||||
return err
|
||||
}
|
||||
|
||||
func cfgErr(format string, args ...any) error {
|
||||
return archererr.Newf(archererr.KindConfig, format, args...)
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
// Package estimate derives a cheap total-size estimate for progress display.
|
||||
//
|
||||
// Only bounded MIN/MAX queries are used; COUNT(*) is never attempted. A query
|
||||
// that fails, times out, or whose metric does not apply silently yields no
|
||||
// estimate, leaving the caller to render rows and rate only.
|
||||
package estimate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"archer/internal/progress"
|
||||
"archer/internal/row"
|
||||
"archer/internal/schema"
|
||||
)
|
||||
|
||||
// Spec describes which metric a run can expose.
|
||||
type Spec struct {
|
||||
Kind string // "day", "key" or "" for none
|
||||
Column string
|
||||
ShardKind schema.ShardKind
|
||||
Unit int64
|
||||
Loc *time.Location
|
||||
}
|
||||
|
||||
// RangeQuery returns MIN and MAX of a column as raw text. Implementations must
|
||||
// honor ctx so the bounded timeout is effective.
|
||||
type RangeQuery func(ctx context.Context, column string) (minRaw, maxRaw sql.NullString, err error)
|
||||
|
||||
// Bounded runs q under a timeout and converts the result into a display
|
||||
// estimate. It returns nil instead of an error on every failure path.
|
||||
func Bounded(ctx context.Context, q RangeQuery, spec Spec, timeout time.Duration) *progress.Estimate {
|
||||
if q == nil || spec.Kind == "" || timeout <= 0 {
|
||||
return nil
|
||||
}
|
||||
qctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
minRaw, maxRaw, err := q(qctx, spec.Column)
|
||||
if err != nil || !minRaw.Valid || !maxRaw.Valid {
|
||||
return nil
|
||||
}
|
||||
|
||||
var lo, hi float64
|
||||
var ok bool
|
||||
switch spec.Kind {
|
||||
case "day":
|
||||
lo, hi, ok = dayRange(minRaw.String, maxRaw.String, spec.ShardKind, spec.Unit, spec.Loc)
|
||||
case "key":
|
||||
lo, hi, ok = keyRange(minRaw.String, maxRaw.String)
|
||||
}
|
||||
if !ok || hi <= lo {
|
||||
return nil
|
||||
}
|
||||
return &progress.Estimate{Kind: spec.Kind, Lo: lo, Hi: hi}
|
||||
}
|
||||
|
||||
// KeyColumn reports the scan key column when it is a single integer column,
|
||||
// which is the only non-date shape a cheap key-range estimate applies to.
|
||||
func KeyColumn(cols, types []string) (string, bool) {
|
||||
if len(cols) != 1 || len(types) != 1 {
|
||||
return "", false
|
||||
}
|
||||
if !row.IsInteger(types[0]) {
|
||||
return "", false
|
||||
}
|
||||
return cols[0], true
|
||||
}
|
||||
|
||||
func dayRange(minRaw, maxRaw string, kind schema.ShardKind, unit int64, loc *time.Location) (float64, float64, bool) {
|
||||
if loc == nil {
|
||||
loc = time.UTC
|
||||
}
|
||||
switch kind {
|
||||
case schema.ShardDate, schema.ShardDateTime:
|
||||
lo, ok1 := daySeconds(minRaw, loc)
|
||||
hi, ok2 := daySeconds(maxRaw, loc)
|
||||
return lo, hi, ok1 && ok2
|
||||
case schema.ShardEpoch:
|
||||
minV, err1 := strconv.ParseInt(strings.TrimSpace(minRaw), 10, 64)
|
||||
maxV, err2 := strconv.ParseInt(strings.TrimSpace(maxRaw), 10, 64)
|
||||
if err1 != nil || err2 != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
lo := dayStart(schema.EpochTime(minV, unit, loc), loc).Unix()
|
||||
hi := dayStart(schema.EpochTime(maxV, unit, loc), loc).Unix()
|
||||
return float64(lo), float64(hi), true
|
||||
}
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
func daySeconds(raw string, loc *time.Location) (float64, bool) {
|
||||
s := strings.TrimSpace(raw)
|
||||
if len(s) < 10 {
|
||||
return 0, false
|
||||
}
|
||||
day := s[:10]
|
||||
if strings.HasPrefix(day, "0000-") {
|
||||
return 0, false
|
||||
}
|
||||
t, err := time.ParseInLocation("2006-01-02", day, loc)
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
return float64(t.Unix()), true
|
||||
}
|
||||
|
||||
func keyRange(minRaw, maxRaw string) (float64, float64, bool) {
|
||||
minV, err1 := strconv.ParseInt(strings.TrimSpace(minRaw), 10, 64)
|
||||
maxV, err2 := strconv.ParseInt(strings.TrimSpace(maxRaw), 10, 64)
|
||||
if err1 != nil || err2 != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
return float64(minV), float64(maxV), true
|
||||
}
|
||||
|
||||
func dayStart(t time.Time, loc *time.Location) time.Time {
|
||||
return time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, loc)
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package estimate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"archer/internal/schema"
|
||||
)
|
||||
|
||||
func validRange(minRaw, maxRaw string) RangeQuery {
|
||||
return func(context.Context, string) (sql.NullString, sql.NullString, error) {
|
||||
return sql.NullString{String: minRaw, Valid: true}, sql.NullString{String: maxRaw, Valid: true}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyColumnApplicability(t *testing.T) {
|
||||
cases := []struct {
|
||||
cols, types []string
|
||||
want bool
|
||||
}{
|
||||
{[]string{"id"}, []string{"bigint"}, true},
|
||||
{[]string{"id"}, []string{"int"}, true},
|
||||
{[]string{"a", "b"}, []string{"bigint", "bigint"}, false},
|
||||
{[]string{"code"}, []string{"varchar"}, false},
|
||||
{[]string{"payload"}, []string{"blob"}, false},
|
||||
{[]string{"bin"}, []string{"varbinary"}, false},
|
||||
{[]string{"ts"}, []string{"datetime"}, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
col, ok := KeyColumn(tc.cols, tc.types)
|
||||
if ok != tc.want {
|
||||
t.Fatalf("%v/%v: got %v want %v", tc.cols, tc.types, ok, tc.want)
|
||||
}
|
||||
if ok && col != tc.cols[0] {
|
||||
t.Fatalf("%v: wrong column %q", tc.cols, col)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBoundedDayRangeFromDates(t *testing.T) {
|
||||
spec := Spec{Kind: "day", Column: "created_at", ShardKind: schema.ShardDate, Loc: time.UTC}
|
||||
got := Bounded(context.Background(), validRange("2025-01-01", "2025-01-11"), spec, time.Second)
|
||||
if got == nil || got.Kind != "day" {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
if d := time.Duration(got.Hi-got.Lo) * time.Second; d != 10*24*time.Hour {
|
||||
t.Fatalf("span = %s", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBoundedDayRangeFromDatetimeAndEpoch(t *testing.T) {
|
||||
dt := Spec{Kind: "day", Column: "c", ShardKind: schema.ShardDateTime, Loc: time.UTC}
|
||||
if got := Bounded(context.Background(), validRange("2025-01-01 10:00:00", "2025-01-03 23:00:00"), dt, time.Second); got == nil {
|
||||
t.Fatal("datetime range unavailable")
|
||||
}
|
||||
ep := Spec{Kind: "day", Column: "ts", ShardKind: schema.ShardEpoch, Unit: 1, Loc: time.UTC}
|
||||
lo := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC).Unix()
|
||||
hi := time.Date(2025, 1, 4, 0, 0, 0, 0, time.UTC).Unix()
|
||||
got := Bounded(context.Background(), validRange(strconv.FormatInt(lo, 10), strconv.FormatInt(hi, 10)), ep, time.Second)
|
||||
if got == nil || got.Hi-got.Lo != 3*86400 {
|
||||
t.Fatalf("epoch range = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBoundedKeyRange(t *testing.T) {
|
||||
spec := Spec{Kind: "key", Column: "id"}
|
||||
got := Bounded(context.Background(), validRange("100", "1100"), spec, time.Second)
|
||||
if got == nil || got.Kind != "key" || got.Lo != 100 || got.Hi != 1100 {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBoundedSkipsUnavailableMetrics(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
q RangeQuery
|
||||
spec Spec
|
||||
}{
|
||||
{"no metric", validRange("1", "10"), Spec{}},
|
||||
{"query error", func(context.Context, string) (sql.NullString, sql.NullString, error) {
|
||||
return sql.NullString{}, sql.NullString{}, context.DeadlineExceeded
|
||||
}, Spec{Kind: "key", Column: "id"}},
|
||||
{"null bounds", func(context.Context, string) (sql.NullString, sql.NullString, error) {
|
||||
return sql.NullString{}, sql.NullString{}, nil
|
||||
}, Spec{Kind: "key", Column: "id"}},
|
||||
{"empty table", validRange("5", "5"), Spec{Kind: "key", Column: "id"}},
|
||||
{"zero date", validRange("0000-00-00", "2025-01-01"), Spec{Kind: "day", ShardKind: schema.ShardDate, Loc: time.UTC}},
|
||||
{"string key", validRange("a", "z"), Spec{Kind: "key", Column: "id"}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := Bounded(context.Background(), tc.q, tc.spec, time.Second); got != nil {
|
||||
t.Fatalf("%s: expected nil, got %+v", tc.name, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBoundedTimeoutDegrades(t *testing.T) {
|
||||
blocked := func(ctx context.Context, _ string) (sql.NullString, sql.NullString, error) {
|
||||
<-ctx.Done()
|
||||
return sql.NullString{}, sql.NullString{}, ctx.Err()
|
||||
}
|
||||
start := time.Now()
|
||||
got := Bounded(context.Background(), blocked, Spec{Kind: "key", Column: "id"}, 20*time.Millisecond)
|
||||
if got != nil {
|
||||
t.Fatalf("timed-out estimate must be nil, got %+v", got)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
t.Fatalf("timeout took too long: %s", elapsed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
// Package lag checks replica delay before each fetch.
|
||||
package lag
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
// Checker waits until a replica's delay is below Max and replication is
|
||||
// running.
|
||||
type Checker struct {
|
||||
db *sql.DB
|
||||
Max time.Duration
|
||||
Interval time.Duration
|
||||
OnWait func(reason string)
|
||||
}
|
||||
|
||||
// Open connects to the replica DSN.
|
||||
func Open(dsn string, max, interval time.Duration) (*Checker, error) {
|
||||
db, err := sql.Open("mysql", dsn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
db.SetMaxOpenConns(1)
|
||||
if err := db.Ping(); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("connect replica: %w", err)
|
||||
}
|
||||
return &Checker{db: db, Max: max, Interval: interval}, nil
|
||||
}
|
||||
|
||||
func (c *Checker) Close() error { return c.db.Close() }
|
||||
|
||||
// Wait blocks until the replica lag is acceptable or ctx is cancelled.
|
||||
func (c *Checker) Wait(ctx context.Context) error {
|
||||
for {
|
||||
reason, ok, err := c.status(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if ok {
|
||||
return nil
|
||||
}
|
||||
if c.OnWait != nil {
|
||||
c.OnWait(reason)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(c.Interval):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Checker) status(ctx context.Context) (string, bool, error) {
|
||||
fields, err := c.showStatus(ctx, "SHOW REPLICA STATUS")
|
||||
if err != nil || len(fields) == 0 {
|
||||
fields, err = c.showStatus(ctx, "SHOW SLAVE STATUS")
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
}
|
||||
if len(fields) == 0 {
|
||||
return "replication is not configured", false, nil
|
||||
}
|
||||
sqlRunning := yes(fields["Replica_SQL_Running"], fields["Slave_SQL_Running"])
|
||||
ioRunning := yes(fields["Replica_IO_Running"], fields["Slave_IO_Running"])
|
||||
if !sqlRunning || !ioRunning {
|
||||
return "replication threads are not running", false, nil
|
||||
}
|
||||
raw := firstNonEmpty(fields["Seconds_Behind_Source"], fields["Seconds_Behind_Master"])
|
||||
if raw == "" || strings.EqualFold(raw, "NULL") {
|
||||
return "replica lag is unknown", false, nil
|
||||
}
|
||||
secs, err := strconv.ParseFloat(raw, 64)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("cannot parse replica lag %q", raw), false, nil
|
||||
}
|
||||
lag := time.Duration(secs * float64(time.Second))
|
||||
if lag > c.Max {
|
||||
return fmt.Sprintf("replica lag %s exceeds %s", lag, c.Max), false, nil
|
||||
}
|
||||
return "", true, nil
|
||||
}
|
||||
|
||||
func (c *Checker) showStatus(ctx context.Context, q string) (map[string]string, error) {
|
||||
rows, err := c.db.QueryContext(ctx, q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
cols, err := rows.Columns()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !rows.Next() {
|
||||
if err := rows.Err(); err != nil && !errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, err
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
vals := make([]sql.RawBytes, len(cols))
|
||||
ptrs := make([]any, len(cols))
|
||||
for i := range vals {
|
||||
ptrs[i] = &vals[i]
|
||||
}
|
||||
if err := rows.Scan(ptrs...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make(map[string]string, len(cols))
|
||||
for i, name := range cols {
|
||||
if vals[i] == nil {
|
||||
out[name] = "NULL"
|
||||
} else {
|
||||
out[name] = string(vals[i])
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func yes(vals ...string) bool {
|
||||
for _, v := range vals {
|
||||
if strings.EqualFold(v, "Yes") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func firstNonEmpty(vals ...string) string {
|
||||
for _, v := range vals {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,707 @@
|
||||
// Package progress renders human-facing run progress for long archives.
|
||||
//
|
||||
// A Reporter receives events from the single-threaded run loop and renders
|
||||
// them from a background ticker, so blocking phases (a slow scan, fsync, lag
|
||||
// waiting) still produce a heartbeat. Rendering never touches scanner or
|
||||
// writer state directly: the run loop copies what it needs into the Reporter.
|
||||
package progress
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// rateWindow is how far back a rate sample may sit to be considered current.
|
||||
const rateWindow = 5 * time.Second
|
||||
|
||||
// Estimate describes the numeric range used for percentage and ETA.
|
||||
// Kind is "day" (position is a day boundary in unix seconds) or "key"
|
||||
// (position is a numeric scan key).
|
||||
type Estimate struct {
|
||||
Kind string
|
||||
Lo, Hi float64
|
||||
}
|
||||
|
||||
// Options configures a Reporter.
|
||||
type Options struct {
|
||||
Out io.Writer // destination, default os.Stderr
|
||||
Enabled bool // whether progress lines are produced at all
|
||||
Live bool // render in place with carriage returns
|
||||
Interval time.Duration // minimum refresh interval, default 1s
|
||||
RowEvery int64 // extra refresh every N rows; 0 disables
|
||||
Width int // live line limit, default 100
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
type sample struct {
|
||||
t time.Time
|
||||
rows int64
|
||||
pos float64
|
||||
hasPos bool
|
||||
}
|
||||
|
||||
type data struct {
|
||||
started time.Time
|
||||
phase string
|
||||
reason string
|
||||
phaseT time.Time
|
||||
base int64
|
||||
read int64
|
||||
pos float64
|
||||
hasPos bool
|
||||
originP float64
|
||||
originT time.Time
|
||||
hasOrig bool
|
||||
estimate *Estimate
|
||||
shard int64
|
||||
hasShard bool
|
||||
bytes int64
|
||||
hasBytes bool
|
||||
shards int64
|
||||
hasShardN bool
|
||||
samples []sample
|
||||
rowMark int64
|
||||
lastDraw time.Time
|
||||
}
|
||||
|
||||
func (d *data) rows() int64 { return d.base + d.read }
|
||||
|
||||
func (d *data) clone() data {
|
||||
c := *d
|
||||
c.samples = append([]sample(nil), d.samples...)
|
||||
return c
|
||||
}
|
||||
|
||||
// Reporter accumulates run metrics and renders them.
|
||||
type Reporter struct {
|
||||
out io.Writer
|
||||
now func() time.Time
|
||||
enabled bool
|
||||
live bool
|
||||
interval time.Duration
|
||||
rowEvery int64
|
||||
width int
|
||||
|
||||
mu sync.Mutex
|
||||
d data
|
||||
started bool
|
||||
running bool
|
||||
stopCh chan struct{}
|
||||
doneCh chan struct{}
|
||||
|
||||
wmu sync.Mutex
|
||||
lastWidth int
|
||||
}
|
||||
|
||||
// New builds a Reporter. A disabled Reporter still carries messages (warnings)
|
||||
// to Out but emits no progress lines.
|
||||
func New(o Options) *Reporter {
|
||||
if o.Out == nil {
|
||||
o.Out = os.Stderr
|
||||
}
|
||||
if o.Now == nil {
|
||||
o.Now = time.Now
|
||||
}
|
||||
if o.Interval <= 0 {
|
||||
o.Interval = time.Second
|
||||
}
|
||||
if o.Width <= 0 {
|
||||
o.Width = 100
|
||||
}
|
||||
return &Reporter{
|
||||
out: o.Out,
|
||||
now: o.Now,
|
||||
enabled: o.Enabled,
|
||||
live: o.Live,
|
||||
interval: o.Interval,
|
||||
rowEvery: o.RowEvery,
|
||||
width: o.Width,
|
||||
stopCh: make(chan struct{}),
|
||||
doneCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// Start begins the refresh goroutine. It is a no-op for a disabled Reporter.
|
||||
func (r *Reporter) Start() {
|
||||
r.mu.Lock()
|
||||
if r.started {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
r.started = true
|
||||
r.initLocked(r.now())
|
||||
if !r.enabled {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
r.running = true
|
||||
r.mu.Unlock()
|
||||
go r.loop()
|
||||
}
|
||||
|
||||
func (r *Reporter) initLocked(now time.Time) {
|
||||
r.d.started = now
|
||||
r.d.phaseT = now
|
||||
r.d.lastDraw = time.Time{}
|
||||
if r.rowEvery > 0 {
|
||||
r.d.rowMark = r.d.read / r.rowEvery
|
||||
}
|
||||
r.pushLocked(now)
|
||||
}
|
||||
|
||||
func (r *Reporter) loop() {
|
||||
t := time.NewTicker(r.interval)
|
||||
defer t.Stop()
|
||||
defer close(r.doneCh)
|
||||
for {
|
||||
select {
|
||||
case <-r.stopCh:
|
||||
return
|
||||
case <-t.C:
|
||||
r.maybeRender(false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SetBase installs the row count carried over from a previous run. Resetting
|
||||
// the sample ring keeps the rate measured from this run's work rather than
|
||||
// jumping by the inherited count on the first chunk.
|
||||
func (r *Reporter) SetBase(n int64) {
|
||||
r.mu.Lock()
|
||||
r.d.base = n
|
||||
r.d.samples = nil
|
||||
r.pushLocked(r.now())
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// AddRows accounts for rows read in this run.
|
||||
func (r *Reporter) AddRows(n int64) {
|
||||
if n == 0 {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.d.read += n
|
||||
now := r.now()
|
||||
r.pushLocked(now)
|
||||
crossed := false
|
||||
if r.rowEvery > 0 {
|
||||
mark := r.d.read / r.rowEvery
|
||||
if mark > r.d.rowMark {
|
||||
r.d.rowMark = mark
|
||||
crossed = true
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if crossed {
|
||||
// The threshold is an extra line at a chunk boundary; draw it inline so
|
||||
// it is not lost if the run finishes before the next tick.
|
||||
r.maybeRender(true)
|
||||
}
|
||||
}
|
||||
|
||||
// SetPosition publishes the current numeric position (day seconds or key).
|
||||
func (r *Reporter) SetPosition(pos float64) {
|
||||
r.mu.Lock()
|
||||
r.d.pos = pos
|
||||
r.d.hasPos = true
|
||||
if !r.d.hasOrig {
|
||||
r.d.originP = pos
|
||||
r.d.originT = r.now()
|
||||
r.d.hasOrig = true
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// SetEstimate publishes or clears the range used for percentage and ETA.
|
||||
func (r *Reporter) SetEstimate(e *Estimate) {
|
||||
r.mu.Lock()
|
||||
r.d.estimate = e
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// SetShard publishes the current shard sequence number.
|
||||
func (r *Reporter) SetShard(seq int64) {
|
||||
r.mu.Lock()
|
||||
r.d.shard = seq
|
||||
r.d.hasShard = true
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// SetBytes publishes bytes written so far.
|
||||
func (r *Reporter) SetBytes(n int64) {
|
||||
r.mu.Lock()
|
||||
r.d.bytes = n
|
||||
r.d.hasBytes = true
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// SetShards publishes the sealed shard count for the done summary.
|
||||
func (r *Reporter) SetShards(n int64) {
|
||||
r.mu.Lock()
|
||||
r.d.shards = n
|
||||
r.d.hasShardN = true
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// Phase records the current phase and an optional waiting reason. Repeating
|
||||
// the same phase (for example lag-wait reported on every poll) keeps the phase
|
||||
// clock running so its duration is visible.
|
||||
func (r *Reporter) Phase(name, reason string) {
|
||||
r.mu.Lock()
|
||||
if r.d.phase != name {
|
||||
r.d.phase = name
|
||||
r.d.phaseT = r.now()
|
||||
}
|
||||
r.d.reason = reason
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// Message writes a complete diagnostic line, first clearing any live line.
|
||||
func (r *Reporter) Message(format string, args ...any) {
|
||||
line := strings.TrimRight(fmt.Sprintf(format, args...), "\n")
|
||||
r.wmu.Lock()
|
||||
defer r.wmu.Unlock()
|
||||
if r.live {
|
||||
r.clearLocked()
|
||||
}
|
||||
fmt.Fprintln(r.out, line)
|
||||
}
|
||||
|
||||
// Stop ends refreshing and clears the live line without printing a summary.
|
||||
func (r *Reporter) Stop() {
|
||||
r.stopTicker()
|
||||
r.wmu.Lock()
|
||||
r.clearLocked()
|
||||
r.wmu.Unlock()
|
||||
}
|
||||
|
||||
// Done prints the final summary line on normal completion.
|
||||
func (r *Reporter) Done() {
|
||||
r.stopTicker()
|
||||
now := r.now()
|
||||
r.mu.Lock()
|
||||
s := r.d.clone()
|
||||
r.mu.Unlock()
|
||||
r.wmu.Lock()
|
||||
defer r.wmu.Unlock()
|
||||
r.clearLocked()
|
||||
if r.enabled {
|
||||
fmt.Fprintln(r.out, buildDone(s, now))
|
||||
}
|
||||
}
|
||||
|
||||
// Interrupted prints the final snapshot line after a signal.
|
||||
func (r *Reporter) Interrupted() {
|
||||
r.stopTicker()
|
||||
now := r.now()
|
||||
r.mu.Lock()
|
||||
s := r.d.clone()
|
||||
r.mu.Unlock()
|
||||
r.wmu.Lock()
|
||||
defer r.wmu.Unlock()
|
||||
r.clearLocked()
|
||||
if r.enabled {
|
||||
fmt.Fprintln(r.out, buildLine(s, now))
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reporter) stopTicker() {
|
||||
r.mu.Lock()
|
||||
if !r.running {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
r.running = false
|
||||
close(r.stopCh)
|
||||
done := r.doneCh
|
||||
r.mu.Unlock()
|
||||
<-done
|
||||
}
|
||||
|
||||
func (r *Reporter) pushLocked(now time.Time) {
|
||||
r.d.samples = append(r.d.samples, sample{
|
||||
t: now,
|
||||
rows: r.d.rows(),
|
||||
pos: r.d.pos,
|
||||
hasPos: r.d.hasPos,
|
||||
})
|
||||
cut := now.Add(-2 * rateWindow)
|
||||
i := 0
|
||||
for i < len(r.d.samples) && r.d.samples[i].t.Before(cut) {
|
||||
i++
|
||||
}
|
||||
if i > 0 {
|
||||
r.d.samples = append(r.d.samples[:0], r.d.samples[i:]...)
|
||||
}
|
||||
if len(r.d.samples) > 256 {
|
||||
r.d.samples = r.d.samples[len(r.d.samples)-256:]
|
||||
}
|
||||
}
|
||||
|
||||
// maybeRender draws a frame. force bypasses the interval throttle so a row
|
||||
// threshold can produce an extra line.
|
||||
func (r *Reporter) maybeRender(force bool) {
|
||||
if !r.enabled {
|
||||
return
|
||||
}
|
||||
now := r.now()
|
||||
r.mu.Lock()
|
||||
if !force && !r.d.lastDraw.IsZero() && now.Sub(r.d.lastDraw) < r.interval {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
r.d.lastDraw = now
|
||||
s := r.d.clone()
|
||||
r.mu.Unlock()
|
||||
|
||||
var line string
|
||||
if r.live {
|
||||
line = r.buildLive(s, now)
|
||||
} else {
|
||||
line = buildLine(s, now)
|
||||
}
|
||||
|
||||
r.wmu.Lock()
|
||||
defer r.wmu.Unlock()
|
||||
if r.live {
|
||||
if line == "" {
|
||||
line = " "
|
||||
}
|
||||
pad := 0
|
||||
if r.lastWidth > len(line) {
|
||||
pad = r.lastWidth - len(line)
|
||||
}
|
||||
fmt.Fprintf(r.out, "\r%s%s", line, strings.Repeat(" ", pad))
|
||||
r.lastWidth = len(line)
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(r.out, line)
|
||||
}
|
||||
|
||||
func (r *Reporter) clearLocked() {
|
||||
if r.live && r.lastWidth > 0 {
|
||||
fmt.Fprintf(r.out, "\r%s\r", strings.Repeat(" ", r.lastWidth))
|
||||
r.lastWidth = 0
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reporter) buildLive(s data, now time.Time) string {
|
||||
// Field priority when narrowing: drop bytes, then the shard number, then
|
||||
// shrink the bar. The line never wraps.
|
||||
type shape struct {
|
||||
bar int
|
||||
bytes bool
|
||||
shard bool
|
||||
}
|
||||
shapes := []shape{
|
||||
{20, true, true},
|
||||
{20, false, true},
|
||||
{20, false, false},
|
||||
{12, false, false},
|
||||
{8, false, false},
|
||||
{0, false, false},
|
||||
}
|
||||
var last string
|
||||
for _, sh := range shapes {
|
||||
last = liveLine(s, now, sh.bar, sh.bytes, sh.shard)
|
||||
if len(last) <= r.width {
|
||||
return last
|
||||
}
|
||||
}
|
||||
return truncate(last, r.width)
|
||||
}
|
||||
|
||||
func liveLine(s data, now time.Time, barWidth int, showBytes, showShard bool) string {
|
||||
var parts []string
|
||||
pct, eta, hasEta, hasEst := calc(s, now)
|
||||
if hasEst {
|
||||
if barWidth > 0 {
|
||||
filled := int(pct*float64(barWidth) + 0.5)
|
||||
parts = append(parts, "["+strings.Repeat("#", filled)+strings.Repeat("-", barWidth-filled)+"]")
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("~%.0f%%", pct*100))
|
||||
if s.estimate.Kind == "day" {
|
||||
cur, total := dayProgress(s)
|
||||
parts = append(parts, fmt.Sprintf("day %d/%d", cur, total))
|
||||
}
|
||||
}
|
||||
parts = append(parts, humanRows(s.rows())+" rows")
|
||||
if rr, ok := rate(s.samples, now, false); ok && rr > 0 {
|
||||
parts = append(parts, humanRate(rr))
|
||||
}
|
||||
if s.phase != "" {
|
||||
parts = append(parts, fmt.Sprintf("%s %s", s.phase, humanDuration(now.Sub(s.phaseT))))
|
||||
}
|
||||
if s.reason != "" {
|
||||
parts = append(parts, "("+s.reason+")")
|
||||
}
|
||||
if showShard && s.hasShard {
|
||||
parts = append(parts, fmt.Sprintf("shard %d", s.shard))
|
||||
}
|
||||
if showBytes && s.hasBytes {
|
||||
parts = append(parts, humanBytes(s.bytes))
|
||||
}
|
||||
if hasEta {
|
||||
parts = append(parts, "eta ~"+humanDuration(eta))
|
||||
}
|
||||
parts = append(parts, "elapsed "+humanDuration(now.Sub(s.started)))
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
func buildLine(s data, now time.Time) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("progress: rows=")
|
||||
b.WriteString(strconv.FormatInt(s.rows(), 10))
|
||||
b.WriteString(" rate=")
|
||||
if rr, ok := rate(s.samples, now, false); ok && rr > 0 {
|
||||
b.WriteString(humanRate(rr))
|
||||
} else {
|
||||
b.WriteString(humanRate(cumulativeRows(s, now)))
|
||||
}
|
||||
b.WriteString(" phase=")
|
||||
if s.phase == "" {
|
||||
b.WriteString("start")
|
||||
} else {
|
||||
b.WriteString(s.phase)
|
||||
}
|
||||
b.WriteString(" phase_elapsed=")
|
||||
b.WriteString(humanDuration(now.Sub(s.phaseT)))
|
||||
b.WriteString(" elapsed=")
|
||||
b.WriteString(humanDuration(now.Sub(s.started)))
|
||||
if s.reason != "" {
|
||||
b.WriteString(" reason=")
|
||||
b.WriteString(strings.Join(strings.Fields(s.reason), "_"))
|
||||
}
|
||||
if s.hasShard {
|
||||
fmt.Fprintf(&b, " shard=%d", s.shard)
|
||||
}
|
||||
if s.hasBytes {
|
||||
fmt.Fprintf(&b, " bytes=%d", s.bytes)
|
||||
}
|
||||
pct, eta, hasEta, hasEst := calc(s, now)
|
||||
if !hasEst {
|
||||
b.WriteString(" estimate=none")
|
||||
return b.String()
|
||||
}
|
||||
fmt.Fprintf(&b, " estimate=%s pct=~%.1f", s.estimate.Kind, pct*100)
|
||||
if s.estimate.Kind == "day" {
|
||||
cur, total := dayProgress(s)
|
||||
fmt.Fprintf(&b, " day=%d/%d", cur, total)
|
||||
}
|
||||
if hasEta {
|
||||
b.WriteString(" eta=~")
|
||||
b.WriteString(humanDuration(eta))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func buildDone(s data, now time.Time) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("done rows=")
|
||||
b.WriteString(strconv.FormatInt(s.rows(), 10))
|
||||
b.WriteString(" elapsed=")
|
||||
b.WriteString(humanDuration(now.Sub(s.started)))
|
||||
if s.hasShardN {
|
||||
fmt.Fprintf(&b, " shards=%d", s.shards)
|
||||
}
|
||||
if s.hasBytes {
|
||||
fmt.Fprintf(&b, " bytes=%d", s.bytes)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// calc returns percentage, ETA and whether an estimate is displayable.
|
||||
func calc(s data, now time.Time) (pct float64, eta time.Duration, hasEta, ok bool) {
|
||||
if s.estimate == nil || !s.hasPos {
|
||||
return 0, 0, false, false
|
||||
}
|
||||
span := s.estimate.Hi - s.estimate.Lo
|
||||
if span <= 0 {
|
||||
return 0, 0, false, false
|
||||
}
|
||||
pct = (s.pos - s.estimate.Lo) / span
|
||||
if pct < 0 {
|
||||
pct = 0
|
||||
}
|
||||
if pct > 1 {
|
||||
pct = 1
|
||||
}
|
||||
if pr, rok := positionRate(s, now); rok && pr > 0 {
|
||||
remaining := s.estimate.Hi - s.pos
|
||||
if remaining < 0 {
|
||||
remaining = 0
|
||||
}
|
||||
secs := remaining / pr
|
||||
if secs >= 0 && secs < 9e9 {
|
||||
eta = time.Duration(secs * float64(time.Second))
|
||||
hasEta = true
|
||||
}
|
||||
}
|
||||
return pct, eta, hasEta, true
|
||||
}
|
||||
|
||||
func dayProgress(s data) (int, int) {
|
||||
const day = 86400.0
|
||||
total := int((s.estimate.Hi-s.estimate.Lo)/day) + 1
|
||||
cur := int((s.pos-s.estimate.Lo)/day) + 1
|
||||
if cur < 1 {
|
||||
cur = 1
|
||||
}
|
||||
if cur > total {
|
||||
cur = total
|
||||
}
|
||||
return cur, total
|
||||
}
|
||||
|
||||
func rate(samples []sample, now time.Time, usePos bool) (float64, bool) {
|
||||
cut := now.Add(-rateWindow)
|
||||
var first *sample
|
||||
for i := range samples {
|
||||
if usePos && !samples[i].hasPos {
|
||||
continue
|
||||
}
|
||||
if !samples[i].t.Before(cut) {
|
||||
first = &samples[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if first == nil {
|
||||
for i := len(samples) - 1; i >= 0; i-- {
|
||||
if !usePos || samples[i].hasPos {
|
||||
first = &samples[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if first == nil || len(samples) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
last := &samples[len(samples)-1]
|
||||
if usePos && !last.hasPos {
|
||||
return 0, false
|
||||
}
|
||||
if first == last {
|
||||
return 0, false
|
||||
}
|
||||
dt := last.t.Sub(first.t).Seconds()
|
||||
if dt <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
var d float64
|
||||
if usePos {
|
||||
d = last.pos - first.pos
|
||||
} else {
|
||||
d = float64(last.rows - first.rows)
|
||||
}
|
||||
if d < 0 {
|
||||
return 0, false
|
||||
}
|
||||
return d / dt, true
|
||||
}
|
||||
|
||||
func positionRate(s data, now time.Time) (float64, bool) {
|
||||
if v, ok := rate(s.samples, now, true); ok {
|
||||
return v, true
|
||||
}
|
||||
if s.hasOrig && now.After(s.originT) {
|
||||
return (s.pos - s.originP) / now.Sub(s.originT).Seconds(), true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func cumulativeRows(s data, now time.Time) float64 {
|
||||
el := now.Sub(s.started).Seconds()
|
||||
if el <= 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(s.read) / el
|
||||
}
|
||||
|
||||
func humanRows(n int64) string {
|
||||
switch {
|
||||
case n >= 1_000_000_000:
|
||||
return fmt.Sprintf("%.2fG", float64(n)/1e9)
|
||||
case n >= 1_000_000:
|
||||
return fmt.Sprintf("%.2fM", float64(n)/1e6)
|
||||
case n >= 10_000:
|
||||
return fmt.Sprintf("%.1fk", float64(n)/1e3)
|
||||
default:
|
||||
return strconv.FormatInt(n, 10)
|
||||
}
|
||||
}
|
||||
|
||||
func humanRate(v float64) string {
|
||||
switch {
|
||||
case v >= 1_000_000:
|
||||
return fmt.Sprintf("%.2fM/s", v/1e6)
|
||||
case v >= 10_000:
|
||||
return fmt.Sprintf("%.1fk/s", v/1e3)
|
||||
case v >= 1:
|
||||
return fmt.Sprintf("%.0f/s", v)
|
||||
default:
|
||||
return "0/s"
|
||||
}
|
||||
}
|
||||
|
||||
func humanBytes(n int64) string {
|
||||
const unit = 1024
|
||||
if n < unit {
|
||||
return fmt.Sprintf("%dB", n)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for v := n / unit; v >= unit; v /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f%ciB", float64(n)/float64(div), "KMGTPE"[exp])
|
||||
}
|
||||
|
||||
func humanDuration(d time.Duration) string {
|
||||
if d < 0 {
|
||||
d = 0
|
||||
}
|
||||
if d < time.Second {
|
||||
return fmt.Sprintf("%dms", d.Milliseconds())
|
||||
}
|
||||
secs := int(d.Seconds())
|
||||
switch {
|
||||
case secs < 60:
|
||||
return fmt.Sprintf("%ds", secs)
|
||||
case secs < 3600:
|
||||
return fmt.Sprintf("%dm%02ds", secs/60, secs%60)
|
||||
default:
|
||||
return fmt.Sprintf("%dh%02dm", secs/3600, (secs%3600)/60)
|
||||
}
|
||||
}
|
||||
|
||||
func truncate(s string, max int) string {
|
||||
if len(s) <= max {
|
||||
return s
|
||||
}
|
||||
if max <= 0 {
|
||||
return ""
|
||||
}
|
||||
runes := []rune(s)
|
||||
if len(runes) <= max {
|
||||
return s
|
||||
}
|
||||
return string(runes[:max])
|
||||
}
|
||||
|
||||
// IsTerminal reports whether w is a character device (a terminal).
|
||||
func IsTerminal(w io.Writer) bool {
|
||||
f, ok := w.(*os.File)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return st.Mode()&os.ModeCharDevice != 0
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package progress
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type clock struct{ t time.Time }
|
||||
|
||||
func (c *clock) now() time.Time { return c.t }
|
||||
func (c *clock) add(d time.Duration) { c.t = c.t.Add(d) }
|
||||
|
||||
func newTestReporter(t *testing.T, live, enabled bool, opts ...func(*Options)) (*Reporter, *bytes.Buffer, *clock) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
clk := &clock{t: time.Unix(1_700_000_000, 0)}
|
||||
o := Options{Out: &buf, Enabled: enabled, Live: live, Interval: time.Second, Now: clk.now}
|
||||
for _, fn := range opts {
|
||||
fn(&o)
|
||||
}
|
||||
r := New(o)
|
||||
r.mu.Lock()
|
||||
r.started = true
|
||||
r.initLocked(clk.t)
|
||||
r.mu.Unlock()
|
||||
return r, &buf, clk
|
||||
}
|
||||
|
||||
func TestLiveLineHasRequiredFields(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, true, true)
|
||||
r.SetEstimate(&Estimate{Kind: "day", Lo: 0, Hi: 10 * 86400})
|
||||
r.Phase("scan", "")
|
||||
r.SetShard(12)
|
||||
r.SetPosition(3 * 86400)
|
||||
r.AddRows(1000)
|
||||
clk.add(time.Second)
|
||||
r.SetPosition(4 * 86400)
|
||||
r.AddRows(1000)
|
||||
r.maybeRender(true)
|
||||
|
||||
out := buf.String()
|
||||
if !strings.HasPrefix(out, "\r") {
|
||||
t.Fatalf("live line must start with carriage return: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "\n") {
|
||||
t.Fatalf("live line must not emit a newline: %q", out)
|
||||
}
|
||||
for _, want := range []string{"rows", "~40%", "day 5/11", "shard 12", "scan", "eta ~", "elapsed"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("live line %q missing %q", out, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLiveLineOmitsEstimateWhenUnavailable(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, true, true)
|
||||
r.Phase("scan", "")
|
||||
r.AddRows(10)
|
||||
r.maybeRender(true)
|
||||
out := buf.String()
|
||||
if strings.Contains(out, "%") || strings.Contains(out, "eta") {
|
||||
t.Fatalf("unexpected estimate in %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "rows") {
|
||||
t.Fatalf("missing rows in %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLineRenderingIsStableKeyValue(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, false, true)
|
||||
r.SetEstimate(&Estimate{Kind: "key", Lo: 0, Hi: 1000})
|
||||
r.Phase("db", "")
|
||||
r.SetPosition(100)
|
||||
r.AddRows(500)
|
||||
clk.add(time.Second)
|
||||
r.SetPosition(500)
|
||||
r.AddRows(500)
|
||||
r.maybeRender(true)
|
||||
|
||||
out := buf.String()
|
||||
if strings.Contains(out, "\r") {
|
||||
t.Fatalf("line mode must not use carriage returns: %q", out)
|
||||
}
|
||||
if !strings.HasSuffix(out, "\n") {
|
||||
t.Fatalf("line mode must end with a newline: %q", out)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"progress: rows=1000", "rate=", "phase=db", "phase_elapsed=", "elapsed=",
|
||||
"estimate=key", "pct=~50.0", "eta=~",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("line %q missing %q", out, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLineRenderingMarksEstimateNone(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, false, true)
|
||||
r.Phase("scan", "")
|
||||
r.AddRows(100)
|
||||
r.maybeRender(true)
|
||||
if out := buf.String(); !strings.Contains(out, "estimate=none") {
|
||||
t.Fatalf("want estimate=none, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntervalThrottle(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, false, true)
|
||||
r.maybeRender(true)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 1 {
|
||||
t.Fatalf("first frame not drawn: %q", buf.String())
|
||||
}
|
||||
clk.add(200 * time.Millisecond)
|
||||
r.maybeRender(false)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 1 {
|
||||
t.Fatalf("throttled frame drawn early: %q", buf.String())
|
||||
}
|
||||
clk.add(time.Second)
|
||||
r.maybeRender(false)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 2 {
|
||||
t.Fatalf("frame after interval not drawn: %q", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRowThresholdRenders(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, false, true, func(o *Options) { o.RowEvery = 100 })
|
||||
r.AddRows(100)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 1 {
|
||||
t.Fatalf("crossing the first threshold did not render: %q", buf.String())
|
||||
}
|
||||
r.AddRows(50)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 1 {
|
||||
t.Fatalf("sub-threshold rows must not render: %q", buf.String())
|
||||
}
|
||||
r.AddRows(50)
|
||||
if n := strings.Count(buf.String(), "\n"); n != 2 {
|
||||
t.Fatalf("crossing the next threshold did not render: %q", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageClearsLiveLine(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, true, true)
|
||||
r.AddRows(1000)
|
||||
r.maybeRender(true)
|
||||
width := r.lastWidth
|
||||
buf.Reset()
|
||||
r.Message("warning: %s", "something")
|
||||
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "\r"+strings.Repeat(" ", width)+"\r") {
|
||||
t.Fatalf("message did not clear the live line: %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "warning: something\n") {
|
||||
t.Fatalf("message missing: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "rows=") {
|
||||
t.Fatalf("live line leaked into message: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShortLineIsPadded(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, true, true)
|
||||
r.SetEstimate(&Estimate{Kind: "day", Lo: 0, Hi: 100 * 86400})
|
||||
r.SetPosition(50 * 86400)
|
||||
r.AddRows(1000)
|
||||
r.maybeRender(true)
|
||||
first := r.lastWidth
|
||||
|
||||
buf.Reset()
|
||||
r.SetEstimate(nil)
|
||||
clk.add(2 * time.Second)
|
||||
r.AddRows(1000)
|
||||
r.maybeRender(true)
|
||||
second := buf.String()
|
||||
if r.lastWidth >= first {
|
||||
t.Fatalf("second frame should be shorter: %d -> %d (%q)", first, r.lastWidth, second)
|
||||
}
|
||||
if !strings.HasSuffix(second, strings.Repeat(" ", first-r.lastWidth)) {
|
||||
t.Fatalf("short frame not padded to previous width: %q", second)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDoneSummary(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, false, true)
|
||||
r.SetShard(3)
|
||||
r.SetShards(3)
|
||||
r.SetBytes(4096)
|
||||
r.AddRows(1500)
|
||||
clk.add(3 * time.Second)
|
||||
r.Done()
|
||||
out := buf.String()
|
||||
for _, want := range []string{"done rows=1500", "elapsed=3s", "shards=3", "bytes=4096"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("done line %q missing %q", out, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInterruptedSnapshot(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, true, true)
|
||||
r.Phase("file", "")
|
||||
r.AddRows(42)
|
||||
r.maybeRender(true)
|
||||
buf.Reset()
|
||||
r.Interrupted()
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "progress: rows=42") {
|
||||
t.Fatalf("final snapshot missing: %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "phase=file") {
|
||||
t.Fatalf("final snapshot missing phase: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisabledReporterIsSilentButMessagesWork(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, false, false)
|
||||
r.AddRows(10)
|
||||
r.maybeRender(true)
|
||||
r.Done()
|
||||
if buf.Len() != 0 {
|
||||
t.Fatalf("disabled reporter emitted progress: %q", buf.String())
|
||||
}
|
||||
r.Message("warning: still visible")
|
||||
if !strings.Contains(buf.String(), "warning: still visible") {
|
||||
t.Fatalf("messages must survive disabled progress: %q", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatedPhaseKeepsClock(t *testing.T) {
|
||||
r, buf, clk := newTestReporter(t, false, true)
|
||||
r.Phase("lag-wait", "reason a")
|
||||
clk.add(3 * time.Second)
|
||||
r.Phase("lag-wait", "reason b")
|
||||
r.maybeRender(true)
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "phase=lag-wait") || !strings.Contains(out, "phase_elapsed=3s") {
|
||||
t.Fatalf("repeated phase reset its clock: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLagWaitShowsReason(t *testing.T) {
|
||||
r, buf, _ := newTestReporter(t, false, true)
|
||||
r.Phase("lag-wait", "replica lag 3s exceeds 1s")
|
||||
r.AddRows(10)
|
||||
r.maybeRender(true)
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "phase=lag-wait") || !strings.Contains(out, "reason=replica_lag_3s_exceeds_1s") {
|
||||
t.Fatalf("lag reason missing: %q", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
// Package purge deletes archived rows from the source table, either by an
|
||||
// explicit key IN list or by key range.
|
||||
package purge
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// Execer is satisfied by *sql.DB, *sql.Tx and *sql.Conn.
|
||||
type Execer interface {
|
||||
ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
|
||||
}
|
||||
|
||||
const maxParams = 1000
|
||||
|
||||
// Purger deletes rows chunk by chunk.
|
||||
type Purger struct {
|
||||
DB *sql.DB
|
||||
DBName string
|
||||
Table string
|
||||
KeyCols []string
|
||||
Where string
|
||||
Bulk bool
|
||||
SafeAutoInc bool
|
||||
AutoIncKey bool
|
||||
KeyIdx []int
|
||||
}
|
||||
|
||||
// New builds a Purger. keyIdx is unused for deletion but kept for symmetry
|
||||
// with the scan configuration.
|
||||
func New(db *sql.DB, dbName, table string, keyCols []string, where string, bulk, safe, autoInc bool) *Purger {
|
||||
return &Purger{
|
||||
DB: db,
|
||||
DBName: dbName,
|
||||
Table: table,
|
||||
KeyCols: keyCols,
|
||||
Where: where,
|
||||
Bulk: bulk,
|
||||
SafeAutoInc: safe,
|
||||
AutoIncKey: autoInc,
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteChunk removes the rows read in this chunk using exec. It returns the
|
||||
// number of affected rows.
|
||||
func (p *Purger) DeleteChunk(ctx context.Context, exec Execer, c *row.Chunk) (int64, error) {
|
||||
if len(c.Keys) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
keys := c.Keys
|
||||
var maxKey *row.KeyPart
|
||||
if p.SafeAutoInc && p.AutoIncKey && len(p.KeyCols) == 1 {
|
||||
v, err := p.currentMax(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
maxKey = v
|
||||
}
|
||||
|
||||
if p.Bulk && maxKey == nil {
|
||||
return p.deleteRange(ctx, exec, c)
|
||||
}
|
||||
return p.deleteInList(ctx, exec, keys, maxKey)
|
||||
}
|
||||
|
||||
func (p *Purger) table() string {
|
||||
return sqlutil.Qualified(p.DBName, p.Table)
|
||||
}
|
||||
|
||||
func (p *Purger) currentMax(ctx context.Context) (*row.KeyPart, error) {
|
||||
q := fmt.Sprintf("SELECT MAX(%s) FROM %s", sqlutil.Ident(p.KeyCols[0]), p.table())
|
||||
var v sql.NullString
|
||||
if err := p.DB.QueryRowContext(ctx, q).Scan(&v); err != nil {
|
||||
return nil, fmt.Errorf("read max auto-increment key: %w", err)
|
||||
}
|
||||
if !v.Valid {
|
||||
return nil, nil
|
||||
}
|
||||
return &row.KeyPart{Val: v.String}, nil
|
||||
}
|
||||
|
||||
func (p *Purger) deleteRange(ctx context.Context, exec Execer, c *row.Chunk) (int64, error) {
|
||||
lo, hi := c.FirstKey, c.LastKey
|
||||
if lo.IsZero() || hi.IsZero() {
|
||||
return 0, nil
|
||||
}
|
||||
var (
|
||||
q string
|
||||
args []any
|
||||
)
|
||||
if len(p.KeyCols) == 1 {
|
||||
q = fmt.Sprintf("DELETE FROM %s WHERE %s BETWEEN ? AND ? AND (%s)",
|
||||
p.table(), sqlutil.Ident(p.KeyCols[0]), p.Where)
|
||||
args = []any{lo[0].DriverValue(), hi[0].DriverValue()}
|
||||
} else {
|
||||
q = fmt.Sprintf("DELETE FROM %s WHERE %s >= %s AND %s <= %s AND (%s)",
|
||||
p.table(), sqlutil.RowTuple(p.KeyCols), sqlutil.ParamTuple(len(p.KeyCols)),
|
||||
sqlutil.RowTuple(p.KeyCols), sqlutil.ParamTuple(len(p.KeyCols)), p.Where)
|
||||
args = append(args, lo.Args()...)
|
||||
args = append(args, hi.Args()...)
|
||||
}
|
||||
res, err := exec.ExecContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("bulk delete: %w", err)
|
||||
}
|
||||
return res.RowsAffected()
|
||||
}
|
||||
|
||||
func (p *Purger) deleteInList(ctx context.Context, exec Execer, keys []row.Key, maxKey *row.KeyPart) (int64, error) {
|
||||
batch := maxParams / len(p.KeyCols)
|
||||
if batch < 1 {
|
||||
batch = 1
|
||||
}
|
||||
var total int64
|
||||
for start := 0; start < len(keys); start += batch {
|
||||
end := start + batch
|
||||
if end > len(keys) {
|
||||
end = len(keys)
|
||||
}
|
||||
var args [][]any
|
||||
for _, k := range keys[start:end] {
|
||||
if maxKey != nil && keyEquals(k[len(k)-1], *maxKey) {
|
||||
continue
|
||||
}
|
||||
args = append(args, k.Args())
|
||||
}
|
||||
if len(args) == 0 {
|
||||
continue
|
||||
}
|
||||
q := fmt.Sprintf("DELETE FROM %s WHERE %s AND (%s)",
|
||||
p.table(), sqlutil.InList(p.KeyCols, len(args)), p.Where)
|
||||
flat := make([]any, 0, len(args)*len(p.KeyCols))
|
||||
for _, a := range args {
|
||||
flat = append(flat, a...)
|
||||
}
|
||||
res, err := exec.ExecContext(ctx, q, flat...)
|
||||
if err != nil {
|
||||
return total, fmt.Errorf("delete chunk: %w", err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return total, err
|
||||
}
|
||||
total += n
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
|
||||
func keyEquals(a, b row.KeyPart) bool {
|
||||
if a.Null || b.Null {
|
||||
return a.Null && b.Null
|
||||
}
|
||||
if a.Val == b.Val {
|
||||
return true
|
||||
}
|
||||
ai, err1 := strconv.ParseInt(a.Val, 10, 64)
|
||||
bi, err2 := strconv.ParseInt(b.Val, 10, 64)
|
||||
return err1 == nil && err2 == nil && ai == bi
|
||||
}
|
||||
|
||||
// DryRunSQL returns representative delete statements.
|
||||
func (p *Purger) DryRunSQL() []string {
|
||||
in := fmt.Sprintf("DELETE FROM %s WHERE %s IN (?, ?) AND (%s)",
|
||||
p.table(), sqlutil.RowTuple(p.KeyCols), p.Where)
|
||||
if len(p.KeyCols) == 1 {
|
||||
in = fmt.Sprintf("DELETE FROM %s WHERE %s IN (?, ?) AND (%s)",
|
||||
p.table(), sqlutil.Ident(p.KeyCols[0]), p.Where)
|
||||
}
|
||||
out := []string{in}
|
||||
if p.Bulk {
|
||||
if len(p.KeyCols) == 1 {
|
||||
out = append(out, fmt.Sprintf("DELETE FROM %s WHERE %s BETWEEN ? AND ? AND (%s) -- --bulk-delete",
|
||||
p.table(), sqlutil.Ident(p.KeyCols[0]), p.Where))
|
||||
} else {
|
||||
out = append(out, fmt.Sprintf("DELETE FROM %s WHERE %s >= %s AND %s <= %s AND (%s) -- --bulk-delete",
|
||||
p.table(), sqlutil.RowTuple(p.KeyCols), sqlutil.ParamTuple(len(p.KeyCols)),
|
||||
sqlutil.RowTuple(p.KeyCols), sqlutil.ParamTuple(len(p.KeyCols)), p.Where))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package row
|
||||
|
||||
import "bytes"
|
||||
|
||||
// csvEncoder emits RFC4180 CSV with a column-name header and CRLF line
|
||||
// endings. NULL is written as the configured token and empty strings as empty
|
||||
// fields.
|
||||
type csvEncoder struct {
|
||||
cols []Column
|
||||
idx []int
|
||||
null string
|
||||
}
|
||||
|
||||
func (e *csvEncoder) Header() []byte {
|
||||
var b bytes.Buffer
|
||||
for n, i := range e.idx {
|
||||
if n > 0 {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
writeCSVField(&b, []byte(e.cols[i].Name))
|
||||
}
|
||||
b.WriteString("\r\n")
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func (e *csvEncoder) Encode(dst []byte, r Row) ([]byte, error) {
|
||||
for n, i := range e.idx {
|
||||
if n > 0 {
|
||||
dst = append(dst, ',')
|
||||
}
|
||||
v := r.Values[i]
|
||||
if v.IsNull() {
|
||||
dst = append(dst, e.null...)
|
||||
continue
|
||||
}
|
||||
dst = appendCSVField(dst, v.Raw)
|
||||
}
|
||||
return append(dst, '\r', '\n'), nil
|
||||
}
|
||||
|
||||
func needsCSVQuote(b []byte) bool {
|
||||
for _, c := range b {
|
||||
switch c {
|
||||
case ',', '"', '\r', '\n':
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func appendCSVField(dst, field []byte) []byte {
|
||||
if !needsCSVQuote(field) {
|
||||
return append(dst, field...)
|
||||
}
|
||||
dst = append(dst, '"')
|
||||
for _, c := range field {
|
||||
if c == '"' {
|
||||
dst = append(dst, '"', '"')
|
||||
continue
|
||||
}
|
||||
dst = append(dst, c)
|
||||
}
|
||||
return append(dst, '"')
|
||||
}
|
||||
|
||||
func writeCSVField(b *bytes.Buffer, field []byte) {
|
||||
b.Write(appendCSVField(nil, field))
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package row
|
||||
|
||||
// dumpEncoder emits MySQL LOAD DATA compatible TSV. Fields are separated by
|
||||
// TAB, rows by LF, NULL is \N and the escape table matches
|
||||
// SELECT ... INTO OUTFILE with the default FIELDS ESCAPED BY '\\'.
|
||||
type dumpEncoder struct {
|
||||
idx []int
|
||||
}
|
||||
|
||||
func (e *dumpEncoder) Header() []byte { return nil }
|
||||
|
||||
func (e *dumpEncoder) Encode(dst []byte, r Row) ([]byte, error) {
|
||||
for n, i := range e.idx {
|
||||
if n > 0 {
|
||||
dst = append(dst, '\t')
|
||||
}
|
||||
v := r.Values[i]
|
||||
if v.IsNull() {
|
||||
dst = append(dst, '\\', 'N')
|
||||
continue
|
||||
}
|
||||
for _, b := range v.Raw {
|
||||
switch b {
|
||||
case '\\':
|
||||
dst = append(dst, '\\', '\\')
|
||||
case '\t':
|
||||
dst = append(dst, '\\', 't')
|
||||
case '\n':
|
||||
dst = append(dst, '\\', 'n')
|
||||
case '\r':
|
||||
dst = append(dst, '\\', 'r')
|
||||
case 0:
|
||||
dst = append(dst, '\\', '0')
|
||||
case '\b':
|
||||
dst = append(dst, '\\', 'b')
|
||||
case 0x1a:
|
||||
dst = append(dst, '\\', 'Z')
|
||||
default:
|
||||
dst = append(dst, b)
|
||||
}
|
||||
}
|
||||
}
|
||||
return append(dst, '\n'), nil
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package row
|
||||
|
||||
import "fmt"
|
||||
|
||||
// Options carries encoder-level knobs shared by all formats.
|
||||
type Options struct {
|
||||
CSVNull string
|
||||
}
|
||||
|
||||
// Encoder appends the textual representation of a row to dst. It is not safe
|
||||
// for concurrent use; each shard owns its own encoder.
|
||||
type Encoder interface {
|
||||
// Header returns bytes that must be written at the start of every shard.
|
||||
Header() []byte
|
||||
// Encode appends one row to dst.
|
||||
Encode(dst []byte, r Row) ([]byte, error)
|
||||
}
|
||||
|
||||
// NewEncoder builds an encoder for the given format. cols describes every
|
||||
// column in the row (select order) and idx selects which of them are emitted,
|
||||
// in output order.
|
||||
func NewEncoder(format string, cols []Column, idx []int, opts Options) (Encoder, error) {
|
||||
if len(idx) == 0 {
|
||||
idx = make([]int, len(cols))
|
||||
for i := range cols {
|
||||
idx[i] = i
|
||||
}
|
||||
}
|
||||
switch format {
|
||||
case "dump":
|
||||
return &dumpEncoder{idx: idx}, nil
|
||||
case "csv":
|
||||
return &csvEncoder{cols: cols, idx: idx, null: opts.CSVNull}, nil
|
||||
case "jsonl":
|
||||
return &jsonlEncoder{cols: cols, idx: idx}, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported format %q", format)
|
||||
}
|
||||
}
|
||||
|
||||
// FormatExt returns the file extension for a format.
|
||||
func FormatExt(format string) string {
|
||||
switch format {
|
||||
case "dump":
|
||||
return "dump"
|
||||
case "csv":
|
||||
return "csv"
|
||||
case "jsonl":
|
||||
return "jsonl"
|
||||
}
|
||||
return format
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package row
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func vals(raw ...any) Row {
|
||||
r := Row{}
|
||||
for _, v := range raw {
|
||||
switch x := v.(type) {
|
||||
case nil:
|
||||
r.Values = append(r.Values, Value{})
|
||||
case string:
|
||||
r.Values = append(r.Values, Value{Raw: []byte(x)})
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func cols(names ...string) []Column {
|
||||
out := make([]Column, len(names))
|
||||
for i, n := range names {
|
||||
out[i] = Column{Name: n, Type: "varchar"}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestDumpEncoding(t *testing.T) {
|
||||
enc, err := NewEncoder("dump", cols("a", "b", "c"), nil, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := vals("x\ty", nil, `a\nb`)
|
||||
got, err := enc.Encode(nil, r)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "x\\ty\t\\N\ta\\\\nb\n"
|
||||
if string(got) != want {
|
||||
t.Fatalf("got %q want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDumpEscapesControlBytes(t *testing.T) {
|
||||
enc, _ := NewEncoder("dump", cols("a"), nil, Options{})
|
||||
r := Row{Values: []Value{{Raw: []byte{'a', 0, '\b', 0x1a, '\r'}}}}
|
||||
got, _ := enc.Encode(nil, r)
|
||||
want := "a\\0\\b\\Z\\r\n"
|
||||
if string(got) != want {
|
||||
t.Fatalf("got %q want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCSVEncoding(t *testing.T) {
|
||||
enc, err := NewEncoder("csv", cols("a", "b", "c", "d"), nil, Options{CSVNull: `\N`})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if h := string(enc.Header()); h != "a,b,c,d\r\n" {
|
||||
t.Fatalf("header %q", h)
|
||||
}
|
||||
r := vals(`a,"b"`, "", nil, "plain")
|
||||
got, _ := enc.Encode(nil, r)
|
||||
want := "\"a,\"\"b\"\"\",,\\N,plain\r\n"
|
||||
if string(got) != want {
|
||||
t.Fatalf("got %q want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCSVNullToken(t *testing.T) {
|
||||
enc, _ := NewEncoder("csv", cols("a"), nil, Options{CSVNull: "NULL"})
|
||||
got, _ := enc.Encode(nil, vals(nil))
|
||||
if string(got) != "NULL\r\n" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONLTypes(t *testing.T) {
|
||||
columns := []Column{
|
||||
{Name: "id", Type: "bigint"},
|
||||
{Name: "price", Type: "decimal"},
|
||||
{Name: "created", Type: "datetime"},
|
||||
{Name: "day", Type: "date"},
|
||||
{Name: "blob", Type: "blob"},
|
||||
{Name: "note", Type: "varchar"},
|
||||
{Name: "missing", Type: "varchar"},
|
||||
}
|
||||
enc, _ := NewEncoder("jsonl", columns, nil, Options{})
|
||||
r := vals("42", "12.30", "2024-01-02 03:04:05", "2024-01-02", "hi", `line "one"`, nil)
|
||||
got, _ := enc.Encode(nil, r)
|
||||
want := `{"id":42,"price":"12.30","created":"2024-01-02T03:04:05Z","day":"2024-01-02","blob":"aGk=","note":"line \"one\"","missing":null}` + "\n"
|
||||
if string(got) != want {
|
||||
t.Fatalf("got %s\nwant %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONLColumnProjection(t *testing.T) {
|
||||
columns := []Column{{Name: "a", Type: "int"}, {Name: "b", Type: "int"}, {Name: "c", Type: "int"}}
|
||||
enc, _ := NewEncoder("jsonl", columns, []int{2, 0}, Options{})
|
||||
got, _ := enc.Encode(nil, vals("1", "2", "3"))
|
||||
if strings.TrimSpace(string(got)) != `{"c":3,"a":1}` {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package row
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"time"
|
||||
)
|
||||
|
||||
// jsonlEncoder emits one JSON object per line. Types are mapped deliberately
|
||||
// so downstream tools see useful values: decimals become strings to preserve
|
||||
// precision, binary columns become base64, date/time columns become strings
|
||||
// and NULL becomes JSON null.
|
||||
type jsonlEncoder struct {
|
||||
cols []Column
|
||||
idx []int
|
||||
}
|
||||
|
||||
func (e *jsonlEncoder) Header() []byte { return nil }
|
||||
|
||||
func (e *jsonlEncoder) Encode(dst []byte, r Row) ([]byte, error) {
|
||||
dst = append(dst, '{')
|
||||
for n, i := range e.idx {
|
||||
if n > 0 {
|
||||
dst = append(dst, ',')
|
||||
}
|
||||
dst = appendJSONString(dst, []byte(e.cols[i].Name))
|
||||
dst = append(dst, ':')
|
||||
dst = appendJSONValue(dst, e.cols[i], r.Values[i])
|
||||
}
|
||||
return append(dst, '}', '\n'), nil
|
||||
}
|
||||
|
||||
func appendJSONValue(dst []byte, col Column, v Value) []byte {
|
||||
if v.IsNull() {
|
||||
return append(dst, "null"...)
|
||||
}
|
||||
switch {
|
||||
case IsInteger(col.Type), IsFloat(col.Type):
|
||||
if looksNumeric(v.Raw) {
|
||||
return append(dst, v.Raw...)
|
||||
}
|
||||
return appendJSONString(dst, v.Raw)
|
||||
case IsDecimal(col.Type):
|
||||
return appendJSONString(dst, v.Raw)
|
||||
case IsBinary(col.Type):
|
||||
dst = append(dst, '"')
|
||||
dst = append(dst, base64.StdEncoding.EncodeToString(v.Raw)...)
|
||||
return append(dst, '"')
|
||||
case IsDate(col.Type):
|
||||
return appendJSONString(dst, v.Raw)
|
||||
case IsDateTime(col.Type):
|
||||
if s, ok := formatRFC3339(v.Raw); ok {
|
||||
return appendJSONString(dst, []byte(s))
|
||||
}
|
||||
return appendJSONString(dst, v.Raw)
|
||||
default:
|
||||
return appendJSONString(dst, v.Raw)
|
||||
}
|
||||
}
|
||||
|
||||
func looksNumeric(b []byte) bool {
|
||||
if len(b) == 0 {
|
||||
return false
|
||||
}
|
||||
for i, c := range b {
|
||||
if (c >= '0' && c <= '9') || c == '-' || c == '+' || c == '.' || c == 'e' || c == 'E' {
|
||||
continue
|
||||
}
|
||||
if i == 0 && c == ' ' {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
var dateTimeLayouts = []string{
|
||||
"2006-01-02 15:04:05.999999999",
|
||||
"2006-01-02 15:04:05",
|
||||
"2006-01-02T15:04:05Z07:00",
|
||||
"2006-01-02T15:04:05.999999999Z07:00",
|
||||
}
|
||||
|
||||
func formatRFC3339(raw []byte) (string, bool) {
|
||||
s := string(raw)
|
||||
for _, layout := range dateTimeLayouts {
|
||||
t, err := time.ParseInLocation(layout, s, time.UTC)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if t.Nanosecond() != 0 {
|
||||
return t.UTC().Format(time.RFC3339Nano), true
|
||||
}
|
||||
return t.UTC().Format(time.RFC3339), true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
const hexDigits = "0123456789abcdef"
|
||||
|
||||
func appendJSONString(dst, s []byte) []byte {
|
||||
dst = append(dst, '"')
|
||||
for _, c := range s {
|
||||
switch c {
|
||||
case '"', '\\':
|
||||
dst = append(dst, '\\', c)
|
||||
case '\n':
|
||||
dst = append(dst, '\\', 'n')
|
||||
case '\r':
|
||||
dst = append(dst, '\\', 'r')
|
||||
case '\t':
|
||||
dst = append(dst, '\\', 't')
|
||||
case '\b':
|
||||
dst = append(dst, '\\', 'b')
|
||||
case '\f':
|
||||
dst = append(dst, '\\', 'f')
|
||||
default:
|
||||
if c < 0x20 {
|
||||
dst = append(dst, '\\', 'u', '0', '0', hexDigits[c>>4], hexDigits[c&0xf])
|
||||
} else {
|
||||
dst = append(dst, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
return append(dst, '"')
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
// Package row defines scanned values, key tuples and the line encoders used
|
||||
// to serialize rows into dump, csv and jsonl shards.
|
||||
package row
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Value is one column value as returned by the MySQL driver with
|
||||
// parseTime disabled. A nil Raw means SQL NULL; an empty non-nil Raw means
|
||||
// the empty string.
|
||||
type Value struct {
|
||||
Raw []byte
|
||||
}
|
||||
|
||||
func (v Value) IsNull() bool { return v.Raw == nil }
|
||||
func (v Value) Bytes() []byte { return v.Raw }
|
||||
func (v Value) String() string { return string(v.Raw) }
|
||||
|
||||
// DriverValue returns the value as a bind parameter.
|
||||
func (v Value) DriverValue() any {
|
||||
if v.Raw == nil {
|
||||
return nil
|
||||
}
|
||||
return v.Raw
|
||||
}
|
||||
|
||||
// KeyPart is a single component of a key tuple. Values are held as text so a
|
||||
// key can be serialized into state.json and rebound to the driver without
|
||||
// carrying driver-specific types.
|
||||
type KeyPart struct {
|
||||
Null bool `json:"null,omitempty"`
|
||||
Val string `json:"val,omitempty"`
|
||||
}
|
||||
|
||||
// DriverValue converts the part into a value accepted by database/sql.
|
||||
func (p KeyPart) DriverValue() any {
|
||||
if p.Null {
|
||||
return nil
|
||||
}
|
||||
return p.Val
|
||||
}
|
||||
|
||||
// Key is a tuple of values identifying a row position in a unique index.
|
||||
type Key []KeyPart
|
||||
|
||||
func KeyOf(values ...string) Key {
|
||||
parts := make(Key, len(values))
|
||||
for i, v := range values {
|
||||
parts[i] = KeyPart{Val: v}
|
||||
}
|
||||
return parts
|
||||
}
|
||||
|
||||
// Args returns the key as driver bind parameters.
|
||||
func (k Key) Args() []any {
|
||||
out := make([]any, len(k))
|
||||
for i, p := range k {
|
||||
out[i] = p.DriverValue()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (k Key) IsZero() bool { return len(k) == 0 }
|
||||
|
||||
func (k Key) Equal(o Key) bool {
|
||||
if len(k) != len(o) {
|
||||
return false
|
||||
}
|
||||
for i := range k {
|
||||
if k[i] != o[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (k Key) String() string {
|
||||
if len(k) == 0 {
|
||||
return "()"
|
||||
}
|
||||
parts := make([]string, len(k))
|
||||
for i, p := range k {
|
||||
if p.Null {
|
||||
parts[i] = "NULL"
|
||||
} else {
|
||||
parts[i] = p.Val
|
||||
}
|
||||
}
|
||||
return "(" + strings.Join(parts, ",") + ")"
|
||||
}
|
||||
|
||||
// FirstLast returns the two parts of a composite (or single) key.
|
||||
func (k Key) FirstLast() (KeyPart, KeyPart) {
|
||||
if len(k) == 0 {
|
||||
return KeyPart{Null: true}, KeyPart{Null: true}
|
||||
}
|
||||
return k[0], k[len(k)-1]
|
||||
}
|
||||
|
||||
// Row holds a scanned row in select-list order.
|
||||
type Row struct {
|
||||
Values []Value
|
||||
}
|
||||
|
||||
// Chunk is a batch of rows produced by one scan step. Keys is parallel to
|
||||
// Rows and holds the unique-index key of each row; Buckets is parallel to
|
||||
// Rows and is populated only in date sharding mode.
|
||||
type Chunk struct {
|
||||
Rows []Row
|
||||
Keys []Key
|
||||
Buckets []string
|
||||
FirstKey Key
|
||||
LastKey Key
|
||||
}
|
||||
|
||||
func (c *Chunk) Len() int { return len(c.Rows) }
|
||||
|
||||
// Column describes one exported column for encoding purposes.
|
||||
type Column struct {
|
||||
Name string
|
||||
Type string // lowercase information_schema DATA_TYPE
|
||||
}
|
||||
|
||||
func isOneOf(t string, set ...string) bool {
|
||||
for _, s := range set {
|
||||
if t == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func IsInteger(t string) bool {
|
||||
return isOneOf(t, "tinyint", "smallint", "mediumint", "int", "integer", "bigint", "year")
|
||||
}
|
||||
|
||||
func IsFloat(t string) bool {
|
||||
return isOneOf(t, "float", "double", "real")
|
||||
}
|
||||
|
||||
func IsDecimal(t string) bool { return isOneOf(t, "decimal", "numeric") }
|
||||
|
||||
func IsBinary(t string) bool {
|
||||
return isOneOf(t, "binary", "varbinary", "tinyblob", "blob", "mediumblob", "longblob", "bit", "geometry", "geometrycollection", "point", "linestring", "polygon", "multipoint", "multilinestring", "multipolygon")
|
||||
}
|
||||
|
||||
func IsDate(t string) bool { return t == "date" }
|
||||
|
||||
func IsDateTime(t string) bool { return isOneOf(t, "datetime", "timestamp") }
|
||||
|
||||
func IsTimeType(t string) bool { return isOneOf(t, "date", "datetime", "timestamp") }
|
||||
@@ -0,0 +1,148 @@
|
||||
package scan
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// DateAScanner uses an index that starts with (shard_column, key...) and
|
||||
// performs keyset pagination over it. NULL rows are scanned in a separate
|
||||
// first phase because row-constructor comparisons do not advance past NULL.
|
||||
type DateAScanner struct {
|
||||
o *Options
|
||||
phase string // null | value
|
||||
cursor row.Key
|
||||
done bool
|
||||
}
|
||||
|
||||
func NewDateA(o Options) *DateAScanner {
|
||||
s := &DateAScanner{o: &o, phase: "null"}
|
||||
if o.Resume != nil {
|
||||
switch o.Resume.Phase {
|
||||
case "null":
|
||||
s.phase = "null"
|
||||
s.cursor = o.Resume.Key
|
||||
case "value":
|
||||
s.phase = "value"
|
||||
s.cursor = o.Resume.Key
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *DateAScanner) Next(ctx context.Context) (*row.Chunk, error) {
|
||||
for {
|
||||
if s.done {
|
||||
return nil, io.EOF
|
||||
}
|
||||
var (
|
||||
c *row.Chunk
|
||||
exhausted bool
|
||||
err error
|
||||
)
|
||||
if s.phase == "null" {
|
||||
c, exhausted, err = s.queryNull(ctx)
|
||||
} else {
|
||||
c, exhausted, err = s.queryValue(ctx)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if c.Len() == 0 {
|
||||
if s.phase == "null" {
|
||||
s.phase = "value"
|
||||
s.cursor = nil
|
||||
continue
|
||||
}
|
||||
s.done = true
|
||||
return nil, io.EOF
|
||||
}
|
||||
if s.phase == "null" {
|
||||
if exhausted {
|
||||
s.phase = "value"
|
||||
s.cursor = nil
|
||||
} else {
|
||||
s.cursor = c.LastKey
|
||||
}
|
||||
} else {
|
||||
s.cursor = s.o.compositeCursor(c.Rows[len(c.Rows)-1])
|
||||
c.LastKey = s.cursor
|
||||
if exhausted {
|
||||
s.done = true
|
||||
}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *DateAScanner) queryNull(ctx context.Context) (*row.Chunk, bool, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.Ident(s.o.Shard.Column))
|
||||
b.WriteString(" IS NULL")
|
||||
var args []any
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(s.o.KeyCols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(s.o.KeyCols))
|
||||
b.WriteString(" LIMIT ?")
|
||||
args = append(args, s.o.ChunkSize)
|
||||
return s.run(ctx, b.String(), args)
|
||||
}
|
||||
|
||||
func (s *DateAScanner) queryValue(ctx context.Context) (*row.Chunk, bool, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.Ident(s.o.Shard.Column))
|
||||
b.WriteString(" IS NOT NULL")
|
||||
cols := append([]string{s.o.Shard.Column}, s.o.KeyCols...)
|
||||
var args []any
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(cols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(cols))
|
||||
b.WriteString(" LIMIT ?")
|
||||
args = append(args, s.o.ChunkSize)
|
||||
c, exhausted, err := s.run(ctx, b.String(), args)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if c.Len() > 0 {
|
||||
c.LastKey = s.o.compositeCursor(c.Rows[len(c.Rows)-1])
|
||||
}
|
||||
return c, exhausted, nil
|
||||
}
|
||||
|
||||
func (s *DateAScanner) run(ctx context.Context, q string, args []any) (*row.Chunk, bool, error) {
|
||||
rows, err := s.o.DB.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
defer rows.Close()
|
||||
c, exhausted, err := s.o.readChunk(rows, s.o.ChunkSize)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if err := s.o.fillBuckets(c); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
return c, exhausted, nil
|
||||
}
|
||||
|
||||
func (s *DateAScanner) ResumeState() CursorState {
|
||||
return CursorState{Phase: s.phase, Key: s.cursor}
|
||||
}
|
||||
|
||||
func (s *DateAScanner) Close() error { return nil }
|
||||
@@ -0,0 +1,350 @@
|
||||
package scan
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/schema"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
const zeroDay = "0000-00-00"
|
||||
|
||||
// DateBScanner enumerates days one at a time and streams each day's rows over
|
||||
// a dedicated connection. It is used when only a single-column index on the
|
||||
// shard column exists.
|
||||
type DateBScanner struct {
|
||||
o *Options
|
||||
conn *sql.Conn
|
||||
|
||||
phase string // null | day
|
||||
rows *sql.Rows
|
||||
cursor row.Key
|
||||
done bool
|
||||
|
||||
daysInited bool
|
||||
hasDays bool
|
||||
zeroPending bool
|
||||
zeroDone bool
|
||||
nextDay time.Time
|
||||
maxDay time.Time
|
||||
curDay string
|
||||
nextDayStr string
|
||||
|
||||
resumeDay string
|
||||
resumeKey row.Key
|
||||
resumeUse bool
|
||||
}
|
||||
|
||||
func NewDateB(ctx context.Context, o Options) (*DateBScanner, error) {
|
||||
conn, err := o.DB.Conn(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acquire streaming connection: %w", err)
|
||||
}
|
||||
s := &DateBScanner{o: &o, conn: conn, phase: "null"}
|
||||
if o.Resume != nil {
|
||||
switch o.Resume.Phase {
|
||||
case "null":
|
||||
s.phase = "null"
|
||||
s.cursor = o.Resume.Key
|
||||
case "day":
|
||||
s.phase = "day"
|
||||
s.resumeDay = o.Resume.DayStart
|
||||
s.resumeKey = o.Resume.Key
|
||||
}
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func (s *DateBScanner) Next(ctx context.Context) (*row.Chunk, error) {
|
||||
for {
|
||||
if s.done {
|
||||
return nil, io.EOF
|
||||
}
|
||||
if s.rows == nil {
|
||||
if err := s.openNext(ctx); err != nil {
|
||||
if err == io.EOF {
|
||||
s.done = true
|
||||
return nil, io.EOF
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
c, exhausted, err := s.o.readChunk(s.rows, s.o.ChunkSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.o.fillBuckets(c); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if c.Len() > 0 {
|
||||
if s.phase == "day" {
|
||||
s.cursor = s.o.compositeCursor(c.Rows[len(c.Rows)-1])
|
||||
} else {
|
||||
s.cursor = c.Keys[len(c.Keys)-1]
|
||||
}
|
||||
c.LastKey = s.cursor
|
||||
}
|
||||
if exhausted {
|
||||
if err := s.rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.rows = nil
|
||||
if s.phase == "null" {
|
||||
s.phase = "day"
|
||||
s.cursor = nil
|
||||
}
|
||||
}
|
||||
if c.Len() == 0 {
|
||||
continue
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *DateBScanner) openNext(ctx context.Context) error {
|
||||
if s.phase == "null" {
|
||||
return s.openNull(ctx)
|
||||
}
|
||||
if !s.daysInited {
|
||||
if err := s.initDays(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if s.zeroPending && !s.zeroDone {
|
||||
s.zeroPending = false
|
||||
s.zeroDone = true
|
||||
s.curDay = zeroDay
|
||||
s.nextDayStr = zeroDay
|
||||
return s.openZero(ctx)
|
||||
}
|
||||
if !s.advanceDay() {
|
||||
return io.EOF
|
||||
}
|
||||
return s.openDay(ctx)
|
||||
}
|
||||
|
||||
func (s *DateBScanner) openNull(ctx context.Context) error {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.Ident(s.o.Shard.Column))
|
||||
b.WriteString(" IS NULL")
|
||||
var args []any
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(s.o.KeyCols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(s.o.KeyCols))
|
||||
return s.query(ctx, b.String(), args)
|
||||
}
|
||||
|
||||
func (s *DateBScanner) openZero(ctx context.Context) error {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.Ident(s.o.Shard.Column))
|
||||
b.WriteString(" < ?")
|
||||
upper := "0001-01-01"
|
||||
if s.o.Shard.Kind == schema.ShardDateTime {
|
||||
upper = "0001-01-01 00:00:00"
|
||||
}
|
||||
args := []any{upper}
|
||||
cols := append([]string{s.o.Shard.Column}, s.o.KeyCols...)
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(cols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(cols))
|
||||
s.nextDayStr = zeroDay
|
||||
return s.query(ctx, b.String(), args)
|
||||
}
|
||||
|
||||
func (s *DateBScanner) openDay(ctx context.Context) error {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
b.WriteString(" AND ")
|
||||
col := sqlutil.Ident(s.o.Shard.Column)
|
||||
b.WriteString(col)
|
||||
b.WriteString(" >= ? AND ")
|
||||
b.WriteString(col)
|
||||
b.WriteString(" < ?")
|
||||
|
||||
var lo, hi any
|
||||
switch s.o.Shard.Kind {
|
||||
case schema.ShardDate:
|
||||
lo = s.curDay
|
||||
hi = s.nextDayStr
|
||||
case schema.ShardDateTime:
|
||||
lo = s.curDay + " 00:00:00"
|
||||
hi = s.nextDayStr + " 00:00:00"
|
||||
case schema.ShardEpoch:
|
||||
lo = schema.TimeToEpoch(s.dayTime(s.curDay), s.o.Shard.Unit)
|
||||
hi = schema.TimeToEpoch(s.dayTime(s.curDay).AddDate(0, 0, 1), s.o.Shard.Unit)
|
||||
}
|
||||
args := []any{lo, hi}
|
||||
cols := append([]string{s.o.Shard.Column}, s.o.KeyCols...)
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(cols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(cols))
|
||||
return s.query(ctx, b.String(), args)
|
||||
}
|
||||
|
||||
func (s *DateBScanner) query(ctx context.Context, q string, args []any) error {
|
||||
rows, err := s.conn.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.rows = rows
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *DateBScanner) dayTime(day string) time.Time {
|
||||
t, _ := time.ParseInLocation("2006-01-02", day, s.o.ShardLoc)
|
||||
return t
|
||||
}
|
||||
|
||||
func (s *DateBScanner) initDays(ctx context.Context) error {
|
||||
s.daysInited = true
|
||||
q := fmt.Sprintf("SELECT MIN(%s), MAX(%s) FROM %s WHERE (%s) AND %s IS NOT NULL",
|
||||
sqlutil.Ident(s.o.Shard.Column), sqlutil.Ident(s.o.Shard.Column),
|
||||
s.o.qualified(), s.o.Where, sqlutil.Ident(s.o.Shard.Column))
|
||||
var minRaw, maxRaw sql.NullString
|
||||
if err := s.conn.QueryRowContext(ctx, q).Scan(&minRaw, &maxRaw); err != nil {
|
||||
return fmt.Errorf("read shard column range: %w", err)
|
||||
}
|
||||
if !minRaw.Valid || !maxRaw.Valid {
|
||||
return nil
|
||||
}
|
||||
switch s.o.Shard.Kind {
|
||||
case schema.ShardDate, schema.ShardDateTime:
|
||||
minDay := s.first10(minRaw.String)
|
||||
maxDay := s.first10(maxRaw.String)
|
||||
zeroExists := strings.HasPrefix(minDay, "0000-")
|
||||
if zeroExists {
|
||||
minDay = "0001-01-01"
|
||||
}
|
||||
s.zeroPending = s.wantZero(zeroExists)
|
||||
if strings.HasPrefix(maxDay, "0000-") {
|
||||
return nil
|
||||
}
|
||||
minT, err := time.ParseInLocation("2006-01-02", minDay, s.o.ShardLoc)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse shard range minimum %q: %w", minRaw.String, err)
|
||||
}
|
||||
maxT, err := time.ParseInLocation("2006-01-02", maxDay, s.o.ShardLoc)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse shard range maximum %q: %w", maxRaw.String, err)
|
||||
}
|
||||
s.minMax(minT, maxT)
|
||||
case schema.ShardEpoch:
|
||||
minV, err := strconv.ParseInt(strings.TrimSpace(minRaw.String), 10, 64)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse shard range minimum: %w", err)
|
||||
}
|
||||
maxV, err := strconv.ParseInt(strings.TrimSpace(maxRaw.String), 10, 64)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse shard range maximum: %w", err)
|
||||
}
|
||||
minT := schema.EpochTime(minV, s.o.Shard.Unit, s.o.ShardLoc)
|
||||
maxT := schema.EpochTime(maxV, s.o.Shard.Unit, s.o.ShardLoc)
|
||||
s.minMax(dayStart(minT, s.o.ShardLoc), dayStart(maxT, s.o.ShardLoc))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *DateBScanner) wantZero(zeroExists bool) bool {
|
||||
if s.resumeDay == "" {
|
||||
return zeroExists
|
||||
}
|
||||
return s.resumeDay == zeroDay
|
||||
}
|
||||
|
||||
func (s *DateBScanner) minMax(minT, maxT time.Time) {
|
||||
s.hasDays = true
|
||||
s.nextDay = minT
|
||||
s.maxDay = maxT
|
||||
if s.resumeDay != "" && s.resumeDay != zeroDay {
|
||||
if d, err := time.ParseInLocation("2006-01-02", s.resumeDay, s.o.ShardLoc); err == nil && d.After(minT) {
|
||||
s.nextDay = d
|
||||
}
|
||||
}
|
||||
if s.resumeDay == zeroDay {
|
||||
s.nextDay = minT
|
||||
}
|
||||
}
|
||||
|
||||
func (s *DateBScanner) advanceDay() bool {
|
||||
if !s.hasDays {
|
||||
return false
|
||||
}
|
||||
for !s.nextDay.After(s.maxDay) {
|
||||
day := s.nextDay.Format("2006-01-02")
|
||||
if s.resumeDay != "" && s.resumeDay != zeroDay && day < s.resumeDay && !s.resumeUse {
|
||||
s.nextDay = s.nextDay.AddDate(0, 0, 1)
|
||||
continue
|
||||
}
|
||||
s.curDay = day
|
||||
s.nextDayStr = s.nextDay.AddDate(0, 0, 1).Format("2006-01-02")
|
||||
s.nextDay = s.nextDay.AddDate(0, 0, 1)
|
||||
if s.resumeDay != "" && day == s.resumeDay && !s.resumeUse {
|
||||
s.cursor = s.resumeKey
|
||||
s.resumeUse = true
|
||||
} else {
|
||||
s.cursor = nil
|
||||
}
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (s *DateBScanner) first10(v string) string {
|
||||
if len(v) < 10 {
|
||||
return v
|
||||
}
|
||||
return v[:10]
|
||||
}
|
||||
|
||||
func dayStart(t time.Time, loc *time.Location) time.Time {
|
||||
return time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, loc)
|
||||
}
|
||||
|
||||
func (s *DateBScanner) ResumeState() CursorState {
|
||||
phase := s.phase
|
||||
day := ""
|
||||
if phase == "day" {
|
||||
day = s.curDay
|
||||
}
|
||||
return CursorState{Phase: phase, Key: s.cursor, DayStart: day}
|
||||
}
|
||||
|
||||
func (s *DateBScanner) Close() error {
|
||||
var firstErr error
|
||||
if s.rows != nil {
|
||||
if err := s.rows.Close(); err != nil {
|
||||
firstErr = err
|
||||
}
|
||||
s.rows = nil
|
||||
}
|
||||
if s.conn != nil {
|
||||
if err := s.conn.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
s.conn = nil
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package scan
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// PKScanner walks the table by its chosen unique key using
|
||||
// key > :last ORDER BY key LIMIT :chunk_size.
|
||||
type PKScanner struct {
|
||||
o *Options
|
||||
cursor row.Key
|
||||
done bool
|
||||
}
|
||||
|
||||
func NewPK(o Options) *PKScanner {
|
||||
s := &PKScanner{o: &o}
|
||||
if o.Resume != nil && (o.Resume.Phase == "" || o.Resume.Phase == "pk") {
|
||||
s.cursor = o.Resume.Key
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *PKScanner) Next(ctx context.Context) (*row.Chunk, error) {
|
||||
if s.done {
|
||||
return nil, io.EOF
|
||||
}
|
||||
q, args := s.query()
|
||||
rows, err := s.o.DB.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
chunk, exhausted, err := s.o.readChunk(rows, s.o.ChunkSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(chunk.Rows) == 0 {
|
||||
s.done = true
|
||||
return nil, io.EOF
|
||||
}
|
||||
if exhausted {
|
||||
s.done = true
|
||||
}
|
||||
s.cursor = chunk.LastKey
|
||||
return chunk, nil
|
||||
}
|
||||
|
||||
func (s *PKScanner) query() (string, []any) {
|
||||
var b strings.Builder
|
||||
b.WriteString(s.o.selectSQL())
|
||||
var args []any
|
||||
if len(s.cursor) > 0 {
|
||||
b.WriteString(" AND ")
|
||||
b.WriteString(sqlutil.GreaterThan(s.o.KeyCols))
|
||||
args = append(args, s.cursor.Args()...)
|
||||
}
|
||||
b.WriteString(" ORDER BY ")
|
||||
b.WriteString(sqlutil.Columns(s.o.KeyCols))
|
||||
b.WriteString(" LIMIT ?")
|
||||
args = append(args, s.o.ChunkSize)
|
||||
return b.String(), args
|
||||
}
|
||||
|
||||
func (s *PKScanner) Close() error { return nil }
|
||||
|
||||
func (s *PKScanner) ResumeState() CursorState {
|
||||
return CursorState{Phase: "pk", Key: s.cursor}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
// Package scan implements keyset (seek) chunked scanning of a source table.
|
||||
package scan
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/schema"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// CursorState is the resumable position of a scanner.
|
||||
type CursorState struct {
|
||||
Phase string
|
||||
Key row.Key
|
||||
DayStart string
|
||||
}
|
||||
|
||||
// Scanner produces chunks in scan order.
|
||||
type Scanner interface {
|
||||
Next(ctx context.Context) (*row.Chunk, error) // io.EOF when exhausted
|
||||
ResumeState() CursorState
|
||||
Close() error
|
||||
}
|
||||
|
||||
// Resume is the persisted scan position.
|
||||
type Resume struct {
|
||||
Phase string // pk|null|value|day
|
||||
Key row.Key
|
||||
DayStart string
|
||||
}
|
||||
|
||||
// Options configures a Scanner.
|
||||
type Options struct {
|
||||
DB *sql.DB
|
||||
DBName string
|
||||
Table string
|
||||
Where string
|
||||
SelectCols []string
|
||||
KeyCols []string
|
||||
KeyIdx []int
|
||||
ChunkSize int
|
||||
|
||||
Shard *schema.ShardSpec // nil outside date mode
|
||||
ShardIdx int // -1 outside date mode
|
||||
ShardLoc *time.Location
|
||||
ZeroDate string
|
||||
Resume *Resume
|
||||
}
|
||||
|
||||
func (o *Options) qualified() string { return sqlutil.Qualified(o.DBName, o.Table) }
|
||||
|
||||
func (o *Options) selectSQL() string {
|
||||
return "SELECT " + sqlutil.Columns(o.SelectCols) +
|
||||
" FROM " + o.qualified() +
|
||||
" WHERE (" + o.Where + ")"
|
||||
}
|
||||
|
||||
func (o *Options) keyOf(r row.Row) row.Key {
|
||||
k := make(row.Key, len(o.KeyIdx))
|
||||
for i, idx := range o.KeyIdx {
|
||||
v := r.Values[idx]
|
||||
if v.IsNull() {
|
||||
k[i] = row.KeyPart{Null: true}
|
||||
} else {
|
||||
k[i] = row.KeyPart{Val: string(v.Raw)}
|
||||
}
|
||||
}
|
||||
return k
|
||||
}
|
||||
|
||||
func (o *Options) compositeCursor(r row.Row) row.Key {
|
||||
k := o.keyOf(r)
|
||||
out := make(row.Key, 0, 1+len(k))
|
||||
v := r.Values[o.ShardIdx]
|
||||
if v.IsNull() {
|
||||
out = append(out, row.KeyPart{Null: true})
|
||||
} else {
|
||||
out = append(out, row.KeyPart{Val: string(v.Raw)})
|
||||
}
|
||||
return append(out, k...)
|
||||
}
|
||||
|
||||
func (o *Options) bucket(r row.Row) (string, error) {
|
||||
return schema.BucketOf(r.Values[o.ShardIdx], o.Shard, o.ShardLoc, o.ZeroDate, o.keyOf(r))
|
||||
}
|
||||
|
||||
// readChunk consumes up to limit rows from an open result set.
|
||||
func (o *Options) readChunk(rows *sql.Rows, limit int) (*row.Chunk, bool, error) {
|
||||
chunk := &row.Chunk{}
|
||||
raw := make([]sql.RawBytes, len(o.SelectCols))
|
||||
ptrs := make([]any, len(raw))
|
||||
for i := range raw {
|
||||
ptrs[i] = &raw[i]
|
||||
}
|
||||
exhausted := false
|
||||
for len(chunk.Rows) < limit {
|
||||
if !rows.Next() {
|
||||
exhausted = true
|
||||
break
|
||||
}
|
||||
if err := rows.Scan(ptrs...); err != nil {
|
||||
return nil, false, fmt.Errorf("scan row: %w", err)
|
||||
}
|
||||
vals := make([]row.Value, len(raw))
|
||||
for i, b := range raw {
|
||||
if b == nil {
|
||||
continue
|
||||
}
|
||||
c := make([]byte, len(b))
|
||||
copy(c, b)
|
||||
vals[i] = row.Value{Raw: c}
|
||||
}
|
||||
chunk.Rows = append(chunk.Rows, row.Row{Values: vals})
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, false, fmt.Errorf("read rows: %w", err)
|
||||
}
|
||||
for i := range chunk.Rows {
|
||||
chunk.Keys = append(chunk.Keys, o.keyOf(chunk.Rows[i]))
|
||||
}
|
||||
if len(chunk.Rows) > 0 {
|
||||
chunk.FirstKey = chunk.Keys[0]
|
||||
chunk.LastKey = chunk.Keys[len(chunk.Keys)-1]
|
||||
}
|
||||
return chunk, exhausted, nil
|
||||
}
|
||||
|
||||
func (o *Options) fillBuckets(c *row.Chunk) error {
|
||||
c.Buckets = make([]string, len(c.Rows))
|
||||
for i := range c.Rows {
|
||||
b, err := o.bucket(c.Rows[i])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c.Buckets[i] = b
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
package schema
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// ShardKind is how a shard column's date is derived from its raw value.
|
||||
type ShardKind int
|
||||
|
||||
const (
|
||||
ShardNone ShardKind = iota
|
||||
ShardDate
|
||||
ShardDateTime
|
||||
ShardEpoch
|
||||
)
|
||||
|
||||
// ShardSpec is the resolved shard column and how to interpret it.
|
||||
type ShardSpec struct {
|
||||
Column string
|
||||
Kind ShardKind
|
||||
Unit int64 // epoch multiplier to nanoseconds: 1e9, 1e6, 1e3
|
||||
Type string
|
||||
}
|
||||
|
||||
// ResolveShardColumn validates --shard-column against the table and the unit
|
||||
// flag. It returns a descriptive error listing candidates when the column is
|
||||
// missing or the type is unsupported.
|
||||
func ResolveShardColumn(ctx context.Context, db *sql.DB, t *Table, column, unitFlag, where string, loc *time.Location) (*ShardSpec, error) {
|
||||
if column == "" {
|
||||
return nil, fmt.Errorf("--shard-column is required for --shard-by date\ncandidate columns:\n%s", describeTimeColumns(t))
|
||||
}
|
||||
c, ok := t.Column(column)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("shard column %q does not exist; candidate columns:\n%s", column, describeTimeColumns(t))
|
||||
}
|
||||
switch c.DataType {
|
||||
case "date":
|
||||
return &ShardSpec{Column: c.Name, Kind: ShardDate, Unit: 1, Type: c.DataType}, nil
|
||||
case "datetime", "timestamp":
|
||||
return &ShardSpec{Column: c.Name, Kind: ShardDateTime, Unit: 1, Type: c.DataType}, nil
|
||||
}
|
||||
if row.IsInteger(c.DataType) {
|
||||
unit, err := ParseEpochUnit(unitFlag)
|
||||
if err != nil {
|
||||
minV, maxV, ok, serr := SampleEpochRange(ctx, db, t, c.Name, where)
|
||||
if serr != nil {
|
||||
return nil, fmt.Errorf("%w (also failed to sample %s: %v)", err, c.Name, serr)
|
||||
}
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w; column %s has no non-NULL values to sample", err, c.Name)
|
||||
}
|
||||
return nil, fmt.Errorf("%w\nsampled %s range: %d .. %d\n seconds: %s .. %s\n milliseconds: %s .. %s\n microseconds: %s .. %s",
|
||||
err, c.Name, minV, maxV,
|
||||
epochString(minV, 1, loc), epochString(maxV, 1, loc),
|
||||
epochString(minV, 1_000, loc), epochString(maxV, 1_000, loc),
|
||||
epochString(minV, 1_000_000, loc), epochString(maxV, 1_000_000, loc))
|
||||
}
|
||||
return &ShardSpec{Column: c.Name, Kind: ShardEpoch, Unit: unit, Type: c.DataType}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("column %q has unsupported type %s; v1 supports DATE, DATETIME, TIMESTAMP and integer epoch columns with --shard-column-type\ncandidate columns:\n%s",
|
||||
c.Name, c.DataType, describeTimeColumns(t))
|
||||
}
|
||||
|
||||
func describeTimeColumns(t *Table) string {
|
||||
var b strings.Builder
|
||||
found := false
|
||||
for _, c := range t.TimeColumns() {
|
||||
found = true
|
||||
fmt.Fprintf(&b, " - %s %s%s\n", c.Name, c.ColumnType, nullableSuffix(c))
|
||||
}
|
||||
for _, c := range t.Columns {
|
||||
if row.IsInteger(c.DataType) {
|
||||
found = true
|
||||
fmt.Fprintf(&b, " - %s %s (integer; requires --shard-column-type)%s\n", c.Name, c.ColumnType, nullableSuffix(c))
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return " (none)"
|
||||
}
|
||||
return strings.TrimRight(b.String(), "\n")
|
||||
}
|
||||
|
||||
func nullableSuffix(c Column) string {
|
||||
if c.Nullable {
|
||||
return " NULL"
|
||||
}
|
||||
return " NOT NULL"
|
||||
}
|
||||
|
||||
// ParseEpochUnit maps --shard-column-type to raw units per second: 1 for
|
||||
// seconds, 1000 for milliseconds, 1000000 for microseconds.
|
||||
func ParseEpochUnit(flag string) (int64, error) {
|
||||
switch strings.ToLower(strings.TrimSpace(flag)) {
|
||||
case "epoch-seconds", "seconds", "second", "s", "1":
|
||||
return 1, nil
|
||||
case "epoch-milliseconds", "epoch-millis", "milliseconds", "millisecond", "ms", "1000":
|
||||
return 1_000, nil
|
||||
case "epoch-microseconds", "epoch-micros", "microseconds", "microsecond", "us", "1000000":
|
||||
return 1_000_000, nil
|
||||
case "":
|
||||
return 0, fmt.Errorf("integer shard column requires --shard-column-type (epoch-seconds, epoch-milliseconds or epoch-microseconds)")
|
||||
default:
|
||||
return 0, fmt.Errorf("unsupported --shard-column-type %q; use epoch-seconds, epoch-milliseconds or epoch-microseconds", flag)
|
||||
}
|
||||
}
|
||||
|
||||
// SampleEpochRange returns MIN/MAX of an integer column under the base where.
|
||||
func SampleEpochRange(ctx context.Context, db *sql.DB, t *Table, col, where string) (int64, int64, bool, error) {
|
||||
q := fmt.Sprintf("SELECT MIN(%s), MAX(%s) FROM %s", sqlutil.Ident(col), sqlutil.Ident(col), sqlutil.Qualified(t.DB, t.Name))
|
||||
if strings.TrimSpace(where) != "" {
|
||||
q += " WHERE (" + where + ")"
|
||||
}
|
||||
var minRaw, maxRaw sql.NullString
|
||||
if err := db.QueryRowContext(ctx, q).Scan(&minRaw, &maxRaw); err != nil {
|
||||
return 0, 0, false, err
|
||||
}
|
||||
if !minRaw.Valid || !maxRaw.Valid {
|
||||
return 0, 0, false, nil
|
||||
}
|
||||
minV, err := strconv.ParseInt(strings.TrimSpace(minRaw.String), 10, 64)
|
||||
if err != nil {
|
||||
return 0, 0, false, fmt.Errorf("parse min %q: %w", minRaw.String, err)
|
||||
}
|
||||
maxV, err := strconv.ParseInt(strings.TrimSpace(maxRaw.String), 10, 64)
|
||||
if err != nil {
|
||||
return 0, 0, false, fmt.Errorf("parse max %q: %w", maxRaw.String, err)
|
||||
}
|
||||
return minV, maxV, true, nil
|
||||
}
|
||||
|
||||
func epochString(v, unit int64, loc *time.Location) string {
|
||||
return EpochTime(v, unit, loc).Format("2006-01-02 15:04:05")
|
||||
}
|
||||
|
||||
// EpochTime converts an epoch value with the given nanoseconds-per-unit to a
|
||||
// time in loc.
|
||||
func EpochTime(v, unit int64, loc *time.Location) time.Time {
|
||||
sec := v / unit
|
||||
frac := v % unit
|
||||
if frac < 0 {
|
||||
sec--
|
||||
frac += unit
|
||||
}
|
||||
nsec := frac * (1_000_000_000 / unit)
|
||||
return time.Unix(sec, nsec).In(loc)
|
||||
}
|
||||
|
||||
// TimeToEpoch is the inverse of EpochTime.
|
||||
func TimeToEpoch(t time.Time, unit int64) int64 {
|
||||
return t.Unix()*unit + int64(t.Nanosecond())/(1_000_000_000/unit)
|
||||
}
|
||||
|
||||
const nullBucket = "_null"
|
||||
const zeroBucket = "0000-00-00"
|
||||
|
||||
// BucketOf maps a raw shard value to its date bucket. key is used only in
|
||||
// error messages.
|
||||
func BucketOf(v row.Value, spec *ShardSpec, loc *time.Location, zeroDate string, key row.Key) (string, error) {
|
||||
if v.IsNull() {
|
||||
return nullBucket, nil
|
||||
}
|
||||
switch spec.Kind {
|
||||
case ShardEpoch:
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(string(v.Raw)), 10, 64)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("row %s: cannot parse epoch value %q in column %s: %w", key, string(v.Raw), spec.Column, err)
|
||||
}
|
||||
return EpochTime(n, spec.Unit, loc).Format("2006-01-02"), nil
|
||||
case ShardDate, ShardDateTime:
|
||||
s := string(v.Raw)
|
||||
if len(s) < 10 {
|
||||
return "", fmt.Errorf("row %s: cannot parse date value %q in column %s", key, s, spec.Column)
|
||||
}
|
||||
day := s[:10]
|
||||
if strings.HasPrefix(day, "0000-") {
|
||||
if zeroDate == "bucket" {
|
||||
return zeroBucket, nil
|
||||
}
|
||||
return "", fmt.Errorf("row %s has zero date %q in column %s; pass --zero-date bucket to route it to date=0000-00-00", key, s, spec.Column)
|
||||
}
|
||||
if _, err := time.Parse("2006-01-02", day); err != nil {
|
||||
return "", fmt.Errorf("row %s: invalid date %q in column %s", key, s, spec.Column)
|
||||
}
|
||||
return day, nil
|
||||
}
|
||||
return "", fmt.Errorf("internal: shard kind not resolved")
|
||||
}
|
||||
|
||||
// BucketLess orders buckets in scan order: NULL first, then zero dates, then
|
||||
// real dates.
|
||||
func BucketLess(a, b string) bool {
|
||||
ca, ka := bucketClass(a)
|
||||
cb, kb := bucketClass(b)
|
||||
if ca != cb {
|
||||
return ca < cb
|
||||
}
|
||||
return ka < kb
|
||||
}
|
||||
|
||||
func bucketClass(b string) (int, string) {
|
||||
switch b {
|
||||
case "":
|
||||
return -1, ""
|
||||
case nullBucket:
|
||||
return 0, ""
|
||||
case zeroBucket:
|
||||
return 1, ""
|
||||
default:
|
||||
return 2, b
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package schema
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
func TestBucketOfDateTime(t *testing.T) {
|
||||
spec := &ShardSpec{Column: "created_at", Kind: ShardDateTime}
|
||||
loc := time.UTC
|
||||
cases := []struct {
|
||||
raw []byte
|
||||
want string
|
||||
}{
|
||||
{[]byte("2025-06-01 13:45:00"), "2025-06-01"},
|
||||
{[]byte("2025-12-31 00:00:00"), "2025-12-31"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, err := BucketOf(row.Value{Raw: c.raw}, spec, loc, "", row.KeyOf("1"))
|
||||
if err != nil || got != c.want {
|
||||
t.Fatalf("%s: got %q err %v want %q", c.raw, got, err, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBucketOfNull(t *testing.T) {
|
||||
spec := &ShardSpec{Column: "c", Kind: ShardDateTime}
|
||||
got, err := BucketOf(row.Value{}, spec, time.UTC, "", row.KeyOf("1"))
|
||||
if err != nil || got != "_null" {
|
||||
t.Fatalf("got %q err %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBucketOfZeroDate(t *testing.T) {
|
||||
spec := &ShardSpec{Column: "c", Kind: ShardDateTime}
|
||||
if _, err := BucketOf(row.Value{Raw: []byte("0000-00-00 00:00:00")}, spec, time.UTC, "", row.KeyOf("7")); err == nil {
|
||||
t.Fatal("expected zero-date error")
|
||||
}
|
||||
got, err := BucketOf(row.Value{Raw: []byte("0000-00-00 00:00:00")}, spec, time.UTC, "bucket", row.KeyOf("7"))
|
||||
if err != nil || got != "0000-00-00" {
|
||||
t.Fatalf("got %q err %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBucketOfEpoch(t *testing.T) {
|
||||
loc, err := time.LoadLocation("UTC")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
unit int64
|
||||
raw string
|
||||
want string
|
||||
}{
|
||||
{1, "1717200000", "2024-06-01"},
|
||||
{1_000, "1717200000000", "2024-06-01"},
|
||||
{1_000_000, "1717200000000000", "2024-06-01"},
|
||||
} {
|
||||
spec := &ShardSpec{Column: "ts", Kind: ShardEpoch, Unit: tc.unit}
|
||||
got, err := BucketOf(row.Value{Raw: []byte(tc.raw)}, spec, loc, "", row.KeyOf("1"))
|
||||
if err != nil || got != tc.want {
|
||||
t.Fatalf("unit %d: got %q err %v want %q", tc.unit, got, err, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBucketLess(t *testing.T) {
|
||||
order := []string{"_null", "0000-00-00", "2024-01-01", "2024-01-02", "2025-01-01"}
|
||||
for i := 0; i+1 < len(order); i++ {
|
||||
if !BucketLess(order[i], order[i+1]) {
|
||||
t.Fatalf("expected %q < %q", order[i], order[i+1])
|
||||
}
|
||||
if BucketLess(order[i+1], order[i]) {
|
||||
t.Fatalf("expected %q not < %q", order[i+1], order[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseEpochUnit(t *testing.T) {
|
||||
cases := map[string]int64{
|
||||
"epoch-seconds": 1,
|
||||
"seconds": 1,
|
||||
"epoch-milliseconds": 1_000,
|
||||
"ms": 1_000,
|
||||
"epoch-microseconds": 1_000_000,
|
||||
"us": 1_000_000,
|
||||
}
|
||||
for in, want := range cases {
|
||||
got, err := ParseEpochUnit(in)
|
||||
if err != nil || got != want {
|
||||
t.Fatalf("%s: got %d err %v want %d", in, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := ParseEpochUnit(""); err == nil {
|
||||
t.Fatal("empty unit should error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEpochRoundTrip(t *testing.T) {
|
||||
loc := time.UTC
|
||||
for _, unit := range []int64{1, 1_000, 1_000_000} {
|
||||
now := time.Date(2024, 6, 1, 12, 30, 15, 0, loc)
|
||||
v := TimeToEpoch(now, unit)
|
||||
back := EpochTime(v, unit, loc)
|
||||
if !back.Equal(now) {
|
||||
t.Fatalf("unit %d: got %s want %s", unit, back, now)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
// Package schema loads MySQL table metadata and answers questions about
|
||||
// columns and indexes needed to choose a scan strategy.
|
||||
package schema
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
// Column is one column of the source table.
|
||||
type Column struct {
|
||||
Name string
|
||||
Ordinal int
|
||||
DataType string
|
||||
ColumnType string
|
||||
Nullable bool
|
||||
Extra string
|
||||
Key string
|
||||
}
|
||||
|
||||
func (c Column) IsAutoIncrement() bool {
|
||||
return strings.Contains(strings.ToLower(c.Extra), "auto_increment")
|
||||
}
|
||||
|
||||
// Index is one index of the source table, in index-column order.
|
||||
type Index struct {
|
||||
Name string
|
||||
Unique bool
|
||||
Columns []string
|
||||
}
|
||||
|
||||
// Table is the loaded metadata of the source table.
|
||||
type Table struct {
|
||||
DB string
|
||||
Name string
|
||||
Columns []Column
|
||||
byName map[string]Column
|
||||
Indexes []Index
|
||||
}
|
||||
|
||||
func (t *Table) Column(name string) (Column, bool) {
|
||||
c, ok := t.byName[strings.ToLower(name)]
|
||||
return c, ok
|
||||
}
|
||||
|
||||
func (t *Table) ColumnNames() []string {
|
||||
out := make([]string, len(t.Columns))
|
||||
for i, c := range t.Columns {
|
||||
out[i] = c.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TimeColumns returns every DATE/DATETIME/TIMESTAMP/IPv4-ish candidate used to
|
||||
// explain to the user which shard columns are available.
|
||||
func (t *Table) TimeColumns() []Column {
|
||||
var out []Column
|
||||
for _, c := range t.Columns {
|
||||
if c.DataType == "date" || c.DataType == "datetime" || c.DataType == "timestamp" {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Load reads column and index metadata from information_schema.
|
||||
func Load(ctx context.Context, db *sql.DB, dbName, table string) (*Table, error) {
|
||||
t := &Table{DB: dbName, Name: table, byName: map[string]Column{}}
|
||||
|
||||
rows, err := db.QueryContext(ctx, `
|
||||
SELECT COLUMN_NAME, ORDINAL_POSITION, LOWER(DATA_TYPE), COLUMN_TYPE, IS_NULLABLE, EXTRA, COLUMN_KEY
|
||||
FROM information_schema.COLUMNS
|
||||
WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?
|
||||
ORDER BY ORDINAL_POSITION`, dbName, table)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load columns: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var c Column
|
||||
var nullable string
|
||||
if err := rows.Scan(&c.Name, &c.Ordinal, &c.DataType, &c.ColumnType, &nullable, &c.Extra, &c.Key); err != nil {
|
||||
return nil, fmt.Errorf("scan column: %w", err)
|
||||
}
|
||||
c.Nullable = strings.EqualFold(nullable, "YES")
|
||||
t.Columns = append(t.Columns, c)
|
||||
t.byName[strings.ToLower(c.Name)] = c
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("load columns: %w", err)
|
||||
}
|
||||
if len(t.Columns) == 0 {
|
||||
return nil, fmt.Errorf("table %s.%s not found or has no columns", dbName, table)
|
||||
}
|
||||
|
||||
idxRows, err := db.QueryContext(ctx, `
|
||||
SELECT INDEX_NAME, NON_UNIQUE, SEQ_IN_INDEX, COLUMN_NAME
|
||||
FROM information_schema.STATISTICS
|
||||
WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?
|
||||
ORDER BY INDEX_NAME, SEQ_IN_INDEX`, dbName, table)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load indexes: %w", err)
|
||||
}
|
||||
defer idxRows.Close()
|
||||
|
||||
byIdx := map[string]*Index{}
|
||||
for idxRows.Next() {
|
||||
var name, col string
|
||||
var nonUnique, seq int
|
||||
if err := idxRows.Scan(&name, &nonUnique, &seq, &col); err != nil {
|
||||
return nil, fmt.Errorf("scan index: %w", err)
|
||||
}
|
||||
if col == "" {
|
||||
continue
|
||||
}
|
||||
ix := byIdx[name]
|
||||
if ix == nil {
|
||||
ix = &Index{Name: name, Unique: nonUnique == 0}
|
||||
byIdx[name] = ix
|
||||
t.Indexes = append(t.Indexes, *ix)
|
||||
}
|
||||
ix.Columns = append(ix.Columns, col)
|
||||
}
|
||||
if err := idxRows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("load indexes: %w", err)
|
||||
}
|
||||
// Rebuild pointers into the slice since append may have copied structs.
|
||||
for i := range t.Indexes {
|
||||
t.Indexes[i] = *byIdx[t.Indexes[i].Name]
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// KeyInfo describes the unique index chosen as scan key.
|
||||
type KeyInfo struct {
|
||||
Name string
|
||||
Columns []string
|
||||
IsPrimary bool
|
||||
}
|
||||
|
||||
func (t *Table) PrimaryKey() *Index {
|
||||
for i := range t.Indexes {
|
||||
if t.Indexes[i].Name == "PRIMARY" {
|
||||
return &t.Indexes[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ChooseKey selects the scan key: an explicitly requested index, else the
|
||||
// primary key, else the first unique index. Every key column must be NOT NULL.
|
||||
func (t *Table) ChooseKey(forced string) (KeyInfo, error) {
|
||||
var chosen *Index
|
||||
switch {
|
||||
case forced != "":
|
||||
for i := range t.Indexes {
|
||||
if strings.EqualFold(t.Indexes[i].Name, forced) {
|
||||
chosen = &t.Indexes[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if chosen == nil {
|
||||
return KeyInfo{}, fmt.Errorf("index %q does not exist on %s", forced, t.Name)
|
||||
}
|
||||
if !chosen.Unique {
|
||||
return KeyInfo{}, fmt.Errorf("index %q is not unique; a unique index is required for seek pagination", forced)
|
||||
}
|
||||
case t.PrimaryKey() != nil:
|
||||
chosen = t.PrimaryKey()
|
||||
default:
|
||||
for i := range t.Indexes {
|
||||
if t.Indexes[i].Unique {
|
||||
chosen = &t.Indexes[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if chosen == nil {
|
||||
return KeyInfo{}, fmt.Errorf("table %s has no primary key or unique index; create one or pass --index", t.Name)
|
||||
}
|
||||
}
|
||||
for _, col := range chosen.Columns {
|
||||
c, ok := t.Column(col)
|
||||
if !ok {
|
||||
return KeyInfo{}, fmt.Errorf("index %q references unknown column %q", chosen.Name, col)
|
||||
}
|
||||
if c.Nullable {
|
||||
return KeyInfo{}, fmt.Errorf("index %q column %q is nullable; seek pagination needs a NOT NULL unique key", chosen.Name, col)
|
||||
}
|
||||
}
|
||||
return KeyInfo{Name: chosen.Name, Columns: chosen.Columns, IsPrimary: chosen.Name == "PRIMARY"}, nil
|
||||
}
|
||||
|
||||
// FindIndexExact returns the index whose columns are exactly cols.
|
||||
func (t *Table) FindIndexExact(cols []string) *Index {
|
||||
for i := range t.Indexes {
|
||||
if equalFoldSlice(t.Indexes[i].Columns, cols) {
|
||||
return &t.Indexes[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// FindIndexPrefix returns the first index whose leading columns are exactly
|
||||
// prefix, and its full column list.
|
||||
func (t *Table) FindIndexPrefix(prefix []string) *Index {
|
||||
for i := range t.Indexes {
|
||||
ix := &t.Indexes[i]
|
||||
if len(ix.Columns) < len(prefix) {
|
||||
continue
|
||||
}
|
||||
if equalFoldSlice(ix.Columns[:len(prefix)], prefix) {
|
||||
return ix
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// HasColumnInIndex reports whether col is the first column of some index.
|
||||
func (t *Table) HasColumnPrefixIndex(col string) *Index {
|
||||
return t.FindIndexPrefix([]string{col})
|
||||
}
|
||||
|
||||
func equalFoldSlice(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if !strings.EqualFold(a[i], b[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// OutputColumns resolves the requested column list (empty means all) and
|
||||
// validates names against the table.
|
||||
func (t *Table) OutputColumns(requested []string) ([]row.Column, error) {
|
||||
if len(requested) == 0 {
|
||||
out := make([]row.Column, len(t.Columns))
|
||||
for i, c := range t.Columns {
|
||||
out[i] = row.Column{Name: c.Name, Type: c.DataType}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
out := make([]row.Column, 0, len(requested))
|
||||
seen := map[string]bool{}
|
||||
for _, name := range requested {
|
||||
c, ok := t.Column(name)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unknown column %q", name)
|
||||
}
|
||||
if seen[c.Name] {
|
||||
continue
|
||||
}
|
||||
seen[c.Name] = true
|
||||
out = append(out, row.Column{Name: c.Name, Type: c.DataType})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package shard
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"github.com/klauspost/compress/zstd"
|
||||
"github.com/ulikunitz/xz"
|
||||
)
|
||||
|
||||
// DefaultLevel is the compression level used when none is specified. It
|
||||
// matches the byte output of archer before --compress-level existed.
|
||||
const DefaultLevel = 3
|
||||
|
||||
// xzReadDictCap is the largest dictionary xz can produce here (level 9).
|
||||
// Readers always allow it so lowering the level with --force-resume does not
|
||||
// make previously written shards unreadable.
|
||||
const xzReadDictCap = 64 << 20
|
||||
|
||||
// Compressor wraps shard byte streams. Every shard is compressed on its own
|
||||
// so a single file can be decompressed and imported without its siblings.
|
||||
// Level is a format-neutral 1..9 scale where 9 is the strongest setting.
|
||||
type Compressor struct {
|
||||
Name string // "none", "gzip", "zstd", "xz"
|
||||
Ext string // "", ".gz", ".zst", ".xz"
|
||||
Level int
|
||||
}
|
||||
|
||||
func NewCompressor(name string, level int) (*Compressor, error) {
|
||||
if level < 1 || level > 9 {
|
||||
return nil, fmt.Errorf("compression level %d out of range 1-9", level)
|
||||
}
|
||||
switch name {
|
||||
case "", "none":
|
||||
return &Compressor{Name: "none", Level: level}, nil
|
||||
case "gzip":
|
||||
return &Compressor{Name: "gzip", Ext: ".gz", Level: level}, nil
|
||||
case "zstd":
|
||||
return &Compressor{Name: "zstd", Ext: ".zst", Level: level}, nil
|
||||
case "xz":
|
||||
return &Compressor{Name: "xz", Ext: ".xz", Level: level}, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported compression %q", name)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Compressor) Wrap(w io.Writer) (io.WriteCloser, error) {
|
||||
switch c.Name {
|
||||
case "gzip":
|
||||
return gzip.NewWriterLevel(w, gzipLevel(c.Level))
|
||||
case "zstd":
|
||||
return zstd.NewWriter(w, zstd.WithEncoderLevel(zstdLevel(c.Level)))
|
||||
case "xz":
|
||||
xw, err := xz.WriterConfig{DictCap: xzDictCap(c.Level)}.NewWriter(w)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return xw, nil
|
||||
default:
|
||||
return nopCloser{w}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Reader returns a decompressing reader for an existing shard.
|
||||
func (c *Compressor) Reader(r io.Reader) (io.ReadCloser, error) {
|
||||
switch c.Name {
|
||||
case "gzip":
|
||||
return gzip.NewReader(r)
|
||||
case "zstd":
|
||||
zr, err := zstd.NewReader(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return zr.IOReadCloser(), nil
|
||||
case "xz":
|
||||
xr, err := xz.ReaderConfig{DictCap: xzReadDictCap}.NewReader(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return io.NopCloser(xr), nil
|
||||
default:
|
||||
return io.NopCloser(r), nil
|
||||
}
|
||||
}
|
||||
|
||||
// gzipLevel passes the neutral level straight through.
|
||||
func gzipLevel(level int) int {
|
||||
return level
|
||||
}
|
||||
|
||||
func zstdLevel(level int) zstd.EncoderLevel {
|
||||
switch {
|
||||
case level <= 2:
|
||||
return zstd.SpeedFastest
|
||||
case level <= 5:
|
||||
return zstd.SpeedDefault
|
||||
case level <= 8:
|
||||
return zstd.SpeedBetterCompression
|
||||
default:
|
||||
return zstd.SpeedBestCompression
|
||||
}
|
||||
}
|
||||
|
||||
// xzDictCap maps the neutral level onto xz dictionary sizes; the library has
|
||||
// no preset parameter, so the dictionary is the only strength knob.
|
||||
func xzDictCap(level int) int {
|
||||
switch level {
|
||||
case 1:
|
||||
return 1 << 20
|
||||
case 2:
|
||||
return 2 << 20
|
||||
case 3, 4:
|
||||
return 4 << 20
|
||||
case 5, 6:
|
||||
return 8 << 20
|
||||
case 7:
|
||||
return 16 << 20
|
||||
case 8:
|
||||
return 32 << 20
|
||||
default:
|
||||
return 64 << 20
|
||||
}
|
||||
}
|
||||
|
||||
type nopCloser struct{ io.Writer }
|
||||
|
||||
func (nopCloser) Close() error { return nil }
|
||||
|
||||
type flushCloser interface {
|
||||
Flush() error
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
package shard
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Entry is one line of manifest.jsonl.
|
||||
type Entry struct {
|
||||
Seq int64 `json:"seq"`
|
||||
Path string `json:"path"`
|
||||
Date string `json:"date,omitempty"`
|
||||
PKLo string `json:"pk_lo"`
|
||||
PKHi string `json:"pk_hi"`
|
||||
Rows int64 `json:"rows"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
ClosedAt string `json:"closed_at"`
|
||||
}
|
||||
|
||||
// Manifest appends sealed-shard records to manifest.jsonl and knows which
|
||||
// shards already exist.
|
||||
type Manifest struct {
|
||||
path string
|
||||
f *os.File
|
||||
seen map[string]bool
|
||||
entries []Entry
|
||||
}
|
||||
|
||||
const manifestName = "manifest.jsonl"
|
||||
|
||||
func OpenManifest(root string) (*Manifest, error) {
|
||||
path := filepath.Join(root, manifestName)
|
||||
m := &Manifest{path: path, seen: map[string]bool{}}
|
||||
if err := m.readExisting(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open manifest: %w", err)
|
||||
}
|
||||
m.f = f
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m *Manifest) readExisting() error {
|
||||
f, err := os.Open(m.path)
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("read manifest: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal([]byte(line), &e); err != nil {
|
||||
continue
|
||||
}
|
||||
m.entries = append(m.entries, e)
|
||||
m.seen[e.Path] = true
|
||||
}
|
||||
return sc.Err()
|
||||
}
|
||||
|
||||
func (m *Manifest) Has(rel string) bool { return m.seen[rel] }
|
||||
|
||||
func (m *Manifest) Entries() []Entry { return m.entries }
|
||||
|
||||
// Append writes one record and fsyncs it. A sealed shard is only considered
|
||||
// complete once its manifest line is durable.
|
||||
func (m *Manifest) Append(e Entry) error {
|
||||
if e.ClosedAt == "" {
|
||||
e.ClosedAt = time.Now().UTC().Format(time.RFC3339)
|
||||
}
|
||||
b, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b = append(b, '\n')
|
||||
if _, err := m.f.Write(b); err != nil {
|
||||
return fmt.Errorf("append manifest: %w", err)
|
||||
}
|
||||
if err := m.f.Sync(); err != nil {
|
||||
return fmt.Errorf("sync manifest: %w", err)
|
||||
}
|
||||
m.seen[e.Path] = true
|
||||
m.entries = append(m.entries, e)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manifest) Close() error {
|
||||
if m.f == nil {
|
||||
return nil
|
||||
}
|
||||
return m.f.Close()
|
||||
}
|
||||
|
||||
var shardNameRE = regexp.MustCompile(`^(\d+)\..*?\.([A-Za-z0-9]+?)(\.gz|\.zst|\.xz)?$`)
|
||||
|
||||
// Adopt scans the output tree for sealed shard files that have no manifest
|
||||
// record (a crash after rename but before the manifest append) and records
|
||||
// them, so resume never overwrites durable data. It returns the maximum seq
|
||||
// seen per bucket.
|
||||
func (m *Manifest) Adopt(root, format string, comp *Compressor) (map[string]int64, error) {
|
||||
maxSeq := map[string]int64{}
|
||||
root = filepath.Clean(root)
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
name := d.Name()
|
||||
if name == manifestName || name == "state.json" || name == "state.json.tmp" ||
|
||||
strings.HasSuffix(name, ".partial") || strings.HasSuffix(name, ".partial.meta") {
|
||||
return nil
|
||||
}
|
||||
mt := shardNameRE.FindStringSubmatch(name)
|
||||
if mt == nil {
|
||||
return nil
|
||||
}
|
||||
if mt[2] != format || mt[3] != comp.Ext {
|
||||
return nil
|
||||
}
|
||||
rel, err := filepath.Rel(root, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
bucket := bucketOfRel(rel)
|
||||
var seq int64
|
||||
fmt.Sscanf(mt[1], "%d", &seq)
|
||||
if seq > maxSeq[bucket] {
|
||||
maxSeq[bucket] = seq
|
||||
}
|
||||
if m.seen[rel] {
|
||||
return nil
|
||||
}
|
||||
rows, err := countRows(path, comp, format)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, size, err := hashFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
entry := Entry{
|
||||
Seq: seq,
|
||||
Path: rel,
|
||||
Date: bucket,
|
||||
Rows: rows,
|
||||
Bytes: size,
|
||||
SHA256: sum,
|
||||
}
|
||||
return m.Append(entry)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("adopt sealed shards: %w", err)
|
||||
}
|
||||
return maxSeq, nil
|
||||
}
|
||||
|
||||
func bucketOfRel(rel string) string {
|
||||
dir := filepath.ToSlash(filepath.Dir(rel))
|
||||
if dir == "." {
|
||||
return ""
|
||||
}
|
||||
parts := strings.Split(dir, "/")
|
||||
for _, p := range parts {
|
||||
if strings.HasPrefix(p, "date=") {
|
||||
return strings.TrimPrefix(p, "date=")
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// MaxSeqByBucket returns the highest seq recorded per bucket.
|
||||
func (m *Manifest) MaxSeqByBucket() map[string]int64 {
|
||||
out := map[string]int64{}
|
||||
for _, e := range m.entries {
|
||||
if e.Seq > out[e.Date] {
|
||||
out[e.Date] = e.Seq
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func hashFile(path string) (string, int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, f)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), n, nil
|
||||
}
|
||||
|
||||
func countRows(path string, comp *Compressor, format string) (int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
r, err := comp.Reader(f)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer r.Close()
|
||||
br := bufio.NewReaderSize(r, 256*1024)
|
||||
var n int64
|
||||
for {
|
||||
_, err := br.ReadBytes('\n')
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n++
|
||||
}
|
||||
if format == "csv" && n > 0 {
|
||||
n--
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// CleanupPartials removes leftover .partial files and their metadata, keeping
|
||||
// keepPartial (if non-empty) and its sidecar.
|
||||
func CleanupPartials(root, keepPartial string) error {
|
||||
var paths []string
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if keepPartial != "" && (path == keepPartial || path == keepPartial+".meta") {
|
||||
return nil
|
||||
}
|
||||
name := d.Name()
|
||||
if strings.HasSuffix(name, ".partial") || strings.HasSuffix(name, ".partial.meta") {
|
||||
paths = append(paths, path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Strings(paths)
|
||||
for _, p := range paths {
|
||||
if err := os.Remove(p); err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package shard
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
// FinalName builds the sealed shard filename:
|
||||
// <seq:06d>.<lo>-<hi>.<format>[.<comp>]
|
||||
func FinalName(seq int64, lo, hi, format, compExt string) string {
|
||||
return fmt.Sprintf("%06d.%s-%s.%s%s", seq, safeNamePart(lo), safeNamePart(hi), format, compExt)
|
||||
}
|
||||
|
||||
// KeyFilePart renders a key tuple into a filename-safe fragment.
|
||||
func KeyFilePart(k row.Key) string {
|
||||
if len(k) == 0 {
|
||||
return "NULL"
|
||||
}
|
||||
parts := make([]string, len(k))
|
||||
for i, p := range k {
|
||||
if p.Null {
|
||||
parts[i] = "NULL"
|
||||
} else {
|
||||
parts[i] = p.Val
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, ",")
|
||||
}
|
||||
|
||||
func safeNamePart(s string) string {
|
||||
if s == "" {
|
||||
return "_"
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
|
||||
b.WriteRune(r)
|
||||
case r == '.', r == '_', r == '-':
|
||||
b.WriteRune(r)
|
||||
default:
|
||||
b.WriteByte('_')
|
||||
}
|
||||
if b.Len() >= 96 {
|
||||
break
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
return "_"
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// BucketDir returns the directory that holds shards for a bucket.
|
||||
func BucketDir(root, bucket string) string {
|
||||
if bucket == "" {
|
||||
return root
|
||||
}
|
||||
return filepath.Join(root, "date="+safeNamePart(bucket))
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package shard
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"archer/internal/atomicfile"
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
// PartialMeta describes the durable prefix of an in-progress shard. It is
|
||||
// written before each chunk is committed, so a resume can truncate the file
|
||||
// back to a consistent offset and continue appending.
|
||||
type PartialMeta struct {
|
||||
Seq int64 `json:"seq"`
|
||||
Bucket string `json:"bucket,omitempty"`
|
||||
Rows int64 `json:"rows"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
FirstKey row.Key `json:"first_key"`
|
||||
LastKey row.Key `json:"last_key"`
|
||||
}
|
||||
|
||||
func partialPath(dir string, seq int64) string {
|
||||
return filepath.Join(dir, fmt.Sprintf("%06d.partial", seq))
|
||||
}
|
||||
|
||||
func partialMetaPath(dir string, seq int64) string {
|
||||
return partialPath(dir, seq) + ".meta"
|
||||
}
|
||||
|
||||
// PartialPath exposes the partial path for tests and diagnostics.
|
||||
func PartialPath(root, bucket string, seq int64) string {
|
||||
return partialPath(BucketDir(root, bucket), seq)
|
||||
}
|
||||
|
||||
// LoadPartialMeta reads the sidecar for an in-progress shard. A missing file
|
||||
// returns (nil, nil).
|
||||
func LoadPartialMeta(root, bucket string, seq int64) (*PartialMeta, error) {
|
||||
path := partialMetaPath(BucketDir(root, bucket), seq)
|
||||
b, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read partial meta: %w", err)
|
||||
}
|
||||
var m PartialMeta
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
return nil, fmt.Errorf("parse partial meta: %w", err)
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func savePartialMeta(dir string, m *PartialMeta) error {
|
||||
path := partialMetaPath(dir, m.Seq)
|
||||
b, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := path + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(b); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := atomicfile.Replace(tmp, path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func removePartialMeta(dir string, seq int64) {
|
||||
os.Remove(partialMetaPath(dir, seq))
|
||||
}
|
||||
@@ -0,0 +1,426 @@
|
||||
// Package shard writes row streams into ordered, independently importable
|
||||
// shard files and records them in manifest.jsonl.
|
||||
package shard
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"archer/internal/atomicfile"
|
||||
"archer/internal/row"
|
||||
"archer/internal/schema"
|
||||
)
|
||||
|
||||
// Options configures a Writer.
|
||||
type Options struct {
|
||||
Dir string
|
||||
Mode string // none|count|date
|
||||
ShardRows int64
|
||||
Format string
|
||||
Compressor *Compressor
|
||||
Columns []row.Column
|
||||
OutIdx []int
|
||||
CSVNull string
|
||||
StartBucket string
|
||||
StartSeq int64
|
||||
}
|
||||
|
||||
// Writer owns the single active shard file.
|
||||
type Writer struct {
|
||||
opt Options
|
||||
man *Manifest
|
||||
seq int64
|
||||
curBucket string
|
||||
cur *shardFile
|
||||
|
||||
totalRows int64
|
||||
shards int64
|
||||
bytes int64
|
||||
}
|
||||
|
||||
type shardFile struct {
|
||||
bucket string
|
||||
seq int64
|
||||
dir string
|
||||
partial string
|
||||
file *os.File
|
||||
cw *countWriter
|
||||
compW io.WriteCloser
|
||||
bw *bufio.Writer
|
||||
enc row.Encoder
|
||||
buf []byte
|
||||
rows int64
|
||||
memberRows int64
|
||||
firstKey row.Key
|
||||
lastKey row.Key
|
||||
}
|
||||
|
||||
type countWriter struct {
|
||||
w io.Writer
|
||||
n int64
|
||||
}
|
||||
|
||||
func (c *countWriter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
c.n += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Open creates a writer. It does not open a shard until the first row arrives.
|
||||
func Open(opt Options, man *Manifest) (*Writer, error) {
|
||||
w := &Writer{opt: opt, man: man, seq: opt.StartSeq, curBucket: opt.StartBucket}
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// Resume reopens an existing partial shard and continues appending to it.
|
||||
func (w *Writer) Resume(m *PartialMeta) error {
|
||||
dir := BucketDir(w.opt.Dir, m.Bucket)
|
||||
path := partialPath(dir, m.Seq)
|
||||
f, err := os.OpenFile(path, os.O_RDWR, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume partial shard: %w", err)
|
||||
}
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if st.Size() < m.Bytes {
|
||||
f.Close()
|
||||
return fmt.Errorf("partial shard %s is shorter (%d bytes) than its metadata (%d bytes); refusing to resume", path, st.Size(), m.Bytes)
|
||||
}
|
||||
if st.Size() > m.Bytes {
|
||||
if err := f.Truncate(m.Bytes); err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("truncate partial shard: %w", err)
|
||||
}
|
||||
}
|
||||
if _, err := f.Seek(m.Bytes, io.SeekStart); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
enc, err := row.NewEncoder(w.opt.Format, w.opt.Columns, w.opt.OutIdx, row.Options{CSVNull: w.opt.CSVNull})
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
sf := &shardFile{
|
||||
bucket: m.Bucket,
|
||||
seq: m.Seq,
|
||||
dir: dir,
|
||||
partial: path,
|
||||
file: f,
|
||||
enc: enc,
|
||||
rows: m.Rows,
|
||||
memberRows: m.Rows,
|
||||
firstKey: m.FirstKey,
|
||||
lastKey: m.LastKey,
|
||||
}
|
||||
if err := sf.startChain(m.Bytes); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := sf.openChain(w.opt.Compressor, 256*1024); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
w.cur = sf
|
||||
w.seq = m.Seq
|
||||
w.curBucket = m.Bucket
|
||||
w.totalRows = m.Rows
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sf *shardFile) startChain(offset int64) error {
|
||||
sf.cw = &countWriter{w: sf.file, n: offset}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sf *shardFile) openChain(comp *Compressor, bufSize int) error {
|
||||
cw, err := comp.Wrap(sf.cw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sf.compW = cw
|
||||
sf.bw = bufio.NewWriterSize(cw, bufSize)
|
||||
sf.memberRows = sf.rows
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Writer) openShard(bucket string) error {
|
||||
if w.cur != nil {
|
||||
return fmt.Errorf("internal: shard already open")
|
||||
}
|
||||
if bucket != w.curBucket {
|
||||
w.seq = 0
|
||||
w.curBucket = bucket
|
||||
}
|
||||
w.seq++
|
||||
dir := BucketDir(w.opt.Dir, bucket)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("create shard directory: %w", err)
|
||||
}
|
||||
path := partialPath(dir, w.seq)
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create partial shard: %w", err)
|
||||
}
|
||||
enc, err := row.NewEncoder(w.opt.Format, w.opt.Columns, w.opt.OutIdx, row.Options{CSVNull: w.opt.CSVNull})
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
sf := &shardFile{bucket: bucket, seq: w.seq, dir: dir, partial: path, file: f, enc: enc}
|
||||
if err := sf.startChain(0); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := sf.openChain(w.opt.Compressor, 256*1024); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if hdr := enc.Header(); len(hdr) > 0 {
|
||||
if _, err := sf.bw.Write(hdr); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
}
|
||||
w.cur = sf
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteChunk routes chunk rows into the active shard, sealing on bucket change
|
||||
// or when --shard-rows is reached. A chunk may span multiple shards.
|
||||
func (w *Writer) WriteChunk(c *row.Chunk) error {
|
||||
for i := range c.Rows {
|
||||
bucket := ""
|
||||
if w.opt.Mode == "date" {
|
||||
bucket = c.Buckets[i]
|
||||
}
|
||||
if w.cur != nil {
|
||||
needSeal := false
|
||||
if bucket != w.cur.bucket {
|
||||
if !schema.BucketLess(w.cur.bucket, bucket) {
|
||||
return fmt.Errorf("rows arrived out of shard order: %q after %q", bucket, w.cur.bucket)
|
||||
}
|
||||
needSeal = true
|
||||
} else if w.opt.ShardRows > 0 && w.cur.rows >= w.opt.ShardRows {
|
||||
needSeal = true
|
||||
}
|
||||
if needSeal {
|
||||
if err := w.seal(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if w.cur == nil {
|
||||
if err := w.openShard(bucket); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := w.cur.writeRow(c.Rows[i]); err != nil {
|
||||
return err
|
||||
}
|
||||
w.cur.trackKey(c.Keys[i])
|
||||
w.totalRows++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sf *shardFile) writeRow(r row.Row) error {
|
||||
var err error
|
||||
sf.buf, err = sf.enc.Encode(sf.buf[:0], r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := sf.bw.Write(sf.buf); err != nil {
|
||||
return fmt.Errorf("write shard row: %w", err)
|
||||
}
|
||||
sf.rows++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sf *shardFile) trackKey(k row.Key) {
|
||||
if sf.firstKey.IsZero() {
|
||||
sf.firstKey = k
|
||||
}
|
||||
sf.lastKey = k
|
||||
}
|
||||
|
||||
// Checkpoint makes the current shard bytes and their metadata durable. It is
|
||||
// called before destructive database work when --purge is enabled.
|
||||
func (w *Writer) Checkpoint() error {
|
||||
if w.cur == nil {
|
||||
return nil
|
||||
}
|
||||
sf := w.cur
|
||||
if sf.rows > sf.memberRows {
|
||||
if err := sf.closeMember(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
meta := &PartialMeta{
|
||||
Seq: sf.seq,
|
||||
Bucket: sf.bucket,
|
||||
Rows: sf.rows,
|
||||
Bytes: sf.cw.n,
|
||||
FirstKey: sf.firstKey,
|
||||
LastKey: sf.lastKey,
|
||||
}
|
||||
if err := savePartialMeta(sf.dir, meta); err != nil {
|
||||
return fmt.Errorf("write partial metadata: %w", err)
|
||||
}
|
||||
if err := sf.openChain(w.opt.Compressor, 256*1024); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sf *shardFile) closeMember() error {
|
||||
if err := sf.bw.Flush(); err != nil {
|
||||
return fmt.Errorf("flush shard: %w", err)
|
||||
}
|
||||
if err := sf.compW.Close(); err != nil {
|
||||
return fmt.Errorf("close shard compressor: %w", err)
|
||||
}
|
||||
if err := sf.file.Sync(); err != nil {
|
||||
return fmt.Errorf("sync shard: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Flush pushes buffered rows to the OS without sealing the shard.
|
||||
func (w *Writer) Flush() error {
|
||||
if w.cur == nil || w.cur.bw == nil {
|
||||
return nil
|
||||
}
|
||||
return w.cur.bw.Flush()
|
||||
}
|
||||
|
||||
// SealCurrent closes and publishes the active shard even if it is not full.
|
||||
func (w *Writer) SealCurrent() error {
|
||||
if w.cur == nil {
|
||||
return nil
|
||||
}
|
||||
if w.cur.rows == 0 {
|
||||
return w.discardCurrent()
|
||||
}
|
||||
return w.seal()
|
||||
}
|
||||
|
||||
func (w *Writer) discardCurrent() error {
|
||||
sf := w.cur
|
||||
sf.file.Close()
|
||||
os.Remove(sf.partial)
|
||||
removePartialMeta(sf.dir, sf.seq)
|
||||
w.cur = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Writer) seal() error {
|
||||
sf := w.cur
|
||||
if sf.rows == 0 {
|
||||
return w.discardCurrent()
|
||||
}
|
||||
if sf.rows > sf.memberRows || sf.compW != nil {
|
||||
if err := sf.closeMember(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := sf.file.Close(); err != nil {
|
||||
return fmt.Errorf("close partial shard: %w", err)
|
||||
}
|
||||
lo := KeyFilePart(sf.firstKey)
|
||||
hi := KeyFilePart(sf.lastKey)
|
||||
name := FinalName(sf.seq, lo, hi, w.opt.Format, w.opt.Compressor.Ext)
|
||||
finalPath := filepath.Join(sf.dir, name)
|
||||
if err := os.Remove(finalPath); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("remove stale shard: %w", err)
|
||||
}
|
||||
if err := atomicfile.Replace(sf.partial, finalPath); err != nil {
|
||||
return fmt.Errorf("publish shard: %w", err)
|
||||
}
|
||||
sum, size, err := hashFile(finalPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("hash shard: %w", err)
|
||||
}
|
||||
rel, err := filepath.Rel(w.opt.Dir, finalPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
entry := Entry{
|
||||
Seq: sf.seq,
|
||||
Path: filepath.ToSlash(rel),
|
||||
Date: sf.bucket,
|
||||
PKLo: lo,
|
||||
PKHi: hi,
|
||||
Rows: sf.rows,
|
||||
Bytes: size,
|
||||
SHA256: sum,
|
||||
}
|
||||
if err := w.man.Append(entry); err != nil {
|
||||
return err
|
||||
}
|
||||
removePartialMeta(sf.dir, sf.seq)
|
||||
w.shards++
|
||||
w.bytes += size
|
||||
w.cur = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close publishes any in-progress shard and closes the manifest.
|
||||
func (w *Writer) Close() error {
|
||||
if err := w.SealCurrent(); err != nil {
|
||||
return err
|
||||
}
|
||||
return w.man.Close()
|
||||
}
|
||||
|
||||
// StatePosition reports the bucket and the last sealed seq within that bucket,
|
||||
// which is what a checkpoint must record for resume.
|
||||
func (w *Writer) StatePosition() (bucket string, sealedSeq int64) {
|
||||
if w.cur != nil {
|
||||
return w.cur.bucket, w.cur.seq - 1
|
||||
}
|
||||
return w.curBucket, w.seq
|
||||
}
|
||||
|
||||
// Abandon closes file handles without sealing the active shard, leaving the
|
||||
// partial and its metadata for a later resume.
|
||||
func (w *Writer) Abandon() error {
|
||||
if w.cur != nil {
|
||||
if w.cur.bw != nil {
|
||||
_ = w.cur.bw.Flush()
|
||||
}
|
||||
_ = w.cur.file.Close()
|
||||
w.cur = nil
|
||||
}
|
||||
return w.man.Close()
|
||||
}
|
||||
|
||||
// Stats returns rows written, shards sealed and bytes sealed.
|
||||
func (w *Writer) Stats() (rows, shards, bytes int64) {
|
||||
return w.totalRows, w.shards, w.bytes
|
||||
}
|
||||
|
||||
// CurrentShard returns the sequence number of the active shard, or the last
|
||||
// one when no shard is currently open.
|
||||
func (w *Writer) CurrentShard() int64 {
|
||||
if w.cur != nil {
|
||||
return w.cur.seq
|
||||
}
|
||||
return w.seq
|
||||
}
|
||||
|
||||
// WrittenBytes returns sealed bytes plus the bytes already written to the
|
||||
// active partial shard.
|
||||
func (w *Writer) WrittenBytes() int64 {
|
||||
n := w.bytes
|
||||
if w.cur != nil && w.cur.cw != nil {
|
||||
n += w.cur.cw.n
|
||||
}
|
||||
return n
|
||||
}
|
||||
@@ -0,0 +1,459 @@
|
||||
package shard
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/klauspost/compress/zstd"
|
||||
|
||||
"archer/internal/row"
|
||||
)
|
||||
|
||||
func testOpts(dir, mode string) Options {
|
||||
return Options{
|
||||
Dir: dir,
|
||||
Mode: mode,
|
||||
Format: "dump",
|
||||
Compressor: &Compressor{Name: "none"},
|
||||
Columns: []row.Column{{Name: "id", Type: "int"}, {Name: "v", Type: "varchar"}},
|
||||
}
|
||||
}
|
||||
|
||||
func chunk(start int, values ...string) *row.Chunk {
|
||||
c := &row.Chunk{}
|
||||
for i, v := range values {
|
||||
id := start + i
|
||||
c.Rows = append(c.Rows, row.Row{Values: []row.Value{
|
||||
{Raw: []byte(itoa(id))},
|
||||
{Raw: []byte(v)},
|
||||
}})
|
||||
c.Keys = append(c.Keys, row.KeyOf(itoa(id)))
|
||||
}
|
||||
if len(c.Keys) > 0 {
|
||||
c.FirstKey = c.Keys[0]
|
||||
c.LastKey = c.Keys[len(c.Keys)-1]
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func itoa(i int) string {
|
||||
if i == 0 {
|
||||
return "0"
|
||||
}
|
||||
var b [20]byte
|
||||
p := len(b)
|
||||
neg := i < 0
|
||||
if neg {
|
||||
i = -i
|
||||
}
|
||||
for i > 0 {
|
||||
p--
|
||||
b[p] = byte('0' + i%10)
|
||||
i /= 10
|
||||
}
|
||||
if neg {
|
||||
p--
|
||||
b[p] = '-'
|
||||
}
|
||||
return string(b[p:])
|
||||
}
|
||||
|
||||
func TestCountShardsOrderAndNames(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, err := OpenManifest(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opt := testOpts(dir, "count")
|
||||
opt.ShardRows = 2
|
||||
w, err := Open(opt, man)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A chunk of 3 rows spans two shards.
|
||||
if err := w.WriteChunk(chunk(1, "a", "b", "c")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.WriteChunk(chunk(4, "d", "e")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"000001.1-2.dump": "1\ta\n2\tb\n",
|
||||
"000002.3-4.dump": "3\tc\n4\td\n",
|
||||
"000003.5-5.dump": "5\te\n",
|
||||
}
|
||||
for name, content := range want {
|
||||
b, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if string(b) != content {
|
||||
t.Fatalf("%s: got %q want %q", name, b, content)
|
||||
}
|
||||
}
|
||||
entries := man.Entries()
|
||||
if len(entries) != 3 {
|
||||
t.Fatalf("manifest entries: %d", len(entries))
|
||||
}
|
||||
if entries[0].PKLo != "1" || entries[0].PKHi != "2" || entries[0].Rows != 2 {
|
||||
t.Fatalf("entry0 %+v", entries[0])
|
||||
}
|
||||
if entries[2].Path != "000003.5-5.dump" || entries[2].SHA256 == "" {
|
||||
t.Fatalf("entry2 %+v", entries[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestDateShardsDirectories(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
opt := testOpts(dir, "date")
|
||||
w, _ := Open(opt, man)
|
||||
c := chunk(1, "a", "b", "c")
|
||||
c.Buckets = []string{"2025-06-01", "2025-06-01", "2025-06-02"}
|
||||
if err := w.WriteChunk(c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "date=2025-06-01", "000001.1-2.dump")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "date=2025-06-02", "000001.3-3.dump")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDateOrderViolation(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
w, _ := Open(testOpts(dir, "date"), man)
|
||||
c := chunk(1, "a", "b")
|
||||
c.Buckets = []string{"2025-06-02", "2025-06-01"}
|
||||
if err := w.WriteChunk(c); err == nil {
|
||||
t.Fatal("expected ordering error")
|
||||
}
|
||||
_ = w.Abandon()
|
||||
}
|
||||
|
||||
func TestResumeAppendsAfterTruncatingGarbage(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
opt := testOpts(dir, "count")
|
||||
opt.ShardRows = 10
|
||||
w, _ := Open(opt, man)
|
||||
if err := w.WriteChunk(chunk(1, "a", "b")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Checkpoint(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
meta, err := LoadPartialMeta(dir, "", 1)
|
||||
if err != nil || meta == nil {
|
||||
t.Fatalf("meta %v %v", meta, err)
|
||||
}
|
||||
if err := w.Abandon(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Simulate bytes from an uncommitted chunk that were flushed but never
|
||||
// checkpointed: resume must truncate them.
|
||||
partial := PartialPath(dir, "", 1)
|
||||
f, err := os.OpenFile(partial, os.O_APPEND|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.WriteString("garbage\n")
|
||||
f.Close()
|
||||
|
||||
man2, _ := OpenManifest(dir)
|
||||
w2, err := Open(opt, man2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w2.Resume(meta); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w2.WriteChunk(chunk(3, "c", "d")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w2.Checkpoint(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w2.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "000001.1-4.dump"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(b) != "1\ta\n2\tb\n3\tc\n4\td\n" {
|
||||
t.Fatalf("got %q", b)
|
||||
}
|
||||
if _, err := os.Stat(partial); !os.IsNotExist(err) {
|
||||
t.Fatal("partial should be gone after seal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestAdoptOrphan(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
orphan := filepath.Join(dir, "000007.9-12.dump")
|
||||
if err := os.WriteFile(orphan, []byte("9\tx\n10\ty\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
comp := &Compressor{Name: "none"}
|
||||
maxSeq, err := man.Adopt(dir, "dump", comp)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if maxSeq[""] != 7 {
|
||||
t.Fatalf("maxSeq %+v", maxSeq)
|
||||
}
|
||||
if len(man.Entries()) != 1 || man.Entries()[0].Rows != 2 {
|
||||
t.Fatalf("entries %+v", man.Entries())
|
||||
}
|
||||
// A second adopt must not duplicate the entry.
|
||||
if _, err := man.Adopt(dir, "dump", comp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(man.Entries()) != 1 {
|
||||
t.Fatalf("entries %+v", man.Entries())
|
||||
}
|
||||
man.Close()
|
||||
}
|
||||
|
||||
func TestSafeNamePart(t *testing.T) {
|
||||
got := safeNamePart("2024-01-01 00:00:00")
|
||||
if strings.ContainsAny(got, `:/\*?"<>|`) {
|
||||
t.Fatalf("unsafe characters kept: %q", got)
|
||||
}
|
||||
if got != "2024-01-01_00_00_00" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if safeNamePart("") != "_" {
|
||||
t.Fatal("empty name part")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompressedShardIsIndependent(t *testing.T) {
|
||||
for _, compName := range []string{"gzip", "zstd", "xz"} {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
comp, err := NewCompressor(compName, DefaultLevel)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opt := testOpts(dir, "none")
|
||||
opt.Compressor = comp
|
||||
w, _ := Open(opt, man)
|
||||
if err := w.WriteChunk(chunk(1, "a", "b")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var path string
|
||||
entries := man.Entries()
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("%s: entries %d", compName, len(entries))
|
||||
}
|
||||
path = filepath.Join(dir, entries[0].Path)
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := comp.Reader(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var b strings.Builder
|
||||
buf := make([]byte, 1024)
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
b.Write(buf[:n])
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
r.Close()
|
||||
f.Close()
|
||||
if b.String() != "1\ta\n2\tb\n" {
|
||||
t.Fatalf("%s: got %q", compName, b.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompressionLevelMapping(t *testing.T) {
|
||||
xzDictMiB := []int{1, 2, 4, 4, 8, 8, 16, 32, 64}
|
||||
for level := 1; level <= 9; level++ {
|
||||
c, err := NewCompressor("gzip", level)
|
||||
if err != nil {
|
||||
t.Fatalf("level %d: %v", level, err)
|
||||
}
|
||||
if c.Level != level || c.Name != "gzip" || c.Ext != ".gz" {
|
||||
t.Fatalf("level %d: %+v", level, c)
|
||||
}
|
||||
if got := gzipLevel(level); got != level {
|
||||
t.Fatalf("gzip level %d -> %d", level, got)
|
||||
}
|
||||
|
||||
var wantZstd zstd.EncoderLevel
|
||||
switch {
|
||||
case level <= 2:
|
||||
wantZstd = zstd.SpeedFastest
|
||||
case level <= 5:
|
||||
wantZstd = zstd.SpeedDefault
|
||||
case level <= 8:
|
||||
wantZstd = zstd.SpeedBetterCompression
|
||||
default:
|
||||
wantZstd = zstd.SpeedBestCompression
|
||||
}
|
||||
if got := zstdLevel(level); got != wantZstd {
|
||||
t.Fatalf("zstd level %d -> %v want %v", level, got, wantZstd)
|
||||
}
|
||||
if got, want := xzDictCap(level), xzDictMiB[level-1]<<20; got != want {
|
||||
t.Fatalf("xz level %d -> %d want %d", level, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
for _, name := range []string{"none", "gzip", "zstd", "xz"} {
|
||||
for _, level := range []int{-1, 0, 10} {
|
||||
if _, err := NewCompressor(name, level); err == nil {
|
||||
t.Fatalf("%s level %d: expected error", name, level)
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := NewCompressor("brotli", DefaultLevel); err == nil {
|
||||
t.Fatal("unsupported compressor must fail")
|
||||
}
|
||||
|
||||
for name, ext := range map[string]string{"none": "", "gzip": ".gz", "zstd": ".zst", "xz": ".xz"} {
|
||||
c, err := NewCompressor(name, DefaultLevel)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if c.Ext != ext {
|
||||
t.Fatalf("%s: ext %q want %q", name, c.Ext, ext)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestXZCheckpointKeepsAllStreams(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
defer man.Close()
|
||||
comp, err := NewCompressor("xz", 9)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opt := testOpts(dir, "none")
|
||||
opt.Compressor = comp
|
||||
w, _ := Open(opt, man)
|
||||
if err := w.WriteChunk(chunk(1, "a", "b")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Checkpoint(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.WriteChunk(chunk(3, "c", "d")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
entries := man.Entries()
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("entries %d", len(entries))
|
||||
}
|
||||
if entries[0].Rows != 4 {
|
||||
t.Fatalf("rows %d", entries[0].Rows)
|
||||
}
|
||||
if !strings.HasSuffix(entries[0].Path, ".dump.xz") {
|
||||
t.Fatalf("path %q", entries[0].Path)
|
||||
}
|
||||
|
||||
f, err := os.Open(filepath.Join(dir, entries[0].Path))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer f.Close()
|
||||
r, err := comp.Reader(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer r.Close()
|
||||
got, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != "1\ta\n2\tb\n3\tc\n4\td\n" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestAdoptXZOrphan(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
man, _ := OpenManifest(dir)
|
||||
defer man.Close()
|
||||
comp, err := NewCompressor("xz", DefaultLevel)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
xw, err := comp.Wrap(&buf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := xw.Write([]byte("9\tx\n10\ty\n")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := xw.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
orphan := filepath.Join(dir, "000007.9-12.dump.xz")
|
||||
if err := os.WriteFile(orphan, buf.Bytes(), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
maxSeq, err := man.Adopt(dir, "dump", comp)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if maxSeq[""] != 7 {
|
||||
t.Fatalf("maxSeq %+v", maxSeq)
|
||||
}
|
||||
entries := man.Entries()
|
||||
if len(entries) != 1 || entries[0].Seq != 7 || entries[0].Rows != 2 {
|
||||
t.Fatalf("entries %+v", entries)
|
||||
}
|
||||
if entries[0].Path != "000007.9-12.dump.xz" {
|
||||
t.Fatalf("path %q", entries[0].Path)
|
||||
}
|
||||
|
||||
// A resumed writer must continue after the adopted seq, not overwrite it.
|
||||
opt := testOpts(dir, "none")
|
||||
opt.Compressor = comp
|
||||
opt.StartSeq = maxSeq[""]
|
||||
w, _ := Open(opt, man)
|
||||
if err := w.WriteChunk(chunk(1, "a")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "000008.1-1.dump.xz")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(orphan); err != nil {
|
||||
t.Fatalf("adopted shard was overwritten: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package sink
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/sqlutil"
|
||||
)
|
||||
|
||||
// Execer is satisfied by *sql.DB, *sql.Tx and *sql.Conn.
|
||||
type Execer interface {
|
||||
ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
|
||||
}
|
||||
|
||||
const defaultMaxPacket = 4 << 20
|
||||
|
||||
// DB inserts chunk rows into the destination table in batches that stay under
|
||||
// max_allowed_packet.
|
||||
type DB struct {
|
||||
DBName string
|
||||
Table string
|
||||
Columns []string
|
||||
Mode string // "", "ignore", "replace"
|
||||
MaxPacket int64
|
||||
OutIdx []int
|
||||
}
|
||||
|
||||
// NewDB reads max_allowed_packet and returns a sink. cols are the output
|
||||
// column names; outIdx maps them to positions in a scanned row.
|
||||
func NewDB(ctx context.Context, db *sql.DB, dbName, table string, cols []string, mode string, outIdx []int) (*DB, error) {
|
||||
maxPacket := int64(defaultMaxPacket)
|
||||
var v sql.NullInt64
|
||||
if err := db.QueryRowContext(ctx, "SELECT @@max_allowed_packet").Scan(&v); err == nil && v.Valid && v.Int64 > 0 {
|
||||
maxPacket = v.Int64
|
||||
}
|
||||
return &DB{
|
||||
DBName: dbName,
|
||||
Table: table,
|
||||
Columns: cols,
|
||||
Mode: mode,
|
||||
MaxPacket: maxPacket,
|
||||
OutIdx: outIdx,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (d *DB) table() string { return sqlutil.Qualified(d.DBName, d.Table) }
|
||||
|
||||
func (d *DB) verb() string {
|
||||
switch d.Mode {
|
||||
case "ignore":
|
||||
return "INSERT IGNORE INTO"
|
||||
case "replace":
|
||||
return "REPLACE INTO"
|
||||
default:
|
||||
return "INSERT INTO"
|
||||
}
|
||||
}
|
||||
|
||||
// InsertChunk writes all chunk rows through exec.
|
||||
func (d *DB) InsertChunk(ctx context.Context, exec Execer, c *row.Chunk) (int64, error) {
|
||||
if len(c.Rows) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
limit := d.MaxPacket * 8 / 10
|
||||
var total int64
|
||||
start := 0
|
||||
for start < len(c.Rows) {
|
||||
end := start
|
||||
var size int64
|
||||
for end < len(c.Rows) {
|
||||
rowSize := int64(8 + len(d.Columns)*4)
|
||||
for _, i := range d.OutIdx {
|
||||
rowSize += int64(len(c.Rows[end].Values[i].Raw))
|
||||
}
|
||||
if end > start && size+rowSize > limit {
|
||||
break
|
||||
}
|
||||
size += rowSize
|
||||
end++
|
||||
}
|
||||
if end == start {
|
||||
end = start + 1
|
||||
}
|
||||
n, err := d.insertBatch(ctx, exec, c.Rows[start:end])
|
||||
if err != nil {
|
||||
return total, err
|
||||
}
|
||||
total += n
|
||||
start = end
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
|
||||
func (d *DB) insertBatch(ctx context.Context, exec Execer, rows []row.Row) (int64, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString(d.verb())
|
||||
b.WriteByte(' ')
|
||||
b.WriteString(d.table())
|
||||
b.WriteString(" (")
|
||||
b.WriteString(sqlutil.Columns(d.Columns))
|
||||
b.WriteString(") VALUES ")
|
||||
tuple := sqlutil.ParamTuple(len(d.Columns))
|
||||
args := make([]any, 0, len(rows)*len(d.Columns))
|
||||
for n, r := range rows {
|
||||
if n > 0 {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
b.WriteString(tuple)
|
||||
for _, i := range d.OutIdx {
|
||||
args = append(args, r.Values[i].DriverValue())
|
||||
}
|
||||
}
|
||||
res, err := exec.ExecContext(ctx, b.String(), args...)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("insert into %s: %w", d.Table, err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, nil
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// DryRunSQL returns a representative insert statement.
|
||||
func (d *DB) DryRunSQL() []string {
|
||||
tuple := sqlutil.ParamTuple(len(d.Columns))
|
||||
return []string{fmt.Sprintf("%s %s (%s) VALUES %s, %s, ...",
|
||||
d.verb(), d.table(), sqlutil.Columns(d.Columns), tuple, tuple)}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
// Package sink contains the file and database write targets.
|
||||
package sink
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"archer/internal/row"
|
||||
"archer/internal/shard"
|
||||
)
|
||||
|
||||
// File writes chunks into shard files.
|
||||
type File struct {
|
||||
W *shard.Writer
|
||||
}
|
||||
|
||||
func (f *File) WriteChunk(_ context.Context, c *row.Chunk) error { return f.W.WriteChunk(c) }
|
||||
func (f *File) Flush(context.Context) error { return f.W.Flush() }
|
||||
func (f *File) Close() error { return f.W.Close() }
|
||||
|
||||
// Checkpoint makes buffered shard data durable before a destructive commit.
|
||||
func (f *File) Checkpoint() error { return f.W.Checkpoint() }
|
||||
@@ -0,0 +1,71 @@
|
||||
// Package sqlutil holds small helpers for building SQL text safely.
|
||||
package sqlutil
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Ident quotes a MySQL identifier.
|
||||
func Ident(s string) string {
|
||||
return "`" + strings.ReplaceAll(s, "`", "``") + "`"
|
||||
}
|
||||
|
||||
// Qualified quotes a database-qualified table name.
|
||||
func Qualified(db, table string) string {
|
||||
if db == "" {
|
||||
return Ident(table)
|
||||
}
|
||||
return Ident(db) + "." + Ident(table)
|
||||
}
|
||||
|
||||
// Columns joins quoted column names with commas.
|
||||
func Columns(cols []string) string {
|
||||
parts := make([]string, len(cols))
|
||||
for i, c := range cols {
|
||||
parts[i] = Ident(c)
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// Placeholders returns n comma separated question marks.
|
||||
func Placeholders(n int) string {
|
||||
if n <= 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Repeat("?, ", n-1) + "?"
|
||||
}
|
||||
|
||||
// RowTuple renders a parenthesized, quoted column list.
|
||||
func RowTuple(cols []string) string {
|
||||
return "(" + Columns(cols) + ")"
|
||||
}
|
||||
|
||||
// ParamTuple renders a parenthesized list of n placeholders.
|
||||
func ParamTuple(n int) string {
|
||||
return "(" + Placeholders(n) + ")"
|
||||
}
|
||||
|
||||
// GreaterThan renders a keyset comparison (cols) > (?, ?, ...). For a single
|
||||
// column it emits the simpler col > ?.
|
||||
func GreaterThan(cols []string) string {
|
||||
if len(cols) == 1 {
|
||||
return Ident(cols[0]) + " > ?"
|
||||
}
|
||||
return RowTuple(cols) + " > " + ParamTuple(len(cols))
|
||||
}
|
||||
|
||||
// InList renders (cols) IN ((?,?),(?,?),...). count is the number of tuples.
|
||||
func InList(cols []string, count int) string {
|
||||
if count == 0 {
|
||||
return "1=0"
|
||||
}
|
||||
tuple := ParamTuple(len(cols))
|
||||
tuples := make([]string, count)
|
||||
for i := range tuples {
|
||||
tuples[i] = tuple
|
||||
}
|
||||
if len(cols) == 1 {
|
||||
return Ident(cols[0]) + " IN (" + Placeholders(count) + ")"
|
||||
}
|
||||
return RowTuple(cols) + " IN (" + strings.Join(tuples, ", ") + ")"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package sqlutil
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestIdentQuoting(t *testing.T) {
|
||||
if got := Ident("a`b"); got != "`a``b`" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if got := Qualified("db", "t"); got != "`db`.`t`" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if got := Qualified("", "t"); got != "`t`" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGreaterThan(t *testing.T) {
|
||||
if got := GreaterThan([]string{"id"}); got != "`id` > ?" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if got := GreaterThan([]string{"a", "b"}); got != "(`a`, `b`) > (?, ?)" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInList(t *testing.T) {
|
||||
if got := InList([]string{"id"}, 3); got != "`id` IN (?, ?, ?)" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if got := InList([]string{"a", "b"}, 2); got != "(`a`, `b`) IN ((?, ?), (?, ?))" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if got := InList([]string{"id"}, 0); got != "1=0" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
// Package version reports build information injected at link time.
|
||||
package version
|
||||
|
||||
import "fmt"
|
||||
|
||||
var (
|
||||
Version = "dev"
|
||||
Commit = "none"
|
||||
Date = "unknown"
|
||||
)
|
||||
|
||||
func String() string {
|
||||
return fmt.Sprintf("archer %s (commit %s, built %s)", Version, Commit, Date)
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-11-06
|
||||
@@ -0,0 +1,136 @@
|
||||
## Context
|
||||
|
||||
现状与动机见 `proposal.md`。与设计直接相关的几个约束:
|
||||
|
||||
- 运行循环是单线程的(`internal/app/app.go` 的 `execute` → `processChunk`),计数分散在 `Stats` 结构体的裸字段里,直接加后台 goroutine 会产生 data race。
|
||||
- 唯一能让"长阻塞期间仍有输出"的办法是后台定时器:`scan.Next`、fsync、`lag.Wait` 都无法在阻塞点内部插入回调。
|
||||
- 扫描器(`internal/scan`)的状态不是并发安全的;进度不能从后台线程直接读扫描器或 writer。
|
||||
- 项目刻意保持零额外依赖(`go.mod` 只有 MySQL 驱动与压缩库),且需兼容 Windows conhost。
|
||||
- 现有 `--progress N` 是 int,默认 0 = 关闭;`--statistics` 与 `--dry-run` 输出在 stdout。
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- 终端里能一眼看到"进行到哪、还剩多久、是不是卡住了"。
|
||||
- 重定向到日志时可按需得到稳定、完整、可 `tail` 的进度行。
|
||||
- 估算成本有上界,估不出来就诚实地不显示。
|
||||
- 进度实现与归档逻辑解耦,可注入时钟与 writer 做单测。
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- 不做机器可读事件流(JSONL)、颜色、多行 TUI。
|
||||
- 不做精确 `COUNT(*)` 或后台异步计数。
|
||||
- 不改变 chunk / 事务 / 文件 / checkpoint 的任何语义。
|
||||
- 不为 `export-all` 脚本实现多表聚合进度(脚本可用 `--progress on` 获取日志行)。
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1:新增 `internal/progress` 作为深模块
|
||||
|
||||
`Reporter` 对外只暴露少量方法:`Start/Stop`、`Phase`、`AddRows`、`SetPosition`、`SetEstimate`、`Message`、`Snapshot`。渲染、TTY 判定、速率窗口、ETA 全部藏在内部。`app` 只负责在正确的时机推送事件,不知道渲染细节。
|
||||
|
||||
替代方案是直接在 `app.go` 里加 ticker 和格式化逻辑,但 `app.go` 已经 600 行且承担编排职责;把渲染测试塞进 `app` 会让集成测试更难写。
|
||||
|
||||
### D2:`--progress` 改为 `auto|on|off|N`,而不是新增一堆布尔开关
|
||||
|
||||
仓库已有 `--progress N` 的调用方(集成测试、脚本),必须保留 `N>0` 的行阈值语义。把 flag 实现为自定义 `flag.Value`:
|
||||
|
||||
- `auto`(默认):TTY 时开启,非 TTY 静默。
|
||||
- `on`:始终开启,非 TTY 时退化为周期行。
|
||||
- `off` / `0`:关闭。
|
||||
- `N>0`:开启并额外按行阈值触发。
|
||||
|
||||
替代方案是保留 int 并用 `-1` 表示 auto;但 `--progress -1` 语义不直观,且 argparse 风格上容易被误读为错误。字符串取值可以把三种意图明确写进 `--help`。
|
||||
|
||||
### D3:进度写 stderr,消息也收口到 Reporter
|
||||
|
||||
stderr 是"给人看的诊断流",stdout 留给 `--dry-run` 和显式请求的 `--statistics`。这也让 `archer ... > result.txt` 不会把进度写进结果文件。
|
||||
|
||||
`app.go` 现有 6 处直接 `fmt.Fprintln(os.Stderr, ...)`(启动提示、指纹警告、purge 警告、中断警告、lag 等待)必须改走 `Reporter.Message`。`Message` 在 live 模式下先清当前行、打印整行消息,再在下一拍重绘;非 live 模式直接打印。`main.go` 的错误输出发生在 `Run` 返回之后,reporter 已停止,不需要改。
|
||||
|
||||
替代方案是让渲染器在检测到行长度变化时自行清行,但那样消息输出顺序无法保证,仍可能撕裂 live 行。
|
||||
|
||||
### D4:TTY 判定与渲染
|
||||
|
||||
用标准库 `os.Stderr.Stat()` 判断 `ModeCharDevice`,不引入 `x/term`。live 模式单行渲染:
|
||||
|
||||
```
|
||||
[###############-----] 58% ~4.12M rows 18.2k/s scan 0.4s shard 12 eta ~1m36s
|
||||
```
|
||||
|
||||
- 只用 `\r` 回到行首,行尾用空格补齐到上一次宽度,不使用 `\x1b[K` 或颜色,保证 conhost 正常。
|
||||
- 行宽目标 ≤ 100 字符;超宽时按优先级裁剪字段(先去掉分片序号与字节数,再缩短 bar),不换行。
|
||||
- 无百分比时省略 bar/百分比/ETA,只留 rows、rate、phase、elapsed。
|
||||
- 非 live 模式固定输出 `progress: rows=... pct=... estimate=... rate=... phase=... elapsed=...`,字段顺序稳定。
|
||||
|
||||
### D5:阶段由主循环推送,Reporter 只记录
|
||||
|
||||
阶段切换点全部在 `app` 内:`probe`(`setupSource`)、`resume`(`prepareFile` 中 `Manifest.Adopt`)、`estimate`、`scan`/`file`/`db`/`checkpoint`(`processChunk` 内)、`finalize`(`finish`)、`lag-wait`(`lag.OnWait`)。Reporter 为每次 `Phase()` 调用累加前一阶段耗时并重置阶段计时。
|
||||
|
||||
扫描器与 writer 的状态**不**由后台线程读取:主循环在 chunk 完成后调用 `SetPosition(bucket, seq, bytes, lastKey)`,把 writer 的 `StatePosition/Stats` 结果和当前键位置复制进快照。这样后台线程只接触 Reporter 内部数据。
|
||||
|
||||
### D6:并发模型:互斥量保护快照
|
||||
|
||||
Reporter 内部一个 `sync.Mutex` 保护 `snapshot` 结构(计数、阶段、阶段开始时间、估算、位置、消息代数)。主循环的 `AddRows/Phase/SetPosition` 都是短临界区;ticker 每次加锁复制一份快照,解锁后渲染。相比把每个字段拆成 `atomic.Int64`,互斥量让"一组字段的一致快照"更容易保证,也避免 `atomic.Value` 里放结构体的分配开销。
|
||||
|
||||
`Stats` 继续由主循环独占,供 `--statistics` 使用;Reporter 不反向写 `Stats`。两者在结束时由 `app` 汇总。
|
||||
|
||||
### D7:估算查询:有界、只读、可放弃
|
||||
|
||||
估算在 `prepare` 之后、进入扫描前执行一次:
|
||||
|
||||
- 日期模式:`SELECT MIN(shard_col), MAX(shard_col) FROM t WHERE (where) AND shard_col IS NOT NULL`。
|
||||
- 非日期模式:仅当扫描键为单列整数类型时执行 `SELECT MIN(key), MAX(key) FROM t WHERE (where)`。
|
||||
- 复合键、字符串/二进制键、或表没有可用索引时直接跳过。
|
||||
|
||||
查询在独立 `*sql.Conn` 上执行,`context.WithTimeout(2s)`。超时或出错则 `SetEstimate(nil)`,此后不显示百分比与 ETA,**不重试**。
|
||||
|
||||
路径 B 的扫描器稍后还会自己算一次 `MIN/MAX`(`initDays`)。这里接受一次重复的索引范围查询,换取 Reporter 与扫描器解耦;该查询与路径 B 首次查询量级相同,不影响正确性。
|
||||
|
||||
替代方案是让扫描器实现 `EstimateProvider` 接口,但那会把并发发布职责扩散到三个扫描器实现里。
|
||||
|
||||
### D8:位置与百分比口径
|
||||
|
||||
- 日期模式:位置取当前 chunk 尾行的 bucket(`chunk.Buckets` 已有),百分比 = 已完成天数 / 总天数;显示 `day 37/120`,`estimate=day`。NULL 与零日期阶段单独处理,未进入正常日期前不显示天数进度。
|
||||
- 非日期模式:位置取 `chunk.LastKey[0]` 的数值,百分比 = (当前键 − 最小键) / (最大键 − 最小键),`estimate=key`。
|
||||
- 续跑时位置来自游标,天然包含此前进度。
|
||||
|
||||
已知口径限制写进 Risks,不在代码里假装精确。
|
||||
|
||||
### D9:速率与 ETA
|
||||
|
||||
Reporter 维护最近约 5 秒的 `(时间, 进度位置)` 样本环,速率用窗口内差分计算;不足两个样本时回退到累计平均。ETA = 剩余量 / 窗口速率,其中剩余量由百分比口径推导(天数或键跨度),而不是行数——因为总行数未知。ETA 与百分比一样带 `~`,且只在 `estimate != none` 时显示。
|
||||
|
||||
### D10:生命周期
|
||||
|
||||
- `execute` 开始时 `Start`,`finish` 与 `interrupt` 结束时 `Stop`。
|
||||
- `Stop` 先停 ticker、清 live 行;正常完成再输出 `done` 行,中断则输出最终快照行。
|
||||
- `interrupt` 的执行顺序:停刷新 → 清行 → 快照 → 现有 `SealCurrent` → `saveState`,退出码不变。
|
||||
- `--dry-run` 不启动 Reporter(没有运行循环)。
|
||||
|
||||
### D11:续跑基线
|
||||
|
||||
`prepareFile` 读到 `state.json` 后,除现有游标外记录 `baseRows = st.RowsDone`。Reporter 的 `rows` 指标显示 `baseRows + 本次 RowsRead`;`saveState` 写入 `baseRows + 本次 RowsCommitted`。`--statistics` 增加"本次 / 累计"两列。
|
||||
|
||||
### D12:不新增依赖
|
||||
|
||||
TTY 判定用 `os.File.Stat`;时间源注入 `func() time.Time`;渲染目标注入 `io.Writer`。单测不需要伪终端,直接用 `bytes.Buffer` + 手动推进时钟。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **估算查询在繁忙库上仍可能拖慢启动** → 2s 上下文超时 + 只在启动时一次 + 绝不 `COUNT(*)`;超时后彻底放弃,不重试。
|
||||
- **键位置百分比在键有空洞或分布不均时偏差大** → 标注 `estimate=key` 与 `~`;日期模式优先按天;任何估算都不影响归档逻辑。
|
||||
- **日期模式的百分比按天跳变** → 显示 `day 37/120` 原文,让操作者看到真实口径;不把天进度伪装成行进度。
|
||||
- **升级后首次续跑旧 checkpoint 时 `rows_done` 仍可能偏低** → 旧版本写入的值只是上次运行的行数;无法还原真实累计。文档说明该限制,不做迁移;新写入的值从此正确。
|
||||
- **live 行与日志交错** → 所有 stderr 消息经 `Reporter.Message`;单测覆盖"消息插入后行内容完整"。
|
||||
- **Windows conhost 不支持复杂控制序列** → 只使用 `\r` + 空格补齐;不引入颜色与光标移动。
|
||||
- **进度从 stdout 转到 stderr 会改变现有调用方** → 集成测试更新;README/spec 明确通道;脚本本来就把 stderr 重定向进日志。
|
||||
- **ticker 增加少量锁竞争** → 每秒一次、临界区仅是字段复制与格式化,开销可忽略。
|
||||
|
||||
## Migration Plan
|
||||
|
||||
- 仅发布新二进制,无数据迁移;`state.json` 版本不变,旧文件可读。
|
||||
- 升级后第一次续跑可能沿用旧的偏低 `rows_done`,完成后即恢复累计语义。
|
||||
- `scripts/export-all.ps1` 的 `-Verify` 在历史 checkpoint 上可能仍报一次差异,重新完成后消失;如需日志内进度,可给 archer 加 `--progress on`。
|
||||
- 回滚到旧二进制不影响运行;旧二进制会按旧语义覆盖 `rows_done`,仅丢失累计值。
|
||||
@@ -0,0 +1,37 @@
|
||||
## Why
|
||||
|
||||
archer 面向的是长时间运行的大表归档/导出,但当前唯一的进度输出(`--progress N`)只在 chunk 完整处理完后按行数阈值打印一行,既没有速率、百分比和 ETA,也不显示当前处于扫描、写文件、写库还是等待副本延迟。路径 B 的单日长查询、fsync 与 `lag.Wait` 期间完全没有输出,操作者无法区分"正在慢"和"已经卡死";续跑后进度与 `state.json` 的行数还会从零重新起算。
|
||||
|
||||
这次改动把进度从"稀疏日志"变成"给人看的实时显示",同时修正续跑计数语义。
|
||||
|
||||
## What Changes
|
||||
|
||||
- 新增默认 `--progress auto`:stderr 是终端时输出单行实时刷新(`\r` 覆盖、空格补齐、不使用 ANSI 清行与颜色,兼容 Windows conhost);非终端时保持静默。
|
||||
- `--progress on|off|auto|N`:`on` 强制开启(非终端时按间隔输出稳定的 key=value 行,便于 `tail` 日志),`N>0` 保留现有行阈值语义,`off`/`0` 关闭。新增 `--progress-interval`(默认 1s)。
|
||||
- 进度行统一写 stderr;`--dry-run` 与 `--statistics` 仍在 stdout。
|
||||
- 显示内容包含 rows、窗口速率、分片序号、已写字节、当前阶段及各阶段耗时、elapsed;在能廉价估算总量时额外显示带 `~` 的百分比与 ETA,并标注估算口径(`day|key`)。
|
||||
- 新增阶段模型:`probe → resume → estimate → scan → file → db → checkpoint → finalize`,`lag-wait` 作为 scan 前的独立阶段;长阻塞期间持续刷新。
|
||||
- 进度开启时,结束打印一行 `done` 汇总;中断时停止刷新、清除 live 行并打印最终快照后再执行现有的封片与 checkpoint。
|
||||
- **BREAKING(行为)**:`state.json` 的 `rows_done` 改为累计值(含续跑前已处理行数)。旧 checkpoint 继续可读;语义变化会影响依赖该字段的校验脚本(修正 `scripts/export-all.ps1 -Verify` 的误报)。
|
||||
- `--statistics` 分开展示"本次运行"与"累计(含续跑)"。
|
||||
- 应用内所有面向人的警告与 lag 提示收口到进度 reporter,避免与 live 行交错。
|
||||
|
||||
非目标:JSONL/机器可读进度、彩色输出、多行 TUI、精确 `COUNT(*)`、后台异步计数、`export-all` 脚本自身的多表进度条(脚本可通过 `--progress on` 获得日志行)。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `progress-reporting`: 进度输出的模式与降级、显示指标与格式、廉价总量估算与回退、阶段模型、中断与完成时的最终快照、续跑基线语义。
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `cli-contract`: `--progress` 的取值与默认值、新增 `--progress-interval`、进度输出通道(stderr)及 TTY/非 TTY 行为。
|
||||
- `archive-core`: 运行循环的阶段标记、lag 等待期间的进度、中断时的最终快照,以及 `state.json` 中 `rows_done` 的累计语义与 `--statistics` 口径。
|
||||
|
||||
## Impact
|
||||
|
||||
- 代码:新增 `internal/progress`;修改 `internal/app/app.go`(阶段标记、消息收口、续跑基线、done/中断快照)、`internal/app/probe.go` 或扫描器(暴露日期/键边界供估算)、`internal/cli/{parse,config,validate}.go`、`internal/checkpoint` 使用方、`internal/lag` 的等待回调。
|
||||
- 输出契约:进度从 stdout 移到 stderr;集成测试中按 stdout 抓 `progress:` 的用例需要更新。
|
||||
- 脚本:`scripts/export-all.ps1` 的 `-Verify` 恢复正确;可选加 `--progress on` 让日志可 `tail -f`。
|
||||
- 依赖:不新增第三方库(TTY 判定使用标准库;不使用 ANSI 清行)。
|
||||
@@ -0,0 +1,71 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 断点续跑
|
||||
|
||||
系统 SHALL 在每个 chunk 提交后原子更新 `state.json`,记录配置指纹与游标位置。`state.json` 中记录的源连接与目标连接 MUST 采用标准 MySQL DSN 形式(去除密码),例如 `user@tcp(host:3306)/db?charset=utf8mb4`。续跑时 MUST 校验源、表、过滤条件、列、分片参数、时区、输出格式、压缩设置,以及源/目标连接的全部规范化参数(含字符集与排序规则)的指纹;不一致时 MUST 拒绝续跑,除非显式传入 `--force-resume`。密码 MUST NOT 计入指纹,也 MUST NOT 写入 `state.json`。压缩档位仅在启用压缩且档位不是默认值 3 时纳入指纹,以保证默认配置的存量任务可以无缝续跑。`rows_done` MUST 表示该输出目录累计已提交的行数(含续跑前已提交的部分),MUST NOT 在续跑后被重置为本次运行的行数。
|
||||
|
||||
#### Scenario: 配置一致时续跑
|
||||
|
||||
- **WHEN** 上次运行在 chunk 中途退出,且本次配置指纹一致
|
||||
- **THEN** 系统从 `state.json` 记录的游标位置继续,不重复处理已提交的 chunk
|
||||
|
||||
#### Scenario: 配置被修改
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出不一致的字段,拒绝开始扫描
|
||||
|
||||
#### Scenario: 压缩档位被修改
|
||||
|
||||
- **WHEN** 续跑时 `--compress-level` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `compress_level` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 默认档位兼容存量任务
|
||||
|
||||
- **WHEN** `state.json` 由引入压缩档位之前的版本写入,且本次使用默认档位 3
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接字符集被修改
|
||||
|
||||
- **WHEN** 续跑时 `--source` 的 `charset` 或 `collation` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `source` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 密码轮换
|
||||
|
||||
- **WHEN** 续跑时仅 `--source` 或 `--dest` 的密码与上次运行不同
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接信息的存储形态
|
||||
|
||||
- **WHEN** 查看任意一次运行生成的 `state.json`
|
||||
- **THEN** `config.source` 与 `config.dest` 为标准 MySQL DSN,且不包含密码明文
|
||||
|
||||
#### Scenario: 续跑后累计行数
|
||||
|
||||
- **WHEN** 首次运行提交 10000 行后中断,随后续跑提交剩余 40000 行
|
||||
- **THEN** 完成后的 `state.json` 中 `rows_done` 为 50000,而不是 40000
|
||||
|
||||
#### Scenario: 校验脚本读取累计值
|
||||
|
||||
- **WHEN** 一次经过续跑的导出完成后,外部脚本读取 `state.json` 的 `rows_done` 并与源表匹配行数比较
|
||||
- **THEN** 两者一致,不产生误报
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 运行阶段顺序
|
||||
|
||||
系统 SHALL 在运行中维护以下阶段:`probe`、`resume`、`estimate`、`scan`、`file`、`db`、`checkpoint`、`finalize`。启用复制延迟保护时,每个 chunk 拉取前 MUST 进入 `lag-wait` 阶段。单个 chunk 的处理顺序 MUST 为 `lag-wait → scan → file → db → checkpoint`。阶段切换 MUST 对进度输出可见,且任一阶段持续期间 MUST NOT 阻断周期性进度刷新。
|
||||
|
||||
#### Scenario: 等待副本延迟
|
||||
|
||||
- **WHEN** 副本延迟超过 `--max-lag`
|
||||
- **THEN** 系统进入 `lag-wait` 阶段并持续输出进度,直到延迟恢复
|
||||
|
||||
#### Scenario: 恢复后进入扫描
|
||||
|
||||
- **WHEN** 延迟恢复到 `--max-lag` 以下
|
||||
- **THEN** 系统离开 `lag-wait`,按 `scan → file → db → checkpoint` 处理下一个 chunk
|
||||
|
||||
#### Scenario: 长时间扫描
|
||||
|
||||
- **WHEN** 路径 B 的单日查询长时间未返回首个 chunk
|
||||
- **THEN** 阶段保持 `scan`,进度刷新不中断,操作者可观察到阶段耗时增长
|
||||
@@ -0,0 +1,68 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 默认值
|
||||
|
||||
系统 SHALL 采用以下默认值:`--chunk-size 2000`、`--shard-by none`、`--shard-rows 0`、`--format dump`、`--compress none`、`--compress-level 3`、`--max-lag 1s`、`--check-interval 1s`、`--retries 1`、`--safe-auto-increment true`、`--progress auto`、`--progress-interval 1s`。
|
||||
|
||||
#### Scenario: 最小可运行配置
|
||||
|
||||
- **WHEN** 用户只提供 `--source`、`--table`、`--where`、`--file`
|
||||
- **THEN** 系统以 2000 行一个 chunk、不分片、dump 格式、不压缩的方式导出
|
||||
|
||||
#### Scenario: 默认进度模式
|
||||
|
||||
- **WHEN** 用户未显式指定 `--progress`
|
||||
- **THEN** stderr 是终端时显示实时进度,否则不输出进度行
|
||||
|
||||
### Requirement: 参数依赖与互斥
|
||||
|
||||
系统 MUST 校验以下规则:`--shard-by date` 必须有 `--shard-column`;`--shard-by count|none` 不得带 `--shard-column`;`--shard-rows` 仅在 `--shard-by count` 或 `date` 时有效;`--bulk-delete` 依赖 `--purge` 且在日期模式下禁止;普通插入、`--insert-ignore`、`--replace` 三者互斥;`--compress-level` MUST 在 1–9 范围内;`--compress none` 与非默认档位同时出现时 MUST 视为配置错误;`--progress` MUST 取值 `auto`、`on`、`off` 或非负整数;`--progress-interval` MUST 为正时长。违反时 MUST 在启动阶段报错。
|
||||
|
||||
#### Scenario: 数量模式误传日期列
|
||||
|
||||
- **WHEN** `--shard-by count` 同时带上 `--shard-column`
|
||||
- **THEN** 系统报错,说明该参数仅在日期模式下有效
|
||||
|
||||
#### Scenario: 未启用 purge 却要求 bulk-delete
|
||||
|
||||
- **WHEN** 用户指定 `--bulk-delete` 但未指定 `--purge`
|
||||
- **THEN** 系统报错,说明 bulk-delete 必须配合 purge
|
||||
|
||||
#### Scenario: 插入模式互斥
|
||||
|
||||
- **WHEN** 用户同时指定 `--insert-ignore` 与 `--replace`
|
||||
- **THEN** 系统报错,要求只能选择一种插入模式
|
||||
|
||||
#### Scenario: 压缩档位越界
|
||||
|
||||
- **WHEN** 用户指定 `--compress-level 0` 或 `--compress-level 10`
|
||||
- **THEN** 系统在启动阶段报错,说明档位必须在 1–9 之间
|
||||
|
||||
#### Scenario: 不压缩却指定非默认档位
|
||||
|
||||
- **WHEN** 用户指定 `--compress none` 与 `--compress-level 5`
|
||||
- **THEN** 系统在启动阶段报错,说明压缩档位仅在启用压缩时有效
|
||||
|
||||
#### Scenario: 非法进度参数
|
||||
|
||||
- **WHEN** 用户指定 `--progress abc`、`--progress -1` 或 `--progress-interval 0s`
|
||||
- **THEN** 系统在启动阶段报错,说明取值要求
|
||||
|
||||
### Requirement: 进度与统计
|
||||
|
||||
`--progress` SHALL 接受 `auto`、`on`、`off` 或非负整数,默认 `auto`。`auto` 时,stderr 是终端则启用实时进度,否则不输出进度行;`on` MUST 始终启用;`off` 与 `0` MUST 关闭;正整数 MUST 额外作为每处理 N 行触发一次的行阈值。`--progress-interval` MUST 控制最小输出间隔,默认 `1s`。所有进度输出 MUST 写入 stderr;`--dry-run` 与 `--statistics` 的输出 MUST 保持在 stdout。启用 `--statistics` 时,系统 MUST 在结束时分别输出本次运行与含续跑的累计读取行数、写入行数、删除行数、分片数与各阶段耗时。
|
||||
|
||||
#### Scenario: 周期进度
|
||||
|
||||
- **WHEN** `--progress 100000`
|
||||
- **THEN** 系统至少每 `--progress-interval` 输出一次进度,并在每处理 100000 行时额外输出一次
|
||||
|
||||
#### Scenario: 终端实时进度
|
||||
|
||||
- **WHEN** stderr 是终端且进度启用
|
||||
- **THEN** 系统在同一行内刷新进度,并输出到 stderr
|
||||
|
||||
#### Scenario: 结束统计
|
||||
|
||||
- **WHEN** 启用 `--statistics`
|
||||
- **THEN** 进程退出前输出汇总统计,并区分本次运行与含续跑的累计值
|
||||
@@ -0,0 +1,138 @@
|
||||
## Purpose
|
||||
|
||||
定义 archer 面向操作者的实时进度显示:何时显示、显示什么、百分比与 ETA 如何估算与降级,以及完成和中断时的最终反馈,使长时间归档任务的运行状态可被直接观察。
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 进度模式与输出通道
|
||||
|
||||
`--progress` SHALL 接受 `auto`、`on`、`off` 或正整数,默认 `auto`。`auto` 时,当 stderr 是终端则启用实时进度,否则 MUST 不输出任何进度行。`on` MUST 始终启用进度。`off` 或 `0` MUST 关闭进度。正整数 MUST 在按间隔输出之外,额外作为"每处理 N 行至少触发一次"的行阈值。
|
||||
|
||||
所有进度输出 MUST 写入 stderr。`--dry-run` 与 `--statistics` 的输出 MUST 保持在 stdout。启用 `--progress-interval` 且值为非正数时 MUST 视为配置错误。
|
||||
|
||||
#### Scenario: 终端下的默认模式
|
||||
|
||||
- **WHEN** 用户未传 `--progress`,且 stderr 是终端
|
||||
- **THEN** 系统显示实时进度
|
||||
|
||||
#### Scenario: 重定向时的默认模式
|
||||
|
||||
- **WHEN** 用户未传 `--progress`,且 stderr 被重定向到文件或管道
|
||||
- **THEN** 系统不输出进度行,只有警告与错误信息
|
||||
|
||||
#### Scenario: 强制开启写入日志
|
||||
|
||||
- **WHEN** 用户传 `--progress on`,且 stderr 被重定向
|
||||
- **THEN** 系统按 `--progress-interval` 向 stderr 输出进度行,每行是一条完整的 key=value 记录
|
||||
|
||||
#### Scenario: 关闭进度
|
||||
|
||||
- **WHEN** 用户传 `--progress off` 或 `--progress 0`
|
||||
- **THEN** 系统不输出任何进度行
|
||||
|
||||
#### Scenario: 非法进度参数
|
||||
|
||||
- **WHEN** 用户传 `--progress abc` 或 `--progress-interval 0s`
|
||||
- **THEN** 系统在连接数据库前以配置错误退出
|
||||
|
||||
### Requirement: 刷新节奏
|
||||
|
||||
系统 MUST 至少每隔 `--progress-interval`(默认 `1s`)输出一次进度,并在启用行阈值时于跨过阈值的 chunk 边界额外输出。无论是否到达间隔或阈值,完成与中断时 MUST 输出一次最终快照。
|
||||
|
||||
#### Scenario: 慢 chunk 仍在刷新
|
||||
|
||||
- **WHEN** 单个 chunk 的扫描或写库耗时超过 `--progress-interval`
|
||||
- **THEN** 系统在该 chunk 处理期间仍按间隔输出进度,不出现长时间静默
|
||||
|
||||
#### Scenario: 行阈值提前触发
|
||||
|
||||
- **WHEN** `--progress 100000`,且两个间隔之间处理行数超过 100000
|
||||
- **THEN** 系统在跨过阈值时额外输出一次进度
|
||||
|
||||
### Requirement: 显示指标
|
||||
|
||||
终端模式下,进度 MUST 以单行原地刷新(使用回车覆盖并以空格补齐,不依赖 ANSI 清行或颜色)。进度行 MUST 包含:已处理行数、当前速率、当前阶段及阶段内耗时、本次运行已用时间。当当前分片存在时 MUST 包含分片序号;写入文件时 MUST 包含已写字节数。存在总量估算时 MUST 额外包含百分比与 ETA,并以 `~` 标注为估算值。
|
||||
|
||||
非终端模式下,每行 MUST 是一条独立、完整、稳定的 key=value 记录,至少包含 `rows`、`rate`、`phase`、`elapsed`;存在估算时包含 `pct`、`estimate` 与 `eta`。记录 MUST NOT 使用回车覆盖。
|
||||
|
||||
#### Scenario: 终端单行刷新
|
||||
|
||||
- **WHEN** stderr 是终端且进度启用
|
||||
- **THEN** 每秒在同一行内更新进度,不向终端累积刷屏
|
||||
|
||||
#### Scenario: 重定向下的一行一记录
|
||||
|
||||
- **WHEN** stderr 被重定向且通过 `--progress on` 启用进度
|
||||
- **THEN** 每行是一条完整记录,不含回车控制字符,可直接 `tail` 或按行解析
|
||||
|
||||
### Requirement: 廉价总量估算与降级
|
||||
|
||||
系统 MUST 只使用有界成本的估算方式,MUST NOT 默认执行 `COUNT(*)`。日期分片模式 SHALL 使用分片列的 `MIN`/`MAX` 得到天数进度,并标注估算口径为 `day`;非日期模式仅在扫描键是单列整数唯一键时使用键的 `MIN`/`MAX` 做位置比例,并标注为 `key`。估算查询 MUST 在独立连接上执行并设有超时上限;超时、失败或口径不适用时,系统 MUST 静默降级为不显示百分比与 ETA,并标注 `estimate=none`。
|
||||
|
||||
百分比与 ETA MUST 带 `~` 前缀。ETA MUST 基于本次运行的推进速率与剩余量计算,MUST NOT 使用"已用时间 ÷ 百分比"。
|
||||
|
||||
#### Scenario: 日期模式按天估算
|
||||
|
||||
- **WHEN** `--shard-by date` 且存在可用的分片列索引
|
||||
- **THEN** 进度显示形如 `day 37/120` 的百分比与 ETA,并标注 `estimate=day`
|
||||
|
||||
#### Scenario: 单列整数键估算
|
||||
|
||||
- **WHEN** 非日期模式,扫描键是单列整数唯一键
|
||||
- **THEN** 进度显示基于键位置的百分比与 ETA,并标注 `estimate=key`
|
||||
|
||||
#### Scenario: 复合键或字符串键
|
||||
|
||||
- **WHEN** 扫描键为复合键或非整数类型
|
||||
- **THEN** 进度不显示百分比与 ETA,并标注 `estimate=none`
|
||||
|
||||
#### Scenario: 估算超时
|
||||
|
||||
- **WHEN** 估算查询在超时上限内未返回
|
||||
- **THEN** 系统放弃本次估算,进度降级为仅显示行数与速率,归档继续执行
|
||||
|
||||
### Requirement: 阶段可见性
|
||||
|
||||
系统 MUST 在进度中显示当前阶段,阶段集合为:`probe`、`resume`、`estimate`、`scan`、`file`、`db`、`checkpoint`、`finalize`、`lag-wait`。阶段内耗时 MUST 随时间递增,使操作者可以区分"正在慢"与"已经卡死"。`lag-wait` 阶段 MUST 显示等待原因。
|
||||
|
||||
#### Scenario: 等待副本延迟
|
||||
|
||||
- **WHEN** 副本延迟超过 `--max-lag`
|
||||
- **THEN** 进度显示 `lag-wait` 阶段与等待原因,而不是只输出一条 `waiting:` 消息
|
||||
|
||||
#### Scenario: 长扫描期间保持刷新
|
||||
|
||||
- **WHEN** 路径 B 的单日查询长时间未返回首个 chunk
|
||||
- **THEN** 阶段保持 `scan`,进度继续按间隔刷新,阶段内耗时持续增长
|
||||
|
||||
### Requirement: 消息与实时行互斥
|
||||
|
||||
在实时进度启用时,任何警告、提示或 lag 等待消息 MUST NOT 与实时行交错。系统 MUST 在输出消息前清除当前实时行,并在下一个刷新周期重新绘制。
|
||||
|
||||
#### Scenario: 警告出现在实时行之后
|
||||
|
||||
- **WHEN** 实时进度运行期间产生一条警告
|
||||
- **THEN** 警告单独占一行,实时行不在同一行上被截断或拼接
|
||||
|
||||
### Requirement: 完成与中断反馈
|
||||
|
||||
进度启用时,正常完成 MUST 输出一行最终汇总,包含累计行数、耗时、分片数与已写字节(存在文件输出时)。收到中断信号时,系统 MUST 先停止刷新、清除实时行并输出最终快照,然后执行现有的封片与 checkpoint 写入,退出码保持现有语义(130)。
|
||||
|
||||
#### Scenario: 正常完成
|
||||
|
||||
- **WHEN** 扫描自然结束且进度启用
|
||||
- **THEN** 在退出前输出一行 `done` 汇总
|
||||
|
||||
#### Scenario: 中断时保留现场
|
||||
|
||||
- **WHEN** 运行期间收到 SIGINT
|
||||
- **THEN** 进度停止并输出最终快照,随后当前分片被封存、checkpoint 被写入,进程以 130 退出
|
||||
|
||||
### Requirement: 续跑基线
|
||||
|
||||
进度中的已处理行数 MUST 包含当前输出目录在续跑前已提交的行数,即 `已处理 = 续跑前累计 + 本次运行读取行数`。续跑后进度 MUST NOT 从零重新起算。
|
||||
|
||||
#### Scenario: 续跑后继续累计
|
||||
|
||||
- **WHEN** 上次运行处理 10000 行后中断,本次续跑处理 40000 行
|
||||
- **THEN** 进度显示已处理行数为 50000,而不是 40000
|
||||
@@ -0,0 +1,52 @@
|
||||
## 1. CLI 参数与校验
|
||||
|
||||
- [x] 1.1 把 `--progress` 从 int 改为自定义 `flag.Value`,支持 `auto|on|off|N`,默认 `auto`
|
||||
- [x] 1.2 新增 `--progress-interval`(默认 `1s`),并接入 `cli.Config`
|
||||
- [x] 1.3 在 `validate.go` 中校验 `--progress` 取值与 `--progress-interval > 0`,错误走 KindConfig
|
||||
- [x] 1.4 更新 `--help` 文案与 `cli_test.go`,覆盖 `auto`、`on`、`off`、`0`、正整数、非法值
|
||||
|
||||
## 2. internal/progress 模块
|
||||
|
||||
- [x] 2.1 定义 `Reporter` 与内部快照结构,用互斥量保护计数、阶段、估算、位置与消息
|
||||
- [x] 2.2 实现 `Start/Stop` 与 ticker 生命周期,支持注入时钟与 `io.Writer`
|
||||
- [x] 2.3 实现 TTY 判定(`os.File.Stat` 的 `ModeCharDevice`)与 live/line 渲染器选择
|
||||
- [x] 2.4 实现 live 单行渲染:`\r` 覆盖、空格补齐、宽度上限与字段优先级裁剪
|
||||
- [x] 2.5 实现非 TTY 的稳定 key=value 行渲染
|
||||
- [x] 2.6 实现 5 秒窗口速率与 ETA(基于本次运行的推进速率,不使用 elapsed/pct)
|
||||
- [x] 2.7 实现 `Message`:清 live 行、整行输出、下一拍重绘
|
||||
- [x] 2.8 实现 `done` 汇总行与中断时的最终快照行
|
||||
- [x] 2.9 单元测试:live/line 输出、间隔节流、行阈值触发、清行补齐、消息不撕裂、速率窗口、ETA 降级
|
||||
|
||||
## 3. 廉价总量估算
|
||||
|
||||
- [x] 3.1 实现有界估算查询:独立连接、`context.WithTimeout(2s)`、失败/超时静默返回无估算
|
||||
- [x] 3.2 日期模式:`MIN/MAX(shard_column)` 得到天范围,发布 `estimate=day`
|
||||
- [x] 3.3 非日期模式:仅单列整数唯一键时用 `MIN/MAX(key)`,发布 `estimate=key`
|
||||
- [x] 3.4 复合键、字符串/二进制键、无索引时跳过并保持 `estimate=none`
|
||||
- [x] 3.5 单测覆盖估算可用、超时降级、口径不适用;集成测试覆盖日期与主键两种口径
|
||||
|
||||
## 4. 运行循环接入
|
||||
|
||||
- [x] 4.1 在 `execute` 启动 Reporter,在 `finish`/`interrupt` 停止
|
||||
- [x] 4.2 推送阶段:`probe`、`resume`(`Manifest.Adopt`)、`estimate`、`scan`、`file`、`db`、`checkpoint`、`finalize`
|
||||
- [x] 4.3 每个 chunk 完成后发布 RowsRead/committed、分片序号、已写字节与当前键位置
|
||||
- [x] 4.4 把 `lag.OnWait` 改为 `lag-wait` 阶段并携带原因,移除裸 `waiting:` 输出
|
||||
- [x] 4.5 把 `app.go` 中直接写 `os.Stderr` 的警告与提示改为 `Reporter.Message`
|
||||
- [x] 4.6 读取 `state.json` 时记录 `baseRows`,进度显示 `base + 本次 RowsRead`
|
||||
- [x] 4.7 `saveState` 写入 `baseRows + 本次 RowsCommitted`,修复续跑后 `rows_done` 重置
|
||||
- [x] 4.8 `--statistics` 分开展示本次运行与含续跑的累计值
|
||||
- [x] 4.9 中断顺序:停刷新 → 清行 → 最终快照 → `SealCurrent` → `saveState`,退出码保持 130
|
||||
|
||||
## 5. 测试与验证
|
||||
|
||||
- [x] 5.1 更新 `internal/app/integration_test.go` 中从 stdout 抓 `progress:` 的用例,改为 stderr 与新模式
|
||||
- [x] 5.2 新增集成测试:续跑后 `state.json` 的 `rows_done` 为累计值
|
||||
- [x] 5.3 新增集成测试:`--progress off` 与默认非 TTY 时 stderr 无进度行
|
||||
- [ ] 5.4 手动验证 Windows conhost 的 live 刷新、重定向日志的 `--progress on` 输出、SIGINT 快照
|
||||
- [ ] 5.5 运行 `go test ./...` 与 `go vet ./...`,确认无 data race(`go test -race` 覆盖 progress 单测)
|
||||
|
||||
## 6. 文档与收尾
|
||||
|
||||
- [x] 6.1 README 增加进度章节:模式、默认行为、通道、估算口径与限制
|
||||
- [x] 6.2 说明 `scripts/export-all.ps1` 可用 `--progress on` 让日志可 tail,并说明 `-Verify` 的续跑语义变化
|
||||
- [x] 6.3 运行 `openspec validate add-realtime-progress --strict` 并修复所有告警
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-11-06
|
||||
@@ -0,0 +1,3 @@
|
||||
# add-archer-archiver
|
||||
|
||||
Go implementation of pt-archiver with ordered count-based and date-based file sharding, file+dest+purge, dump/csv/jsonl + gzip/zstd
|
||||
@@ -0,0 +1,187 @@
|
||||
## Context
|
||||
|
||||
这是一个全新的 Go 项目,没有既有代码需要兼容。需求见 `proposal.md`,行为契约见 `specs/`。
|
||||
|
||||
关键约束来自目标环境:MySQL/MariaDB、可能是有活跃写入的生产大表、单次导出规模从百万到数十亿行。因此设计围绕三件事展开:**内存恒定**(不能把结果集读进内存)、**扫描不退化**(不能出现 O(n²))、**产物可分段导入**(分片连续有序)。
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- 单趟、流式、内存恒定的扫描与导出。
|
||||
- 分片在扫描顺序上连续、有序、无重叠,文件名反映真实主键范围。
|
||||
- 同一套提交流水线同时支持 file、dest、purge。
|
||||
- 崩溃后可从 checkpoint 续跑,语义明确为 at-least-once。
|
||||
- 无合适索引时快速失败,而不是静默变慢。
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- 并行导出(多 worker)。
|
||||
- 文件侧 exactly-once。
|
||||
- Parquet / Arrow 输出。
|
||||
- 非 MySQL 数据源。
|
||||
- PXC Flow Control 与插件机制。
|
||||
|
||||
## Decisions
|
||||
|
||||
### 决策 1:seek 分块,chunk 即事务
|
||||
|
||||
每个 chunk 用 `key > :last ORDER BY key LIMIT :chunk_size` 向前 seek,不使用 `OFFSET`。`--chunk-size` 同时是 SELECT 批量、事务大小和 checkpoint 粒度,**不引入 `--txn-size`**。
|
||||
|
||||
理由:pt-archiver 把 fetch 和 transaction 解耦后,出现“一个事务跨多个 chunk / 一个 chunk 拆多个事务”的组合复杂度;而 `OFFSET` 在 InnoDB 上是线性扫描,深分页会退化。统一粒度后,续跑、lag 检查、崩溃语义都只有一个边界。
|
||||
|
||||
**备选**:保留 `--txn-size` 做 parity — 放弃,收益不足以抵消复杂度。
|
||||
|
||||
### 决策 2:分片是 file sink 内部的行级路由,不是独立 sink
|
||||
|
||||
```
|
||||
Scanner ──chunk──▶ FileSink ──row──▶ ShardWriter ──▶ .partial ──seal──▶ final + manifest
|
||||
│
|
||||
├──▶ DBSink
|
||||
└──▶ Purger
|
||||
```
|
||||
|
||||
`FileSink.WriteChunk` 逐行交给 `ShardWriter`,由它判断是否封片。因为 `--chunk-size` 与 `--shard-rows` 正交,一个 chunk 可以跨多个分片;这是正常路径,不是特例。
|
||||
|
||||
**关键推论**:三种分片模式都保证扫描顺序等于分片顺序,因此**同一时刻只需要一个打开的分片文件**。不需要 LRU 句柄池,也没有 fd 上限风险。这是选择“有序扫描”换来的最大简化。
|
||||
|
||||
### 决策 3:按数量分片采用流式滚动
|
||||
|
||||
`--shard-by count` 每写满 `--shard-rows` 行封片,分片边界由扫描顺序自然产生。**不预计算边界**。
|
||||
|
||||
理由:预计算需要额外一趟扫描或 `OFFSET` 采样,边界状态还要按分片存进 checkpoint;流式滚动只需一个游标,且边界天然连续。代价是文件名在封片前未知,用 `.partial` + 封片时重命名解决。
|
||||
|
||||
**备选**:预计算边界以支持并行 — 留给 v2。
|
||||
|
||||
### 决策 4:日期分片的双扫描路径
|
||||
|
||||
启动时探测索引,按能力选择:
|
||||
|
||||
| 探测结果 | 路径 | 事务 | purge 连接 |
|
||||
|---|---|---|---|
|
||||
| `(shard_column, pk)` 联合索引 | A:keyset 分块 | 每 chunk 短事务 | 同连接 |
|
||||
| 仅 `shard_column` 单列索引 | B:逐日流式 | 长游标,无 chunk 事务 | 第二连接 |
|
||||
| 都没有 | fail fast(`--allow-full-scan` 可覆盖) | — | — |
|
||||
|
||||
路径 A 的续点条件是行构造器比较:
|
||||
|
||||
```sql
|
||||
WHERE (<base where>)
|
||||
AND (shard_col, pk) > (?, ?)
|
||||
ORDER BY shard_col, pk
|
||||
LIMIT ?
|
||||
```
|
||||
|
||||
路径 B 每天一条查询,用 `rows.Next()` 流式读完:
|
||||
|
||||
```sql
|
||||
WHERE (<base where>)
|
||||
AND shard_col >= ? AND shard_col < ?
|
||||
ORDER BY shard_col, pk
|
||||
```
|
||||
|
||||
选择依据:单列索引的物理顺序已经是 `(shard_col, pk)`,所以路径 B 的**输出有序**;但它的续点无法 seek,只能整日流式读。反过来,路径 A 能用联合索引精确 seek,但要求用户多建一个索引。检测驱动让两种用户都能运行,而不是硬性要求联合索引。
|
||||
|
||||
**NULL 处理**:行构造器比较遇到 NULL 结果恒为 NULL,无法推进游标。因此当存在 `date=_null` 桶时,先单独扫描 `shard_col IS NULL` 的行,再扫描非 NULL 行;两段都保持有序。
|
||||
|
||||
### 决策 5:日期列必须显式指定,类型白名单
|
||||
|
||||
`--shard-by date` 必须给 `--shard-column`,即使表中只有一个日期列。类型仅允许 `DATE`、`DATETIME`、`TIMESTAMP`,以及显式声明单位的整数 epoch。整数列缺单位时采样 min/max,展示三种解释下的日期范围,但**绝不自动选择**。
|
||||
|
||||
理由:同一张表常有多个日期列;`BIGINT` 列可能是 ID、金额或时间戳,秒与毫秒相差 1000 倍且转换后仍落在合法日期区间。静默猜错的代价是整批分片归属错误,且很难事后发现。
|
||||
|
||||
### 决策 6:时区模型显式化
|
||||
|
||||
| 列类型 | 日期归属 |
|
||||
|---|---|
|
||||
| `DATE` / `DATETIME` | 原值,不做换算 |
|
||||
| `TIMESTAMP` | 连接会话 `time_zone` |
|
||||
| epoch 整数 | `--shard-time-zone`,默认 UTC |
|
||||
|
||||
`--shard-time-zone` 与 DSN 的 `time_zone` 都计入配置指纹。续跑时不一致直接拒绝,因为同一行可能落到相邻日期分片,拼出的文件内部不一致。
|
||||
|
||||
### 决策 7:driver 使用 `parseTime=false`
|
||||
|
||||
所有列先按原始字节取回。dump/CSV 需要的就是 MySQL 原始文本;`parseTime=true` 会让 `0000-00-00` 直接报错,也会引入 `time.Time` 往返的格式漂移。分片时只对 `shard_column` 单独解析日期部分,JSONL 的类型化在编码层按列元数据完成。
|
||||
|
||||
### 决策 8:提交顺序与崩溃语义
|
||||
|
||||
```
|
||||
chunk:
|
||||
① FileSink 写行(缓冲)
|
||||
② DBSink 插入
|
||||
③ Purger 删除
|
||||
④ 提交(同实例单事务 / 跨实例分别提交)
|
||||
⑤ flush 文件 → seal 分片 → 追加 manifest
|
||||
⑥ 原子更新 state.json
|
||||
```
|
||||
|
||||
同实例时 ③④ 与 ② 在同一个事务里;跨实例时分别提交并重试。任一步失败都不推进 checkpoint。
|
||||
|
||||
一致性等级为 **at-least-once**:文件写入与数据库提交不在同一个原子域内,崩溃重跑可能重复追加。这是与 pt-archiver 对齐的取舍;文件侧 exactly-once 需要 pending journal,属于 v2。
|
||||
|
||||
**备选**:pending journal — 每 chunk 多一次 fsync,吞吐下降明显,当前需求不要求。
|
||||
|
||||
### 决策 9:删除策略
|
||||
|
||||
默认 IN 列表,按 1000 个参数分批。`--bulk-delete` 改为区间删除,并显式附上 `--where`;仅允许在 `none`/`count` 模式下使用。日期模式下 chunk 的主键不连续(迟到数据会让区间跨日期),区间删除会误伤其他分片的数据,因此直接拒绝。
|
||||
|
||||
**备选**:始终区间删除 — 性能收益在大 chunk 时才明显,但引入并发写入误删风险,不作为默认。
|
||||
|
||||
### 决策 10:checkpoint 与配置指纹
|
||||
|
||||
`state.json` 原子写(tmp + fsync + rename),记录 `version`、`run_id`、`config_hash`、模式、`last_key`、`last_date`、`last_shard_seq`、`rows_done`。
|
||||
|
||||
指纹输入:source 的 host/port/db/user、table、where、columns、shard 参数、时区、format、compress、dest、purge、bulk-delete。**排除密码**,允许凭据轮换。
|
||||
|
||||
### 决策 11:包结构
|
||||
|
||||
```
|
||||
cmd/archer/main.go
|
||||
internal/
|
||||
cli/ 参数解析、校验、互斥规则
|
||||
schema/ 表元数据、索引探测、列类型
|
||||
scan/ seek 分块迭代器(路径 A / B / 主键)
|
||||
row/ 行值、编码器(dump/csv/jsonl)
|
||||
sink/ FileSink、DBSink
|
||||
shard/ ShardWriter、命名、manifest
|
||||
purge/ IN 列表 / 区间删除
|
||||
checkpoint/ state.json
|
||||
lag/ 复制延迟监控
|
||||
```
|
||||
|
||||
核心接口:
|
||||
|
||||
```go
|
||||
type Chunk struct {
|
||||
Rows []Row
|
||||
LastKey Key
|
||||
}
|
||||
|
||||
type Scanner interface {
|
||||
Next(ctx context.Context) (Chunk, error)
|
||||
}
|
||||
|
||||
type Sink interface {
|
||||
WriteChunk(ctx context.Context, c Chunk) error
|
||||
Flush(ctx context.Context) error
|
||||
Close() error
|
||||
}
|
||||
```
|
||||
|
||||
依赖:`github.com/go-sql-driver/mysql`、`github.com/klauspost/compress/zstd`,gzip 用标准库。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **日期模式 B 持有长读快照** → 大表上 undo 可能膨胀,且无法在扫描期间用同连接删除。缓解:路径 A 优先;提供 `--max-lag`;文档说明长事务影响。
|
||||
- **at-least-once 会产生重复行** → 重跑后文件可能重复。缓解:manifest 记录已封分片;文档明确;导入端可用 `INSERT IGNORE` 去重。
|
||||
- **`--bulk-delete` 并发误删** → 扫描期间落在区间内且满足 `--where` 的新行会被删。缓解:默认关闭;启用时打印警告;日期模式禁止。
|
||||
- **分片文件名延迟确定** → 长时间运行的大分片在封片前没有最终文件名。缓解:`state.json` 记录当前 seq 与起始主键;`.partial` 可识别。
|
||||
- **时区配置漂移** → 同一表在不同机器分片结果不同。缓解:时区进入指纹;续跑时不一致即拒绝。
|
||||
- **无索引时误开全表扫描** → 缓解:默认 fail fast,`--allow-full-scan` 必须显式开启。
|
||||
- **单条 INSERT/DELETE 超过 `max_allowed_packet`** → 缓解:按字节预算自动拆分为多条语句。
|
||||
- **并发写入导致扫描期间的行集合漂移** → 与本工具语义一致(只保证读到的行),文档说明不提供快照一致性。
|
||||
|
||||
## Migration Plan
|
||||
|
||||
全新项目,无数据迁移。首个版本只读能力(`--file` / `--dest`)可安全试跑;`--purge` 需显式开启。建议上线流程:先 `--dry-run`,再只导出核对文件,最后再启用 `--purge`。回滚即停止使用新二进制;已产生的导出文件不受影响。
|
||||
@@ -0,0 +1,41 @@
|
||||
## Why
|
||||
|
||||
`pt-archiver` 是归档大表的可靠方案,但它的文件导出只在启动时计算一次文件名,无法把一次归档拆成多个可独立导入的文件;它也不支持按数据自身的日期或行数做有序分片。同时它是 Perl 工具,部署和二次开发成本较高。
|
||||
|
||||
需要一个 Go 原生的归档工具:复用 pt-archiver 的 seek 式分块扫描与 file + dest + purge 语义,并补齐“导出文件可有序分片”这一能力,让大表导出结果能按日期或按行数切成连续、无重叠、可并行导入的文件。
|
||||
|
||||
## What Changes
|
||||
|
||||
- 新增 Go CLI `archer`,对齐 pt-archiver 的核心归档语义:按唯一索引 seek 分块,支持 file、dest 表、purge 三件套。
|
||||
- 新增导出分片能力:
|
||||
- `--shard-by count`:按固定行数流式封片,分片沿扫描顺序连续、有序、主键范围不重叠。
|
||||
- `--shard-by date`:按用户显式指定的日期/时间列分片,每个文件只含一个日期。
|
||||
- `--shard-by none`:不分片,输出单文件。
|
||||
- 新增日期分片的两种扫描路径:存在 `(shard_column, pk)` 联合索引时走 keyset 分块(短事务);仅存在 `shard_column` 单列索引时走逐日流式(长游标 + 第二连接删除)。
|
||||
- 新增输出格式 `dump`(TSV,NULL=`\N`)、`csv`(RFC4180 + 表头)、`jsonl`(类型化 JSON),支持每分片独立 `gzip` 或 `zstd` 压缩。
|
||||
- 新增 `manifest.jsonl` 与 `state.json`:记录分片边界、行数、校验和与断点,支持崩溃续跑;一致性等级为 at-least-once。
|
||||
- 新增安全控制:`--dry-run`、`--max-lag` 复制延迟等待、`--safe-auto-increment`、`--bulk-delete` 的适用性限制、续跑时配置指纹校验。
|
||||
- **BREAKING**: 新项目,无既有接口需要兼容。`--chunk-size` 取代 pt-archiver 的 `--limit`,语义为“每次 SELECT 的行数”;`--limit` 仅作为隐藏别名。
|
||||
|
||||
**非目标(v1 不做)**:PostgreSQL 等非 MySQL 数据源、Parquet 输出、字符串日期列分片、按预计算边界并行导出、文件侧 exactly-once、PXC Flow Control 适配。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `archive-core`: 扫描与归档执行——按唯一索引 seek 分块、file/dest/purge 三件套提交顺序、事务边界、复制延迟等待、安全护栏与断点续跑。
|
||||
- `file-sharding`: 分片路由与产物管理——`none`/`count`/`date` 三种分片模式、有序连续不变量、文件命名、句柄管理、`manifest.jsonl` 与 `state.json`。
|
||||
- `output-formats`: 行编码与压缩——dump/csv/jsonl 的精确格式规则、NULL 与类型处理、每分片独立 gzip/zstd。
|
||||
- `cli-contract`: 命令行契约——参数集合与默认值、互斥与依赖校验、日期列类型与索引探测、错误信息与 `--dry-run` 输出。
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
无。这是新项目,`openspec/specs/` 下没有既有能力。
|
||||
|
||||
## Impact
|
||||
|
||||
- **代码**:新建 Go module 与目录结构(`cmd/archer`、`internal/{cli,schema,scan,row,sink,shard,purge,checkpoint}`)。
|
||||
- **依赖**:`github.com/go-sql-driver/mysql`;`github.com/klauspost/compress`(zstd)。其余用标准库。
|
||||
- **运行环境**:MySQL 5.7+/8.0 与 MariaDB 10.x;不引入服务端组件。
|
||||
- **输出**:新增用户指定目录下的分片文件、`manifest.jsonl`、`state.json`。
|
||||
- **数据安全**:启用 `--purge` 会删除源表数据,需要显式确认;`--bulk-delete` 在日期模式或被拒绝。
|
||||
@@ -0,0 +1,150 @@
|
||||
## Purpose
|
||||
|
||||
定义 archer 如何按唯一索引分块读取源表,并以固定顺序完成文件写入、目标表写入与源行删除,同时提供复制延迟保护、安全护栏和可中断续跑能力。
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 唯一索引 seek 分块扫描
|
||||
|
||||
系统 SHALL 以源表的一个唯一索引(优先主键)作为扫描游标。每个 chunk 的查询 MUST 使用“索引键大于上一次尾值”的条件、按索引列 `ORDER BY`、并带 `LIMIT chunk-size`;系统 MUST NOT 使用 `OFFSET` 翻页。返回行数小于 `chunk-size` 时,系统 SHALL 结束扫描。
|
||||
|
||||
#### Scenario: 主键连续
|
||||
|
||||
- **WHEN** 源表主键为连续的 1..N,且 `--chunk-size 2000`
|
||||
- **THEN** 系统按 2000 行一个 chunk 依次读取,直到最后一个不足 2000 行的 chunk
|
||||
|
||||
#### Scenario: 主键存在空洞
|
||||
|
||||
- **WHEN** 主键序列为 1,2,100,101,且 `--chunk-size 2`
|
||||
- **THEN** 第一个 chunk 返回 1,2,第二个 chunk 从 `id > 2` 继续并返回 100,101,不会跳过或重复
|
||||
|
||||
#### Scenario: 无可用唯一索引
|
||||
|
||||
- **WHEN** 源表既没有主键也没有唯一索引,且用户未指定索引
|
||||
- **THEN** 系统在开始扫描前报错退出,并提示需要唯一索引
|
||||
|
||||
### Requirement: chunk 即事务
|
||||
|
||||
每个 chunk SHALL 构成唯一的事务边界,`--chunk-size` 同时决定 SELECT 批量、事务大小和 checkpoint 粒度。系统 SHALL NOT 提供跨 chunk 的事务,也 SHALL NOT 提供独立的 `--txn-size` 参数。
|
||||
|
||||
#### Scenario: 源表与目标表在同一实例
|
||||
|
||||
- **WHEN** `--source` 与 `--dest` 指向同一 MySQL 实例,且同时启用 dest 与 purge
|
||||
- **THEN** 目标表写入与源行删除在同一个事务中提交,二者要么同时生效要么同时回滚
|
||||
|
||||
#### Scenario: 源表与目标表在不同实例
|
||||
|
||||
- **WHEN** `--source` 与 `--dest` 指向不同实例
|
||||
- **THEN** 系统使用两个连接分别提交,并在任一步失败时按 `--retries` 重试;已提交的目标表写入可能在重跑时重复,行为为 at-least-once
|
||||
|
||||
### Requirement: 三件套提交顺序
|
||||
|
||||
当 file、dest、purge 中任意多项同时启用时,系统 MUST 对每个 chunk 按固定顺序执行:写文件、写目标表、删除源行、提交事务并更新 checkpoint。任何前置步骤失败时,系统 MUST NOT 执行后续步骤,也 MUST NOT 推进 checkpoint。
|
||||
|
||||
#### Scenario: 文件写入失败
|
||||
|
||||
- **WHEN** 写文件返回错误
|
||||
- **THEN** 系统不写目标表、不删除源行、不推进 checkpoint,并以非零码退出
|
||||
|
||||
#### Scenario: 目标表写入失败
|
||||
|
||||
- **WHEN** 文件写入成功但目标表插入失败
|
||||
- **THEN** 系统删除源行,回滚本次事务,不推进 checkpoint,并以非零码退出
|
||||
|
||||
### Requirement: 输出目标约束
|
||||
|
||||
系统 SHALL 要求 `--file`、`--dest`、`--purge` 至少启用一个,并且 `--where` 必填。三者可以任意组合,包括只导出文件、只删除、或文件与目标表同时输出。
|
||||
|
||||
#### Scenario: 只删不写
|
||||
|
||||
- **WHEN** 用户只指定 `--purge` 和 `--where`
|
||||
- **THEN** 系统仅按 chunk 删除匹配行,不生成任何文件
|
||||
|
||||
#### Scenario: 文件与目标表同时输出
|
||||
|
||||
- **WHEN** 用户同时指定 `--file` 与 `--dest`
|
||||
- **THEN** 同一批行既写入分片文件,也写入目标表
|
||||
|
||||
### Requirement: 目标表写入模式
|
||||
|
||||
目标表写入 SHALL 支持普通插入、`INSERT IGNORE` 与 `REPLACE` 三种模式,三者互斥。系统 MUST 使用多行批量插入,并在单条语句可能超过 `max_allowed_packet` 时自动拆分为多条语句。
|
||||
|
||||
#### Scenario: 默认插入遇到重复主键
|
||||
|
||||
- **WHEN** 使用默认插入模式,且目标表已存在相同主键
|
||||
- **THEN** 该 chunk 失败并回滚,系统报告重复键错误
|
||||
|
||||
#### Scenario: ignore 模式遇到重复主键
|
||||
|
||||
- **WHEN** 使用 `--insert-ignore`
|
||||
- **THEN** 重复行被跳过,其余行写入成功,chunk 正常提交
|
||||
|
||||
### Requirement: 删除语义
|
||||
|
||||
默认情况下,系统 MUST 以主键 IN 列表删除当前 chunk 实际读取到的行,并将过长的列表分批。启用 `--bulk-delete` 时,系统 SHALL 改为按首尾主键区间删除,并 MUST 在语句中附带原始 `--where` 条件。`--bulk-delete` 依赖 `--purge`,且在日期分片模式下 MUST 被拒绝。
|
||||
|
||||
#### Scenario: 默认 IN 列表删除
|
||||
|
||||
- **WHEN** 一个 chunk 读取到主键 1001..3000,且 `--where "created_at < '2026-01-01'"`
|
||||
- **THEN** 系统生成 `DELETE ... WHERE id IN (1001,...,3000)`,不删除未被读取的行
|
||||
|
||||
#### Scenario: 区间删除附带回填条件
|
||||
|
||||
- **WHEN** 启用 `--bulk-delete`,chunk 首尾主键为 1001 与 3000
|
||||
- **THEN** 系统生成 `DELETE ... WHERE id BETWEEN 1001 AND 3000 AND (<where>)`
|
||||
|
||||
#### Scenario: 日期模式启用区间删除
|
||||
|
||||
- **WHEN** `--shard-by date` 与 `--bulk-delete` 同时出现
|
||||
- **THEN** 系统在启动校验阶段报错退出,不执行任何查询
|
||||
|
||||
### Requirement: 复制延迟保护
|
||||
|
||||
当指定 `--check-replica-lag` 时,系统 SHALL 在每次拉取前检查该副本的延迟;延迟超过 `--max-lag` 或复制未运行时,系统 MUST 暂停并在 `--check-interval` 后重试。
|
||||
|
||||
#### Scenario: 延迟超限
|
||||
|
||||
- **WHEN** `--max-lag 1s`,副本延迟为 5s
|
||||
- **THEN** 系统暂停拉取,直到延迟降到 1s 以下
|
||||
|
||||
#### Scenario: 副本复制中断
|
||||
|
||||
- **WHEN** 被检查的副本复制线程未运行
|
||||
- **THEN** 系统持续等待并周期性重试,不继续归档
|
||||
|
||||
### Requirement: 安全护栏
|
||||
|
||||
系统 SHALL 默认启用 `--safe-auto-increment`,禁止通过 purge 删除当前最大自增主键行。`--dry-run` 时系统 MUST 只打印将要执行的 SQL 与实际生效配置,不连接写入、不创建或修改任何文件。
|
||||
|
||||
#### Scenario: 保护最大自增行
|
||||
|
||||
- **WHEN** purge 范围包含源表当前 `MAX(pk)`,且 `--safe-auto-increment` 为默认值
|
||||
- **THEN** 该行不被删除,系统记录一条警告
|
||||
|
||||
#### Scenario: dry-run 无副作用
|
||||
|
||||
- **WHEN** 用户指定 `--dry-run`
|
||||
- **THEN** 系统打印每条 SQL 与解析后的配置后退出,输出目录不产生任何文件
|
||||
|
||||
### Requirement: 断点续跑
|
||||
|
||||
系统 SHALL 在每个 chunk 提交后原子更新 `state.json`,记录配置指纹与游标位置。续跑时 MUST 校验源、表、过滤条件、列、分片参数与时区的指纹;不一致时 MUST 拒绝续跑,除非显式传入 `--force-resume`。
|
||||
|
||||
#### Scenario: 配置一致时续跑
|
||||
|
||||
- **WHEN** 上次运行在 chunk 中途退出,且本次配置指纹一致
|
||||
- **THEN** 系统从 `state.json` 记录的游标位置继续,不重复处理已提交的 chunk
|
||||
|
||||
#### Scenario: 配置被修改
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出不一致的字段,拒绝开始扫描
|
||||
|
||||
### Requirement: 一致性等级
|
||||
|
||||
文件输出的一致性等级 SHALL 为 at-least-once:崩溃后重跑可能向分片文件重复追加已处理的行,系统 MUST 在文档与启动输出中明确该行为。
|
||||
|
||||
#### Scenario: 崩溃后重跑
|
||||
|
||||
- **WHEN** 文件已写入但 checkpoint 尚未更新时进程崩溃,随后以相同配置重跑
|
||||
- **THEN** 系统从上一个已提交游标继续,文件可能出现重复行,数据不丢失
|
||||
@@ -0,0 +1,98 @@
|
||||
## Purpose
|
||||
|
||||
定义 archer 的命令行参数、默认值、参数之间的依赖与互斥规则,以及启动校验、错误信息和 dry-run 行为。
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 必填参数
|
||||
|
||||
系统 MUST 要求提供 `--source` 与 `--where`,并且 `--file`、`--dest`、`--purge` 至少启用一个。缺少任一前置条件时,系统 MUST 在连接数据库前报错退出。
|
||||
|
||||
#### Scenario: 缺少过滤条件
|
||||
|
||||
- **WHEN** 用户未提供 `--where`
|
||||
- **THEN** 系统立即报错,提示 `--where` 必填,不建立数据库连接
|
||||
|
||||
#### Scenario: 没有任何输出目标
|
||||
|
||||
- **WHEN** 用户既未提供 `--file`、`--dest`,也未启用 `--purge`
|
||||
- **THEN** 系统报错,提示至少需要一种输出目标
|
||||
|
||||
### Requirement: 默认值
|
||||
|
||||
系统 SHALL 采用以下默认值:`--chunk-size 2000`、`--shard-by none`、`--shard-rows 0`、`--format dump`、`--compress none`、`--max-lag 1s`、`--check-interval 1s`、`--retries 1`、`--safe-auto-increment true`、`--progress 0`。
|
||||
|
||||
#### Scenario: 最小可运行配置
|
||||
|
||||
- **WHEN** 用户只提供 `--source`、`--table`、`--where`、`--file`
|
||||
- **THEN** 系统以 2000 行一个 chunk、不分片、dump 格式、不压缩的方式导出
|
||||
|
||||
### Requirement: 参数依赖与互斥
|
||||
|
||||
系统 MUST 校验以下规则:`--shard-by date` 必须有 `--shard-column`;`--shard-by count|none` 不得带 `--shard-column`;`--shard-rows` 仅在 `--shard-by count` 或 `date` 时有效;`--bulk-delete` 依赖 `--purge` 且在日期模式下禁止;普通插入、`--insert-ignore`、`--replace` 三者互斥。违反时 MUST 在启动阶段报错。
|
||||
|
||||
#### Scenario: 数量模式误传日期列
|
||||
|
||||
- **WHEN** `--shard-by count` 同时带上 `--shard-column`
|
||||
- **THEN** 系统报错,说明该参数仅在日期模式下有效
|
||||
|
||||
#### Scenario: 未启用 purge 却要求 bulk-delete
|
||||
|
||||
- **WHEN** 用户指定 `--bulk-delete` 但未指定 `--purge`
|
||||
- **THEN** 系统报错,说明 bulk-delete 必须配合 purge
|
||||
|
||||
#### Scenario: 插入模式互斥
|
||||
|
||||
- **WHEN** 用户同时指定 `--insert-ignore` 与 `--replace`
|
||||
- **THEN** 系统报错,要求只能选择一种插入模式
|
||||
|
||||
### Requirement: 日期列探测与错误提示
|
||||
|
||||
启动时,系统 SHALL 读取表元数据并判断 `--shard-column` 是否存在、类型是否受支持、是否可为 NULL。缺少 `--shard-column` 时 MUST 列出所有候选日期/时间列及其类型。整数列缺少单位声明时 MUST 采样最小/最大值,并展示按秒、毫秒、微秒解释后的日期范围。
|
||||
|
||||
#### Scenario: 列出候选列
|
||||
|
||||
- **WHEN** 用户指定 `--shard-by date` 但未给 `--shard-column`
|
||||
- **THEN** 错误信息包含表内所有 DATE/DATETIME/TIMESTAMP 列的列名、类型与可空性
|
||||
|
||||
#### Scenario: epoch 单位提示
|
||||
|
||||
- **WHEN** 分片列为 BIGINT 且未声明单位
|
||||
- **THEN** 错误信息展示采样范围以及三种单位解释下的日期范围,不自动选择
|
||||
|
||||
### Requirement: dry-run
|
||||
|
||||
`--dry-run` 时,系统 MUST 打印解析后的配置、索引探测结果、最终生效的分片与格式参数,以及第一条将要执行的 SELECT 和每类写入语句,然后退出。系统 MUST NOT 建立写入连接、创建目录或写文件。
|
||||
|
||||
#### Scenario: dry-run 输出 SQL
|
||||
|
||||
- **WHEN** 用户传入 `--dry-run`
|
||||
- **THEN** 标准输出包含解析后的配置和将要执行的 SQL,进程以 0 退出
|
||||
|
||||
### Requirement: 进度与统计
|
||||
|
||||
当 `--progress N` 大于 0 时,系统 SHALL 每处理 N 行输出一次进度,包含已处理行数、当前分片序号与耗时。启用 `--statistics` 时,系统 MUST 在结束时输出读取行数、写入行数、删除行数、分片数与各阶段耗时。
|
||||
|
||||
#### Scenario: 周期进度
|
||||
|
||||
- **WHEN** `--progress 100000`
|
||||
- **THEN** 每处理 100000 行打印一行进度信息
|
||||
|
||||
#### Scenario: 结束统计
|
||||
|
||||
- **WHEN** 启用 `--statistics`
|
||||
- **THEN** 进程退出前输出汇总统计
|
||||
|
||||
### Requirement: 退出码
|
||||
|
||||
系统 SHALL 在成功完成时返回 0;配置错误、连接错误、写入错误或校验失败时返回非零退出码。被信号中断时 MUST 完成当前 chunk 的清理并返回非零退出码。
|
||||
|
||||
#### Scenario: 配置错误
|
||||
|
||||
- **WHEN** 参数校验失败
|
||||
- **THEN** 进程以非零退出码结束,且不产生部分输出文件
|
||||
|
||||
#### Scenario: 信号中断
|
||||
|
||||
- **WHEN** 进程收到 SIGINT
|
||||
- **THEN** 系统停止拉取新 chunk,关闭并 flush 当前分片,写入 checkpoint,然后以非零退出码结束
|
||||
@@ -0,0 +1,151 @@
|
||||
## Purpose
|
||||
|
||||
定义导出文件的分片模式、顺序不变量、命名规则、句柄管理与分片清单,确保大规模导出的结果可被分段、有序、独立地导入。
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 分片模式
|
||||
|
||||
系统 SHALL 支持三种分片模式:`none`(不分片,单文件)、`count`(按固定行数封片)、`date`(按用户指定的日期或时间列封片)。分片模式 SHALL 由 `--shard-by` 选择,默认 `none`。
|
||||
|
||||
#### Scenario: 不分片
|
||||
|
||||
- **WHEN** `--shard-by none`
|
||||
- **THEN** 本次扫描的全部行写入同一个文件
|
||||
|
||||
#### Scenario: 按数量分片
|
||||
|
||||
- **WHEN** `--shard-by count --shard-rows 5000000`
|
||||
- **THEN** 每写满 500 万行封一个文件,最后一个文件可少于 500 万行
|
||||
|
||||
#### Scenario: 按日期分片
|
||||
|
||||
- **WHEN** `--shard-by date --shard-column created_at`
|
||||
- **THEN** 每个文件只包含 `created_at` 落在同一个日期的行
|
||||
|
||||
### Requirement: 有序连续不变量
|
||||
|
||||
对于任何分片模式,系统 MUST 保证:分片按扫描顺序编号;相邻分片的主键范围连续且不重叠;所有分片合并后等于本次扫描读取的全部行;每个分片内部的行顺序与扫描顺序一致;文件名中的首尾主键为该分片实际写入的第一行与最后一行主键。
|
||||
|
||||
#### Scenario: 分片边界连续
|
||||
|
||||
- **WHEN** 一次扫描读取主键 1..2500,且 `--shard-rows 1000`
|
||||
- **THEN** 生成三个分片,主键范围分别为 1-1000、1001-2000、2001-2500,无重叠也无空洞
|
||||
|
||||
#### Scenario: 主键存在空洞时仍连续
|
||||
|
||||
- **WHEN** 扫描读取到主键 1,2,100,101,且 `--shard-rows 2`
|
||||
- **THEN** 分片范围分别为 1-2 与 100-101,文件名反映实际首尾主键
|
||||
|
||||
### Requirement: 按行数流式封片
|
||||
|
||||
在 `count` 模式下,系统 SHALL 在写入过程中逐行判断当前分片是否已满,写满 `--shard-rows` 行后立即封片。`--chunk-size` 与 `--shard-rows` SHALL 相互独立,系统 MUST 支持一个 chunk 跨越多个分片,且不要求两者存在整除关系。
|
||||
|
||||
#### Scenario: chunk 跨越分片边界
|
||||
|
||||
- **WHEN** `--chunk-size 2000` 且 `--shard-rows 2500`
|
||||
- **THEN** 第二个 chunk 的前 1500 行写入第一个分片,后 500 行写入第二个分片
|
||||
|
||||
#### Scenario: 文件在关闭时才确定名称
|
||||
|
||||
- **WHEN** 一个分片正在写入
|
||||
- **THEN** 该文件在关闭前以 `.partial` 形式存在,关闭并 flush 后重命名为包含实际首尾主键的最终文件名
|
||||
|
||||
### Requirement: 日期分片的列契约
|
||||
|
||||
`--shard-by date` 时,系统 MUST 要求显式提供 `--shard-column`;即使表中只有一个日期列也不得自动选择。该列类型 MUST 属于 `DATE`、`DATETIME`、`TIMESTAMP` 或整数类型(epoch,需显式声明单位)。其他类型 MUST 被拒绝。
|
||||
|
||||
#### Scenario: 缺少分片列
|
||||
|
||||
- **WHEN** `--shard-by date` 但未提供 `--shard-column`
|
||||
- **THEN** 系统报错,并列出表中所有候选的日期/时间列及其类型与可空性
|
||||
|
||||
#### Scenario: 多个日期列
|
||||
|
||||
- **WHEN** 表中有 `created_at`、`updated_at`、`paid_at` 三个日期列,用户指定 `--shard-column created_at`
|
||||
- **THEN** 系统只按 `created_at` 分片,其他日期列作为普通导出列
|
||||
|
||||
#### Scenario: 整数时间戳列未声明单位
|
||||
|
||||
- **WHEN** `--shard-column created_at` 指向 `BIGINT` 列,且未提供 `--shard-column-type`
|
||||
- **THEN** 系统报错,采样该列最小/最大值,并分别展示按秒、毫秒、微秒解释后的日期范围供用户选择
|
||||
|
||||
#### Scenario: 不支持的列类型
|
||||
|
||||
- **WHEN** `--shard-column` 指向 `VARCHAR` 或 `TIME` 列
|
||||
- **THEN** 系统报错,并说明 v1 仅支持日期时间类型与显式声明的 epoch 整数
|
||||
|
||||
### Requirement: 日期值归属规则
|
||||
|
||||
系统 MUST 按列语义决定日期归属:`DATE` 与 `DATETIME` 使用原值,不做时区换算;`TIMESTAMP` 按连接会话时区换算;epoch 整数按 `--shard-time-zone`(默认 UTC)换算。NULL 值默认落入 `date=_null` 分片;零日期默认报错并给出行主键,可通过 `--zero-date bucket` 改为落入 `date=0000-00-00` 分片。`--shard-time-zone` MUST 计入配置指纹。
|
||||
|
||||
#### Scenario: NULL 日期值
|
||||
|
||||
- **WHEN** 某行的 `created_at` 为 NULL,且使用默认配置
|
||||
- **THEN** 该行写入 `date=_null` 分片,扫描顺序与分片顺序仍保持一致
|
||||
|
||||
#### Scenario: 零日期值
|
||||
|
||||
- **WHEN** 某行的 `created_at` 为 `0000-00-00 00:00:00`,且未指定 `--zero-date`
|
||||
- **THEN** 系统报错并输出该行主键,不生成静默的错误分片
|
||||
|
||||
#### Scenario: 时区改变后续跑
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致
|
||||
- **THEN** 系统拒绝续跑,除非显式传入 `--force-resume`
|
||||
|
||||
### Requirement: 日期扫描路径选择
|
||||
|
||||
系统 SHALL 在启动时探测分片列的索引,并据此选择扫描路径:存在 `(shard_column, pk)` 联合索引时使用 keyset 分块(短事务);仅存在 `shard_column` 单列索引时使用逐日流式读取(长游标,purge 使用独立连接);两者都不存在时 MUST fail fast,除非显式传入 `--allow-full-scan`。
|
||||
|
||||
#### Scenario: 存在联合索引
|
||||
|
||||
- **WHEN** 表上存在 `(created_at, id)` 联合索引
|
||||
- **THEN** 系统按 `(created_at, id)` 做 keyset 分块,每个 chunk 独立提交
|
||||
|
||||
#### Scenario: 仅存在单列索引
|
||||
|
||||
- **WHEN** 表上只有 `created_at` 单列索引
|
||||
- **THEN** 系统按日期逐日发出查询并流式读取,日期内按 `created_at, id` 排序;删除操作使用第二条源连接
|
||||
|
||||
#### Scenario: 没有任何相关索引
|
||||
|
||||
- **WHEN** 分片列上既无联合索引也无单列索引
|
||||
- **THEN** 系统报错并提示创建索引,或由用户显式传入 `--allow-full-scan` 以接受全表扫描
|
||||
|
||||
### Requirement: 文件命名与目录布局
|
||||
|
||||
分片文件 MUST 按 `<输出目录>/[date=<YYYY-MM-DD>/]<seq>.<pk_lo>-<pk_hi>.<format>[.<compress>]` 命名,`seq` 为从 1 开始的六位零填充序号。日期模式 MUST 按日期建立子目录。输出目录下同时生成 `manifest.jsonl` 与 `state.json`。
|
||||
|
||||
#### Scenario: 数量分片命名
|
||||
|
||||
- **WHEN** 第一个分片主键范围为 1..5000000,格式为 dump,压缩为 zstd
|
||||
- **THEN** 文件名为 `000001.1-5000000.dump.zst`
|
||||
|
||||
#### Scenario: 日期分片布局
|
||||
|
||||
- **WHEN** `--shard-by date` 导出的首个日期为 2025-06-01
|
||||
- **THEN** 文件位于 `date=2025-06-01/` 子目录下,序号在该日期内从 000001 开始
|
||||
|
||||
### Requirement: 分片清单
|
||||
|
||||
系统 SHALL 在每次分片关闭后向 `manifest.jsonl` 追加一行并 flush,记录序号、相对路径、日期(如适用)、首尾主键、行数、字节数与 SHA-256。清单 MUST 在进程崩溃后仍可用于识别已完成的分片。
|
||||
|
||||
#### Scenario: 分片关闭后写入清单
|
||||
|
||||
- **WHEN** 一个分片完成 flush 并重命名
|
||||
- **THEN** `manifest.jsonl` 追加一条包含 seq、path、pk_lo、pk_hi、rows、bytes、sha256、closed_at 的记录
|
||||
|
||||
#### Scenario: 崩溃后识别已完成分片
|
||||
|
||||
- **WHEN** 进程在写完两个分片后崩溃
|
||||
- **THEN** `manifest.jsonl` 中保留这两条完整记录,残留的 `.partial` 文件不进入清单
|
||||
|
||||
### Requirement: 分片文件原子性
|
||||
|
||||
每个分片 MUST 先写入同目录下的 `.partial` 临时文件,成功 flush 后原子重命名为最终文件名。每个分片 MUST 独立压缩,使单个文件可以脱离其他分片单独导入。
|
||||
|
||||
#### Scenario: 崩溃留下临时文件
|
||||
|
||||
- **WHEN** 进程在分片写入过程中崩溃
|
||||
- **THEN** 目录中留下 `.partial` 文件,最终文件名对应的文件不存在,续跑时系统忽略或清理该临时文件
|
||||
@@ -0,0 +1,75 @@
|
||||
## Purpose
|
||||
|
||||
定义行数据到文件的编码规则与压缩方式,确保导出产物可以被 MySQL、PostgreSQL、ClickHouse、Spark 等常见工具直接导入。
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: dump 格式
|
||||
|
||||
`--format dump` 时,系统 MUST 以制表符分隔字段、换行符结束行、`\N` 表示 NULL、反斜杠转义特殊字符,并且不写表头。转义规则 MUST 与 MySQL `SELECT ... INTO OUTFILE` 一致:反斜杠、制表符、换行、回车、NUL、退格、Ctrl-Z 均需转义。
|
||||
|
||||
#### Scenario: 默认 dump 输出
|
||||
|
||||
- **WHEN** 一行包含 `a\tb` 与 NULL
|
||||
- **THEN** 输出为 `a\\tb\t\\N\n`,可被 `LOAD DATA INFILE` 以默认参数读回
|
||||
|
||||
#### Scenario: 数值与日期列
|
||||
|
||||
- **WHEN** 列类型为 DECIMAL、DATETIME
|
||||
- **THEN** 直接输出 MySQL 返回的文本表示,不做重格式化
|
||||
|
||||
### Requirement: CSV 格式
|
||||
|
||||
`--format csv` 时,系统 MUST 输出 RFC4180 兼容内容:逗号分隔、首行为列名表头、含逗号/引号/CR/LF 的字段用双引号包裹、字段内双引号写为两个双引号、行尾为 CRLF。NULL 默认输出 `\N`,可通过 `--csv-null` 覆盖;空字符串输出为空字段。
|
||||
|
||||
#### Scenario: 字段包含逗号与引号
|
||||
|
||||
- **WHEN** 字段值为 `a,"b"`
|
||||
- **THEN** 输出为 `"a,""b"""`
|
||||
|
||||
#### Scenario: 表头
|
||||
|
||||
- **WHEN** 生成任意 CSV 分片
|
||||
- **THEN** 每个文件首行都是列名列表,顺序与 `--columns` 一致
|
||||
|
||||
### Requirement: JSONL 格式
|
||||
|
||||
`--format jsonl` 时,系统 MUST 每行输出一个 JSON 对象,键为列名。类型映射 MUST 为:DATETIME/TIMESTAMP 输出 RFC3339 字符串;DECIMAL 输出字符串以保留精度;BLOB/BINARY 输出 base64 字符串;NULL 输出 JSON `null`;整数与浮点输出 JSON 数字。
|
||||
|
||||
#### Scenario: 类型映射
|
||||
|
||||
- **WHEN** 一行包含 DECIMAL(10,2)=12.30、DATETIME、NULL
|
||||
- **THEN** 对应字段分别为 `"12.30"`、RFC3339 时间字符串、`null`
|
||||
|
||||
#### Scenario: 每行独立可解析
|
||||
|
||||
- **WHEN** 读取 JSONL 分片的任意一行
|
||||
- **THEN** 该行可以被标准 JSON 解析器独立解析,不依赖前后行
|
||||
|
||||
### Requirement: 压缩
|
||||
|
||||
系统 SHALL 支持 `--compress none|gzip|zstd`,默认 `none`。每个分片 MUST 独立压缩,压缩后的文件扩展名分别为 `.gz` 与 `.zst`。gzip 与 zstd 的默认压缩级别 MUST 为 3(吞吐优先)。
|
||||
|
||||
#### Scenario: gzip 分片
|
||||
|
||||
- **WHEN** `--compress gzip`
|
||||
- **THEN** 每个分片是一个独立的 gzip 流,文件名为 `*.gz`
|
||||
|
||||
#### Scenario: zstd 分片
|
||||
|
||||
- **WHEN** `--compress zstd`
|
||||
- **THEN** 每个分片是一个独立的 zstd 帧,文件名为 `*.zst`
|
||||
|
||||
### Requirement: 可导入性
|
||||
|
||||
系统 MUST 保证每种格式的产物可被至少一种常见工具直接导入:dump 可被 MySQL `LOAD DATA INFILE` 默认参数导入;csv 可被 PostgreSQL `COPY ... CSV HEADER`、ClickHouse `CSVWithNames`、Pandas/Spark 默认 CSV 读取器导入;jsonl 可被 ClickHouse `JSONEachRow` 及常见流式 JSON 工具导入。
|
||||
|
||||
#### Scenario: 压缩文件导入
|
||||
|
||||
- **WHEN** 产物为 `000001.1-1000.csv.gz`
|
||||
- **THEN** 可以通过 `gunzip -c` 或 `zcat` 管道交给目标数据库导入,无需改写文件内容
|
||||
|
||||
#### Scenario: dump 直接导入 MySQL
|
||||
|
||||
- **WHEN** 使用 dump 格式且不压缩
|
||||
- **THEN** `LOAD DATA INFILE '000001.1-1000.dump' INTO TABLE t` 在列顺序一致时可成功执行
|
||||
@@ -0,0 +1,96 @@
|
||||
## 1. 项目骨架
|
||||
|
||||
- [x] 1.1 初始化 Go module,建立 `cmd/archer` 与 `internal/{cli,schema,scan,row,sink,shard,purge,checkpoint,lag}` 目录
|
||||
- [x] 1.2 引入 `go-sql-driver/mysql` 与 `klauspost/compress/zstd`,锁定版本
|
||||
- [x] 1.3 定义 `Chunk`、`Scanner`、`Sink` 接口与共享的 `Row`、`Key` 类型
|
||||
- [x] 1.4 建立最小 `main.go` 与版本号输出
|
||||
|
||||
## 2. CLI 与配置校验
|
||||
|
||||
- [x] 2.1 实现全部参数解析与默认值(`--chunk-size 2000`、`--shard-by none`、`--format dump`、`--compress none` 等)
|
||||
- [x] 2.2 实现必填校验:`--source`、`--where`、至少一个输出目标
|
||||
- [x] 2.3 实现依赖与互斥校验:date 必须带 `--shard-column`;count/none 禁止;`--bulk-delete` 依赖 `--purge` 且日期模式禁止;insert/ignore/replace 互斥
|
||||
- [x] 2.4 实现配置标准化与 `config_hash` 计算(排除密码)
|
||||
- [x] 2.5 实现 `--dry-run` 输出:解析后配置、索引探测结果、首条 SELECT 与写入 SQL
|
||||
|
||||
## 3. Schema 元数据与索引探测
|
||||
|
||||
- [x] 3.1 从 `information_schema` 读取列名、类型、可空性、列顺序
|
||||
- [x] 3.2 实现唯一索引发现,优先主键,支持用户强制指定索引
|
||||
- [x] 3.3 实现分片列索引探测:联合索引 / 单列索引 / 无索引三种结果
|
||||
- [x] 3.4 实现日期列候选枚举与错误提示
|
||||
- [x] 3.5 实现整数 epoch 列的 min/max 采样与三种单位解释提示
|
||||
|
||||
## 4. 扫描器
|
||||
|
||||
- [x] 4.1 实现主键 seek 分块扫描(`key > :last ORDER BY key LIMIT :chunk_size`)
|
||||
- [x] 4.2 实现主键空洞与尾 chunk 的处理,返回准确 `LastKey`
|
||||
- [x] 4.3 实现日期路径 A:`(shard_col, pk)` 行构造器 keyset 分块
|
||||
- [x] 4.4 实现日期路径 B:逐日枚举 + 长游标流式读取
|
||||
- [x] 4.5 实现 NULL 日期先扫、非 NULL 后扫的两段扫描
|
||||
- [x] 4.6 接入 `--max-lag` 拉取前检查与 `--check-interval` 等待
|
||||
|
||||
## 5. 行编码与压缩
|
||||
|
||||
- [x] 5.1 实现 dump 编码器(TAB、`\N`、与 MySQL 一致的转义表)
|
||||
- [x] 5.2 实现 CSV 编码器(RFC4180、表头、引号规则、`--csv-null`)
|
||||
- [x] 5.3 实现 JSONL 编码器(时间、DECIMAL、BLOB 的类型映射)
|
||||
- [x] 5.4 实现每分片独立的 gzip 与 zstd 压缩包装器
|
||||
- [x] 5.5 用固定样例验证三种格式可被对应工具直接导入
|
||||
|
||||
## 6. 分片写入与清单
|
||||
|
||||
- [x] 6.1 实现 `ShardWriter`:单活跃分片、逐行写入、按 `--shard-rows` 封片
|
||||
- [x] 6.2 实现 `.partial` 写入与封片时的 fsync + 原子重命名
|
||||
- [x] 6.3 实现文件名规则:`[date=<d>/]<seq>.<lo>-<hi>.<format>[.<comp>]`
|
||||
- [x] 6.4 实现日期分片的目录布局与日期切换时封片
|
||||
- [x] 6.5 实现 `manifest.jsonl` 追加与 flush(seq/path/date/pk_lo/pk_hi/rows/bytes/sha256)
|
||||
- [x] 6.6 实现崩溃后残留 `.partial` 的识别与清理
|
||||
|
||||
## 7. 目标表写入
|
||||
|
||||
- [x] 7.1 实现多行批量 INSERT,按 `max_allowed_packet` 自动拆分
|
||||
- [x] 7.2 实现 `--insert-ignore` 与 `--replace`
|
||||
- [x] 7.3 实现同实例单事务提交
|
||||
- [x] 7.4 实现跨实例分别提交与 `--retries` 重试
|
||||
|
||||
## 8. 删除
|
||||
|
||||
- [x] 8.1 实现默认 IN 列表删除与 1000 参数分批
|
||||
- [x] 8.2 实现 `--bulk-delete` 区间删除并附带 `--where`
|
||||
- [x] 8.3 实现 `--safe-auto-increment` 保护最大主键行
|
||||
- [x] 8.4 实现日期路径 B 的独立 purge 连接
|
||||
- [x] 8.5 实现日期模式拒绝 `--bulk-delete` 的启动校验
|
||||
|
||||
## 9. Checkpoint 与续跑
|
||||
|
||||
- [x] 9.1 实现 `state.json` 原子写(tmp + fsync + rename)
|
||||
- [x] 9.2 记录并校验 `config_hash`、游标、当前分片序号、已处理行数
|
||||
- [x] 9.3 实现续跑:从游标恢复、跳过已封分片、清理未完成临时文件
|
||||
- [x] 9.4 实现配置不一致时的拒绝与 `--force-resume` 覆盖
|
||||
- [x] 9.5 实现信号中断时 flush 当前分片并写 checkpoint
|
||||
|
||||
## 10. 进度、统计与错误
|
||||
|
||||
- [x] 10.1 实现 `--progress N` 周期进度输出
|
||||
- [x] 10.2 实现 `--statistics` 结束汇总
|
||||
- [x] 10.3 统一退出码与错误分类(配置、连接、写入、校验、中断)
|
||||
|
||||
## 11. 集成测试与验收
|
||||
|
||||
- [x] 11.1 搭建 MySQL 测试环境与固定数据夹具
|
||||
- [x] 11.2 验证分片有序连续不变量:范围无重叠、合并等于全集、文件名首尾主键正确
|
||||
- [x] 11.3 验证 chunk 跨分片边界(`chunk-size` 与 `shard-rows` 非整除)
|
||||
- [x] 11.4 验证日期路径 A 与路径 B 的输出一致
|
||||
- [x] 11.5 验证 NULL、零日期、epoch 秒/毫秒/微秒的分片归属
|
||||
- [x] 11.6 验证崩溃续跑不丢数据、manifest 与 state 一致
|
||||
- [x] 11.7 验证 dump/csv/jsonl 分别可被 MySQL、Postgres/ClickHouse 导入
|
||||
- [x] 11.8 验证 gzip/zstd 分片可独立解压导入
|
||||
- [x] 11.9 验证 `--dry-run`、互斥校验、无索引 fail-fast 的错误信息
|
||||
|
||||
## 12. 文档与发布
|
||||
|
||||
- [x] 12.1 编写 README:安装、最小示例、分片模式、导入方法
|
||||
- [x] 12.2 编写安全说明:`--purge`、`--bulk-delete`、at-least-once、时区
|
||||
- [x] 12.3 编写与 pt-archiver 的参数对照表
|
||||
- [x] 12.4 打包 Windows/Linux 二进制与版本信息
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-11-06
|
||||
@@ -0,0 +1,3 @@
|
||||
# add-compress-level-and-xz
|
||||
|
||||
新增 xz 压缩格式,并引入统一 1-9 压缩档位 --compress-level
|
||||
@@ -0,0 +1,85 @@
|
||||
# 设计:xz 压缩与统一压缩档位
|
||||
|
||||
## 依赖选择
|
||||
|
||||
采用 `github.com/ulikunitz/xz`(纯 Go、BSD-3、读写齐全)。
|
||||
|
||||
| 方案 | 结论 |
|
||||
|---|---|
|
||||
| `klauspost/compress` | 不支持 xz |
|
||||
| cgo 绑定 liblzma | 破坏交叉编译与"纯 Go 构建"约束 |
|
||||
| 外部 `xz` 进程 | 额外运行时依赖、错误处理与安全面变大 |
|
||||
| `ulikunitz/xz` | 唯一可行,本机 module cache 已有 v0.5.17 |
|
||||
|
||||
## 档位模型:1–9 是语义档位,不是原生级别
|
||||
|
||||
zstd 库没有连续 1–22 级别,只有 4 个 `EncoderLevel`;xz 库没有 preset 参数。因此 1–9 被定义为**跨格式语义档位**,各格式显式映射,9 表示该格式最强:
|
||||
|
||||
| level | gzip (flate) | zstd | xz 字典 |
|
||||
|---|---|---|---|
|
||||
| 1 | 1 | SpeedFastest | 1 MiB |
|
||||
| 2 | 2 | SpeedFastest | 2 MiB |
|
||||
| 3 | 3 | SpeedDefault | 4 MiB |
|
||||
| 4 | 4 | SpeedDefault | 4 MiB |
|
||||
| 5 | 5 | SpeedDefault | 8 MiB |
|
||||
| 6 | 6 | SpeedBetterCompression | 8 MiB |
|
||||
| 7 | 7 | SpeedBetterCompression | 16 MiB |
|
||||
| 8 | 8 | SpeedBetterCompression | 32 MiB |
|
||||
| 9 | 9 | SpeedBestCompression | 64 MiB |
|
||||
|
||||
默认 3:gzip 与现行 `NewWriterLevel(3)` 一致;zstd 的 `SpeedDefault` 与现行代码一致;xz 无历史行为。
|
||||
|
||||
xz 映射只设置 `WriterConfig.DictCap`,其余使用库默认(CRC64、hash-table matcher)。这与 `xz -N` 的字典表一致,但不比特级复刻 xz-utils 的 preset(matcher 不同,压缩率略低),文档需说明产物仍是标准 xz。
|
||||
|
||||
## xz 读取的字典上限
|
||||
|
||||
`ulikunitz/xz` 的 reader 默认 `DictCap = 8 MiB`;读取 level 9(64 MiB 字典)的流会失败。
|
||||
|
||||
**决定:xz reader 固定使用 64 MiB 上限,而不是按当前 level 推导。**
|
||||
|
||||
原因:
|
||||
|
||||
1. 本工具可能产出的最大字典就是 64 MiB;
|
||||
2. `--force-resume` 允许改配置续跑。若按当前 level 推导,从 9 降到 1 后读取旧分片会因上限不足而失败;
|
||||
3. 读侧只出现在续跑与清单补录路径,不在主写入热路径,一次 64 MiB 缓冲可接受。
|
||||
|
||||
## checkpoint 会产生多 stream
|
||||
|
||||
`Writer.Checkpoint()` 关闭当前压缩流、记录偏移,再在偏移处开启新流。gzip 多 member、zstd 多 frame、xz 多 stream 都是合法拼接;`ulikunitz/xz` 的 `SingleStream` 默认 `false`,会解码全部 stream,`xz -dc` 同样如此。
|
||||
|
||||
**要求:必须有测试覆盖"checkpoint 后的 xz 分片能被完整读回",防止 `--purge` 场景下静默丢行。**
|
||||
|
||||
## 指纹兼容性
|
||||
|
||||
现有 `state.json` 的指纹里没有 `compress_level`。无条件加字段会让本功能上线前启动的任务全部续跑失败。
|
||||
|
||||
**决定:仅当 `compress != none && level != 3` 时把 `compress_level` 加入指纹。**
|
||||
|
||||
- 老任务(无键)+ 新进程默认 3 → 哈希输入逐字节不变,无缝续跑;
|
||||
- 非默认档位才出现新键,`DiffFields` 会显示 `compress_level: "5" -> (absent)` 并拒绝;
|
||||
- 从 5 改回 3 也因键消失被识别。
|
||||
|
||||
`compress == none` 时档位不参与哈希,因为不影响产物。
|
||||
|
||||
## none 与 level 的组合
|
||||
|
||||
`compress == none && level != 3` → 配置错误;`level == 3`(默认值)静默接受。这样既能拒绝无意义组合,又不会惩罚"只是没传"的用户。
|
||||
|
||||
## 崩溃恢复识别 `.xz`
|
||||
|
||||
`manifest.go` 的 `shardNameRE` 目前只列 `\.gz|\.zst`,且 `Adopt` 用 `mt[3] != comp.Ext` 过滤。不加 `\.xz` 时,崩溃在 rename 与 manifest 追加之间的 xz 分片不会被计入 `maxSeq`,续跑会重算序号并覆盖已封分片。
|
||||
|
||||
**决定:正则与扩展名判断都识别 `.xz`,并补一条针对性测试。**
|
||||
|
||||
## Compressor 结构与扩展名收敛
|
||||
|
||||
`Compressor` 从 `{Name, Ext}` 扩展为 `{Name, Ext, Level}`,构造签名改为 `NewCompressor(name string, level int)`。gzip/zstd/xz 都从同一个对象取 `Ext`。
|
||||
|
||||
`app/dryrun.go` 的 `compressExt()` 是第二份扩展名表,改成复用 `Compressor.Ext`,避免新增格式时漏改一处。
|
||||
|
||||
## 被否决的方案
|
||||
|
||||
- **把 zstd 暴露为 1–22**:与 gzip/xz 的 1–9 不一致,校验和文档复杂度上升;zstd `best` 已足够,空间优先交给 xz。
|
||||
- **按当前 level 推导 reader 字典**:与 `--force-resume` 降级续跑冲突。
|
||||
- **xz 使用默认 8 MiB 字典**:level 3 与 xz preset 3 不一致,且高档位无法表达。
|
||||
- **无条件把档位写入指纹**:打断存量任务续跑。
|
||||
@@ -0,0 +1,34 @@
|
||||
## Why
|
||||
|
||||
当前只支持 `none|gzip|zstd`,且压缩级别写死为 3,用户无法在"吞吐优先"与"空间优先"之间取舍。xz 能显著降低长期归档的存储成本,但压缩吞吐远低于 gzip/zstd;把它作为一个显式选项,同时把级别做成统一档位,才能让三种压缩器在同一语义下被选择。
|
||||
|
||||
## What Changes
|
||||
|
||||
- 新增 `--compress xz`,每个分片是一个独立 xz 流,扩展名为 `.xz`。
|
||||
- 新增 `--compress-level 1..9`,默认 3;1–9 是跨格式统一档位,按格式映射到各自的压缩参数,9 表示该格式最强。默认档位 3 与现有行为一致。
|
||||
- `--compress none` 与非默认 `--compress-level` 同时出现时视为配置错误。
|
||||
- 配置指纹:仅当 `compress != none` 且 `level != 3` 时把 `compress_level` 纳入指纹,保证本功能之前启动的任务仍能无缝续跑;非默认档位变化仍会被识别并拒绝。
|
||||
- 崩溃恢复扫描必须识别 `.xz` 分片,避免重算序号覆盖已封分片。
|
||||
- 收敛扩展名逻辑:dry-run 复用 `Compressor.Ext`,删除重复的 `compressExt` 表。
|
||||
- 引入纯 Go 依赖 `github.com/ulikunitz/xz`。
|
||||
- **BREAKING**: 无。未设置 `--compress-level` 时,gzip/zstd 的产物与当前版本一致。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
(无)
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `output-formats`: 压缩 Requirement 从 3 种压缩器扩展为 4 种,并定义 1–9 档位到各格式的映射与 xz 场景。
|
||||
- `cli-contract`: 默认值中加入 `--compress-level 3`,参数依赖规则加入 `none` 与非默认档位的冲突校验。
|
||||
- `archive-core`: 断点续跑指纹纳入非默认压缩档位,并明确档位变化时的拒绝行为。
|
||||
- `file-sharding`: 文件命名与分片清单覆盖 `.xz`,并明确崩溃后按压缩扩展名识别已封分片。
|
||||
|
||||
## Impact
|
||||
|
||||
- 代码:`internal/cli/{config,parse,validate,hash}.go`、`internal/shard/{compress,manifest}.go`、`internal/app/{app,dryrun}.go`
|
||||
- 依赖:`go.mod` / `go.sum` 新增 `github.com/ulikunitz/xz`
|
||||
- 文档:`README.md` 压缩章节、档位表与 xz 导入示例
|
||||
- 测试:`internal/shard/writer_test.go`、`internal/app/integration_test.go`、`internal/cli` 校验与指纹测试
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 断点续跑
|
||||
|
||||
系统 SHALL 在每个 chunk 提交后原子更新 `state.json`,记录配置指纹与游标位置。续跑时 MUST 校验源、表、过滤条件、列、分片参数、时区、输出格式与压缩设置的指纹;不一致时 MUST 拒绝续跑,除非显式传入 `--force-resume`。压缩档位仅在启用压缩且档位不是默认值 3 时纳入指纹,以保证默认配置的存量任务可以无缝续跑。
|
||||
|
||||
#### Scenario: 配置一致时续跑
|
||||
|
||||
- **WHEN** 上次运行在 chunk 中途退出,且本次配置指纹一致
|
||||
- **THEN** 系统从 `state.json` 记录的游标位置继续,不重复处理已提交的 chunk
|
||||
|
||||
#### Scenario: 配置被修改
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出不一致的字段,拒绝开始扫描
|
||||
|
||||
#### Scenario: 压缩档位被修改
|
||||
|
||||
- **WHEN** 续跑时 `--compress-level` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `compress_level` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 默认档位兼容存量任务
|
||||
|
||||
- **WHEN** `state.json` 由引入压缩档位之前的版本写入,且本次使用默认档位 3
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 默认值
|
||||
|
||||
系统 SHALL 采用以下默认值:`--chunk-size 2000`、`--shard-by none`、`--shard-rows 0`、`--format dump`、`--compress none`、`--compress-level 3`、`--max-lag 1s`、`--check-interval 1s`、`--retries 1`、`--safe-auto-increment true`、`--progress 0`。
|
||||
|
||||
#### Scenario: 最小可运行配置
|
||||
|
||||
- **WHEN** 用户只提供 `--source`、`--table`、`--where`、`--file`
|
||||
- **THEN** 系统以 2000 行一个 chunk、不分片、dump 格式、不压缩的方式导出
|
||||
|
||||
### Requirement: 参数依赖与互斥
|
||||
|
||||
系统 MUST 校验以下规则:`--shard-by date` 必须有 `--shard-column`;`--shard-by count|none` 不得带 `--shard-column`;`--shard-rows` 仅在 `--shard-by count` 或 `date` 时有效;`--bulk-delete` 依赖 `--purge` 且在日期模式下禁止;普通插入、`--insert-ignore`、`--replace` 三者互斥;`--compress-level` MUST 在 1–9 范围内;`--compress none` 与非默认档位同时出现时 MUST 视为配置错误。违反时 MUST 在启动阶段报错。
|
||||
|
||||
#### Scenario: 数量模式误传日期列
|
||||
|
||||
- **WHEN** `--shard-by count` 同时带上 `--shard-column`
|
||||
- **THEN** 系统报错,说明该参数仅在日期模式下有效
|
||||
|
||||
#### Scenario: 未启用 purge 却要求 bulk-delete
|
||||
|
||||
- **WHEN** 用户指定 `--bulk-delete` 但未指定 `--purge`
|
||||
- **THEN** 系统报错,说明 bulk-delete 必须配合 purge
|
||||
|
||||
#### Scenario: 插入模式互斥
|
||||
|
||||
- **WHEN** 用户同时指定 `--insert-ignore` 与 `--replace`
|
||||
- **THEN** 系统报错,要求只能选择一种插入模式
|
||||
|
||||
#### Scenario: 压缩档位越界
|
||||
|
||||
- **WHEN** 用户指定 `--compress-level 0` 或 `--compress-level 10`
|
||||
- **THEN** 系统在启动阶段报错,说明档位必须在 1–9 之间
|
||||
|
||||
#### Scenario: 不压缩却指定非默认档位
|
||||
|
||||
- **WHEN** 用户指定 `--compress none` 与 `--compress-level 5`
|
||||
- **THEN** 系统在启动阶段报错,说明压缩档位仅在启用压缩时有效
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 文件命名与目录布局
|
||||
|
||||
分片文件 MUST 按 `<输出目录>/[date=<YYYY-MM-DD>/]<seq>.<pk_lo>-<pk_hi>.<format>[.<compress>]` 命名,`seq` 为从 1 开始的六位零填充序号。日期模式 MUST 按日期建立子目录。输出目录下同时生成 `manifest.jsonl` 与 `state.json`。
|
||||
|
||||
#### Scenario: 数量分片命名
|
||||
|
||||
- **WHEN** 第一个分片主键范围为 1..5000000,格式为 dump,压缩为 zstd
|
||||
- **THEN** 文件名为 `000001.1-5000000.dump.zst`
|
||||
|
||||
#### Scenario: xz 分片命名
|
||||
|
||||
- **WHEN** 第一个分片主键范围为 1..5000000,格式为 dump,压缩为 xz
|
||||
- **THEN** 文件名为 `000001.1-5000000.dump.xz`
|
||||
|
||||
#### Scenario: 日期分片布局
|
||||
|
||||
- **WHEN** `--shard-by date` 导出的首个日期为 2025-06-01
|
||||
- **THEN** 文件位于 `date=2025-06-01/` 子目录下,序号在该日期内从 000001 开始
|
||||
|
||||
### Requirement: 分片清单
|
||||
|
||||
系统 SHALL 在每次分片关闭后向 `manifest.jsonl` 追加一行并 flush,记录序号、相对路径、日期(如适用)、首尾主键、行数、字节数与 SHA-256。清单 MUST 在进程崩溃后仍可用于识别已完成的分片。对于已重命名但尚无清单记录的分片,续跑 MUST 按当前输出格式与压缩扩展名识别、补录并计入序号,不得覆盖。
|
||||
|
||||
#### Scenario: 分片关闭后写入清单
|
||||
|
||||
- **WHEN** 一个分片完成 flush 并重命名
|
||||
- **THEN** `manifest.jsonl` 追加一条包含 seq、path、pk_lo、pk_hi、rows、bytes、sha256、closed_at 的记录
|
||||
|
||||
#### Scenario: 崩溃后识别已完成分片
|
||||
|
||||
- **WHEN** 进程在写完两个分片后崩溃
|
||||
- **THEN** `manifest.jsonl` 中保留这两条完整记录,残留的 `.partial` 文件不进入清单
|
||||
|
||||
#### Scenario: xz 分片在清单追加前崩溃
|
||||
|
||||
- **WHEN** 一个 `.xz` 分片完成重命名后、写 `manifest.jsonl` 前进程崩溃
|
||||
- **THEN** 续跑扫描识别该分片,补录清单,后续序号从其后继续,不覆盖该文件
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 压缩
|
||||
|
||||
系统 SHALL 支持 `--compress none|gzip|zstd|xz`,默认 `none`。每个分片 MUST 独立压缩,压缩后的文件扩展名分别为 `.gz`、`.zst` 与 `.xz`。系统 SHALL 支持 `--compress-level 1..9`,默认 3;档位是跨格式统一语义,数值越大压缩率越高、吞吐越低,9 表示该压缩器的最强档位。gzip 与 zstd 在默认档位下的字节输出 MUST 与引入档位前保持一致。xz 分片 MUST 是标准 xz 流,可被通用 xz 工具解码。
|
||||
|
||||
#### Scenario: gzip 分片
|
||||
|
||||
- **WHEN** `--compress gzip`
|
||||
- **THEN** 每个分片是一个独立的 gzip 流,文件名为 `*.gz`
|
||||
|
||||
#### Scenario: zstd 分片
|
||||
|
||||
- **WHEN** `--compress zstd`
|
||||
- **THEN** 每个分片是一个独立的 zstd 帧,文件名为 `*.zst`
|
||||
|
||||
#### Scenario: xz 分片
|
||||
|
||||
- **WHEN** `--compress xz`
|
||||
- **THEN** 每个分片是一个独立的 xz 流,文件名为 `*.xz`
|
||||
|
||||
#### Scenario: xz 分片可被标准工具解码
|
||||
|
||||
- **WHEN** 产物为 `000001.1-1000.dump.xz`
|
||||
- **THEN** `xz -dc` 可将其还原为与该次扫描行内容一致的 dump 文本
|
||||
|
||||
#### Scenario: 默认档位保持既有产物
|
||||
|
||||
- **WHEN** 用户未指定 `--compress-level`
|
||||
- **THEN** gzip 与 zstd 使用档位 3,产物与引入该参数前的版本一致
|
||||
|
||||
#### Scenario: 各档位对三种压缩器均可用
|
||||
|
||||
- **WHEN** 用户指定 `--compress gzip|zstd|xz` 与任意 `--compress-level 1..9`
|
||||
- **THEN** 系统接受配置并产出对应扩展名的独立压缩分片
|
||||
@@ -0,0 +1,42 @@
|
||||
## 1. 依赖与压缩器
|
||||
|
||||
- [x] 1.1 `go get github.com/ulikunitz/xz@v0.5.17`,更新 `go.mod` / `go.sum`
|
||||
- [x] 1.2 `Compressor` 增加 `Level` 字段;`NewCompressor(name string, level int)` 接受档位
|
||||
- [x] 1.3 实现档位映射:gzip 直传 1–9;zstd 按 1-2 fastest / 3-5 default / 6-8 better / 9 best;xz 按字典表 1/2/4/4/8/8/16/32/64 MiB
|
||||
- [x] 1.4 xz writer 使用 `xz.WriterConfig{DictCap: ...}`;reader 固定 `DictCap 64 MiB` 并保留多 stream 解码
|
||||
|
||||
## 2. CLI
|
||||
|
||||
- [x] 2.1 `cli/config.go`:`CompressLevel int`,默认 3
|
||||
- [x] 2.2 `cli/parse.go`:注册 `--compress-level`,help 标注 1–9
|
||||
- [x] 2.3 `cli/validate.go`:档位必须在 1–9;`--compress none` 且档位非 3 时报配置错误
|
||||
- [x] 2.4 `cli/hash.go`:仅当 `compress != none && level != 3` 时加入 `compress_level`
|
||||
|
||||
## 3. 装配与扩展名收敛
|
||||
|
||||
- [x] 3.1 `app/app.go`:用 `NewCompressor(cfg.Compress, cfg.CompressLevel)` 构造
|
||||
- [x] 3.2 `app/dryrun.go`:输出压缩档位;删除 `compressExt()`,改用 `Compressor.Ext`
|
||||
|
||||
## 4. 崩溃恢复
|
||||
|
||||
- [x] 4.1 `shard/manifest.go`:`shardNameRE` 增加 `\.xz`,`Adopt` 能识别 xz 分片并计入序号
|
||||
|
||||
## 5. 测试
|
||||
|
||||
- [x] 5.1 档位映射边界单测(含 gzip/zstd/xz 与非法档位)
|
||||
- [x] 5.2 `shard/writer_test.go` 的压缩往返循环加入 `xz`
|
||||
- [x] 5.3 新增:`Checkpoint()` 后多 stream xz 分片能被完整读回
|
||||
- [x] 5.4 新增:无 manifest 记录的 `.xz` 分片可被 `Adopt` 补录且不影响序号
|
||||
- [x] 5.5 CLI 测试:非法档位、`none` + 非默认档位、默认档位的指纹向后兼容、非默认档位变化被拒绝
|
||||
- [x] 5.6 `app/integration_test.go`:压缩循环加入 `xz`,`decompressFile` 增加 xz 分支
|
||||
|
||||
## 6. 文档
|
||||
|
||||
- [x] 6.1 `README.md`:压缩选项加入 xz 与 `--compress-level`,补档位表、xz 导入示例(`xz -dc`)与 xz 吞吐/空间定位说明
|
||||
|
||||
## 7. 验证
|
||||
|
||||
- [x] 7.1 `go test ./...`
|
||||
- [x] 7.2 `go vet ./...`
|
||||
- [x] 7.3 `openspec validate add-compress-level-and-xz --strict`
|
||||
- [ ] 7.4 如有 MySQL 环境,跑一次 `ARCHER_TEST_DSN=... go test ./internal/app -run TestIntegration -v`
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-11-06
|
||||
@@ -0,0 +1,3 @@
|
||||
# normalize-state-config
|
||||
|
||||
state.json 的 config.source/dest 改用标准 MySQL DSN(去密码),并修复 charset 等参数未纳入指纹的问题
|
||||
@@ -0,0 +1,75 @@
|
||||
## Context
|
||||
|
||||
当前实现(见 proposal.md - Why):
|
||||
|
||||
- `cli.Config.Fingerprint()`(internal/cli/hash.go:17)把 `dsnFingerprint(src)` 与 `dsnFingerprint(dst)` 的结果放进 `fields` map,该 map 既用于计算 `config_hash`,也原样写入 `state.json` 的 `config`(internal/app/app.go:492)。
|
||||
- `dsnFingerprint`(internal/cli/hash.go:69)手工拼接 `Net|Addr|DBName|User` 和 `Params`。
|
||||
- `normalize()`(internal/cli/config.go:86)统一设置 `ParseTime=false`,`Loc` 为空时置为 UTC。
|
||||
- `redactDSN`(internal/app/dryrun.go:155)已经采用"复制配置、清空密码、`FormatDSN()`"的做法。
|
||||
|
||||
约束:指纹必须保持可复现;密码不得落盘;旧 `state.json` 必须仍能读取(版本号不变)。
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- `config.source` / `config.dest` 使用可逆、可转义、生态通用的标准 DSN。
|
||||
- 指纹覆盖所有可能改变导出结果的连接参数,尤其是 `charset` 与 `collation`。
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- 不拆分"哈希输入"与"展示快照";两者继续共用同一份 `fields` map。
|
||||
- 不改变 `state.json` 的 schema 版本,不做旧指纹的兼容计算。
|
||||
- 不改变 `--source` / `--dest` 的输入语法(仍然是 MySQL DSN)。
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. 用 `FormatDSN()` 生成指纹片段
|
||||
|
||||
```go
|
||||
func dsnFingerprint(c *mysql.Config) string {
|
||||
if c == nil {
|
||||
return ""
|
||||
}
|
||||
redacted := *c // 值拷贝,保留未导出字段(charsets 等)
|
||||
redacted.Passwd = ""
|
||||
return redacted.FormatDSN()
|
||||
}
|
||||
```
|
||||
|
||||
理由:
|
||||
|
||||
- `charset`、`collation`、`compress`、`timeTruncate` 是驱动的未导出字段,**只有 `FormatDSN` 能序列化它们**(dsn.go:318、322 等),手工枚举字段必然继续漏检。
|
||||
- `FormatDSN` 对库名做 `PathEscape`、对参数做 `QueryEscape`、对参数 key 排序(dsn.go:379-396),输出确定且无歧义。
|
||||
- 与 `redactDSN` 共用同一套序列化逻辑,`--dry-run` 与 `state.json` 的写法一致。
|
||||
|
||||
备选方案:
|
||||
|
||||
- 继续竖线格式、补上 `charset`:仍是非标准写法,未导出字段没有稳定的读取入口。
|
||||
- 嵌套 JSON 对象:需要改 `State.Config` 类型与 `DiffFields`,收益是更精确的 diff;本次不取,记录为后续方向。
|
||||
|
||||
### 2. 用值拷贝而不是原地清空密码
|
||||
|
||||
`mysql.Config` 含未导出字段,值拷贝会一起带上,`FormatDSN` 因而仍能输出 `charset`。原地清空 `Passwd` 会污染调用方的配置,直接排除。
|
||||
|
||||
### 3. 保持 `state.json` `Version = 1`
|
||||
|
||||
指纹变化会让旧任务一次性失配。不升版本的理由:`checkpoint.Load` 对未知版本直接报错,会让已 `complete` 的目录也无法重入;而保持版本 1 时,`DiffFields` 会把 `source` 的旧竖线串与新 DSN 并列展示,用户能直观看懂差异,必要时加 `--force-resume` 即可。
|
||||
|
||||
### 4. 密码继续排除在指纹之外
|
||||
|
||||
`Passwd` 清空后再序列化。口令轮换不应使续跑失效,这是现有契约。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- `FormatDSN` 会连带覆盖 `tls`、`timeout` 等不影响产物字节的参数,这类修改也会触发续跑失配 → 方向是"宁可多报";确实需要时用 `--force-resume`。若未来噪声过大,再改为语义白名单。
|
||||
- 旧 `state.json` 需要一次性 `--force-resume` → 在变更说明中标注为 BREAKING,且错误信息本身会展示新旧写法。
|
||||
- `Params` 中若有自定义敏感参数会被写入 `state.json` → 与现状相同(原实现同样写入 `Params`),本次不扩大范围。
|
||||
- 哈希输入与展示快照仍然耦合,未来再改 state 展示格式仍会触发失配 → 接受为已知债务,不在本次处理。
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. 修改 `dsnFingerprint` 并补充单元测试。
|
||||
2. 验证新 `state.json` 中 `config.source` 为标准 DSN 且不含密码。
|
||||
3. 对存量任务:首轮续跑加 `--force-resume`;已完成任务不受影响。
|
||||
4. 回滚:还原 `dsnFingerprint` 即可;已写入的新格式 `config` 只是普通字符串,仍可读取,但指纹会再次失配,需要 `--force-resume`。
|
||||
@@ -0,0 +1,27 @@
|
||||
## Why
|
||||
|
||||
`state.json` 的 `config.source` / `config.dest` 目前由 `dsnFingerprint`(internal/cli/hash.go)用竖线拼成 `tcp|host:3306|db|user`。这既不是 MySQL 生态的标准写法,也无法转义;更严重的是它只读取 `Net/Addr/DBName/User/Params`,导致 `charset`、`collation` 等被驱动收进结构体字段的参数**完全不参与指纹校验**——续跑时把 `charset=utf8mb4` 改成其他编码不会被发现,同一份导出会出现混合编码的分片。
|
||||
|
||||
## What Changes
|
||||
|
||||
- `state.json` 中的 `config.source` / `config.dest` 改为标准 MySQL DSN(去掉密码),例如 `root@tcp(s1.void01.com:3306)/lexis?charset=utf8mb4`。
|
||||
- 配置指纹改用同一份规范化 DSN 计算,覆盖驱动解析出的全部连接参数(密码除外),从而把 `charset`、`collation`、`loc` 等纳入续跑校验。
|
||||
- 与 `--dry-run` 的展示格式(`redactDSN`,internal/app/dryrun.go)保持一致,工具的"用户可见连接串"只有一种写法。
|
||||
- **BREAKING**:指纹取值变化,旧版本写入的 `state.json` 会被判定为配置不同。续跑旧任务需要一次性传入 `--force-resume`;`state.json` 版本号不变(仍为 1),旧文件仍可读取并展示差异。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
(无)
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `archive-core`: 断点续跑要求中,明确配置指纹必须覆盖源/目标连接的规范化参数(含字符集与排序规则),并规定 `state.json` 中连接信息的存储形态为标准 DSN。
|
||||
|
||||
## Impact
|
||||
|
||||
- 代码:`internal/cli/hash.go`(`dsnFingerprint` 改为复制配置、清空 `Passwd`、调用 `FormatDSN`)。
|
||||
- 测试:`internal/cli/cli_test.go` 增加 charset/参数变化的指纹断言与 `state.json` 快照断言。
|
||||
- 数据:`state.json` 的 `config` 内容形态变化,触发一次性续跑校验失败。
|
||||
- 文档:README 中关于 `state.json` 记录的描述无需改动;`--dry-run` 输出不变。
|
||||
@@ -0,0 +1,40 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 断点续跑
|
||||
|
||||
系统 SHALL 在每个 chunk 提交后原子更新 `state.json`,记录配置指纹与游标位置。`state.json` 中记录的源连接与目标连接 MUST 采用标准 MySQL DSN 形式(去除密码),例如 `user@tcp(host:3306)/db?charset=utf8mb4`。续跑时 MUST 校验源、表、过滤条件、列、分片参数、时区、输出格式、压缩设置,以及源/目标连接的全部规范化参数(含字符集与排序规则)的指纹;不一致时 MUST 拒绝续跑,除非显式传入 `--force-resume`。密码 MUST NOT 计入指纹,也 MUST NOT 写入 `state.json`。压缩档位仅在启用压缩且档位不是默认值 3 时纳入指纹,以保证默认配置的存量任务可以无缝续跑。
|
||||
|
||||
#### Scenario: 配置一致时续跑
|
||||
|
||||
- **WHEN** 上次运行在 chunk 中途退出,且本次配置指纹一致
|
||||
- **THEN** 系统从 `state.json` 记录的游标位置继续,不重复处理已提交的 chunk
|
||||
|
||||
#### Scenario: 配置被修改
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出不一致的字段,拒绝开始扫描
|
||||
|
||||
#### Scenario: 压缩档位被修改
|
||||
|
||||
- **WHEN** 续跑时 `--compress-level` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `compress_level` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 默认档位兼容存量任务
|
||||
|
||||
- **WHEN** `state.json` 由引入压缩档位之前的版本写入,且本次使用默认档位 3
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接字符集被修改
|
||||
|
||||
- **WHEN** 续跑时 `--source` 的 `charset` 或 `collation` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `source` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 密码轮换
|
||||
|
||||
- **WHEN** 续跑时仅 `--source` 或 `--dest` 的密码与上次运行不同
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接信息的存储形态
|
||||
|
||||
- **WHEN** 查看任意一次运行生成的 `state.json`
|
||||
- **THEN** `config.source` 与 `config.dest` 为标准 MySQL DSN,且不包含密码明文
|
||||
@@ -0,0 +1,17 @@
|
||||
## 1. 指纹生成
|
||||
|
||||
- [x] 1.1 将 `internal/cli/hash.go` 的 `dsnFingerprint` 改为:`nil` 返回空串;否则值拷贝 `mysql.Config`、清空 `Passwd`、返回 `FormatDSN()`
|
||||
- [x] 1.2 更新 `Fingerprint` 与 `dsnFingerprint` 的注释,说明输出为标准 DSN、覆盖驱动全部已解析参数、密码除外
|
||||
|
||||
## 2. 测试
|
||||
|
||||
- [x] 2.1 新增单元测试:`charset` 或 `collation` 不同时指纹不同
|
||||
- [x] 2.2 新增单元测试:仅密码不同时指纹相同
|
||||
- [x] 2.3 新增单元测试:指纹中不含密码明文,`dest` 为空时片段为空串
|
||||
- [x] 2.4 新增断言:`state.json` 的 `config.source` 等于 `redactDSN` 输出的标准 DSN(可复用现有 app 测试夹具)
|
||||
|
||||
## 3. 验证
|
||||
|
||||
- [x] 3.1 运行 `go test ./...`(无数据库环境下跳过集成测试)
|
||||
- [ ] 3.2 用同一 `--source` 分别以不同密码、不同 `charset` 运行 `--dry-run` 与续跑,确认前者指纹一致、后者报错并展示 `source` 差异
|
||||
- [ ] 3.3 用旧版 `state.json` 续跑一次,确认报错信息可读,且 `--force-resume` 后正常继续并改写为新格式
|
||||
@@ -0,0 +1,32 @@
|
||||
schema: spec-driven
|
||||
|
||||
# Project context (optional)
|
||||
# This is shown to AI when creating artifacts.
|
||||
# Add your tech stack, conventions, style guides, domain knowledge, etc.
|
||||
# Example:
|
||||
# context: |
|
||||
# Tech stack: TypeScript, React, Node.js
|
||||
# We use conventional commits
|
||||
# Domain: e-commerce platform
|
||||
|
||||
# Per-artifact rules (optional)
|
||||
# Add custom rules for specific artifacts.
|
||||
# Example:
|
||||
# rules:
|
||||
# proposal:
|
||||
# - Keep proposals under 500 words
|
||||
# - Always include a "Non-goals" section
|
||||
# tasks:
|
||||
# - Break tasks into chunks of max 2 hours
|
||||
|
||||
# Per-operation guidance (optional)
|
||||
# Add advisory guidance for how apply and archive work should be conducted.
|
||||
# This is separate from artifact rules above.
|
||||
# Example:
|
||||
# operations:
|
||||
# apply:
|
||||
# guidance:
|
||||
# - Keep test summaries concise
|
||||
# archive:
|
||||
# guidance:
|
||||
# - Summarize the archive outcome before finishing
|
||||
@@ -0,0 +1,177 @@
|
||||
# archive-core Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
定义 archer 如何按唯一索引分块读取源表,并以固定顺序完成文件写入、目标表写入与源行删除,同时提供复制延迟保护、安全护栏和可中断续跑能力。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 唯一索引 seek 分块扫描
|
||||
|
||||
系统 SHALL 以源表的一个唯一索引(优先主键)作为扫描游标。每个 chunk 的查询 MUST 使用“索引键大于上一次尾值”的条件、按索引列 `ORDER BY`、并带 `LIMIT chunk-size`;系统 MUST NOT 使用 `OFFSET` 翻页。返回行数小于 `chunk-size` 时,系统 SHALL 结束扫描。
|
||||
|
||||
#### Scenario: 主键连续
|
||||
|
||||
- **WHEN** 源表主键为连续的 1..N,且 `--chunk-size 2000`
|
||||
- **THEN** 系统按 2000 行一个 chunk 依次读取,直到最后一个不足 2000 行的 chunk
|
||||
|
||||
#### Scenario: 主键存在空洞
|
||||
|
||||
- **WHEN** 主键序列为 1,2,100,101,且 `--chunk-size 2`
|
||||
- **THEN** 第一个 chunk 返回 1,2,第二个 chunk 从 `id > 2` 继续并返回 100,101,不会跳过或重复
|
||||
|
||||
#### Scenario: 无可用唯一索引
|
||||
|
||||
- **WHEN** 源表既没有主键也没有唯一索引,且用户未指定索引
|
||||
- **THEN** 系统在开始扫描前报错退出,并提示需要唯一索引
|
||||
|
||||
### Requirement: chunk 即事务
|
||||
|
||||
每个 chunk SHALL 构成唯一的事务边界,`--chunk-size` 同时决定 SELECT 批量、事务大小和 checkpoint 粒度。系统 SHALL NOT 提供跨 chunk 的事务,也 SHALL NOT 提供独立的 `--txn-size` 参数。
|
||||
|
||||
#### Scenario: 源表与目标表在同一实例
|
||||
|
||||
- **WHEN** `--source` 与 `--dest` 指向同一 MySQL 实例,且同时启用 dest 与 purge
|
||||
- **THEN** 目标表写入与源行删除在同一个事务中提交,二者要么同时生效要么同时回滚
|
||||
|
||||
#### Scenario: 源表与目标表在不同实例
|
||||
|
||||
- **WHEN** `--source` 与 `--dest` 指向不同实例
|
||||
- **THEN** 系统使用两个连接分别提交,并在任一步失败时按 `--retries` 重试;已提交的目标表写入可能在重跑时重复,行为为 at-least-once
|
||||
|
||||
### Requirement: 三件套提交顺序
|
||||
|
||||
当 file、dest、purge 中任意多项同时启用时,系统 MUST 对每个 chunk 按固定顺序执行:写文件、写目标表、删除源行、提交事务并更新 checkpoint。任何前置步骤失败时,系统 MUST NOT 执行后续步骤,也 MUST NOT 推进 checkpoint。
|
||||
|
||||
#### Scenario: 文件写入失败
|
||||
|
||||
- **WHEN** 写文件返回错误
|
||||
- **THEN** 系统不写目标表、不删除源行、不推进 checkpoint,并以非零码退出
|
||||
|
||||
#### Scenario: 目标表写入失败
|
||||
|
||||
- **WHEN** 文件写入成功但目标表插入失败
|
||||
- **THEN** 系统删除源行,回滚本次事务,不推进 checkpoint,并以非零码退出
|
||||
|
||||
### Requirement: 输出目标约束
|
||||
|
||||
系统 SHALL 要求 `--file`、`--dest`、`--purge` 至少启用一个,并且 `--where` 必填。三者可以任意组合,包括只导出文件、只删除、或文件与目标表同时输出。
|
||||
|
||||
#### Scenario: 只删不写
|
||||
|
||||
- **WHEN** 用户只指定 `--purge` 和 `--where`
|
||||
- **THEN** 系统仅按 chunk 删除匹配行,不生成任何文件
|
||||
|
||||
#### Scenario: 文件与目标表同时输出
|
||||
|
||||
- **WHEN** 用户同时指定 `--file` 与 `--dest`
|
||||
- **THEN** 同一批行既写入分片文件,也写入目标表
|
||||
|
||||
### Requirement: 目标表写入模式
|
||||
|
||||
目标表写入 SHALL 支持普通插入、`INSERT IGNORE` 与 `REPLACE` 三种模式,三者互斥。系统 MUST 使用多行批量插入,并在单条语句可能超过 `max_allowed_packet` 时自动拆分为多条语句。
|
||||
|
||||
#### Scenario: 默认插入遇到重复主键
|
||||
|
||||
- **WHEN** 使用默认插入模式,且目标表已存在相同主键
|
||||
- **THEN** 该 chunk 失败并回滚,系统报告重复键错误
|
||||
|
||||
#### Scenario: ignore 模式遇到重复主键
|
||||
|
||||
- **WHEN** 使用 `--insert-ignore`
|
||||
- **THEN** 重复行被跳过,其余行写入成功,chunk 正常提交
|
||||
|
||||
### Requirement: 删除语义
|
||||
|
||||
默认情况下,系统 MUST 以主键 IN 列表删除当前 chunk 实际读取到的行,并将过长的列表分批。启用 `--bulk-delete` 时,系统 SHALL 改为按首尾主键区间删除,并 MUST 在语句中附带原始 `--where` 条件。`--bulk-delete` 依赖 `--purge`,且在日期分片模式下 MUST 被拒绝。
|
||||
|
||||
#### Scenario: 默认 IN 列表删除
|
||||
|
||||
- **WHEN** 一个 chunk 读取到主键 1001..3000,且 `--where "created_at < '2026-01-01'"`
|
||||
- **THEN** 系统生成 `DELETE ... WHERE id IN (1001,...,3000)`,不删除未被读取的行
|
||||
|
||||
#### Scenario: 区间删除附带回填条件
|
||||
|
||||
- **WHEN** 启用 `--bulk-delete`,chunk 首尾主键为 1001 与 3000
|
||||
- **THEN** 系统生成 `DELETE ... WHERE id BETWEEN 1001 AND 3000 AND (<where>)`
|
||||
|
||||
#### Scenario: 日期模式启用区间删除
|
||||
|
||||
- **WHEN** `--shard-by date` 与 `--bulk-delete` 同时出现
|
||||
- **THEN** 系统在启动校验阶段报错退出,不执行任何查询
|
||||
|
||||
### Requirement: 复制延迟保护
|
||||
|
||||
当指定 `--check-replica-lag` 时,系统 SHALL 在每次拉取前检查该副本的延迟;延迟超过 `--max-lag` 或复制未运行时,系统 MUST 暂停并在 `--check-interval` 后重试。
|
||||
|
||||
#### Scenario: 延迟超限
|
||||
|
||||
- **WHEN** `--max-lag 1s`,副本延迟为 5s
|
||||
- **THEN** 系统暂停拉取,直到延迟降到 1s 以下
|
||||
|
||||
#### Scenario: 副本复制中断
|
||||
|
||||
- **WHEN** 被检查的副本复制线程未运行
|
||||
- **THEN** 系统持续等待并周期性重试,不继续归档
|
||||
|
||||
### Requirement: 安全护栏
|
||||
|
||||
系统 SHALL 默认启用 `--safe-auto-increment`,禁止通过 purge 删除当前最大自增主键行。`--dry-run` 时系统 MUST 只打印将要执行的 SQL 与实际生效配置,不连接写入、不创建或修改任何文件。
|
||||
|
||||
#### Scenario: 保护最大自增行
|
||||
|
||||
- **WHEN** purge 范围包含源表当前 `MAX(pk)`,且 `--safe-auto-increment` 为默认值
|
||||
- **THEN** 该行不被删除,系统记录一条警告
|
||||
|
||||
#### Scenario: dry-run 无副作用
|
||||
|
||||
- **WHEN** 用户指定 `--dry-run`
|
||||
- **THEN** 系统打印每条 SQL 与解析后的配置后退出,输出目录不产生任何文件
|
||||
|
||||
### Requirement: 断点续跑
|
||||
|
||||
系统 SHALL 在每个 chunk 提交后原子更新 `state.json`,记录配置指纹与游标位置。`state.json` 中记录的源连接与目标连接 MUST 采用标准 MySQL DSN 形式(去除密码),例如 `user@tcp(host:3306)/db?charset=utf8mb4`。续跑时 MUST 校验源、表、过滤条件、列、分片参数、时区、输出格式、压缩设置,以及源/目标连接的全部规范化参数(含字符集与排序规则)的指纹;不一致时 MUST 拒绝续跑,除非显式传入 `--force-resume`。密码 MUST NOT 计入指纹,也 MUST NOT 写入 `state.json`。压缩档位仅在启用压缩且档位不是默认值 3 时纳入指纹,以保证默认配置的存量任务可以无缝续跑。
|
||||
|
||||
#### Scenario: 配置一致时续跑
|
||||
|
||||
- **WHEN** 上次运行在 chunk 中途退出,且本次配置指纹一致
|
||||
- **THEN** 系统从 `state.json` 记录的游标位置继续,不重复处理已提交的 chunk
|
||||
|
||||
#### Scenario: 配置被修改
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出不一致的字段,拒绝开始扫描
|
||||
|
||||
#### Scenario: 压缩档位被修改
|
||||
|
||||
- **WHEN** 续跑时 `--compress-level` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `compress_level` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 默认档位兼容存量任务
|
||||
|
||||
- **WHEN** `state.json` 由引入压缩档位之前的版本写入,且本次使用默认档位 3
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接字符集被修改
|
||||
|
||||
- **WHEN** 续跑时 `--source` 的 `charset` 或 `collation` 与 `state.json` 记录不一致,且未传 `--force-resume`
|
||||
- **THEN** 系统报错并列出 `source` 字段的变化,拒绝开始扫描
|
||||
|
||||
#### Scenario: 密码轮换
|
||||
|
||||
- **WHEN** 续跑时仅 `--source` 或 `--dest` 的密码与上次运行不同
|
||||
- **THEN** 指纹一致,系统正常续跑
|
||||
|
||||
#### Scenario: 连接信息的存储形态
|
||||
|
||||
- **WHEN** 查看任意一次运行生成的 `state.json`
|
||||
- **THEN** `config.source` 与 `config.dest` 为标准 MySQL DSN,且不包含密码明文
|
||||
|
||||
### Requirement: 一致性等级
|
||||
|
||||
文件输出的一致性等级 SHALL 为 at-least-once:崩溃后重跑可能向分片文件重复追加已处理的行,系统 MUST 在文档与启动输出中明确该行为。
|
||||
|
||||
#### Scenario: 崩溃后重跑
|
||||
|
||||
- **WHEN** 文件已写入但 checkpoint 尚未更新时进程崩溃,随后以相同配置重跑
|
||||
- **THEN** 系统从上一个已提交游标继续,文件可能出现重复行,数据不丢失
|
||||
@@ -0,0 +1,110 @@
|
||||
# cli-contract Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
定义 archer 的命令行参数、默认值、参数之间的依赖与互斥规则,以及启动校验、错误信息和 dry-run 行为。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 必填参数
|
||||
|
||||
系统 MUST 要求提供 `--source` 与 `--where`,并且 `--file`、`--dest`、`--purge` 至少启用一个。缺少任一前置条件时,系统 MUST 在连接数据库前报错退出。
|
||||
|
||||
#### Scenario: 缺少过滤条件
|
||||
|
||||
- **WHEN** 用户未提供 `--where`
|
||||
- **THEN** 系统立即报错,提示 `--where` 必填,不建立数据库连接
|
||||
|
||||
#### Scenario: 没有任何输出目标
|
||||
|
||||
- **WHEN** 用户既未提供 `--file`、`--dest`,也未启用 `--purge`
|
||||
- **THEN** 系统报错,提示至少需要一种输出目标
|
||||
|
||||
### Requirement: 默认值
|
||||
|
||||
系统 SHALL 采用以下默认值:`--chunk-size 2000`、`--shard-by none`、`--shard-rows 0`、`--format dump`、`--compress none`、`--compress-level 3`、`--max-lag 1s`、`--check-interval 1s`、`--retries 1`、`--safe-auto-increment true`、`--progress 0`。
|
||||
|
||||
#### Scenario: 最小可运行配置
|
||||
|
||||
- **WHEN** 用户只提供 `--source`、`--table`、`--where`、`--file`
|
||||
- **THEN** 系统以 2000 行一个 chunk、不分片、dump 格式、不压缩的方式导出
|
||||
|
||||
### Requirement: 参数依赖与互斥
|
||||
|
||||
系统 MUST 校验以下规则:`--shard-by date` 必须有 `--shard-column`;`--shard-by count|none` 不得带 `--shard-column`;`--shard-rows` 仅在 `--shard-by count` 或 `date` 时有效;`--bulk-delete` 依赖 `--purge` 且在日期模式下禁止;普通插入、`--insert-ignore`、`--replace` 三者互斥;`--compress-level` MUST 在 1–9 范围内;`--compress none` 与非默认档位同时出现时 MUST 视为配置错误。违反时 MUST 在启动阶段报错。
|
||||
|
||||
#### Scenario: 数量模式误传日期列
|
||||
|
||||
- **WHEN** `--shard-by count` 同时带上 `--shard-column`
|
||||
- **THEN** 系统报错,说明该参数仅在日期模式下有效
|
||||
|
||||
#### Scenario: 未启用 purge 却要求 bulk-delete
|
||||
|
||||
- **WHEN** 用户指定 `--bulk-delete` 但未指定 `--purge`
|
||||
- **THEN** 系统报错,说明 bulk-delete 必须配合 purge
|
||||
|
||||
#### Scenario: 插入模式互斥
|
||||
|
||||
- **WHEN** 用户同时指定 `--insert-ignore` 与 `--replace`
|
||||
- **THEN** 系统报错,要求只能选择一种插入模式
|
||||
|
||||
#### Scenario: 压缩档位越界
|
||||
|
||||
- **WHEN** 用户指定 `--compress-level 0` 或 `--compress-level 10`
|
||||
- **THEN** 系统在启动阶段报错,说明档位必须在 1–9 之间
|
||||
|
||||
#### Scenario: 不压缩却指定非默认档位
|
||||
|
||||
- **WHEN** 用户指定 `--compress none` 与 `--compress-level 5`
|
||||
- **THEN** 系统在启动阶段报错,说明压缩档位仅在启用压缩时有效
|
||||
|
||||
### Requirement: 日期列探测与错误提示
|
||||
|
||||
启动时,系统 SHALL 读取表元数据并判断 `--shard-column` 是否存在、类型是否受支持、是否可为 NULL。缺少 `--shard-column` 时 MUST 列出所有候选日期/时间列及其类型。整数列缺少单位声明时 MUST 采样最小/最大值,并展示按秒、毫秒、微秒解释后的日期范围。
|
||||
|
||||
#### Scenario: 列出候选列
|
||||
|
||||
- **WHEN** 用户指定 `--shard-by date` 但未给 `--shard-column`
|
||||
- **THEN** 错误信息包含表内所有 DATE/DATETIME/TIMESTAMP 列的列名、类型与可空性
|
||||
|
||||
#### Scenario: epoch 单位提示
|
||||
|
||||
- **WHEN** 分片列为 BIGINT 且未声明单位
|
||||
- **THEN** 错误信息展示采样范围以及三种单位解释下的日期范围,不自动选择
|
||||
|
||||
### Requirement: dry-run
|
||||
|
||||
`--dry-run` 时,系统 MUST 打印解析后的配置、索引探测结果、最终生效的分片与格式参数,以及第一条将要执行的 SELECT 和每类写入语句,然后退出。系统 MUST NOT 建立写入连接、创建目录或写文件。
|
||||
|
||||
#### Scenario: dry-run 输出 SQL
|
||||
|
||||
- **WHEN** 用户传入 `--dry-run`
|
||||
- **THEN** 标准输出包含解析后的配置和将要执行的 SQL,进程以 0 退出
|
||||
|
||||
### Requirement: 进度与统计
|
||||
|
||||
当 `--progress N` 大于 0 时,系统 SHALL 每处理 N 行输出一次进度,包含已处理行数、当前分片序号与耗时。启用 `--statistics` 时,系统 MUST 在结束时输出读取行数、写入行数、删除行数、分片数与各阶段耗时。
|
||||
|
||||
#### Scenario: 周期进度
|
||||
|
||||
- **WHEN** `--progress 100000`
|
||||
- **THEN** 每处理 100000 行打印一行进度信息
|
||||
|
||||
#### Scenario: 结束统计
|
||||
|
||||
- **WHEN** 启用 `--statistics`
|
||||
- **THEN** 进程退出前输出汇总统计
|
||||
|
||||
### Requirement: 退出码
|
||||
|
||||
系统 SHALL 在成功完成时返回 0;配置错误、连接错误、写入错误或校验失败时返回非零退出码。被信号中断时 MUST 完成当前 chunk 的清理并返回非零退出码。
|
||||
|
||||
#### Scenario: 配置错误
|
||||
|
||||
- **WHEN** 参数校验失败
|
||||
- **THEN** 进程以非零退出码结束,且不产生部分输出文件
|
||||
|
||||
#### Scenario: 信号中断
|
||||
|
||||
- **WHEN** 进程收到 SIGINT
|
||||
- **THEN** 系统停止拉取新 chunk,关闭并 flush 当前分片,写入 checkpoint,然后以非零退出码结束
|
||||
@@ -0,0 +1,163 @@
|
||||
# file-sharding Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
定义导出文件的分片模式、顺序不变量、命名规则、句柄管理与分片清单,确保大规模导出的结果可被分段、有序、独立地导入。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 分片模式
|
||||
|
||||
系统 SHALL 支持三种分片模式:`none`(不分片,单文件)、`count`(按固定行数封片)、`date`(按用户指定的日期或时间列封片)。分片模式 SHALL 由 `--shard-by` 选择,默认 `none`。
|
||||
|
||||
#### Scenario: 不分片
|
||||
|
||||
- **WHEN** `--shard-by none`
|
||||
- **THEN** 本次扫描的全部行写入同一个文件
|
||||
|
||||
#### Scenario: 按数量分片
|
||||
|
||||
- **WHEN** `--shard-by count --shard-rows 5000000`
|
||||
- **THEN** 每写满 500 万行封一个文件,最后一个文件可少于 500 万行
|
||||
|
||||
#### Scenario: 按日期分片
|
||||
|
||||
- **WHEN** `--shard-by date --shard-column created_at`
|
||||
- **THEN** 每个文件只包含 `created_at` 落在同一个日期的行
|
||||
|
||||
### Requirement: 有序连续不变量
|
||||
|
||||
对于任何分片模式,系统 MUST 保证:分片按扫描顺序编号;相邻分片的主键范围连续且不重叠;所有分片合并后等于本次扫描读取的全部行;每个分片内部的行顺序与扫描顺序一致;文件名中的首尾主键为该分片实际写入的第一行与最后一行主键。
|
||||
|
||||
#### Scenario: 分片边界连续
|
||||
|
||||
- **WHEN** 一次扫描读取主键 1..2500,且 `--shard-rows 1000`
|
||||
- **THEN** 生成三个分片,主键范围分别为 1-1000、1001-2000、2001-2500,无重叠也无空洞
|
||||
|
||||
#### Scenario: 主键存在空洞时仍连续
|
||||
|
||||
- **WHEN** 扫描读取到主键 1,2,100,101,且 `--shard-rows 2`
|
||||
- **THEN** 分片范围分别为 1-2 与 100-101,文件名反映实际首尾主键
|
||||
|
||||
### Requirement: 按行数流式封片
|
||||
|
||||
在 `count` 模式下,系统 SHALL 在写入过程中逐行判断当前分片是否已满,写满 `--shard-rows` 行后立即封片。`--chunk-size` 与 `--shard-rows` SHALL 相互独立,系统 MUST 支持一个 chunk 跨越多个分片,且不要求两者存在整除关系。
|
||||
|
||||
#### Scenario: chunk 跨越分片边界
|
||||
|
||||
- **WHEN** `--chunk-size 2000` 且 `--shard-rows 2500`
|
||||
- **THEN** 第二个 chunk 的前 1500 行写入第一个分片,后 500 行写入第二个分片
|
||||
|
||||
#### Scenario: 文件在关闭时才确定名称
|
||||
|
||||
- **WHEN** 一个分片正在写入
|
||||
- **THEN** 该文件在关闭前以 `.partial` 形式存在,关闭并 flush 后重命名为包含实际首尾主键的最终文件名
|
||||
|
||||
### Requirement: 日期分片的列契约
|
||||
|
||||
`--shard-by date` 时,系统 MUST 要求显式提供 `--shard-column`;即使表中只有一个日期列也不得自动选择。该列类型 MUST 属于 `DATE`、`DATETIME`、`TIMESTAMP` 或整数类型(epoch,需显式声明单位)。其他类型 MUST 被拒绝。
|
||||
|
||||
#### Scenario: 缺少分片列
|
||||
|
||||
- **WHEN** `--shard-by date` 但未提供 `--shard-column`
|
||||
- **THEN** 系统报错,并列出表中所有候选的日期/时间列及其类型与可空性
|
||||
|
||||
#### Scenario: 多个日期列
|
||||
|
||||
- **WHEN** 表中有 `created_at`、`updated_at`、`paid_at` 三个日期列,用户指定 `--shard-column created_at`
|
||||
- **THEN** 系统只按 `created_at` 分片,其他日期列作为普通导出列
|
||||
|
||||
#### Scenario: 整数时间戳列未声明单位
|
||||
|
||||
- **WHEN** `--shard-column created_at` 指向 `BIGINT` 列,且未提供 `--shard-column-type`
|
||||
- **THEN** 系统报错,采样该列最小/最大值,并分别展示按秒、毫秒、微秒解释后的日期范围供用户选择
|
||||
|
||||
#### Scenario: 不支持的列类型
|
||||
|
||||
- **WHEN** `--shard-column` 指向 `VARCHAR` 或 `TIME` 列
|
||||
- **THEN** 系统报错,并说明 v1 仅支持日期时间类型与显式声明的 epoch 整数
|
||||
|
||||
### Requirement: 日期值归属规则
|
||||
|
||||
系统 MUST 按列语义决定日期归属:`DATE` 与 `DATETIME` 使用原值,不做时区换算;`TIMESTAMP` 按连接会话时区换算;epoch 整数按 `--shard-time-zone`(默认 UTC)换算。NULL 值默认落入 `date=_null` 分片;零日期默认报错并给出行主键,可通过 `--zero-date bucket` 改为落入 `date=0000-00-00` 分片。`--shard-time-zone` MUST 计入配置指纹。
|
||||
|
||||
#### Scenario: NULL 日期值
|
||||
|
||||
- **WHEN** 某行的 `created_at` 为 NULL,且使用默认配置
|
||||
- **THEN** 该行写入 `date=_null` 分片,扫描顺序与分片顺序仍保持一致
|
||||
|
||||
#### Scenario: 零日期值
|
||||
|
||||
- **WHEN** 某行的 `created_at` 为 `0000-00-00 00:00:00`,且未指定 `--zero-date`
|
||||
- **THEN** 系统报错并输出该行主键,不生成静默的错误分片
|
||||
|
||||
#### Scenario: 时区改变后续跑
|
||||
|
||||
- **WHEN** 续跑时 `--shard-time-zone` 与 `state.json` 记录不一致
|
||||
- **THEN** 系统拒绝续跑,除非显式传入 `--force-resume`
|
||||
|
||||
### Requirement: 日期扫描路径选择
|
||||
|
||||
系统 SHALL 在启动时探测分片列的索引,并据此选择扫描路径:存在 `(shard_column, pk)` 联合索引时使用 keyset 分块(短事务);仅存在 `shard_column` 单列索引时使用逐日流式读取(长游标,purge 使用独立连接);两者都不存在时 MUST fail fast,除非显式传入 `--allow-full-scan`。
|
||||
|
||||
#### Scenario: 存在联合索引
|
||||
|
||||
- **WHEN** 表上存在 `(created_at, id)` 联合索引
|
||||
- **THEN** 系统按 `(created_at, id)` 做 keyset 分块,每个 chunk 独立提交
|
||||
|
||||
#### Scenario: 仅存在单列索引
|
||||
|
||||
- **WHEN** 表上只有 `created_at` 单列索引
|
||||
- **THEN** 系统按日期逐日发出查询并流式读取,日期内按 `created_at, id` 排序;删除操作使用第二条源连接
|
||||
|
||||
#### Scenario: 没有任何相关索引
|
||||
|
||||
- **WHEN** 分片列上既无联合索引也无单列索引
|
||||
- **THEN** 系统报错并提示创建索引,或由用户显式传入 `--allow-full-scan` 以接受全表扫描
|
||||
|
||||
### Requirement: 文件命名与目录布局
|
||||
|
||||
分片文件 MUST 按 `<输出目录>/[date=<YYYY-MM-DD>/]<seq>.<pk_lo>-<pk_hi>.<format>[.<compress>]` 命名,`seq` 为从 1 开始的六位零填充序号。日期模式 MUST 按日期建立子目录。输出目录下同时生成 `manifest.jsonl` 与 `state.json`。
|
||||
|
||||
#### Scenario: 数量分片命名
|
||||
|
||||
- **WHEN** 第一个分片主键范围为 1..5000000,格式为 dump,压缩为 zstd
|
||||
- **THEN** 文件名为 `000001.1-5000000.dump.zst`
|
||||
|
||||
#### Scenario: xz 分片命名
|
||||
|
||||
- **WHEN** 第一个分片主键范围为 1..5000000,格式为 dump,压缩为 xz
|
||||
- **THEN** 文件名为 `000001.1-5000000.dump.xz`
|
||||
|
||||
#### Scenario: 日期分片布局
|
||||
|
||||
- **WHEN** `--shard-by date` 导出的首个日期为 2025-06-01
|
||||
- **THEN** 文件位于 `date=2025-06-01/` 子目录下,序号在该日期内从 000001 开始
|
||||
|
||||
### Requirement: 分片清单
|
||||
|
||||
系统 SHALL 在每次分片关闭后向 `manifest.jsonl` 追加一行并 flush,记录序号、相对路径、日期(如适用)、首尾主键、行数、字节数与 SHA-256。清单 MUST 在进程崩溃后仍可用于识别已完成的分片。对于已重命名但尚无清单记录的分片,续跑 MUST 按当前输出格式与压缩扩展名识别、补录并计入序号,不得覆盖。
|
||||
|
||||
#### Scenario: 分片关闭后写入清单
|
||||
|
||||
- **WHEN** 一个分片完成 flush 并重命名
|
||||
- **THEN** `manifest.jsonl` 追加一条包含 seq、path、pk_lo、pk_hi、rows、bytes、sha256、closed_at 的记录
|
||||
|
||||
#### Scenario: 崩溃后识别已完成分片
|
||||
|
||||
- **WHEN** 进程在写完两个分片后崩溃
|
||||
- **THEN** `manifest.jsonl` 中保留这两条完整记录,残留的 `.partial` 文件不进入清单
|
||||
|
||||
#### Scenario: xz 分片在清单追加前崩溃
|
||||
|
||||
- **WHEN** 一个 `.xz` 分片完成重命名后、写 `manifest.jsonl` 前进程崩溃
|
||||
- **THEN** 续跑扫描识别该分片,补录清单,后续序号从其后继续,不覆盖该文件
|
||||
|
||||
### Requirement: 分片文件原子性
|
||||
|
||||
每个分片 MUST 先写入同目录下的 `.partial` 临时文件,成功 flush 后原子重命名为最终文件名。每个分片 MUST 独立压缩,使单个文件可以脱离其他分片单独导入。
|
||||
|
||||
#### Scenario: 崩溃留下临时文件
|
||||
|
||||
- **WHEN** 进程在分片写入过程中崩溃
|
||||
- **THEN** 目录中留下 `.partial` 文件,最终文件名对应的文件不存在,续跑时系统忽略或清理该临时文件
|
||||
@@ -0,0 +1,97 @@
|
||||
# output-formats Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
定义行数据到文件的编码规则与压缩方式,确保导出产物可以被 MySQL、PostgreSQL、ClickHouse、Spark 等常见工具直接导入。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: dump 格式
|
||||
|
||||
`--format dump` 时,系统 MUST 以制表符分隔字段、换行符结束行、`\N` 表示 NULL、反斜杠转义特殊字符,并且不写表头。转义规则 MUST 与 MySQL `SELECT ... INTO OUTFILE` 一致:反斜杠、制表符、换行、回车、NUL、退格、Ctrl-Z 均需转义。
|
||||
|
||||
#### Scenario: 默认 dump 输出
|
||||
|
||||
- **WHEN** 一行包含 `a\tb` 与 NULL
|
||||
- **THEN** 输出为 `a\\tb\t\\N\n`,可被 `LOAD DATA INFILE` 以默认参数读回
|
||||
|
||||
#### Scenario: 数值与日期列
|
||||
|
||||
- **WHEN** 列类型为 DECIMAL、DATETIME
|
||||
- **THEN** 直接输出 MySQL 返回的文本表示,不做重格式化
|
||||
|
||||
### Requirement: CSV 格式
|
||||
|
||||
`--format csv` 时,系统 MUST 输出 RFC4180 兼容内容:逗号分隔、首行为列名表头、含逗号/引号/CR/LF 的字段用双引号包裹、字段内双引号写为两个双引号、行尾为 CRLF。NULL 默认输出 `\N`,可通过 `--csv-null` 覆盖;空字符串输出为空字段。
|
||||
|
||||
#### Scenario: 字段包含逗号与引号
|
||||
|
||||
- **WHEN** 字段值为 `a,"b"`
|
||||
- **THEN** 输出为 `"a,""b"""`
|
||||
|
||||
#### Scenario: 表头
|
||||
|
||||
- **WHEN** 生成任意 CSV 分片
|
||||
- **THEN** 每个文件首行都是列名列表,顺序与 `--columns` 一致
|
||||
|
||||
### Requirement: JSONL 格式
|
||||
|
||||
`--format jsonl` 时,系统 MUST 每行输出一个 JSON 对象,键为列名。类型映射 MUST 为:DATETIME/TIMESTAMP 输出 RFC3339 字符串;DECIMAL 输出字符串以保留精度;BLOB/BINARY 输出 base64 字符串;NULL 输出 JSON `null`;整数与浮点输出 JSON 数字。
|
||||
|
||||
#### Scenario: 类型映射
|
||||
|
||||
- **WHEN** 一行包含 DECIMAL(10,2)=12.30、DATETIME、NULL
|
||||
- **THEN** 对应字段分别为 `"12.30"`、RFC3339 时间字符串、`null`
|
||||
|
||||
#### Scenario: 每行独立可解析
|
||||
|
||||
- **WHEN** 读取 JSONL 分片的任意一行
|
||||
- **THEN** 该行可以被标准 JSON 解析器独立解析,不依赖前后行
|
||||
|
||||
### Requirement: 压缩
|
||||
|
||||
系统 SHALL 支持 `--compress none|gzip|zstd|xz`,默认 `none`。每个分片 MUST 独立压缩,压缩后的文件扩展名分别为 `.gz`、`.zst` 与 `.xz`。系统 SHALL 支持 `--compress-level 1..9`,默认 3;档位是跨格式统一语义,数值越大压缩率越高、吞吐越低,9 表示该压缩器的最强档位。gzip 与 zstd 在默认档位下的字节输出 MUST 与引入档位前保持一致。xz 分片 MUST 是标准 xz 流,可被通用 xz 工具解码。
|
||||
|
||||
#### Scenario: gzip 分片
|
||||
|
||||
- **WHEN** `--compress gzip`
|
||||
- **THEN** 每个分片是一个独立的 gzip 流,文件名为 `*.gz`
|
||||
|
||||
#### Scenario: zstd 分片
|
||||
|
||||
- **WHEN** `--compress zstd`
|
||||
- **THEN** 每个分片是一个独立的 zstd 帧,文件名为 `*.zst`
|
||||
|
||||
#### Scenario: xz 分片
|
||||
|
||||
- **WHEN** `--compress xz`
|
||||
- **THEN** 每个分片是一个独立的 xz 流,文件名为 `*.xz`
|
||||
|
||||
#### Scenario: xz 分片可被标准工具解码
|
||||
|
||||
- **WHEN** 产物为 `000001.1-1000.dump.xz`
|
||||
- **THEN** `xz -dc` 可将其还原为与该次扫描行内容一致的 dump 文本
|
||||
|
||||
#### Scenario: 默认档位保持既有产物
|
||||
|
||||
- **WHEN** 用户未指定 `--compress-level`
|
||||
- **THEN** gzip 与 zstd 使用档位 3,产物与引入该参数前的版本一致
|
||||
|
||||
#### Scenario: 各档位对三种压缩器均可用
|
||||
|
||||
- **WHEN** 用户指定 `--compress gzip|zstd|xz` 与任意 `--compress-level 1..9`
|
||||
- **THEN** 系统接受配置并产出对应扩展名的独立压缩分片
|
||||
|
||||
### Requirement: 可导入性
|
||||
|
||||
系统 MUST 保证每种格式的产物可被至少一种常见工具直接导入:dump 可被 MySQL `LOAD DATA INFILE` 默认参数导入;csv 可被 PostgreSQL `COPY ... CSV HEADER`、ClickHouse `CSVWithNames`、Pandas/Spark 默认 CSV 读取器导入;jsonl 可被 ClickHouse `JSONEachRow` 及常见流式 JSON 工具导入。
|
||||
|
||||
#### Scenario: 压缩文件导入
|
||||
|
||||
- **WHEN** 产物为 `000001.1-1000.csv.gz`
|
||||
- **THEN** 可以通过 `gunzip -c` 或 `zcat` 管道交给目标数据库导入,无需改写文件内容
|
||||
|
||||
#### Scenario: dump 直接导入 MySQL
|
||||
|
||||
- **WHEN** 使用 dump 格式且不压缩
|
||||
- **THEN** `LOAD DATA INFILE '000001.1-1000.dump' INTO TABLE t` 在列顺序一致时可成功执行
|
||||
@@ -0,0 +1,24 @@
|
||||
param(
|
||||
[string]$Version = "0.1.0",
|
||||
[string]$Commit = $(if ($env:ARCHER_COMMIT) { $env:ARCHER_COMMIT } else { "local" })
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$root = Split-Path -Parent $PSScriptRoot
|
||||
$date = (Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ")
|
||||
$ldflags = "-s -w -X archer/internal/version.Version=$Version -X archer/internal/version.Commit=$Commit -X archer/internal/version.Date=$date"
|
||||
|
||||
New-Item -ItemType Directory -Force -Path (Join-Path $root "dist") | Out-Null
|
||||
Push-Location $root
|
||||
try {
|
||||
go build -trimpath -ldflags $ldflags -o dist/archer_windows_amd64.exe ./cmd/archer
|
||||
$env:GOOS = "linux"
|
||||
$env:GOARCH = "amd64"
|
||||
go build -trimpath -ldflags $ldflags -o dist/archer_linux_amd64 ./cmd/archer
|
||||
Remove-Item Env:GOOS -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:GOARCH -ErrorAction SilentlyContinue
|
||||
Write-Host "built dist/archer_windows_amd64.exe and dist/archer_linux_amd64 ($Version, $Commit)"
|
||||
}
|
||||
finally {
|
||||
Pop-Location
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
version="${VERSION:-0.1.0}"
|
||||
commit="${COMMIT:-local}"
|
||||
date="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
root="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
ldflags="-s -w -X archer/internal/version.Version=${version} -X archer/internal/version.Commit=${commit} -X archer/internal/version.Date=${date}"
|
||||
|
||||
mkdir -p "$root/dist"
|
||||
cd "$root"
|
||||
GOOS=windows GOARCH=amd64 go build -trimpath -ldflags "$ldflags" -o dist/archer_windows_amd64.exe ./cmd/archer
|
||||
GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "$ldflags" -o dist/archer_linux_amd64 ./cmd/archer
|
||||
echo "built dist/archer_windows_amd64.exe and dist/archer_linux_amd64 (${version}, ${commit})"
|
||||
@@ -0,0 +1,213 @@
|
||||
@echo off
|
||||
chcp 65001 >nul 2>&1
|
||||
setlocal EnableDelayedExpansion
|
||||
|
||||
REM =====================================================================
|
||||
REM export-all.cmd
|
||||
REM
|
||||
REM 用 archer 把某个 MySQL 库的「所有表」导出为按行数分片 + gzip 压缩的 dump。
|
||||
REM
|
||||
REM 用法:
|
||||
REM scripts\export-all.cmd
|
||||
REM 使用下面「配置区」的默认值
|
||||
REM
|
||||
REM scripts\export-all.cmd DSN OUTDIR SHARD_ROWS CHUNK_SIZE
|
||||
REM 例: scripts\export-all.cmd ^
|
||||
REM "root:pass@tcp(10.0.0.5:3306)/app" "D:\dump" 500000 5000
|
||||
REM
|
||||
REM 产物:
|
||||
REM <OUTDIR>\<表名>\000001.<lo>-<hi>.dump.gz 分片文件(按主键连续、不重叠)
|
||||
REM <OUTDIR>\<表名>\manifest.jsonl 每片的行数/字节/sha256
|
||||
REM <OUTDIR>\<表名>\state.json 断点续跑状态
|
||||
REM <OUTDIR>\<表名>.log 该表 archer 的完整输出
|
||||
REM
|
||||
REM 断点续跑:
|
||||
REM 用同样的参数再跑一遍即可,archer 从 state.json 继续;
|
||||
REM state.json 里 complete=true 的表会被跳过。
|
||||
REM 想重导某张表,先删掉它的目录。
|
||||
REM
|
||||
REM 依赖:
|
||||
REM dist\archer_windows_amd64.exe (scripts\build.ps1 构建)
|
||||
REM mysql.exe (只用来枚举表名,需在 PATH 里)
|
||||
REM =====================================================================
|
||||
|
||||
REM ------------------------------ 配置区 ------------------------------
|
||||
REM 源库连接。改动源库时,SRC 与下面 5 个变量要一起改
|
||||
REM (mysql 客户端要从后者取参数去枚举表名)。
|
||||
set "SRC=root:xyTyfxhn4h@tcp(s1.void01.com:3306)/lexis"
|
||||
set "DB=lexis"
|
||||
set "DB_HOST=s1.void01.com"
|
||||
set "DB_PORT=3306"
|
||||
set "DB_USER=root"
|
||||
set "DB_PASS=xyTyfxhn4h"
|
||||
|
||||
REM 输出根目录(每张表会在其下建一个同名子目录)
|
||||
set "OUTDIR=%~dp0..\export"
|
||||
|
||||
REM 每个分片文件的行数;小于表行数时只会得到 1 个分片
|
||||
set "SHARD_ROWS=1000000"
|
||||
|
||||
REM 每次 SELECT / 事务 / checkpoint 的行数,与分片大小无关
|
||||
set "CHUNK_SIZE=5000"
|
||||
|
||||
REM 输出格式 dump|csv|jsonl 与压缩方式 none|gzip|zstd
|
||||
set "FMT=dump"
|
||||
set "COMPRESS=gzip"
|
||||
|
||||
REM 只导指定表(逗号分隔);留空表示导全部
|
||||
set "ONLY_TABLES="
|
||||
|
||||
REM 排除表名通配符,如 tmp_* REM_*;留空表示不排除
|
||||
set "EXCLUDE="
|
||||
REM --------------------------------------------------------------------
|
||||
|
||||
REM 命令行覆盖
|
||||
if not "%~1"=="" set "SRC=%~1"
|
||||
if not "%~2"=="" set "OUTDIR=%~2"
|
||||
if not "%~3"=="" set "SHARD_ROWS=%~3"
|
||||
if not "%~4"=="" set "CHUNK_SIZE=%~4"
|
||||
|
||||
set "EXE=%~dp0..\dist\archer_windows_amd64.exe"
|
||||
if not exist "%EXE%" (
|
||||
echo [错误] 找不到 archer: %EXE%
|
||||
echo 先运行 scripts\build.ps1 构建
|
||||
goto :fail
|
||||
)
|
||||
|
||||
where mysql.exe >nul 2>&1
|
||||
if errorlevel 1 (
|
||||
echo [错误] PATH 中找不到 mysql.exe,无法枚举表名
|
||||
echo 把 mysql 的 bin 目录加进 PATH,或用 ARCHER 逐表手写命令
|
||||
goto :fail
|
||||
)
|
||||
|
||||
if not exist "%OUTDIR%" mkdir "%OUTDIR%" >nul 2>&1
|
||||
if not exist "%OUTDIR%" (
|
||||
echo [错误] 无法创建输出目录: %OUTDIR%
|
||||
goto :fail
|
||||
)
|
||||
|
||||
REM mysql 用环境变量传密码,避免出现在命令行/进程列表里
|
||||
set "MYSQL_PWD=%DB_PASS%"
|
||||
set "MYSQL_CMD=mysql.exe --protocol=TCP -h%DB_HOST% -P%DB_PORT% -u%DB_USER% --default-character-set=utf8mb4 -N -B"
|
||||
|
||||
echo.
|
||||
echo archer : %EXE%
|
||||
echo schema : %DB% (%DB_HOST%:%DB_PORT%)
|
||||
echo out dir : %OUTDIR%
|
||||
echo shard : %FMT% + %COMPRESS%, %SHARD_ROWS% rows/shard, chunk %CHUNK_SIZE%
|
||||
echo.
|
||||
|
||||
REM ---------------------------- 表清单 ----------------------------
|
||||
if defined ONLY_TABLES (
|
||||
set "TABLE_LIST=%ONLY_TABLES:,= %"
|
||||
set "TABLE_LIST=!TABLE_LIST: =!|"
|
||||
) else (
|
||||
set "TABLE_LIST="
|
||||
for /f "usebackq delims=" %%t in (`%MYSQL_CMD% -e "SELECT table_name FROM information_schema.tables WHERE table_schema='%DB%' AND table_type='BASE TABLE' ORDER BY table_name" 2^>nul`) do (
|
||||
set "TABLE_LIST=!TABLE_LIST!%%t|"
|
||||
)
|
||||
)
|
||||
|
||||
if "!TABLE_LIST!"=="" (
|
||||
echo [错误] 没有取到任何表,检查库名/账号/网络
|
||||
goto :fail
|
||||
)
|
||||
|
||||
REM ---------------------------- 逐表导出 ----------------------------
|
||||
set "OK=0"
|
||||
set "SKIP=0"
|
||||
set "FAILED="
|
||||
set /a IDX=0
|
||||
|
||||
for %%t in ("!TABLE_LIST:|=" "!") do (
|
||||
set "T=%%~t"
|
||||
if not "!T!"=="" (
|
||||
call :one_table "!T!"
|
||||
)
|
||||
)
|
||||
|
||||
echo.
|
||||
if defined FAILED (
|
||||
echo 失败的表: %FAILED%
|
||||
echo.
|
||||
echo 完成: 成功 %OK%,跳过 %SKIP%,失败若干
|
||||
goto :fail
|
||||
)
|
||||
echo 完成: 成功 %OK%,跳过 %SKIP%,失败 0
|
||||
endlocal
|
||||
exit /b 0
|
||||
|
||||
REM ====================================================================
|
||||
REM 单表导出
|
||||
REM ====================================================================
|
||||
:one_table
|
||||
set "T=%~1"
|
||||
set /a IDX+=1
|
||||
|
||||
if defined EXCLUDE (
|
||||
REM 简单通配:把 * 替换成批处理的 .*
|
||||
set "PAT=!EXCLUDE:*=.!"
|
||||
for %%p in (!PAT!) do (
|
||||
for /f "delims=" %%m in ("!T!") do (
|
||||
if "%%m"=="%%m" (
|
||||
REM 用 findstr 做通配匹配
|
||||
)
|
||||
)
|
||||
)
|
||||
echo !T! | findstr /r /i /c:"^!EXCLUDE:\*=.!$" >nul 2>&1
|
||||
if not errorlevel 1 (
|
||||
echo [!IDX!] !T! 跳过(被 EXCLUDE 匹配)
|
||||
goto :eof
|
||||
)
|
||||
)
|
||||
|
||||
set "TDIR=%OUTDIR%\!T!"
|
||||
|
||||
REM 已完成则跳过
|
||||
if exist "!TDIR!\state.json" (
|
||||
findstr /c:"\"complete\": true" "!TDIR!\state.json" >nul 2>&1
|
||||
if not errorlevel 1 (
|
||||
echo [!IDX!] !T! 跳过(已完成)
|
||||
set /a SKIP+=1
|
||||
goto :eof
|
||||
)
|
||||
echo [!IDX!] !T! 续跑
|
||||
)
|
||||
|
||||
set "LOG=%OUTDIR%\!T!.log"
|
||||
set "START=%TIME%"
|
||||
|
||||
"%EXE%" ^
|
||||
--source "%SRC%" ^
|
||||
--table "!T!" ^
|
||||
--where "1=1" ^
|
||||
--file "!TDIR!" ^
|
||||
--format %FMT% ^
|
||||
--compress %COMPRESS% ^
|
||||
--shard-by count ^
|
||||
--shard-rows %SHARD_ROWS% ^
|
||||
--chunk-size %CHUNK_SIZE% ^
|
||||
--statistics > "!LOG!" 2>&1
|
||||
|
||||
if errorlevel 1 (
|
||||
echo [!IDX!] !T! 失败,日志: !LOG!
|
||||
for /f "delims=" %%l in ('findstr /c:"archer:" "!LOG!" 2^>nul') do echo %%l
|
||||
set "FAILED=!FAILED! !T!"
|
||||
goto :eof
|
||||
)
|
||||
|
||||
REM 统计分片数
|
||||
set "SHARDS=0"
|
||||
if exist "!TDIR!\manifest.jsonl" (
|
||||
for /f %%n in ('type "!TDIR!\manifest.jsonl" 2^>nul ^| find /c /v ""') do set "SHARDS=%%n"
|
||||
)
|
||||
|
||||
echo [!IDX!] !T! 完成: !SHARDS! 片
|
||||
set /a OK+=1
|
||||
goto :eof
|
||||
|
||||
REM ====================================================================
|
||||
:fail
|
||||
endlocal
|
||||
exit /b 1
|
||||
@@ -0,0 +1,336 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
用 archer 导出某个 MySQL 库中的全部表,按行数分片并压缩。
|
||||
|
||||
.DESCRIPTION
|
||||
自动遍历 information_schema 里指定库的所有基础表,逐表调用 archer:
|
||||
--shard-by count --format dump --compress gzip
|
||||
每张表输出到独立目录 <OutDir>\<Schema>\<table>,因为 archer 的
|
||||
manifest.jsonl / state.json 是「按输出目录」维护的,混在一起会互相覆盖。
|
||||
|
||||
断点续跑:archer 自己会在同目录 + 同配置下续跑,脚本默认跳过
|
||||
state.json 中 complete=true 的表。需要重导某张表时用 -Restart。
|
||||
|
||||
.PARAMETER Source
|
||||
源库 DSN,形如 user:pass@tcp(host:3306)/db。默认取环境变量 ARCHER_SOURCE。
|
||||
|
||||
.PARAMETER Schema
|
||||
要导出的库名。默认取 DSN 中的库名。
|
||||
|
||||
.PARAMETER OutDir
|
||||
输出根目录。默认 <仓库根>\export。
|
||||
|
||||
.PARAMETER ShardRows
|
||||
每个分片文件的行数,默认 1000000。表行数小于该值时只有 1 个分片。
|
||||
|
||||
.PARAMETER ChunkSize
|
||||
每次 SELECT / 事务 / checkpoint 的行数,默认 5000。
|
||||
|
||||
.PARAMETER Format
|
||||
分片格式 dump|csv|jsonl,默认 dump(LOAD DATA 兼容 TSV)。
|
||||
|
||||
.PARAMETER Compress
|
||||
压缩算法 none|gzip|zstd,默认 gzip。
|
||||
|
||||
.PARAMETER Tables
|
||||
只导这些表(表名数组),默认全会话库内所有基础表。
|
||||
|
||||
.PARAMETER Exclude
|
||||
排除的表名通配符,如 'tmp_*'。
|
||||
|
||||
.PARAMETER Retries
|
||||
单表失败后的重试次数,默认 3(0 表示不重试)。archer 的 state.json 会续跑,
|
||||
因此重试是安全的:Windows 上偶发 state.json rename "Access is denied"
|
||||
(杀毒/索引器短暂占用句柄),重试一次即恢复。
|
||||
|
||||
.PARAMETER Restart
|
||||
删除已存在的表输出目录后重新导出(默认会跳过已完成 / 续跑未完成的表)。
|
||||
|
||||
.PARAMETER DryRun
|
||||
只打印将要执行的 archer 命令,不实际执行。
|
||||
|
||||
.PARAMETER Verify
|
||||
导出后逐表比对 state.json 的 rows_done 与源表 count(*)。
|
||||
注意 rows_done 现在是累计值(含续跑前已提交的行数),因此续跑过的表也能与源表
|
||||
正确对齐;升级前的旧 checkpoint 只记录上次运行行数,首次校验可能仍报一次差异,
|
||||
重新完成后即恢复。
|
||||
|
||||
.PARAMETER ProgressMode
|
||||
archer 的 --progress 模式:auto|on|off,默认 auto。脚本把 stdout+stderr 重定向进
|
||||
<OutDir>\<Schema>.<table>.log,重定向时 auto 不输出进度;设为 on 可让日志每秒
|
||||
追加一行完整进度,便于 Get-Content -Wait / tail 观察长任务。
|
||||
|
||||
.EXAMPLE
|
||||
.\export-all.ps1 -Source 'root:pass@tcp(s1.void01.com:3306)/lexis'
|
||||
|
||||
.EXAMPLE
|
||||
.\export-all.ps1 -Source 'root:pass@tcp(s1.void01.com:3306)/lexis' `
|
||||
-OutDir D:\dump -ShardRows 250000 -Verify
|
||||
|
||||
.EXAMPLE
|
||||
.\export-all.ps1 -Source 'root:pass@tcp(s1.void01.com:3306)/lexis' `
|
||||
-ProgressMode on
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Source = $env:ARCHER_SOURCE,
|
||||
[string]$Schema = '',
|
||||
[string]$OutDir = '',
|
||||
[int]$ShardRows = 1000000,
|
||||
[int]$ChunkSize = 5000,
|
||||
[ValidateSet('dump', 'csv', 'jsonl')]
|
||||
[string]$Format = 'dump',
|
||||
[ValidateSet('none', 'gzip', 'zstd')]
|
||||
[string]$Compress = 'gzip',
|
||||
[string[]]$Tables = @(),
|
||||
[string]$Exclude = '',
|
||||
[string]$Archer = '',
|
||||
[string]$Mysql = '',
|
||||
[int]$Retries = 3,
|
||||
[ValidateSet('auto', 'on', 'off')]
|
||||
[string]$ProgressMode = 'auto',
|
||||
[switch]$Restart,
|
||||
[switch]$DryRun,
|
||||
[switch]$Verify
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 路径与可执行文件
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
$root = Split-Path -Parent $PSScriptRoot
|
||||
|
||||
if (-not $Archer) {
|
||||
$candidates = @(
|
||||
(Join-Path $root 'dist\archer_windows_amd64.exe'),
|
||||
(Join-Path $root 'dist\archer.exe')
|
||||
)
|
||||
$Archer = $candidates | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $Archer) {
|
||||
$cmd = Get-Command archer.exe -ErrorAction SilentlyContinue
|
||||
if ($cmd) { $Archer = $cmd.Source }
|
||||
}
|
||||
}
|
||||
if (-not $Archer -or -not (Test-Path $Archer)) {
|
||||
throw "找不到 archer 可执行文件,请先运行 scripts\build.ps1,或用 -Archer 指定路径"
|
||||
}
|
||||
|
||||
if (-not $Mysql) {
|
||||
$cmd = Get-Command mysql.exe -ErrorAction SilentlyContinue
|
||||
if ($cmd) { $Mysql = $cmd.Source }
|
||||
elseif (Test-Path 'C:\portable-apps\mysql-8.3.0-winx64\bin\mysql.exe') {
|
||||
$Mysql = 'C:\portable-apps\mysql-8.3.0-winx64\bin\mysql.exe'
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $Source) {
|
||||
throw "必须提供 -Source 'user:pass@tcp(host:3306)/db'(或设置环境变量 ARCHER_SOURCE)"
|
||||
}
|
||||
|
||||
# user:pass@tcp(host:port)/db
|
||||
$m = [regex]::Match($Source, '^(?<user>[^:@/]*)(:(?<pass>[^@]*))?@tcp\((?<host>[^:)]+)(:(?<port>\d+))?\)/(?<db>[^?]*)')
|
||||
if (-not $m.Success) {
|
||||
throw "无法解析 DSN,期望形如 user:pass@tcp(host:3306)/db,实际: $Source"
|
||||
}
|
||||
$dbUser = $m.Groups['user'].Value
|
||||
$dbPass = $m.Groups['pass'].Value
|
||||
$dbHost = $m.Groups['host'].Value
|
||||
$dbPort = if ($m.Groups['port'].Success) { $m.Groups['port'].Value } else { '3306' }
|
||||
if (-not $Schema) { $Schema = $m.Groups['db'].Value }
|
||||
if (-not $Schema) { throw "DSN 中没有库名,且未提供 -Schema" }
|
||||
|
||||
if (-not $OutDir) { $OutDir = Join-Path $root 'export' }
|
||||
$OutDir = [System.IO.Path]::GetFullPath($OutDir)
|
||||
$schemaDir = Join-Path $OutDir $Schema
|
||||
# 日志写在 $OutDir 下,先确保它存在
|
||||
New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# mysql 客户端查询(只用于列举表名与校验行数)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
function Invoke-MysqlQuery {
|
||||
param([Parameter(Mandatory)][string]$Sql)
|
||||
|
||||
if (-not $Mysql -or -not (Test-Path $Mysql)) {
|
||||
throw "找不到 mysql.exe,无法查询 information_schema;请用 -Tables 显式列出表名,或用 -Mysql 指定路径"
|
||||
}
|
||||
$prev = $env:MYSQL_PWD
|
||||
$env:MYSQL_PWD = $dbPass
|
||||
try {
|
||||
$lines = & $Mysql --protocol=TCP "--host=$dbHost" "--port=$dbPort" `
|
||||
"--user=$dbUser" --default-character-set=utf8mb4 -N -B -e $Sql 2>&1
|
||||
$code = $LASTEXITCODE
|
||||
}
|
||||
finally {
|
||||
$env:MYSQL_PWD = $prev
|
||||
}
|
||||
if ($code -ne 0) {
|
||||
throw "mysql 查询失败(exit $code): $($lines -join ' ')"
|
||||
}
|
||||
# 过滤 mysql 客户端自身的 [Warning] 行
|
||||
return @($lines | Where-Object { "$_" -notmatch '^mysql: \[Warning\]' })
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 收集表清单
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# -File 调用时 -Tables a,b 会作为一个字符串传入,这里按逗号拆开
|
||||
$Tables = @($Tables | ForEach-Object { $_ -split ',' } | ForEach-Object { $_.Trim() } | Where-Object { $_ })
|
||||
|
||||
if ($Tables.Count -gt 0) {
|
||||
$tables = $Tables
|
||||
}
|
||||
else {
|
||||
if ($DryRun) { Write-Host "解析表清单..." }
|
||||
$tables = Invoke-MysqlQuery ("SELECT table_name FROM information_schema.tables " +
|
||||
"WHERE table_schema = '$Schema' AND table_type = 'BASE TABLE' ORDER BY table_name")
|
||||
}
|
||||
if ($Exclude) {
|
||||
$tables = $tables | Where-Object { $_ -notlike $Exclude }
|
||||
}
|
||||
$tables = @($tables | Where-Object { $_ })
|
||||
if ($tables.Count -eq 0) { throw "库 $Schema 中没有匹配的表" }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主流程
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Write-Host ''
|
||||
Write-Host "archer : $Archer"
|
||||
Write-Host "schema : $Schema ($dbHost`:$dbPort)"
|
||||
Write-Host "out dir : $schemaDir"
|
||||
Write-Host "shard : count / $ShardRows rows, chunk $ChunkSize, $Format + $Compress"
|
||||
Write-Host "tables : $($tables.Count)"
|
||||
Write-Host ''
|
||||
|
||||
$failed = @()
|
||||
$skipped = @()
|
||||
$ok = @()
|
||||
$idx = 0
|
||||
|
||||
foreach ($table in $tables) {
|
||||
$idx++
|
||||
$prefix = "[$idx/$($tables.Count)] $table"
|
||||
$tableDir = Join-Path $schemaDir $table
|
||||
$stateFile = Join-Path $tableDir 'state.json'
|
||||
|
||||
# 已完成则跳过(archer 对 complete=true 的目录会直接退出,这里提前判断避免无谓启动)
|
||||
if (-not $Restart -and (Test-Path $stateFile)) {
|
||||
try {
|
||||
$state = Get-Content -Raw -Path $stateFile | ConvertFrom-Json
|
||||
}
|
||||
catch { $state = $null }
|
||||
if ($state -and $state.complete) {
|
||||
Write-Host "$prefix 跳过(已完成,rows_done=$($state.rows_done))"
|
||||
$skipped += $table
|
||||
continue
|
||||
}
|
||||
Write-Host "$prefix 续跑(rows_done=$($state.rows_done))"
|
||||
}
|
||||
|
||||
if ($Restart -and (Test-Path $tableDir)) {
|
||||
Write-Host "$prefix -Restart:删除 $tableDir"
|
||||
Remove-Item -Recurse -Force $tableDir
|
||||
}
|
||||
|
||||
$argv = @(
|
||||
'--source', $Source,
|
||||
'--table', $table,
|
||||
'--where', '1=1',
|
||||
'--file', $tableDir,
|
||||
'--format', $Format,
|
||||
'--compress', $Compress,
|
||||
'--shard-by', 'count',
|
||||
'--shard-rows', "$ShardRows",
|
||||
'--chunk-size', "$ChunkSize",
|
||||
'--progress', $ProgressMode,
|
||||
'--statistics'
|
||||
)
|
||||
|
||||
if ($DryRun) {
|
||||
Write-Host "$prefix"
|
||||
Write-Host (" " + $Archer + " " + ($argv -join ' '))
|
||||
$ok += $table
|
||||
continue
|
||||
}
|
||||
|
||||
$log = Join-Path $OutDir "$Schema.$table.log"
|
||||
$sw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
# archer 会往 stderr 打提示;PowerShell 在 EAP=Stop 下会把原生 stderr 当成终止性错误,
|
||||
# 这里临时降级为 Continue,并把 stdout+stderr 一起落日志。
|
||||
$prevEap = $ErrorActionPreference
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$code = 0
|
||||
try {
|
||||
$attempt = 0
|
||||
while ($true) {
|
||||
$attempt++
|
||||
& $Archer @argv 2>&1 | Out-File -FilePath $log -Encoding utf8
|
||||
$code = $LASTEXITCODE
|
||||
if ($code -eq 0 -or $attempt -gt $Retries) { break }
|
||||
Write-Host "$prefix 第 $attempt 次失败(exit $code),重试中(archer 自动续跑)..." -ForegroundColor Yellow
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
}
|
||||
finally {
|
||||
$ErrorActionPreference = $prevEap
|
||||
}
|
||||
$sw.Stop()
|
||||
|
||||
if ($code -ne 0) {
|
||||
Write-Host "$prefix 失败(exit $code,耗时 $([int]$sw.Elapsed.TotalSeconds)s)—— 见 $log" -ForegroundColor Red
|
||||
Get-Content $log -Tail 5 | ForEach-Object { Write-Host " $_" }
|
||||
$failed += [pscustomobject]@{ Table = $table; Code = $code; Log = $log }
|
||||
continue
|
||||
}
|
||||
|
||||
# 汇总本次分片情况
|
||||
$manifest = Join-Path $tableDir 'manifest.jsonl'
|
||||
$shards = 0; $rows = 0; $bytes = 0
|
||||
if (Test-Path $manifest) {
|
||||
foreach ($line in Get-Content $manifest) {
|
||||
if (-not $line.Trim()) { continue }
|
||||
$e = $line | ConvertFrom-Json
|
||||
$shards++
|
||||
$rows += [int64]$e.rows
|
||||
$bytes += [int64]$e.bytes
|
||||
}
|
||||
}
|
||||
Write-Host ("$prefix 完成:{0} 分片, {1} 行, {2:N1} MB, {3}s" -f `
|
||||
$shards, $rows, ($bytes / 1MB), [int]$sw.Elapsed.TotalSeconds)
|
||||
$ok += $table
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
if ($Verify -and -not $DryRun -and $ok.Count -gt 0) {
|
||||
Write-Host ''
|
||||
Write-Host '== 行数校验 =='
|
||||
foreach ($table in $ok) {
|
||||
$stateFile = Join-Path (Join-Path $schemaDir $table) 'state.json'
|
||||
if (-not (Test-Path $stateFile)) { continue }
|
||||
$state = Get-Content -Raw -Path $stateFile | ConvertFrom-Json
|
||||
$want = [int64]$state.rows_done
|
||||
$got = [int64](Invoke-MysqlQuery "SELECT COUNT(*) FROM ``$Schema``.``$table``" | Select-Object -First 1)
|
||||
if ($want -eq $got) {
|
||||
Write-Host (" {0,-34} {1,10} OK" -f $table, $got)
|
||||
}
|
||||
else {
|
||||
Write-Host (" {0,-34} 导出 {1} / 源 {2} <== DIFF" -f $table, $want, $got) -ForegroundColor Yellow
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host ''
|
||||
Write-Host ("完成:成功 {0},跳过 {1},失败 {2};输出 $schemaDir" -f $ok.Count, $skipped.Count, $failed.Count)
|
||||
if ($failed.Count -gt 0) {
|
||||
$failed | ForEach-Object { Write-Host (" 失败 {0} (exit {1}) 日志 {2}" -f $_.Table, $_.Code, $_.Log) }
|
||||
exit 1
|
||||
}
|
||||
exit 0
|
||||
Reference in new issue
Block a user