terrylica/cc-skills

tick-collection-ops

Operate and troubleshoot the MT5 tick collection system.

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/terrylica/cc-skills --skill tick-collection-ops

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

MT5 Tick Collection Operations

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.
## ⚠️ HOST SUPERSEDED — 2026-08-07 MetaTrader 5 no longer runs on <gpu-host>. It runs exclusively on a broker VPS (Tailscale host mt5-host, <tailscale-ip>, Windows), on a demo account. The Mac-local Wine bench was deleted 2026-08-06; <gpu-host>'s Wine terminal + its xvfb/xfce/x11vnc desktop and fxview-sidecar are being decommissioned. On the VPS the topology is Windows scheduled tasks, not Linux systemd: | Concern | <gpu-host> (retired) | Broker VPS (current) | | --------------- | --------------------------------------- | ---------------------------------------------------------------- | | Terminal | mt5.service under Wine, DISPLAY=:99 | terminal64.exe, native Windows, real session | | Tick rings | /dev/shm/tick_ring_<SYM> | C:\odb-tickrings\tick_ring_<SYM> | | Ring → bus | — | odb-tapegateway.exe → task OdbTickRingToNatsTapeGateway | | Bus | — | nats-server.exe :4222 → task OdbTapeBusNatsJetStreamServer | | Orders | — | order-service :50060 → task ODB-MT5-OrderService | | Symbols | EUR/GBP/XAU/XAG | EUR/GBP/XAU only (silver excluded per directive 2026-05-26) | | Service control | systemctl | Get-ScheduledTask / Start-ScheduledTask | Reach it with ssh <mt5-host>. PowerShell over SSH eats $_ under shell quoting — pass commands base64-encoded via powershell -NoProfile -EncodedCommand <base64-utf16le>. Everything below still describes the Linux/Wine deployment. It is retained because the EA, the tick_writer.dll ring format and the gap-detection reasoning are unchanged and still authoritative — only the HOST and the service manager moved. Do not use it to stand up a new Linux MT5. Canonical: <research-repo>/docs/adr/<date>-retire-<host>-as-a-metatrader-host.md and docs/decision-register-<topic>.md.

Operate, monitor, and troubleshoot the zero-gap tick collection system running on Linux via Wine.

When to Use This Skill

Use this skill when:

  • Deploying or restarting tick collection on <gpu-host>
  • Diagnosing tick gaps or missing data
  • Troubleshooting Wine or MT5 issues
  • Understanding systemd service topology
  • Investigating crash recovery files
  • Checking tick collection health

System Architecture

TickCollector EA (MQL5)  -->  tick_writer.dll (Rust)  -->  Parquet files
     OnTick() callback         C-ABI FFI                   ZSTD compressed
     CopyTicksRange()          Buffered writes              One file per day
     Watermark dedup            Row group flush
  • TickCollector EA runs on MT5 via Wine on headless Linux (<gpu-host>)
  • EA captures every tick via OnTick() callback + OnTimer() fallback (1-second interval)
  • Ticks buffered in EA, flushed to Rust DLL (tick_writer.dll) via C-ABI FFI
  • Rust DLL writes Parquet files with ZSTD level 3 compression
  • One EA instance per symbol per chart -- this is a CRITICAL constraint (single symbol per EA)

Production Topology

ResourcePath
Git repo~/eon/mql5/
Wine prefix~/.mt5/
MT5 install~/.mt5/drive_c/Program Files/MetaTrader 5/
EA source{repo}/mql5_ea/TickCollector.mq5
DLL (Windows){mt5}/MQL5/Libraries/tick_writer.dll
Tick data{mt5}/tick_data/ (symlinked to ODB cache)
Systemd unitssystemd user units: xvfb, xfce, x11vnc, mt5

Systemd Service Chain

xvfb (virtual framebuffer)
  --> xfce (desktop environment)
    --> x11vnc (VNC access)
      --> mt5 (MetaTrader 5 via Wine)

All are user-level systemd units. MT5 depends on the display stack. The chain must start in order -- xvfb first, mt5 last.

Daily File Rotation

  • New Parquet file created each trading day
  • Filename pattern: {SYMBOL}_{YYYYMMDD}.parquet
  • EA detects day change and calls tw_close (finalize current file) + tw_init (open new file)
  • Trading day runs Sunday open to Monday open (forex hours)

Crash Recovery

On EA restart:

  1. EA calls tw_get_last_time_msc to read resume watermark from the last Parquet file footer
  2. Watermark is the timestamp of the last tick written -- resume from there
  3. If an existing file is found for today, new file gets _1, _2 suffix (not overwrite)
  4. All recovery segments are valid, complete Parquet files with footers
  5. Resume watermark prevents duplicate ticks after restart
  6. Gap between shutdown and restart is backfilled via CopyTicksRange from the watermark

The Rust DLL renames corrupt/incomplete files to _partial suffix rather than attempting recovery.

Gap Detection

Use DuckDB to check for gaps between consecutive ticks:

sql
WITH ticks AS (
    SELECT time_msc,
           lead(time_msc) OVER (ORDER BY time_msc) - time_msc AS gap_ms
    FROM read_parquet('{base_path}/FXVIEW_{SYMBOL}/{YYYY}/{SYMBOL}_{YYYYMMDD}.parquet')
)
SELECT time_msc, gap_ms,
       make_timestamp(time_msc * 1000) AS ts
FROM ticks
WHERE gap_ms > 60000  -- gaps > 1 minute
ORDER BY gap_ms DESC;

Interpreting results:

  • Weekend gaps are normal (Friday close to Sunday open)
  • Mid-session gaps indicate collection interruption
  • Gaps < 1 minute are normal during low-activity periods
  • Check systemd journal for correlating restart events

Wine Gotchas

These are CRITICAL anti-patterns -- violations cause silent crashes or data loss:

  • NEVER use `%I64u` or `%I64d` format specifiers in MQL5 -- crashes Wine silently. Use IntegerToString() instead.
  • Wine cannot follow symlinks for file access -- use physical file copies for EA and DLL deployment
  • Headless Linux requires Xvfb virtual framebuffer -- MT5 GUI needs a display even when no monitor is connected
  • compile.sh uses Wine CLI for compilation -- no MetaEditor GUI needed
  • MetaEditor returns exit code 1 even on success under Wine -- check compile log instead
  • Compile log is UTF-16LE -- use iconv -f UTF-16LE -t UTF-8 to read it
  • `DISPLAY=:99` must be set (Xvfb virtual display number)

Deployment Pipeline

Use /mql5:mql5-ship slash command for full deployment:

  • --ea-only for EA source changes
  • --dll-only for Rust DLL changes
  • Full pipeline: commit -> push -> pull on <gpu-host> -> compile -> validate

Detailed deployment steps (SSH commands, file copies, compilation) are in the headless-mt5-remote local skill. Not duplicated here to avoid drift.

Monitoring Commands

Check MT5 service status

bash
systemctl --user status mt5

Check today's tick files

bash
ls -la ~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/$(date +%Y)/

Count ticks in latest file

bash
duckdb -c "SELECT count(*) FROM read_parquet('path/to/file.parquet')"

Check for recovery segments

bash
ls ~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/$(date +%Y)/*_[0-9].parquet 2>/dev/null

View systemd service chain

bash
systemctl --user list-units --type=service | grep -E "xvfb|xfce|x11vnc|mt5"

Check MT5 journal for errors

bash
journalctl --user -u mt5 --since "1 hour ago" --no-pager

Scope Boundary

  • This skill: Operations, monitoring, troubleshooting
  • `headless-mt5-remote` local skill: Deployment steps (SSH, file copy, compilation)
  • `fxview-parquet-consumer` skill: Data consumption patterns (schema, DuckDB queries)

Troubleshooting

IssueCauseSolution
MT5 service won't startDisplay stack not runningStart xvfb first, then xfce, x11vnc, mt5
EA not collecting ticksSymbol not in Market WatchOpen chart for symbol in MT5, re-attach EA
Wine crash on PrintFormatUsing %I64u/%I64d specifiersReplace with IntegerToString()
DLL load failureMissing tick_writer.dllCopy DLL to MQL5/Libraries/ (physical copy)
Gaps in tick dataEA restart or network issueCheck journalctl, verify recovery segments exist
Recovery file \12Normal after restartAll segments are valid, union in DuckDB
Compile log unreadableUTF-16LE encodingUse iconv -f UTF-16LE -t UTF-8
Xvfb not runningService crashed or not enabledsystemctl --user start xvfb

Post-Execution Reflection

After this skill completes, check before closing:

  1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
  2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
  3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

aus demselben Repository

Weitere Skills

Alle Skills
terrylica
Community

academic-pdf-to-gfm

Convert academic PDF papers to GitHub-renderable GFM markdown with math equations. TRIGGERS - PDF, GitHub markdown, math

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

adr-code-traceability

Add ADR references to code for traceability. TRIGGERS - ADR traceability, code reference, document decision in code.

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

arxiv-source-first-paper-ingest

Read an arXiv paper faithfully by fetching the authors' own LaTeX source instead of OCR-ing the PDF. Extracts every display equation with its label, environment and enclosing proposition, renders each to MathML, and proves usability by compiling each formula with a real TeX engine. Use when you need a paper's formulas exactly — to replicate a method, implement a loss function or custom objective, verify a quoted number, or check a claim against what the authors actually wrote. TRIGGERS - read th

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

azure-provision

Bootstrap Azure automation when interactive az login is blocked by a Conditional Access policy (error 530035, "Device state: Unregistered"). Mints a service principal through the Azure Portal session ONCE — using Microsoft Graph over a captured token, not DOM clicking — then provisions resource groups, Cognitive Services / AI Language / Speech resources and their keys entirely via az/REST, writing key AND endpoint together into the SCS vault. Use when az login fails but the portal works, when mo

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.