terrylica/cc-skills

arxiv-source-first-paper-ingest

Read an arXiv paper faithfully by fetching the authors' own LaTeX source instead of OCR-ing the PDF.

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/terrylica/cc-skills --skill arxiv-source-first-paper-ingest

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

arXiv source-first paper ingest

Self-Evolving skill. If a step here fails or the tooling drifts, fix this file in the same change. The Post-Execution Reflection at the bottom says what to check.

The whole idea in one line: for an arXiv paper, the PDF is a rendering of something you can get losslessly, so do not read the rendering.

arxiv.org/e-print/<id> serves the authors' own LaTeX. It carries the exact formulas, their \labels, and — critically — which formulas are stated propositions rather than intermediate steps inside a proof. None of that survives into a PDF, so no vision model can recover it.

Measured on one real paper, OCR of the PDF reached 0.958 mean token similarity to the source and only 49 % of its formulas compiled, against 98 % for the source. Full numbers, including a systematic model defect that similarity cannot see, in `../../references/OCR-VERSUS-AUTHOR-LATEX-GROUND-TRUTH.md`.


Step 1 — fetch the source, not the PDF

Identify yourself; arXiv asks for it and rate-limits anonymous bulk access.

bash
ID=2605.00501
UA="your-project/1.0 (mailto:you@example.com)"
mkdir -p /tmp/paper && cd /tmp/paper
curl -sSL -A "$UA" "https://export.arxiv.org/api/query?id_list=$ID" -o meta.xml
curl -sSL -A "$UA" "https://arxiv.org/e-print/$ID" -o eprint.tar.gz
mkdir -p src && tar xzf eprint.tar.gz -C src

If tar fails the e-print may be a single gzipped .tex — gunzip -c eprint.tar.gz > src/main.tex. Papers withdrawn or submitted as PDF-only have no source; that is when you OCR, and the unlimited-ocr plugin is the tool.

Step 2 — extract every display equation, with the structure that gives it authority

The tools live beside this file, under tools/ in this skill directory.

bash
SKILL_DIR="$(dirname "$0")"   # or the directory containing this SKILL.md
cargo run --release \
  --manifest-path "$SKILL_DIR/tools/arxiv-latex-display-math-extractor/Cargo.toml" \
  -- /tmp/paper/src > /tmp/paper/equations.json

Each record carries label, environment_name, enclosing_theorem_environment, enclosing_theorem_title, is_boxed_by_authors, the verbatim LaTeX, and MathML.

Read `enclosing_theorem_environment` before you trust a formula. A formula inside proposition is a stated result; the same-looking formula inside proof is a step in an argument. Implementing the second as though it were the first is a silent error. is_boxed_by_authors is a strong signal too — authors box the result they want you to take away.

Step 3 — prove the formulas are usable, not merely similar-looking

bash
bun run "$SKILL_DIR/tools/compile_each_formula_with_real_tex_engine.ts"

Requires TeX Live. Every formula compiles as its own minimal document, so one failure cannot cascade.

Do this even when the source is your only input. It catches author macros your extraction context lacks, and it is the only check that answers "can I use this" rather than "does this look right".


Why not the obvious alternatives

ApproachWhy it loses to the source
OCR the PDF with a vision modelLossy by construction. Measured 49 % compile rate, and it invents fused commands like \neqi that read plausibly.
pdftotext / PyMuPDF text layerCannot express mathematics at all — a displayed equation is glyph positioning, not a string. Returns broken fragments.
Ask a language model to recall the paperNo provenance, and the failure mode is confident fabrication of specific numbers.
ar5iv.org/abs/<id>Genuinely good, and it is LaTeXML over this same source. Use it to READ; use this to EXTRACT machine-readable records.

Optional: whole-document conversion

For prose as well as formulas, latexml (the converter behind arXiv's own HTML) produces semantic MathML, and pandoc -f latex -t html --mathml is far faster and usually sufficient. Both are complements to step 2, not replacements: neither emits one record per equation with its proposition context.


Post-Execution Reflection

After this skill completes, check before closing:

  1. Did the e-print unpack? If arXiv served a shape not handled above (single .tex, PDF-only,

withdrawn), record the shape and the branch taken here so the next run does not rediscover it.

  1. Did any formula fail to compile? An author macro means the extraction context needs that

preamble; a genuine failure in source LaTeX is rare and worth noting in the reference document.

  1. Did you fall back to OCR? Record why the source was unavailable — that list is the real

boundary between this skill and unlimited-ocr.

同じリポジトリから

関連する Skills

すべての Skills