security-talent/bugskill-ai

crawl

Deep web crawling using hakrawler and gospider for subdomain discovery, endpoint extraction, and JavaScript analysis.

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/security-talent/bugskill-ai --skill crawl

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

Web Crawling Skill

Overview

This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.

When to Use This Skill

Use this skill when users request:

  • Crawling or spidering a URL or domain
  • Subdomain discovery and enumeration
  • Endpoint and API discovery
  • JavaScript file extraction and analysis
  • Web application reconnaissance or mapping
  • Bug bounty scoping and enumeration
  • Attack surface discovery

Trigger keywords: crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance

Quick Start

The primary tool is scripts/crawl.py, which provides a unified interface to both hakrawler and gospider:

bash
# Basic crawl - creates crawl_example.com_TIMESTAMP/
python scripts/crawl.py https://example.com

# Deep crawl with subdomain discovery
python scripts/crawl.py https://example.com --depth 3 --subdomains

# Crawl with JS analysis
python scripts/crawl.py https://example.com --js-analysis

# Use only one tool
python scripts/crawl.py https://example.com --tool hakrawler

# Custom output directory and timeout
python scripts/crawl.py https://example.com --output-dir ./results --timeout 600

Note: The script automatically creates timestamped output directories following the pattern crawl_{DOMAIN}_{timestamp}/ unless a custom output directory is specified.

Tool Installation

If the crawling tools are not installed, use the installation helper:

bash
bash scripts/install_tools.sh

Or install manually:

bash
go install github.com/hakluke/hakrawler@latest
go install github.com/jaeles-project/gospider@latest
export PATH=$PATH:~/go/bin

Verify with: python scripts/crawl.py --install

Workflow

1. Initial Setup

Before crawling, ensure tools are installed:

bash
# Check if tools are installed
hakrawler -h && gospider -h

# If not, run the installer
bash scripts/install_tools.sh

2. Execute Crawl

Run the crawl script with appropriate options based on user requirements:

python
# For subdomain discovery
python scripts/crawl.py https://example.com --subdomains --depth 2

# For comprehensive endpoint discovery
python scripts/crawl.py https://example.com --depth 3 --js-analysis

# For quick reconnaissance
python scripts/crawl.py https://example.com --tool hakrawler --depth 1

3. Process Results

The script automatically categorizes and saves results to an organized directory structure:

Output structure:

crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized URL lists
│   ├── urls.txt         # All discovered URLs
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash command logs (stdout/stderr)
│   ├── hakrawler.log   # hakrawler execution log
│   └── gospider.log    # gospider execution log
├── summary.json         # Comprehensive JSON summary
└── REPORT.md           # Human-readable markdown report

4. Analyze Results

After crawling, analyze the results based on the user's goals:

bash
# Read summary
cat crawl_example.com_*/summary.json

# Review markdown report
cat crawl_example.com_*/REPORT.md

# Review subdomains
cat crawl_example.com_*/results/subdomains.txt

# Check for interesting endpoints
grep -i "admin\|api\|login\|upload" crawl_example.com_*/results/urls.txt

# Find JavaScript files for further analysis
cat crawl_example.com_*/results/js_files.txt

# Check bash command logs if needed
cat crawl_example.com_*/logs/hakrawler.log
cat crawl_example.com_*/logs/gospider.log

Use Cases

Subdomain Discovery

When users want to discover subdomains:

bash
python scripts/crawl.py https://example.com --subdomains --depth 2

Then read the results/subdomains.txt file from the output directory to provide the list of discovered subdomains, or review the REPORT.md for a formatted overview.

Endpoint Extraction

When users need to extract all endpoints:

bash
python scripts/crawl.py https://example.com --depth 3 --js-analysis

The script will identify and categorize:

  • API endpoints (paths containing /api/, /v1/, /graphql, etc.)
  • Regular web pages
  • Static resources
  • Parameters for further testing

JavaScript Analysis

When users want to find JavaScript files for secret extraction or endpoint discovery:

bash
python scripts/crawl.py https://example.com --js-analysis

The results/js_files.txt output can be passed to other tools like:

  • jsluice (use the jsluice skill for deep JS analysis)
  • SecretFinder
  • LinkFinder

Example integration with jsluice skill:

bash
# After crawling, feed JS files to jsluice
python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com --urls-file crawl_example.com_*/results/js_files.txt

Bug Bounty Reconnaissance

For comprehensive bug bounty reconnaissance:

bash
# Initial deep crawl with all options
python scripts/crawl.py https://target.com --subdomains --js-analysis --depth 3 --timeout 600

Then analyze the results to identify:

  1. Interesting subdomains for further testing
  2. API endpoints for authentication/authorization testing
  3. JavaScript files for secret extraction
  4. Parameters for injection testing

Tool Comparison

hakrawler

  • Strengths: Fast, simple, good for quick enumeration
  • Best for: Initial reconnaissance, quick scans
  • Limitations: Less comprehensive than gospider

gospider

  • Strengths: More thorough, JSON output, better JS analysis
  • Best for: Deep crawling, comprehensive mapping
  • Limitations: Slower than hakrawler

Default behavior: Use both tools (--tool both) for comprehensive coverage.

Advanced Usage

Integration with Other Skills

This skill integrates well with other reconnaissance skills:

  1. After crawling, use jsluice skill for JavaScript endpoint extraction:
bash
   # Feed JS files to jsluice for deep analysis
   python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com \
     --urls-file crawl_example.com_*/results/js_files.txt
  1. Feed subdomains to subdomain takeover tools
bash
   cat crawl_example.com_*/results/subdomains.txt | subdomain-takeover-tool
  1. Pass API endpoints to authentication testing tools
bash
   cat crawl_example.com_*/results/api_endpoints.txt | api-testing-tool
  1. Use parameters list for injection testing
bash
   cat crawl_example.com_*/results/parameters.txt

Custom Filtering

After crawling, apply custom filtering based on user needs:

bash
# Find specific file types
grep "\.pdf$\|\.xlsx$\|\.docx$" crawl_example.com_*/results/urls.txt

# Find admin panels
grep -i "admin\|dashboard\|panel\|management" crawl_example.com_*/results/urls.txt

# Find upload functionality
grep -i "upload\|file" crawl_example.com_*/results/urls.txt

# Filter by extension using the extensions.json file
jq '.js' crawl_example.com_*/results/extensions.json

Troubleshooting

Tools Not Found

If crawl.py reports missing tools:

  1. Run scripts/install_tools.sh
  2. Verify ~/go/bin is in PATH: echo $PATH
  3. Add to PATH if needed: export PATH=$PATH:~/go/bin

Timeout Issues

For large targets that timeout:

  • Increase timeout: --timeout 900
  • Reduce depth: --depth 1
  • Use one tool: --tool hakrawler
  • Crawl subdomains separately

Permission Denied

Make scripts executable:

bash
chmod +x scripts/crawl.py scripts/install_tools.sh

Best Practices

  1. Start shallow, go deep: Begin with depth 1-2, then increase if needed
  2. Use both tools: Default --tool both provides best coverage
  3. Enable subdomain discovery for comprehensive reconnaissance
  4. Review reports first: Check REPORT.md for human-readable overview, then summary.json for detailed statistics
  5. Categorize findings: Use the categorized output files in results/ to prioritize testing
  6. Check bash logs: If tools fail or produce unexpected results, review logs/hakrawler.log and logs/gospider.log for stdout/stderr
  7. Respect scope: Only crawl targets within authorized scope
  8. Monitor resources: Large crawls can be resource-intensive, use timeouts appropriately
  9. Combine with other tools: Chain crawl results into specialized tools for deeper analysis (especially jsluice for JS files)
  10. Archive outputs: Each crawl creates a timestamped directory - keep these for historical tracking and comparison

Resources

scripts/

  • crawl.py - Main Python wrapper for hakrawler and gospider with:
  • Unified interface for both tools
  • Automatic bash command logging (stdout/stderr)
  • Result categorization and organization
  • Timestamped output directories
  • JSON summary and markdown report generation
  • install_tools.sh - Bash script to install hakrawler and gospider via Go

references/

  • tool_comparison.md - Detailed comparison of hakrawler vs gospider features, performance, and use cases

Output Directory Reference

Every crawl creates a structured output directory:

crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized results
│   ├── urls.txt         # All URLs found
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # Detected API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash execution logs
│   ├── hakrawler.log   # Command, stdout, stderr, return code
│   └── gospider.log    # Command, stdout, stderr, return code
├── summary.json         # Complete crawl statistics and metadata
└── REPORT.md           # Human-readable markdown report

Key features:

  • All bash commands are logged with full stdout/stderr for debugging
  • Results are categorized for easy filtering and analysis
  • Summary includes tool execution logs for transparency
  • Markdown reports provide immediate insights
  • Timestamped directories enable historical tracking
同じリポジトリから

関連する Skills

すべての Skills
security-talent
コミュニティ

401-403-bypass-techniques

401/403 bypass playbook. Use when encountering access-denied responses on admin panels, API endpoints, or restricted paths. Covers path manipulation, HTTP method tampering, header injection, protocol downgrade, and automated bypass tools.

1時間の導入数
1
GitHub Stars
15
更新日
9月26日
security-talent
コミュニティ

active-directory-acl-abuse

Active Directory ACL abuse playbook. Use when exploiting misconfigured AD permissions including GenericAll, WriteDACL, DCSync rights, shadow credentials, LAPS reading, GPO abuse, and BloodHound-guided attack paths.

1時間の導入数
1
GitHub Stars
15
更新日
9月26日
security-talent
コミュニティ

active-directory-certificate-services

AD Certificate Services attack playbook. Use when targeting misconfigured AD CS for privilege escalation via ESC1-ESC13 template abuse, NTLM relay to enrollment, CA officer abuse, and certificate-based persistence.

1時間の導入数
1
GitHub Stars
15
更新日
9月26日
security-talent
コミュニティ

active-directory-kerberos-attacks

Kerberos attack playbook for Active Directory. Use when targeting AD authentication via AS-REP roasting, Kerberoasting, golden/silver/diamond tickets, delegation abuse, or pass-the-ticket attacks.

1時間の導入数
1
GitHub Stars
15
更新日
9月26日