Skill_Seekers

Carries strong trust indicators from repository metadata

No Content Extracted?

• Check your main_content selector
• Try: article, main, div[role="main"]

Data Exists But Won't Use It?

# Force re-scrape
rm -rf output/myframework_data/
skill-seekers create --config configs/myframework.json

Categories Not Good?

Edit the config categories section with better keywords.

Want to Update Docs?

# Delete old data and re-scrape
rm -rf output/godot_data/
skill-seekers create --config configs/godot.json

Enhancement Not Working?

# Check if API key is set
echo $ANTHROPIC_API_KEY

# Try LOCAL mode instead (uses Claude Code Max, no API key needed)
skill-seekers enhance output/react/ --mode LOCAL

# Monitor background enhancement status
skill-seekers enhance-status output/react/ --watch

GitHub Rate Limit Issues?

# Set a GitHub token (5000 req/hour vs 60/hour anonymous)
export GITHUB_TOKEN=ghp_your_token_here

# Or configure multiple profiles
skill-seekers config --github

Skill Seekers is the data layer for AI systems. It transforms 18 source types—documentation websites, GitHub repositories, PDFs, videos, Jupyter Notebooks, Word/EPUB/AsciiDoc documents, OpenAPI specs, PowerPoint presentations, RSS feeds, man pages, Confluence wikis, Notion pages, Slack/Discord exports, and more—into structured knowledge assets for every AI target:

Complete documentation: docs/README.md

Instead of spending days on manual preprocessing, Skill Seekers:

1. Ingests — docs, GitHub repos, local codebases, PDFs, videos, notebooks, wikis, and 10+ more source types
2. Analyzes — deep AST parsing, pattern detection, API extraction
3. Structures — categorized reference files with metadata
4. Enhances — AI-powered SKILL.md generation (Claude, Gemini, or local)
5. Exports — 16 platform-specific formats from one asset

🔄 Unified Multi-Source Scraping

• ✅ Combine Multiple Sources - Mix documentation + GitHub + PDF in one skill
• ✅ Conflict Detection - Automatically finds discrepancies between docs and code
• ✅ Intelligent Merging - Rule-based or AI-powered conflict resolution
• ✅ Transparent Reporting - Side-by-side comparison with ⚠️ warnings
• ✅ Documentation Gap Analysis - Identifies outdated docs and undocumented features
• ✅ Single Source of Truth - One skill showing both intent (docs) and reality (code)
• ✅ Backward Compatible - Legacy single-source configs still work

🤖 Multi-LLM Platform Support

• ✅ 12 LLM Platforms - Claude AI, Google Gemini, OpenAI ChatGPT, MiniMax AI, Generic Markdown, OpenCode, Kimi (Moonshot AI), DeepSeek AI, Qwen (Alibaba), OpenRouter, Together AI, Fireworks AI
• ✅ Universal Scraping - Same documentation works for all platforms
• ✅ Platform-Specific Packaging - Optimized formats for each LLM
• ✅ One-Command Export - --target flag selects platform
• ✅ Optional Dependencies - Install only what you need
• ✅ 100% Backward Compatible - Existing Claude workflows unchanged

# Claude (default - no changes needed!)
skill-seekers package output/react/
skill-seekers upload react.zip

# Google Gemini
pip install skill-seekers[gemini]
skill-seekers package output/react/ --target gemini
skill-seekers upload react-gemini.tar.gz --target gemini

# OpenAI ChatGPT
pip install skill-seekers[openai]
skill-seekers package output/react/ --target openai
skill-seekers upload react-openai.zip --target openai

# MiniMax AI
pip install skill-seekers[minimax]
skill-seekers package output/react/ --target minimax
skill-seekers upload react-minimax.zip --target minimax

# Generic Markdown (universal export)
skill-seekers package output/react/ --target markdown
# Use the markdown files directly in any LLM

The optional AI enhancement step (used by create, scan, and enhance) does not require an Anthropic key. You have three ways to power it:

1. Use a subscription you already pay for — no API credits at all (LOCAL agent mode)

Skill Seekers can shell out to a coding-agent CLI you're already logged into, so enhancement runs on your existing plan instead of metered API tokens:

skill-seekers create <source> --agent codex     # OpenAI Codex CLI → your ChatGPT Plus
skill-seekers create <source> --agent claude    # Claude Code      → your Claude Pro/Max

Supported agents: claude, codex, copilot, opencode, kimi, and custom (pair --agent custom with --agent-cmd "<your-cli> ..." to drive any other tool).

2. Any OpenAI-compatible provider (OpenRouter, Groq, Cerebras, Mistral, NVIDIA NIM, …)

All of these expose an OpenAI-compatible /v1 endpoint. Point Skill Seekers at one with three env vars — it detects OPENAI_API_KEY, and the OpenAI SDK honors OPENAI_BASE_URL automatically:

export OPENAI_API_KEY="<your provider key>"
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"   # provider endpoint (see table)
export OPENAI_MODEL="<a model that provider offers>"     # required — default gpt-4o won't exist elsewhere
skill-seekers create <source>

Provider detection picks the first API-key env var it finds (ANTHROPIC_API_KEY → GOOGLE_API_KEY → OPENAI_API_KEY → MOONSHOT_API_KEY). Set SKILL_SEEKER_PROVIDER to force a specific provider, or make sure the higher-priority keys are unset.

3. Claude-compatible endpoints (e.g. GLM, proxies)

export ANTHROPIC_API_KEY="your-key"
export ANTHROPIC_BASE_URL="https://your-claude-compatible-endpoint/v1"

Google Gemini (GOOGLE_API_KEY) and Kimi/Moonshot (MOONSHOT_API_KEY) are also supported natively. See Environment Variables Reference for the full list, including per-provider model overrides.

Installation:

# Install with Gemini support
pip install skill-seekers[gemini]

# Install with OpenAI support
pip install skill-seekers[openai]

# Install with MiniMax support
pip install skill-seekers[minimax]

# Install with all LLM platforms
pip install skill-seekers[all-llms]

🔗 RAG Framework Integrations

• ✅ LangChain Documents - Direct export to Document format with page_content + metadata

  • Perfect for: QA chains, retrievers, vector stores, agents
  • Example: LangChain RAG Pipeline
  • Guide: LangChain Integration

• ✅ LlamaIndex TextNodes - Export to TextNode format with unique IDs + embeddings

  • Perfect for: Query engines, chat engines, storage context
  • Example: LlamaIndex Query Engine
  • Guide: LlamaIndex Integration

• ✅ Pinecone-Ready Format - Optimized for vector database upsert

  • Perfect for: Production vector search, semantic search, hybrid search
  • Example: Pinecone Upsert
  • Guide: Pinecone Integration

Quick Export:

# LangChain Documents (JSON)
skill-seekers package output/django --target langchain
# → output/django-langchain.json

# LlamaIndex TextNodes (JSON)
skill-seekers package output/django --target llama-index
# → output/django-llama-index.json

# Markdown (Universal)
skill-seekers package output/django --target markdown
# → output/django-markdown/SKILL.md + references/

Complete RAG Pipeline Guide: RAG Pipelines Documentation

🧠 AI Coding Assistant Integrations

Transform any framework documentation into expert coding context for 4+ AI assistants:

• ✅ Cursor IDE - Generate .cursorrules for AI-powered code suggestions

  • Perfect for: Framework-specific code generation, consistent patterns
  • Works with: Cursor IDE (VS Code fork)
  • Guide: Cursor Integration
  • Example: Cursor React Skill

• ✅ Windsurf - Customize Windsurf's AI assistant context with .windsurfrules

  • Perfect for: IDE-native AI assistance, flow-based coding
  • Works with: Windsurf IDE by Codeium
  • Guide: Windsurf Integration
  • Example: Windsurf FastAPI Context

• ✅ Cline (VS Code) - System prompts + MCP for VS Code agent

  • Perfect for: Agentic code generation in VS Code
  • Works with: Cline extension for VS Code
  • Guide: Cline Integration
  • Example: Cline Django Assistant

• ✅ Continue.dev - Context servers for IDE-agnostic AI

  • Perfect for: Multi-IDE environments (VS Code, JetBrains, Vim), custom LLM providers
  • Works with: Any IDE with Continue.dev plugin
  • Guide: Continue Integration
  • Example: Continue Universal Context

Quick Export for AI Coding Tools:

# For any AI coding assistant (Cursor, Windsurf, Cline, Continue.dev)
skill-seekers create --config configs/django.json
skill-seekers package output/django --target claude  # or --target markdown

# Copy to your project (example for Cursor)
cp output/django-claude/SKILL.md my-project/.cursorrules

# Or for Windsurf
cp output/django-claude/SKILL.md my-project/.windsurf/rules/django.md

# Or for Cline
cp output/django-claude/SKILL.md my-project/.clinerules

# Or for Continue.dev (HTTP server)
python examples/continue-dev-universal/context_server.py
# Configure in ~/.continue/config.json

Integration Hub: All AI System Integrations

🌊 Three-Stream GitHub Architecture

• ✅ Triple-Stream Analysis - Split GitHub repos into Code, Docs, and Insights streams
• ✅ Unified Codebase Analyzer - Works with GitHub URLs AND local paths
• ✅ C3.x as Analysis Depth - Choose 'basic' (1-2 min) or 'c3x' (20-60 min) analysis
• ✅ Enhanced Router Generation - GitHub metadata, README quick start, common issues
• ✅ Issue Integration - Top problems and solutions from GitHub issues
• ✅ Smart Routing Keywords - GitHub labels weighted 2x for better topic detection

Three Streams Explained:

• Stream 1: Code - Deep C3.x analysis (patterns, examples, guides, configs, architecture)
• Stream 2: Docs - Repository documentation (README, CONTRIBUTING, docs/*.md)
• Stream 3: Insights - Community knowledge (issues, labels, stars, forks)

from skill_seekers.cli.unified_codebase_analyzer import UnifiedCodebaseAnalyzer

# Analyze GitHub repo with all three streams
analyzer = UnifiedCodebaseAnalyzer()
result = analyzer.analyze(
    source="https://github.com/facebook/react",
    depth="c3x",  # or "basic" for fast analysis
    fetch_github_metadata=True
)

# Access code stream (C3.x analysis)
print(f"Design patterns: {len(result.code_analysis['c3_1_patterns'])}")
print(f"Test examples: {result.code_analysis['c3_2_examples_count']}")

# Access docs stream (repository docs)
print(f"README: {result.github_docs['readme'][:100]}")

# Access insights stream (GitHub metadata)
print(f"Stars: {result.github_insights['metadata']['stars']}")
print(f"Common issues: {len(result.github_insights['common_problems'])}")

See complete documentation: Three-Stream Implementation Summary

🔐 Smart Rate Limit Management & Configuration

• ✅ Multi-Token Configuration System - Manage multiple GitHub accounts (personal, work, OSS)
  • Secure config storage at ~/.config/skill-seekers/config.json (600 permissions)
  • Per-profile rate limit strategies: prompt, wait, switch, fail
  • Configurable timeout per profile (default: 30 min, prevents indefinite waits)
  • Smart fallback chain: CLI arg → Env var → Config file → Prompt
  • API key management for Claude, Gemini, OpenAI
• ✅ Interactive Configuration Wizard - Beautiful terminal UI for easy setup
  • Browser integration for token creation (auto-opens GitHub, etc.)
  • Token validation and connection testing
  • Visual status display with color coding
• ✅ Intelligent Rate Limit Handler - No more indefinite waits!
  • Upfront warning about rate limits (60/hour vs 5000/hour)
  • Real-time detection from GitHub API responses
  • Live countdown timers with progress
  • Automatic profile switching when rate limited
  • Four strategies: prompt (ask), wait (countdown), switch (try another), fail (abort)
• ✅ Resume Capability - Continue interrupted jobs
  • Auto-save progress at configurable intervals (default: 60 sec)
  • List all resumable jobs with progress details
  • Auto-cleanup of old jobs (default: 7 days)
• ✅ CI/CD Support - Non-interactive mode for automation
  • --non-interactive flag fails fast without prompts
  • --profile flag to select specific GitHub account
  • Clear error messages for pipeline logs

Quick Setup:

# One-time configuration (5 minutes)
skill-seekers config --github

# Use specific profile for private repos
skill-seekers create mycompany/private-repo --profile work

# CI/CD mode (fail fast, no prompts)
skill-seekers create owner/repo --non-interactive

# Resume interrupted job
skill-seekers resume --list
skill-seekers resume github_react_20260117_143022

Rate Limit Strategies Explained:

• prompt (default) - Ask what to do when rate limited (wait, switch, setup token, cancel)
• wait - Automatically wait with countdown timer (respects timeout)
• switch - Automatically try next available profile (for multi-account setups)
• fail - Fail immediately with clear error (perfect for CI/CD)

🎯 Bootstrap Skill - Self-Hosting

Generate skill-seekers as a skill to use within your AI agent (Claude Code, Kimi, Codex, etc.):

# Generate the skill
./scripts/bootstrap_skill.sh

# Install to Claude Code
cp -r output/skill-seekers ~/.claude/skills/

What you get:

• ✅ Complete skill documentation - All CLI commands and usage patterns
• ✅ CLI command reference - Every tool and its options documented
• ✅ Quick start examples - Common workflows and best practices
• ✅ Auto-generated API docs - Code analysis, patterns, and examples

🔐 Private Config Repositories

• ✅ Git-Based Config Sources - Fetch configs from private/team git repositories
• ✅ Multi-Source Management - Register unlimited GitHub, GitLab, Bitbucket repos
• ✅ Team Collaboration - Share custom configs across 3-5 person teams
• ✅ Enterprise Support - Scale to 500+ developers with priority-based resolution
• ✅ Secure Authentication - Environment variable tokens (GITHUB_TOKEN, GITLAB_TOKEN)
• ✅ Intelligent Caching - Clone once, pull updates automatically
• ✅ Offline Mode - Work with cached configs when offline

🤖 Codebase Analysis (C3.x)

C3.4: Configuration Pattern Extraction with AI Enhancement

• ✅ 9 Config Formats - JSON, YAML, TOML, ENV, INI, Python, JavaScript, Dockerfile, Docker Compose
• ✅ 7 Pattern Types - Database, API, logging, cache, email, auth, server configurations
• ✅ AI Enhancement - Optional dual-mode AI analysis (API + LOCAL)
  • Explains what each config does
  • Suggests best practices and improvements
  • Security analysis - Finds hardcoded secrets, exposed credentials
• ✅ Auto-Documentation - Generates JSON + Markdown documentation of all configs
• ✅ MCP Integration - extract_config_patterns tool with enhancement support

C3.3: AI-Enhanced How-To Guides

• ✅ Comprehensive AI Enhancement - Transforms basic guides into professional tutorials
• ✅ 5 Automatic Improvements - Step descriptions, troubleshooting, prerequisites, next steps, use cases
• ✅ Dual-Mode Support - API mode (Claude API) or LOCAL mode (Claude Code CLI)
• ✅ No API Costs with LOCAL Mode - FREE enhancement using your Claude Code Max plan
• ✅ Quality Transformation - 75-line templates → 500+ line comprehensive guides

Usage:

# Quick analysis (1-2 min, basic features only)
skill-seekers scan tests/ --quick

# Comprehensive analysis with AI (20-60 min, all features)
skill-seekers scan tests/ --comprehensive

# With AI enhancement
skill-seekers scan tests/ --enhance

Full Documentation: docs/features/HOW_TO_GUIDES.md

🔄 Enhancement Workflow Presets

Reusable YAML-defined enhancement pipelines that control how AI transforms your raw documentation into a polished skill.

• ✅ 5 Bundled Presets — default, minimal, security-focus, architecture-comprehensive, api-documentation
• ✅ User-Defined Presets — add custom workflows to ~/.config/skill-seekers/workflows/
• ✅ Multiple Workflows — chain two or more workflows in one command
• ✅ Fully Managed CLI — list, inspect, copy, add, remove, and validate workflows

# Apply a single workflow
skill-seekers create ./my-project --enhance-workflow security-focus

# Chain multiple workflows (applied in order)
skill-seekers create ./my-project \
  --enhance-workflow security-focus \
  --enhance-workflow minimal

# Manage presets
skill-seekers workflows list                          # List all (bundled + user)
skill-seekers workflows show security-focus           # Print YAML content
skill-seekers workflows copy security-focus           # Copy to user dir for editing
skill-seekers workflows add ./my-workflow.yaml        # Install a custom preset
skill-seekers workflows remove my-workflow            # Remove a user preset
skill-seekers workflows validate security-focus       # Validate preset structure

# Copy multiple at once
skill-seekers workflows copy security-focus minimal api-documentation

# Add multiple files at once
skill-seekers workflows add ./wf-a.yaml ./wf-b.yaml

# Remove multiple at once
skill-seekers workflows remove my-wf-a my-wf-b

YAML preset format:

name: security-focus
description: "Security-focused review: vulnerabilities, auth, data handling"
version: "1.0"
stages:
  - name: vulnerabilities
    type: custom
    prompt: "Review for OWASP top 10 and common security vulnerabilities..."
  - name: auth-review
    type: custom
    prompt: "Examine authentication and authorisation patterns..."
    uses_history: true

⚡ Performance & Scale

• ✅ Async Mode - 2-3x faster scraping with async/await (use --async flag)
• ✅ Large Documentation Support - Handle 10K-40K+ page docs with intelligent splitting
• ✅ Router/Hub Skills - Intelligent routing to specialized sub-skills
• ✅ Parallel Scraping - Process multiple skills simultaneously
• ✅ Checkpoint/Resume - Never lose progress on long scrapes
• ✅ Caching System - Scrape once, rebuild instantly

🤖 Agent-Agnostic Skill Generation

• ✅ Multi-Agent Support - Generate skills for Claude, Kimi, Codex, Copilot, OpenCode, or any custom agent via --agent flag
• ✅ Custom Agent Commands - Use --agent-cmd to specify a custom agent CLI command for enhancement
• ✅ Universal Flags - --agent and --agent-cmd available on all commands (create, scrape, github, pdf, etc.)

📦 Marketplace Pipeline

• ✅ Publish to Marketplace - Publish skills to Claude Code plugin marketplace repos
• ✅ End-to-End Pipeline - From documentation source to published marketplace entry

✅ Quality Assurance

• ✅ Fully Tested - 3,700+ tests with comprehensive coverage