https://homelab.nbkelley.com/tags/pipeline/ Recent content in Pipeline on homelab Hugo en Sat, 02 May 2026 00:00:00 +0000 https://homelab.nbkelley.com/docs/ai/wiki-pipeline-scripts/ Sat, 02 May 2026 00:00:00 +0000 https://homelab.nbkelley.com/docs/ai/wiki-pipeline-scripts/ <h1 id="wiki-pipeline-scripts">Wiki Pipeline Scripts<a class="anchor" href="#wiki-pipeline-scripts">#</a></h1> <h2 id="what-was-established">What Was Established<a class="anchor" href="#what-was-established">#</a></h2> <p>Eight Python scripts in <code>/opt/wiki/homelab/scripts/</code> implement the full wiki pipeline: file conversion, document ingestion, conversation crystallization (standard, DeepSeek, and Claude formats), shared LLM infrastructure, wiki health-checking, and knowledge-graph integration. All scripts were ported from the work wiki pipeline (itself developed 2026-04-21 → 2026-04-26) with homelab-specific infrastructure baked in.</p> <p><code>crystallize.py</code> (Claude format) uses a two-step LLM approach: gemma4:e2b cleans, qwen3.6:35b crystallizes. <code>crystallize_deepseek.py</code> skips gemma — JSON parsing is handled deterministically in Python (<code>load_conversation</code> + <code>_clean_text</code>), so only qwen is needed.</p>