DG6 Swarm — AI Content & Media Pipeline

A Python pipeline that turns a domain and starting keyword into site copy, articles and visual assets. Specialized GPU workers handle each stage, starting and stopping according to queued work.

About DG6 Swarm — AI Content & Media Pipeline

I started DG6 Swarm in mid-2025 to connect the different jobs involved in generating website content: finding related topics, writing articles, creating images, organizing the results and handling translations.

Each job needs different models and resources. DG6 Swarm gives them separate workers and coordinates the work from a small, always-on controller.

From a keyword to content and images

A run starts with a domain and a seed keyword. The pipeline finds related keywords, generates homepage copy and articles, and produces titles, descriptions and image prompts.

It then assigns article tags, writes supporting tag-page copy and generates images. Separate workers can also create logos, favicons and avatars.

Text and relationships are stored in PostgreSQL. Images go into S3-compatible object storage, where later stages can access them without passing large files through the controller.

Giving each job the resources it needs

The main engineering work was coordinating these stages across different GPU workers.

Prefect manages the workflow. Docker packages the environments for text generation, embeddings, tagging, images and translation. An autoscaler watches the queues and starts RunPod workers when work is waiting.

Each worker pool has its own GPU preferences and limits. Workers shut down after an idle period, so GPU machines do not need to stay running between batches.

That separation also means I can change an image-generation environment without rebuilding the text-generation workers.

Beyond the generation step

The project grew from single-machine and multi-GPU runners into a controller with independently managed worker pools.

I added batch configuration, chunked processing, content checks, exports and cleanup utilities. These are the parts that make it possible to inspect what a run produced and manage the records and files afterwards.

The stack combines Python, Prefect, Docker and RunPod with vLLM for text generation, FAISS for keyword retrieval and Stable Diffusion for visual assets.

Current state

DG6 Swarm is a personal engineering project with the controller, worker separation, storage handoff and queue-driven scaling implemented.

Translation workflows are also included. The move to PostgreSQL left part of the article-translation worker needing further migration work.

Release history

Version 0.1721 Jun 2025

More software

PHP library / CLI

PicoWebP

PHP Version 1.0.1

A WebP encoder written in PHP, for hosts without GD, Imagick or cwebp. Converts PNGs and baseline JPEGs, preserves transparency and embedded colour profiles, and supports resumable encoding for background jobs.