Building Offline-First AI Tools: Lessons from Kreative Core Vision

When we set out to build Vision — an AI-powered footage search tool for Premiere Pro — the first decision was the hardest: cloud or local?

Cloud would've been easier. Use OpenAI's API, slap a nice UI on it, charge by the search. Two months of development, tops.

But we kept coming back to one question: Would we use this ourselves?

The answer was no. Uploading hours of client footage to someone else's server felt wrong. Paying per-search for something that should run locally felt worse.

So we went the harder route: 100% offline, GPU-accelerated AI. Here's what we learned.

Challenge 1: Model Selection (Not All AI Runs Locally)

OpenAI's GPT-4 Vision is incredible — but it's cloud-only. We needed models that could:

Run on consumer GPUs (RTX 3060 and up)
Process images in <200ms each
Understand semantic queries ("find sad moments")

After testing dozens of models, we landed on:

CLIP (OpenAI)

CLIP connects images and text. You give it an image and a phrase ("person smiling"), and it returns a similarity score. Perfect for semantic search.

            // Simplified CLIP inference (ONNX Runtime)
            const session = await ort.InferenceSession.create('clip-vit-base.onnx');
            const imageEmbedding = await session.run({ pixel_values: imageData });
            const textEmbedding = await session.run({ input_ids: tokenizedQuery });

            const similarity = cosineSimilarity(imageEmbedding, textEmbedding);
            // 0.85+ = strong match
        

Tradeoff: CLIP is fast but not great at fine-grained details. It gets "smiling person" right 90% of the time, but struggles with "person wearing red hat."

InsightFace (for Face Recognition)

CLIP doesn't recognize specific people. InsightFace does. It generates a "face embedding" — a 512-number fingerprint unique to each person.

Key Insight: We cluster faces using DBSCAN. The plugin shows "Person 1", "Person 2", etc. User renames once ("Person 1" → "Alex"), and Vision updates all clips automatically.

Challenge 2: ONNX Runtime Is Not Plug-and-Play

ONNX (Open Neural Network Exchange) is supposed to make models portable. In theory, export from PyTorch → run anywhere.

In practice, it's messier:

GPU Providers: CUDA for NVIDIA, DirectML for AMD/Intel. Each needs separate setup.
Quantization: Full FP32 models are huge (400MB+). INT8 quantization cuts size by 75% but needs careful calibration.
Input Shapes: Many models expect fixed 224×224 images. Dynamic shapes require model surgery.

            Lesson Learned: Don't assume ONNX "just works." Budget 2-3 weeks for optimization. Test on
            low-end GPUs early.
        

Challenge 3: User Expectations (They Want Cloud Speed Offline)

Users don't care about technical constraints. They expect:

Instant results (like Google search)
No setup (download → works)
Low resource usage (Premiere already uses 16GB RAM)

Our first prototype analyzed a 30-minute sequence in ~12 minutes. Users hated it. "Why not just scrub manually?"

How We Fixed It: Smart Sampling

Instead of analyzing every frame (1800 frames per minute), we:

Detect scene changes (OpenCV)
Extract 1 keyframe per scene
Run AI only on keyframes

Result: 30-minute sequence → ~200 keyframes → 8-minute analysis (vs. 12 minutes).

Bonus: Users can opt into "Deep Scan" (every 3rd frame) for precise searches. Most stick with default.

Challenge 4: Distribution Hell

Adobe plugins typically ship as ZXP files (~10-50MB). Our first build was 600MB. Why?

ONNX Runtime binaries: 150MB
CLIP model (FP32): 400MB
InsightFace model: 50MB

Adobe Exchange has a 200MB limit. We couldn't ship it.

Solution: Progressive Download

Ship a minimal "launcher" plugin (20MB). On first run, it downloads AI models from GitHub Releases:

            // Model downloader (simplified)
            async function downloadModel(modelName) {
            const url = `https://github.com/kreativecore/models/releases/download/v1/${modelName}.onnx`;
            const response = await fetch(url);
            const buffer = await response.arrayBuffer();

            fs.writeFileSync(`models/${modelName}.onnx`, buffer);
            console.log(`✅ ${modelName} ready`);
            }
        

User Experience: Install plugin → Panel shows "Downloading AI models (200MB)..." → Progress bar → Done. 3 minutes, once ever.

Challenge 5: Debugging Is Brutal

CEP panels don't have Chrome DevTools by default. Debugging AI inference errors looked like:

Error: RuntimeException: Invalid tensor shape

...with no stack trace. No tensor values. No nothing.

Our Debugging Stack

Enable remote debugging: CEP flag .debug in manifest
Log everything: Tensor shapes, inference times, memory usage
Unit test models separately: Node.js script outside Premiere

The Node.js test script saved us. We could iterate on model loading 10x faster than restarting Premiere every time.

What Would We Do Differently?

1. Start with Smaller Models

We went straight for CLIP ViT-B/32 (best accuracy). Should've started with ViT-B/16 (80% accuracy, 50% faster). Optimize for UX first, accuracy second.

2. Test Low-End Hardware Sooner

We developed on RTX 4090s. When beta testers with GTX 1660s reported crashes, we had to scramble. Lesson: Test on minimum spec hardware from day one.

3. Build Auto-Update Into V1

AI models improve monthly. We should've built model auto-update from the start, not as a "later feature."

The Payoff: What Users Say

Despite the challenges, going offline-first was the right call. Beta feedback:

"I can't believe this runs on my laptop. Feels like magic."
"No monthly fees? I'm switching from [cloud competitor]."
"Finally, something that respects my NDA projects."

The technical complexity is invisible to users. They just see: install, analyze, search. Done.

            Key Takeaway: Offline-first is harder to build, but it creates a defensible moat. Cloud
            tools compete on pricing. Offline tools compete on trust and speed — harder to replicate.
        

For Other Developers Building Offline AI

If you're considering local AI for your plugin:

Good use cases: Repetitive, high-volume tasks (transcription, tagging, search)
Bad use cases: One-off creative generation (where users expect cutting-edge models)
Hybrid approach: Offline by default, cloud as opt-in for advanced features

And remember: User experience beats technical purity. If cloud is 10x faster for your use case, swallow your pride and use cloud. But if you can make offline work, the loyalty and trust you build are irreplaceable.