Last week I saw the same tweet shared in three different Slack channels. NullClaw: a 678 KB binary, ~1 MB RAM, sub-2ms boot, 22+ AI provider support, MIT license, written in ~45,000 lines of Zig. Impressive. Every time someone shared it, the question was the same: “Can we use this?”

That question sounds simple. It isn’t. It carries hidden assumptions nobody says out loud: that open-source means free, that having tests means it’s been tested, that working in a demo means it’ll work in production. And that’s where the problems start.

678 KB, 22 Providers, Zero Guarantees

I won’t pretend NullClaw isn’t impressive — it is. A 678 KB binary handling 22+ AI providers with sub-2ms boot times is a remarkable piece of engineering. But there’s a massive gap between “this is impressive” and “this is production-ready for enterprise.”

In our experience at IQ Source, ~80% of open-source tools that clients bring us for evaluation don’t survive a serious review. Not because they’re bad tools. Because they weren’t designed for what the client wants to do with them.

What’s Not in the README

Every time we evaluate an open-source tool, we run the same exercise: separate what you see from what you actually get.

What You See	What You Get
MIT License, free	No SLA, maintainer can walk away tomorrow
2,738 tests	Written by the same person, no independent audit
22+ providers	22+ API keys, 22+ attack surfaces
~45,000 lines of Zig	Tiny talent pool — who maintains this?
”Multi-layer sandboxing”	Claims you can’t verify without a security review
Active community	Dependency on a stranger’s weekend motivation

Three risks we see repeatedly:

Supply Chain Attacks

In 2024, a developer inserted a backdoor into XZ Utils — a compression library nobody pays attention to but that lives on nearly every Linux server (CVE-2024-3094). He spent two years building trust in the project before acting. If it can happen to critical infrastructure used by millions of servers, it can happen to any AI tool your team downloads from GitHub on a Tuesday afternoon.

Abandonment Risk

According to the Tidelift open-source maintainer survey, 60% of unpaid maintainers work alone on their projects. A job change, burnout, a life decision — and your critical dependency is left without support. There’s no SLA to call. No contract to protect you.

Hidden Infrastructure Costs

A 678 KB binary sounds like zero operational cost. But 22 providers mean 22 sets of credentials to rotate, 22 usage policies to track, 22 failure points to diagnose when something breaks at 3 AM. The binary is lightweight. The operations aren’t.

If you’re evaluating AI vendors, we wrote about the criteria we use in AI vendor selection for B2B.

The Problem with Code Nobody Understands

There’s another trend we’re seeing more and more: vibe coding. If you haven’t heard the term, it’s when someone uses an AI assistant to generate code — accepting suggestion after suggestion without really understanding what each line does or why.

A prospect showed us a tool their team built in two days with an AI assistant recently. It worked. They wanted it in production. We asked three questions and the conversation shifted completely:

1. What happens when this function receives unexpected data?
2. Can you explain the authentication logic line by line?
3. Where’s the input validation before sending data to the external API?

Silence. Not because they didn’t know technology. Because they’d never actually read the code they were about to deploy.

You Don’t Understand What You’re Deploying

Code you can’t explain is code you can’t debug at 2 AM. You can’t extend it when requirements change. You can’t justify it to an auditor. And if the AI model that generated it changes versions or becomes unavailable, you can’t reproduce it.

Vulnerabilities the Model Doesn’t See

AI-generated code introduces patterns that standard security scanners miss. Incomplete input validation, subtle race conditions, secrets hardcoded that “work in dev.” OWASP published a Top 10 specifically for LLM applications identifying these vectors — and most teams doing vibe coding haven’t read it.

For more on how we evaluate AI-generated code security, we have a dedicated article on AI code security in enterprise contexts.

Accelerated Technical Debt

Code that works but wasn’t designed. Speed in week one, cost multiplier by month six. We’ve seen teams build an MVP in days with AI and then spend months rewriting it when they needed to add an integration the original design didn’t account for. Generated code tends to solve the immediate problem without considering how things will evolve.

Compliance Doesn’t Accept “The AI Did It”

SOC 2, ISO 27001, HIPAA — all require clear accountability chains. Who wrote this code? Who reviewed it? What process ensures it meets security controls? “A language model generated it and it seemed to work” isn’t an answer any auditor will accept.

Checklist: 8 Questions Before You Adopt

At IQ Source we use these questions every time a client brings us a tool or AI-generated code for evaluation. If you can’t confidently answer more than two, the tool isn’t production-ready.

1. Who owns this dependency? — If the bus factor is 1, your risk is high. Check who maintains the project, how many active contributors it has, and whether there’s an organization or company behind it.

2. What happens when it breaks at 3 AM? — Is there a support channel? Documented response times? Or do you depend on someone seeing your GitHub issue?

3. Has the code been independently audited? — Check for a SECURITY.md file, a bug bounty program, or published audit reports. Tests written by the same developer don’t count as an audit.

4. Can your team read, debug, and modify this code? — If it’s in a language nobody on your team knows well (Zig, Rust, etc.), you’re creating talent lock-in. When something breaks, you need people who can get into the code.

5. What’s the license — really? — MIT today doesn’t guarantee MIT tomorrow. HashiCorp switched to Business Source License. Redis moved to SSPL. Read the full license and assess what happens if it changes.

6. Where do your data and prompts go? — With 22 providers you have 22 different privacy policies. Which ones retain data? Which train on your prompts? Who’s responsible for a breach?

7. What’s the exit plan? — If you need to migrate tomorrow, what’s the cost? Is there a standard export format? Or are you building on an abstraction that only exists in this tool?

8. Who reviews AI-generated code before it ships? — Not whether it gets reviewed, but who, how, and against what criteria. “I tested it and it works” is not code review.

What We Do at IQ Source

I’m not going to turn this into a sales pitch. But I want to be transparent about how we handle this with clients, because I think the process matters.

When a client brings us an open-source tool or AI-generated code, the first thing we do is run the 8-question checklist. It’s a hard filter. Roughly 4 out of 5 tools that come through don’t pass for production use — not because they’re bad, but because the gap between “works on my machine” and “works in enterprise production with auditing, support, and continuity” is enormous.

For the ones that pass, we design the integration architecture: real sandboxing, monitoring, fallbacks, data governance. It’s not just “install and connect.”

And for AI-generated code, we run targeted reviews looking for patterns models typically get wrong: missing input validation, race conditions, hardcoded assumptions that work in dev but blow up in production.

Open-source AI tools and vibe coding are the recipe. What we bring is the chef’s judgment.

---

If your team is evaluating open-source AI tools or shipping AI-generated code to production, we can run a risk assessment on the specific tools in your stack. Schedule the assessment — 30 minutes that might save you months.