Section 1: The problem that AI creates

Your company just enabled “AI” features. You can now upload PDFs, Excel, CSVs, Slack conversations, directly into Claude to analyze data, extract insights, and summarize documents in seconds. There’s going to be a great productivity boost across all departments. In fact, through MCP, Claude can now work directly with all the tools you use: Gmail, Jira, SnowFlake(DB), and in fact, through integration with Claude code, it has access to your entire private code base, as well as your company machine’s file systems (trigger happy devs).

Your organization has never been more “open”.

However, there’s also a latent risk: what if Claude has access to all those data? What kind of legal and compliance protections are there to curb sensitive data leakage?

Section 2: How this problem is being resolved right now

This problem is being solved from multiple angles. I’ll try to highlight 3.

Resolution 1: by Claude itself, contracts and audits

What Claude Provides

There are many things that Claude Enterprise does to help protect its clients’ business data.

  • Claude Enterprise, by default, does not use inputs and outputs to train models.
  • Claude’s internal policy on data access - access restricted to a small set of approved reviewers, and orgs can choose to have access transparency audit logs
    • “By default, no Anthropic personnel can read your retained conversations. Human review can occur only through a controlled access path—for example, when content is flagged by our automated trust and safety systems for potential harm. These reviews can only be performed by a small set of approved reviewers. Every instance of access is recorded in a tamper-proof log that reviewers cannot suppress or modify. After 30 days, the data is deleted automatically, except in the rare cases where it’s been flagged by our automated trust and safety systems or we’re legally required to keep it. Eligible organizations also have the option to add customer-managed encryption keys and access transparency audit logs.” - Anthropic
  • Claude Platform is not ZDR-eligible - only Claude Code
  • Customer managed encryption Keys (CMEK)
    • Data encrypted at rest, every action Anthropic performs with key is recorded in Cloud Provider’s audit logs
    • But we don’t know the picture in practice: there’s a small chance that anthropic regularly accesses the content to scan for certain things, and justifies that as a use case.
  • US only inference
  • Zero Data Retention (for claude code):
    • Anthropic does not store customer prompts or responses at rest after API response is returned (no logging).
    • Model becomes purely a runtime thing.
    • Applies for Claude Code - available to qualified accounts and requires separate enablement by Anthropic
  • ZDR also available for products that use my commercial organization’s API keys
Where Claude falls short:
  • If you don’t implement CMEK, Claude conversation histories are not end-to-end encrypted
    • Server-side access: Because it is not end-to-end encrypted, Anthropic’s servers process and store the text. Anthropic employees cannot view your private chats by default, but they retain the technical ability to access them for safety policy enforcement or legal compliance
    • So if you accidentally paste API keys or secrets into Claude, you need to rotate them.
  • Chats are retained in Claude’s DB so conversations can be resumed.
  • If a chat is flagged by automated trust and safety system as violating usage policy, inputs and outputs are retained up to 2 years and classification scores up to 7 years.
  • If you do implement CMEK, once a key is attached to a workspace, it can’t be detached or swapped. Switching to a different key requires creating a new workspace with new keys and migrating your data.
  • Claude connectors: Internal data
  • Currently, certain frontier models are not available via ZDR: Covered models like Fable 5 and Mythos - due to risk of exploitation.
The safest approach
  • Would be to set up CMEK, with transparent audit logs. That way, you have total control over access over your data, can revoke it any time, and need to “trust” that Claude would not access it - they will be less incentivized to access it since all actions will be logged and must be justified (if requested).
  • However, it’s possible that many organizations don’t audit their transparency access logs frequently enough - or can’t separate the false positives (support/safety violation requests) from the true positives (unauthorized access) - this is probably full time work!
  • Claude Code is protected via ZDR
Conclusions for this resolution method:
  • This is trust-based security - if you trust the provider (Claude), then it’s okay.
  • But this is not absolute secrecy - it’s a probabilistic model, although for 95% of your data, you can be close to 95% certain they won’t be intercepted and read without a clause.
  • For a large portion of enterprises, I think they will be quite okay with this level of data security protections - provided that they’re willing to pay:
    • Rough estimate (Medium org):
      • Seat Bills: $20/seat/month, before useage. For 600 seats, that’s $12,000/month, billed at $144,000 a year.
      • Usage bills: Very rough math: if 600 developers averaged $6/day on ~21 working days, that’s about $75,000/month in usage; at $25/day it would be over $300,000/month.
    • Rough Estimate (15 people)
  • 64k to 98k per year
  • For larger enterprises, it’s millions per year. Src: Reddit

At 800 users, rough math based on rep’s own estimates: • Seat fees alone: $192,000/year • Mid-case consumption: ~$912,000/year • Total mid-case: ~$1,100,000/year If Team allowed 800 seats at $25 flat: $240,000/year That’s a gap of $336K to $1.4M per year depending on usage, and the only reason we’re in the expensive lane is because we have too many users for the cheaper one.

Resolution 2: Claude/ChatGPT running on cloud infra

  • Services like AWS Dedicated Local Zones (used by SG government)
    • Google Vortex
    • Claude via AWS Bedrock
    • OpenAI via Azure
  • Models hosted by Big Cloud - data stays within AWS env, no training on data, AWS-native identity and access-control, data retention within your own infra
  • Tenant isolation, access control. For many organizations, this is a great solution.
  • Pros: Provide even stricter data policies, using infra you’re familiar with
  • Cons: Cost.
AWS Bedrock Cost estimation - sonnet and opus in parenthesis

Resolution 3: Security harnesses around Claude code

  • Leveraging ZDR of Claude code, there can be security guardrails (prompts injected into every context) such that the AI detects hardcoded secrets before you make a request, always update .gitignore to exclude .env files, flags patterns resembling secrets.
  • When using Claude code on Containers or Kubernetes:
    • never run as root,
    • no secrets in image layers
    • explicit confirmation for sensitive operations: disabling security controls, handling raw private key material
Unresolved issues:
  • Auto-mode bypasses a lot of these.

Resolution 4: Sandbox cloud environments for coding agents

  • Cloud agents –> allows the agent to run on a sandbox machine
    • replicate all the apps you have
    • The actual repo and commands - ran in the cloud
  • Grokbot -

Conclusions for Section 2 - how the problem is being resolved

  • To a large extent, having secure operational protocols by Claude is one way this can be resolved: clients may get the best of both worlds: advanced models, and data security.
  • However, there’s a large cost to that.
  • Also, it does not cater to clients with very confidential requirements.
  • My bet is that there is actually a high chance that a security incident may happen, after which companies may not trust that much data to Claude.
    • Esp with advanced AI models doing the hacking, AI euphoria and usage, code practices with bad security, and “auto-mode”.
  • Running big AI on trusted cloud is also a great solution - to have control over your data flows.

Section 3: Where is the place for local models?

Given that there is a solution for the data security problem, this shrinks the use case of local models and private AI, especially for enterprises with the budget. If you have the budget, you would want to go for a trusted solution.

However, for many enterprises, they will be priced out of Claude’s tier - never fully able to capture the gains of frontier models. This is especially the case nowadays where to squeeze best performance out of models, you need to create harnesses and sub-agents, which are very very costly.

Second, there’s a class of enterprises with stricter data requirements, where they need exposure to AI capabilities but really can’t trust big cloud, or big AI. “Sovereign AI” class. In the event of a security incident involving BigAI, this class will grow.

Third, there are certain tail winds in terms of where AI personalization will eventually go.

  • On the enterprise side, to fully capture AI gains, AI systems will need to be more deeply integrated into OS and file systems - imagine Claude code, but entirely locally ran. This is a truly agentic system.
  • On the consumer side, it’s the same dynamic: I am willing to bet on a future where AI is deeply integrated with personal details, and the only way to protect that is for data to stay local - locally hosted machines with everything else acting as remotes.

So there is a space for another kind of AI company - one that uses local models, the killer product is a completely secure personal assistant that is able to do everything for you.

In conclusion, these factors will account for a demand for local models:

  • Enterprise:
    • Cost control
    • data privacy and security (esp in light of security event)
  • Retail:
    • Personal AI assistants demand a privacy model that didn’t exist before. It requires a new OS. This OS consists of a local server (think always-on ;) with various devices - watch, phone, glasses - as remote controls.

Section 4: How feasible are local models

References: https://www.modemguides.com/blogs/ai-infrastructure/best-open-source-llms-hardware-april-2026 https://www.layer3labs.io/open-weights/best-open-weights-ai-models https://www.digitalapplied.com/blog/best-open-weight-coding-models-self-host-hardware-match-2026

Here’s the kicker: if you get a $9k 128GB M5 MAX Mac, you can run last year’s frontier reasoning model running at reading speed, plus very fast mid-size models. For truly frontier-scale open weights you’d still need a desktop with more memory or a multi-GPU box.

What that buys you, from most to least ambitious:

~280B-class MoE at the top. DeepSeek V4 Flash (284B, MIT) is the best all-round pick at 39 tok/s on the M5 Max. That’s near-frontier reasoning at a comfortable reading speed, which is the headline case for this machine. Similarly,Qwen3-235B-A22B at ~3-bit (~100GB of weights) is exactly the kind of model that fits in 128GB but not 96GB.Aggressive quantization and tight memory, so expect some quality loss and less room for long context.

~120B MoE, the sweet spot. A 120B-class MoE model like GPT-OSS 120B (~65GB) or Qwen3.5 122B-A10B (~72GB) fits comfortably in the M5 Max’s 128GB unified memory, leaving plenty of headroom for context. These run around 15–30 tok/s and are probably the best quality-per-hassle on this hardware.

Dense 70B, if you need a specific one. Published leaderboard runs show ~15-18 tokens/sec on dense 70B models at Q4, and you could even run 70B at Q8 for better fidelity. But the current 120B-class MoE models deliver noticeably higher quality for a similar or lower memory cost, thanks to activating only a small fraction of their total parameters per token.

30B-class MoE for daily driving. Something like Qwen3.6-35B-A3B runs at 80+ tok/s with room to spare, fast enough to feel like a cloud API.

Two caveats. The M5’s big gain is prompt processing, not generation: Apple’s “up to 4x” is the best-case short-context, low-precision figure; at long contexts the real gain settles nearer 2x, and decode speed is still bandwidth-bound. And it still won’t run the biggest open models — even the 128GB tier leaves Llama 4 Maverick (400B) to a cloud API, and DeepSeek V4-Pro, GLM-5.2 and similar are out of reach. Wait for the M5 Ultra Mac Studio (expected around October 2026) if you want a desktop with the memory to run the big frontier MoEs that do not fit a laptop at all.

Section 5: A bet worth considering

Everyone is betting on more and more data centers required for AI to grow. That may be true.

But what if what’s needed is a set of harnesses to operate with open source models in order to squeeze out every last bit of performance for them?

What if the best way to do it is to look at the tasks that’s required, find a way to do it cheaply, and securely? I find the open model space very interesting to explore, and encompasses enormous commercial potential.

Section 6: Next steps: Benchmark specs to performance

  • We need to test open source models and create a benchmark for performance based on what specs our GPUs are.