mahangu@wordpress:~$ cd ~/ && ls -la

mahangu@wordpress:~/blog$ cat

Inkling 2 Mini + Spacefast = ❤️

Inkling 2 Mini + Spacefast = ❤️


Make it with Spacefast. Everything your AI needs to build your next idea. A hill at dusk, where someone sits with a glowing laptop.

Inkling Mini 2

I have wanted a decent local coding model on my laptop for a while. When a Inkling 2 Mini thread showed up on Hacker News, it was full of people comparing quants, llama.cpp patches, and runtimes. I did not have the time or patience to work through all of that myself, so I pointed Claude Code at the thread and asked it to find the best setup for my 64GB M4 Max MacBook Pro and install it.

It worked! 🎉 I now have a great little local model running in pi, and it is currently helping me update Bird or Not, my Sri Lankan bird quiz, which runs on Spacefast. A free model on my laptop and a free home on the web for what it builds is a combination I am very happy with.

Spacefast

Spacefast is where your AI agent publishes what it builds. My colleagues at Automattic launched it recently, and Bird or Not has lived there from the start. I describe what I want, the agent writes the code, and Spacefast puts it on the web at birdornot.com. I never had to set up a server.

Our CEO Matt wrote up the Spacefast story a couple of days ago, and one line in it describes this setup well:

I believe the future is much more “bring your own model”/BYOM than trying to outpace the progress of the frontier labs.

That is what I now have. The model is free to download and it runs on my own laptop, so my code and my prompts stay on my machine. The code for BirdOrNot is GPL v2. Spacefast hosts the site, and its free tier is very generous.

I think a free local model + Spacefast is a great story for the open web. A hobby project like this used to need a paid API for the model and a hosting bill for the site. Now a laptop and a free account cover both. I did use Claude, which is a hosted model, to do the setup, but day to day the local model now does the work.

What we ended up with

  • oMLX 0.8.2 serves the model, and it starts when I log in.
  • The model is mlx-community/Inkling-2-Mini-oQ4e-mtp, which is about 17GB on disk.
  • pi as the harness, with reasoning effort set to low.
  • The context window is 65,536 tokens.
  • oMLX keeps up to 20GB of conversation cache on the SSD.
  • The pi-vcc extension handles auto-compaction in pi.
  • The pi-output-policy extension, in compact mode, caps large tool results and saves the full text to a file. A Wikipedia page that used to add 11,915 tokens to the conversation now adds 2,778, and that task went from 63 seconds to 9.

What made the difference

We started with Ollama and it ran fine. Three things made the setup much faster after that.

  1. Low Power Mode was on. Turning it off took short prompts from about 21 tokens per second to about 50. If your local model feels slow on a MacBook, check this first.
  2. oMLX was quicker on follow-up turns. With 44k tokens of conversation already loaded, a follow-up turn took 2.7 seconds on oMLX and 7.2 seconds on Ollama. oMLX also saves the conversation cache to the SSD, so after a server restart a 45k-token conversation came back in 5.9 seconds. Without the cache, the same reload took 94.5 seconds.
  3. pi was summarising my conversation over and over. I blamed the server for some very slow turns, but Claude found seven compactions in a single pi session file. Once the conversation passed about 49k tokens, pi kept asking the model to summarise it, and a local model is slow at that.

Auto-compaction setup

pi compacts a conversation automatically when it gets within 16,384 tokens of the model’s context window. With my 65,536-token window, compaction starts at about 49k tokens.

By default, pi asks the model itself to write the summary. The pi-vcc extension replaces that step and builds the summary without calling the model. I added it to the packages list in ~/.pi/agent/settings.json, and told pi to keep the most recent 8,000 tokens of the conversation as they are:

{
"packages": ["npm:@sting8k/pi-vcc@0.11.2"],
"compaction": {
"modelOverrides": {
"omlx/Inkling-2-Mini-oQ4e-mtp": { "keepRecentTokens": 8000 }
}
}
}

pi-vcc has its own settings in ~/.pi/agent/pi-vcc-config.json. I use hosted models in pi too, so I list those providers under skipForProviders and they keep pi’s normal compaction:

{
"overrideDefaultCompaction": true,
"smartKeepTail": true,
"continueAfterThresholdCompact": true,
"skipForProviders": ["anthropic", "openai", "google"]
}

Is it any good?

I asked Claude to run the full HumanEval benchmark at low effort with one attempt per problem, and the model passed 159 of 164 problems. It averaged about 44 tokens per second on those short prompts. Speed drops as the conversation grows, so at 45k tokens of context I get about 20 tokens per second.

I also wanted to know whether the quantized model loses quality. Thinking Machines trained the model natively in 4-bit, with the last four layers in 8-bit. The oQ4e build keeps most of those 4-bit weights as they are and only shrinks the last four layers to 4-bit. I asked Claude to compare it with an oQ8e build that keeps those layers at 8-bit, and the two scored within noise: 31 and 30 of 48 coding samples passed. The oQ8e build also took 45% longer to read a 48k-token prompt (106 seconds against 73), so I stayed with oQ4e.

I first set the context window to 81,920 tokens, and that was too much for 64GB. A long session ran out of memory at about 68k tokens, when the model fetched a long web page in one go. I went back to 65,536, which means pi compacts a session at about 49k tokens.

Get started with your agent

If you want the same setup, paste this into Claude Code, Codex, or whichever coding agent you use. I have linked the Hacker News thread in the prompt so your agent can mine it for other hardware.

Set up Inkling 2 Mini as a local coding model on this Mac.

First read this Hacker News thread and the sources it links to, because the
right setup depends on my hardware:
https://news.ycombinator.com/item?id=50581260

Check my chip, RAM, and free disk space before you choose anything. Tell me
what you plan to install and wait for my OK.

This is what worked on a 64GB M4 Max MacBook Pro. Use it as a starting point
and adjust it for my machine:

- Server: oMLX 0.8.2, installed with `brew install jundot/omlx/omlx`
- Model: mlx-community/Inkling-2-Mini-oQ4e-mtp (about 17GB)
- Serve command:
  omlx serve --host 127.0.0.1 --port 8000 --max-concurrent-requests 2 \
    --paged-ssd-cache-dir ~/.omlx/ssd-cache.noindex \
    --paged-ssd-cache-max-size 20GB --hot-cache-max-size 0
- Run the server as a launchd agent so it starts at login.
- Context window: 65,536 tokens. With 81,920, a long session ran out of memory
  at about 68k tokens, so do not go higher on 64GB.
- Reasoning effort: low
- Coding agent: pi, with the model added as an OpenAI-compatible provider at
  http://127.0.0.1:8000/v1
- Turn Low Power Mode off. It halved my speed.

Set up auto-compaction in pi so the local model never has to summarise the
conversation itself:

- Add "npm:@sting8k/pi-vcc@0.11.2" to "packages" in ~/.pi/agent/settings.json.
- In the same file, set compaction.modelOverrides for this model to
  { "keepRecentTokens": 8000 }.
- In ~/.pi/agent/pi-vcc-config.json, set "overrideDefaultCompaction": true,
  "smartKeepTail": true, and "continueAfterThresholdCompact": true.
- If I use hosted models in pi too, list their providers under
  "skipForProviders" so they keep pi's normal compaction.
- pi compacts when the conversation gets within 16,384 tokens of the context
  window, so tell me the token count where compaction will start.

Cap large tool results, so one long web page or command output cannot flood
the conversation:

- Run `pi install npm:@vanillagreen/pi-output-policy`.
- In ~/.pi/agent/settings.json, set "policyMode": "compact" for it under
  kendex.extensionManager.config["@vanillagreen/pi-output-policy"].
- Fetch a long web page in pi before and after, and tell me how many tokens
  each fetch added to the conversation.

Keep the server on 127.0.0.1 only.

After setup, measure and report: tokens per second on a short prompt and on a
long one, how long a follow-up turn takes, and whether the Mac is swapping.

If I have less than 64GB of RAM, or I am not on Apple Silicon, use the thread
to find a setup that fits and tell me what you changed.

Then put what you build on Spacefast

Once your local model has built something, give it a home on the web.

You do not need an account to try it. Just point your agent at spacefast.com/setup.md.

Let me know what you build! I’d love to check it out.


← prev post

Leave a Reply

Your email address will not be published. Required fields are marked *