Carbon – An experimental Apple Silicon DSL for infernce

(github.com)

3 points | by alazarmb 4 hours ago ago

2 comments

  • alazarmb 4 hours ago ago

    While model-specific inference engine vs general debate is quite prevalent, I asked if the real bottleneck is the primitive, I think the real answer lies in redesigning a new primitive!

    With agents, a mlx-alternative didn't seem too ambitious even for a weekend project!

    Carbon is a tile-oriented first Python DSL, so the inference engine up-top(Swiss -> https://github.com/Lazarus-931/swiss), can be kept simple, yet deliver on-par performance as model specific engines like strata/ds4!

    The most promising thing is, we can continuously iterate carbon's internals to keep swiss clean, performative and simple.

    On an M4 Mac mini (16 GB), at batch sizes 1, 2 and 4 and contexts of 2K–16K, Swiss on Qwen3.5-9B ran prefill at 1.07× and decode at 1.00× the speed of mlx-lm on average, and prefill at 1.08× and decode at 1.08× the speed of mlx-vlm. On Qwen3-8B it was 1.07× and 1.05× against mlx-lm, and 1.08× and 1.05× against mlx-vlm. On Qwen3-4B it was 1.10× and 1.07× against mlx-lm, and 1.08× and 1.07× against mlx-vlm. On LFM3-1.7B it was 1.15× and 1.08× against mlx-vlm; mlx-lm does not support LFM3. On Gemma 4 12B, at batch 1 only, it was 1.20× and 1.02× against mlx-lm, and 1.18× and 1.02× against mlx-vlm.

    With speculative decoding on the same M4 (batch 1, prompts of up to 1.5K tokens, greedy), Swiss runs Gemma 4 12B with its bf16 assistant at 1.17–1.30× the speed of mlx-vlm with the same drafter, and with 4-bit drafters it reaches up to 2.7× its own plain decode, for example 41–57 tok/s on Qwen3.5-9B with DFlash against 29–35 tok/s for mlx-vlm with DFlash.

    I think this was a fun experiment, and hopefully will now explore further agent driven optimizations in carbonIR. I think there is more efficiency we can squeeze out!

    More models will be included once/if hardware to run them on, feel free to contribute as well!

    I'd appreciate feedback or ideas!

    • alazarmb 4 hours ago ago

      note:

      >Carbon is a tile-oriented first Python DSL, so the inference engine up-top(Swiss -> https://github.com/Lazarus-931/swiss), can be kept simple, yet deliver on-par performance as model specific engines like strata/ds4!

      might be misleading, but this is a more of 'could be' given the slightly higher performance we can squeeze out from the silicon with carbon