Article · MarkTechPost · Saturday 7 November

How to Run NVIDIA Dory 4B with llama.cpp: Loop by Loop Guide

Explore a comprehensive coding guide for NVIDIA's Dory 1.0 4B looped language model. Learn how to build the pinned llama.cpp branch, analyze loop-by-loop compute costs, test memory caching, and run paired evaluations.

Read at marktechpost.com The opening of the article at marktechpost.com.

In this story

A tool or model opens in the Heidelberg AI Curriculum: what it does, what it costs, how to start. A repo opens on GitHub.

More from MarkTechPost

All stories

ArticleMarkTechPost

Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.

13 h ago

ArticleMarkTechPost

Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster

Zyphra released ZYFA, open-source FP8 and BF16 prefill attention kernels for AMD Instinct MI355X. On real model activations, its FP8 kernels run 1.16-1.76x faster than AMD’s AITER, with 62-78% lower error at head dimension 128.

1 day ago

ArticleMarkTechPost2 sources

Thinking Machines Releases Inkling 2 Small, a 594B MoE With 17B Active

Thinking Machines Lab released Inkling 2 Small, a 594B-parameter MoE with 17B active parameters. It scores 31 on the Artificial Analysis Intelligence Index and runs on Tinker, with Apache 2.0 weights planned.

Repothinking-machines-lab/tinker-cookbook

2 days ago

ArticleMarkTechPost

Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI released Together Compiler, an agent-driven loop that keeps re-optimizing the inference forward pass as traffic shifts. It beat the best InferenceX configurations by up to 2.6x TPS in under 24 hours per model.

3 days ago