Inference economics

Thinking about small-model serving economics

Wrote about serving small models that sit idle most of the time, and what it costs to keep a model loaded in memory.

Inference economics, Resource scheduling, Small models

Notes & links

From a public post. Public post ↗ · Link to this entry