I think it's almost never worth it now. An up to date LLM with a good few-shot prompt, or an agentic LLM will likely outperform a small finetuned LLM, at a lower cost with more flexibility.
Moreover, it's very seldom that you have enough high-quality data to do a meaningful fine-tune.
I have successfully finetuned a small Gemma to play chess badly, and that worked. Mostly because chess data is plentiful and because my aim was to not have perfect answers. I'm self hosting the model inference because running it in the cloud would be too costly.
I also finetuned small LLMs using too small datasets of business specific human conversations and the results were disappointing.
Thanks for your insights. Can you share more about your experience in self hosting the model inference to reduce cost or make it even zero if possible ?
I fine-tuned a model for my first AI web app back in 2019. It created plausible reviews of artwork from prompts. They were totally hallucinated though and had nothing to do with actual images. It was interesting for language and bias analysis above all else. But, it was enough of a novelty that I got thousands of users. I have considered going through the whole process from scratch with Mistral to make a bespoke model and correlate it with a modern vision model. But, I don't want to spend any more money on this project. Hosting an unmonetized public facing AI app for 7 years was not cheap.
I've trained two small models, one I attempted to train it to be an expert (decent?) hearts (card game) player, and the other I trained on dream interpretation texts, I wanted to see if a model could interpret dreams reasonably well if it were trained on all the literature in existence (that I could find anyway). The hearts experiment was a general failure because the game of hearts as played by humans just has too many strategies to consider. I trained it purely on games (~20,000 hands of hearts in the RL dataset I generated) and the trained model did about as good as any decently written procedural game logic would have. the dream analyst training was a bit more successful. I can hand the model a dream log and it will offer some interesting insights. are they accurate? hard to say, it's a very subjective type of training, but definitely a better experiment than the hearts expert training run. in both cases the biggest and most time intensive part is generating the training dataset. building the LoRa is the easy part.
I was using Qwen3.5:2b models for both, running on Dell Pro Max GB10 Cuda,128GB.
There are allegedly multi-tenant LoRA offerings in the works which would change the hosting-pricing constraints considerably. Keeping my fingers crossed they materialize
I personally would only do it if I need it to solve a highly specialized problem that requires every inch of privacy I can get. Otherwise there are better options out there
I think it's almost never worth it now. An up to date LLM with a good few-shot prompt, or an agentic LLM will likely outperform a small finetuned LLM, at a lower cost with more flexibility.
Moreover, it's very seldom that you have enough high-quality data to do a meaningful fine-tune.
I have successfully finetuned a small Gemma to play chess badly, and that worked. Mostly because chess data is plentiful and because my aim was to not have perfect answers. I'm self hosting the model inference because running it in the cloud would be too costly.
I also finetuned small LLMs using too small datasets of business specific human conversations and the results were disappointing.
Thanks for your insights. Can you share more about your experience in self hosting the model inference to reduce cost or make it even zero if possible ?
I fine-tuned a model for my first AI web app back in 2019. It created plausible reviews of artwork from prompts. They were totally hallucinated though and had nothing to do with actual images. It was interesting for language and bias analysis above all else. But, it was enough of a novelty that I got thousands of users. I have considered going through the whole process from scratch with Mistral to make a bespoke model and correlate it with a modern vision model. But, I don't want to spend any more money on this project. Hosting an unmonetized public facing AI app for 7 years was not cheap.
https://artreviewgenerator.com/
I've trained two small models, one I attempted to train it to be an expert (decent?) hearts (card game) player, and the other I trained on dream interpretation texts, I wanted to see if a model could interpret dreams reasonably well if it were trained on all the literature in existence (that I could find anyway). The hearts experiment was a general failure because the game of hearts as played by humans just has too many strategies to consider. I trained it purely on games (~20,000 hands of hearts in the RL dataset I generated) and the trained model did about as good as any decently written procedural game logic would have. the dream analyst training was a bit more successful. I can hand the model a dream log and it will offer some interesting insights. are they accurate? hard to say, it's a very subjective type of training, but definitely a better experiment than the hearts expert training run. in both cases the biggest and most time intensive part is generating the training dataset. building the LoRa is the easy part.
I was using Qwen3.5:2b models for both, running on Dell Pro Max GB10 Cuda,128GB.
Interesting experiment. My favorite is the dream one. Can you share more about datasets you got to train on for this?
That seems interesting
It's worth it if a) you have a decent sample size of data for your problem and b) you have a cost-effective infra to host it.
Notably the latter is more of the bottleneck, particularly with the price race-to-zero with models such as GPT-6 Luna.
There are allegedly multi-tenant LoRA offerings in the works which would change the hosting-pricing constraints considerably. Keeping my fingers crossed they materialize
I personally would only do it if I need it to solve a highly specialized problem that requires every inch of privacy I can get. Otherwise there are better options out there
Maybe when you want to learn the process for when/if you ever need to fine-tune a larger model?
It's faster to make iterate when you're toying around with a 1B model than a 27B one.
that's a very good reason to try this!