I tested the following models. All models were tested using ollama on my MacBook Air M1 with 16 GB RAM and often with Firefox opened with too many tabs. Mainly, I used it for computer/programming related questions, but not for agentic tasks or IDE integration.

gemma4:e4b

It takes quite some time to get answers. And the factual knowledge answer fails.

In the test of summarizing PDFs of scientific articles, it worked quite well. It didn't do facual mistakes and got the gist of the paper. When I asked it to transform the PDF to Markdown, it did not what I wanted, but also wrote a summarized paper. It told me it did so, but ignored my initial request. During its so-called "thinking" process, it decided to summarize, so I guess it was not just running into the context window's limit. A more explicit prompt seemingly helps, but infact is also just not mentioning that it doesn't transforms the PDF to Markdown, but writes a bit more lengthy:

You are a good tool to transform from one data type to another. Transform this PDF to Markdown. Keep the structure and text as is, translate the equations to LaTeX equations within Markdown and only skip the images if you can’t handle them. But place the captions at an appropriate place between two paragraphs, e.g. if the first paragraph refers to figure 1 then place the caption after the first paragraph.

When it comes to coding, the probably easier task of a well known language and task failed after three attempt. It got some things correct, but failed badly when trying to correct its errors. It is very likely caused by not downloading the provided API documentation and thus just guessing the API's structures and responses. I tried to use ollama's command line interface, hoping it could do web access there, but it didn't succeed either.

As expected, when it comes to EPICS coding, it totally fails and can't follow a scheme to complete.

To conclude, it can write proper English sentences that seem to make sense and it even can deal with PDFs quite well. But when it would have to have a look at web resources or have profound background "knowledge"/training data for a specific field, it can't help. Also translating from English to German doesn't work too well, so that one has to thoroughly read through its translated text and check for typos.

gemma4:12b

As the missing web fetch capabilities—caused by ollama without login, not by the model—I added the ReST documentation of the Indico API to my Python coding prompt. Within two error correction attempt, this model got my task to work. That is quite impressive as some free as in beer models on the web, couldn't solve it in a similar situation.

However, when it comes to not well know codes—EPICS—, this model failed as well. Some aspects it got right to my surprise, but overall the provided code doesn't look like EPICS IOCs at all.

The PDF summary was quite ok. Similarly to the e4b model, it resited to convert the PDF to Markdown, but the more elaborated prompt worked.

Surprisingly, the factual "knowledge" in my test case was right. Finding the answer took again very long, but the answer was correct.

gemma3:1b

Deutsch nicht kann Grammatik. Ok, that wasn't proper German, but that's the point. It doesn't answer with correct Grammar.

Summarising a PDF of a paper about energy storage systems for accelerator facilities worked not well enough to use it, but reading the abstract, conclusion and captions might give you more and more correct information than the summary. However, I wouldn't have expected even that, as it requires to analyse the PDF at least.

However, it also halucinates probably easily knowable things like how to uninstall Ollama models and invents URLs.

gemma3:4b

The coding task, even with added documentation file, couldn't be succeeded.

The PDF's summary at least kind of worked. It was mostly right, so this task might be a task that this model might be used for. In practice, I would still read the abstract, conclusion, headings, introduction and figure's and table's caption to get the same information. Though this model was comparibly fast in contrast to the larger gemma4 models, on my hardware, I would still be faster with reading the aforementioned parts rather than generating the summary (which I would have to read then also).

So though these small models are still surprisingly good in comparison to models from some years ago which ran on the same hardware, but barely managed to form correct sentences, I am still searching for a good use-case for them. Maybe they are good for agentic tasks with tooling, though my experiments in that fieldweren't successful either some time ago.

One use-case might be to create README files and automated documentation or metadata extraction for many directories full of simulation research data.

qwen3.5

As this model got some parts right and some parts right, I gave it another round of correction, especially, because it added debug code at the second attempt. All in all, it managed to get it right in four itterations. Not bad for 6.6 GB instead of 7.6 GB (Gemini 3:12B) or even 9.6 GB (Gemini 3:E4B).

As this model was quite okayish, I once again tested the EPICS task. It looks a bit better than Gemma, especially it knows the different record types. Interestingly, though it knows the different record types, it does not know the syntax of a record itself.

Also the factual question was answered correctly.

The summary of the journal article and the conversion to Markdown with an elaborated prompt were also successful.

qwen2.5-coder

This model which was supposed to be good at coding, failed with the coding task, though it interestingly succeeded with the factual question. Maybe the improvement in the model 2.5 to 3 was in the overall coding capabilities. I would have thought that 4.7 GB is too small for the factual knowledge.

In comparison with the smaller, faster models, it doesn't improve enough, so I guess, I'll delete this model very soon.

The PDF task also just went kind of okay, but didn't convert the PDF correctly to Markdown.

qwen3-vl:4b

This vision language model probably doesn't perform better if not worse than the qwen3:4b model for the given tasks. Therefore, I add some vision tasks as well. Both generating as well as analysing tasks.

Tell me what one can see in the attached plot and give me table with all x and y values.

and

Generate me a picture of a kitten sitting on a power cord coming from wind mills and photovoltaics. Photorealistic, futuristic, blue tone.

Turns out: It can't generate images and thinks about suggesting Midjourney and Stable diffusion and wonders if I know of its inabilities to generate images. Usually, I use diffusion bee for image generation anyway, though “usually” I don't do it at all, because I don't like that aesthetics.

The image analysis part worked well, already at “Thinking” time, but it can't extract values from a plot. That means, that it is probably fine to use it for alternative text generation, but not well suited for “reverse engineering” plots from (old) articles where the data is not accessible.

The fact question was answered correctly.

The code generation was stuck in the "Thinking" process and seemed to ran out of context window.

As it tends to overthink, I'll ditch this model in favour of e.g. Gemma 3 which is as small as this one.

qwen3:8b

“Thinking” about the Python code took 18 minutes which is quite long. The larger models were faster which is suprising to me. However, this is similar to qwen3-vl:4b overthinking. At least it got it right at the third shot or two corrections.

Also the PDF to Markdown worked surprisingly well. It even got some minor spacing errors other models had right.

qwen3:4b

As this is the smallests model (2.5 GB), besides gemma3:1b (815 MB), I still want to test if gets the “faster” tasks of answering my factual question and writing the Python code right or not.

“Knowledge” question solved correctly.

It got the API part working correctly on first try which I did not expect. But it failed to get my intend and required me to provide the event ID rather than the full URL. I tried if it gets my intend from the error message with my input. It did not, but tried to improve on the error message and the how to use the script text. Nevertheless, I could tell it my intend again and got the correct solution. Other models also don't accept the URL, but require them to be put into the __main__ block, which I accepted as a result, so with this metrics, it got it immediately right. That is quite impressive for this small model.

As it performed that well, I once again tried to see if EPICS IOCs can be written by local models. And yes, it got closest to the record syntax so far—but still not close enough.

Maybe I was just lucky with my tasks, but in this test the 4b model performed better than the 8b model.

gpt-oss:20b

This model doesn't work at all. It stopps with a complaint about too little resources.

Further models to evaluate

rnj-1:8b

I just read a review or somehow stumbled upon them, but don't know them. But they claim to be good at code and science, so maybe they perform well in the test.

mistral-3

The European/French alternative

deepseek-r1:7b

The first Chinese SOTA model in its small incarnation.

granite4.2:8b

IBM's small model.

nemotron-3-nano:4b

NVIDIA's model

ornith-1.5:9b

I haven't heard of them yet: https://ollama.com/library/ornith-1.5

lfm2.5

Not heard of either.

olmo-3:7b

Research LLM/SLM.

OCR models

• glm-ocr
• deepseek-ocr:3b

Summary

Except for some special use cases where it is sufficient to write text, the smaller models are not very satisfying though the larger models are very slow. However, the larger models give reasonable correct answers to my typical tasks, I would want to test to give to LLMs. Maybe, I manage to do some agentic coding tasks with different local models and write another blog post about that.

So my model to go probably will be Qwen 3.5 9B as it is capable of solving most of the given tasks, feels a bit faster than Gemma 4 12B and is about 1 GB smaller on disk. Furthermore, there is a project called MTPLX that optimises some quantisized models especially for individual Macs. There Qwen 3.5 9B is also the suggested model for 16 GB Macs.

Prompts:

Python coding:

Write me a script that gets the beginning and end of an event and its location for a given indico page. (API documentation at https://docs.getindico.io/en/stable/http-api/exporters/event/ also attached) e.g. I provide https://indico.ph.liv.ac.uk/event/2025/overview and it gets Beginning: 2026-03-30 08:30:00Z End: 2026-04-01 18:00:00Z Location: University of Liverpool Theatre 2, Teaching Hub 502 Liverpool L69 7ZP

Also I give three attempts, by putting two outputs/tracebacks into the next prompt if the first attemt fails. I don't put any further context nor wording around the output.

Claude Sonnet 5 with high thinking solves this task at first try. Mistral does not. Neither without being logged in and thus using "vibe" in the fast mode, nor being logged in and enableing "thinking" and "code-interpreter". It only worked out after some back and forth where I also followed Mistral Le Chat's instructions—not like in the benchmark.

EPICS coding:

You are an EPICS control system expert and know how to use the OPC-UA module very well. Please write me the records, templates, substitution files and startup file for a He measurement PLC. The values are not all in the correct units, so for those int that are internal only (Property prefixed with Internal) have to be devided by 1000 and than exposed as a PV without the Internal prefix. e.g. :InternalFlow returns (int)12345, then you have to write the CALC records to also have :Flow returning (float)1.2345. The IP of the PLC is 192.0.2.42. The names of the data fields, their type and the corresponding property name are listed in the following table. The IOC has got the prefix A:BS: and device name HeGauge:01 where the full PV name then is something like A:BS:HeGauge:01:Description:Info for the property Description:Info. | Data field | Type | Property | EGU | |------------|------|----------|-----| | 349_Helium_Durchflussmenge | float | Flow:Volume | m^3 | | 349_Helium_Durchfluss | int | InternalFlow | (m^3/h)/1000 | | InternalFlow | float | Flow | (m^3/h) | | 349_Helium_Temperatur | int | InternalTemp | (K)/100 | | InternalTemp | float | Temp | K | | 349_Helium_Druck | int | InternalPressure | (bar)/100 | | InternalPressure | float | Pressure | bar | | "349_Helium_Geraet OK" | bool | Status:Ok | | | "349_Helium_Ausserhalb der Spezifikation" | bool | Status:OoSpecs | | | "349_Helium_Funktionsprüfung" | bool | Status:Check | | | "349_Helium_Fehler" | bool | Status:Error | |

PDF summary:

Attached PDF

Please write me a summary of the paper.

Transform this PDF to Markdown.

You are a good tool to transform from one data type to another. Transform this PDF to Markdown. Keep the structure and text as is, translate the equations to LaTeX equations within Markdown and only skip the images if you can’t handle them. But place the captions at an appropriate place between two paragraphs, e.g. if the first paragraph refers to figure 1 then place the caption after the first paragraph.

Factual knowledge:

Where was the parliament of Germany (FRG) in the 1990ies, before and after?

The online models GPT-5.6 Luna, Mistral Vibe small, or Anthropic Claude's Haiku 4.5, for example, get it right and know that it moved from Bonn to Berlin. Most likely that is because they do search the web for it, but maybe also because they read it in their training data.