colin/nix-files
1
0
Fork 1
Code Issues Pull Requests Projects Releases Wiki Activity

ollama: ship deepseek-coder-v2-16b-lite-instruct-q5_1,qwen2_5-32b-instruct-q2_K

Browse Source
This commit is contained in:
Colin
2024-10-13 18:56:05 +00:00
parent 7f5b262801
commit 455b05e99a
5 changed files with 33 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
modules/services/ollama.nix
View File

@@ -24,6 +24,7 @@ let
codestral-22b
# deepseek-coder-7b # subpar to deepseek-coder-v2 in nearly every way
deepseek-coder-v2-16b # GREAT balance between speed and code quality. code is superior to qwen2_5 in some ways, and inferior in others
deepseek-coder-v2-16b-lite-instruct-q5_1 # higher-res version of default 16b (so, better answers?)
# falcon2-11b # code examples are lacking
# gemma2-9b # fast, but not great for code
# glm4-9b # it generates invalid code
@@ -40,6 +41,7 @@ let
# phi3_5-3b # generates invalid code
# qwen2_5-7b # notably less quality than 32b (i.e. generates invalid code)
qwen2_5-14b # *almost* same quality to 32b variant, but faster
qwen2_5-32b-instruct-q2_K # lower-res version of default 32b (so, faster?)
qwen2_5-32b # generates 3~5 words/sec, but notably more accurate than coder-7b
# qwen2_5-coder-7b # fast, and concise, but generates invalid code
# solar-pro-22b # generates invalid code