mariano/nvi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

langfuse generation spans with token usage; switch llm to vllm (qwen2.5-coder-7b on mcrndeb)

Browse Source
This commit is contained in:
buenosairesam
2026-06-03 10:33:22 -03:00
parent bd34c8c5ce
commit 61494362a3
7 changed files with 103 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
ctrl/k8s/base/configmap.yaml
View File

@@ -13,7 +13,13 @@ data:
# LLM. Supported providers: groq, anthropic, openai.
# Secrets (*_API_KEY) live in the Secret built by kustomize from .env.
LLM_PROVIDER: "groq"
# Flip LLM_PROVIDER to switch — the other configs stay so we can A/B easily.
LLM_PROVIDER: "openai"
# Local vLLM serving Qwen2.5-Coder-7B-Instruct-AWQ on mcrndeb.
OPENAI_BASE_URL: "http://mcrndeb:11000/v1"
OPENAI_MODEL: "Qwen/Qwen2.5-Coder-7B-Instruct-AWQ"
GROQ_MODEL: "llama-3.3-70b-versatile"
ANTHROPIC_MODEL: "claude-sonnet-4-6"