在本地部署大模型或构建 RAG 系统时，你可能遇到这些痛点：
• 某些工具 只支持 OpenAI API（如 Neo4j GenAI、部分 Spring AI 配置、某些前端库等）
• 但你想用 Ollama 或本地模型
• 不同 LLM API 格式各不相同，调用麻烦
• 想做 统一 API、日志、限流、负载均衡、代理管理
这就是 LiteLLM 存在的意义。
笔者就是在做Neo4j GenAI的过程中，遇到了不支持Ollama embedding 的问题,所以才转向LiteLLM需求帮助.
LiteLLM 是什么？
LiteLLM = LLM API 统一代理层（Proxy + Router）
LiteLLM 让所有大模型都“看起来像 OpenAI API”。
LiteLLM 能把以下模型：
• OpenAI GPT 全系列
• Azure OpenAI
• Anthropic Claude
• Google Gemini
• HuggingFace
• Ollama（本地模型）
• Mistral
• Groq
• DeepSeek
• Llama.cpp
• 甚至自建模型后台
全部统一为 OpenAI API 标准格式：
POST /v1/chat/completions
POST /v1/embeddings
这意味着：
只要某个系统支持 OpenAI API，就能被 LiteLLM 兼容扩展到所有模型。
LiteLLM 能解决哪些核心问题？
1. 让不支持 Ollama 的系统使用 Ollama
例如：
• Neo4j GenAI 不支持 Ollama provider
• Spring AI 某些模块不支持 Ollama
• LangChain OpenAI-only 模式也不能直接用 Ollama
但通过 LiteLLM：
RETURN genai.vector.encode(
"embedding_field",
"OpenAi", //代理openAI
{
model: "dummy",
endpoint: "http://localhost:4000"
}
);
Neo4j 会认为你访问的是 OpenAI
实际请求链路如下：
Neo4j GenAI
|
| (OpenAI API 请求)
v
LiteLLM Proxy <-- 你本地搭建的 OpenAI API 兼容服务
|
| (Ollama embedding API)
v
Ollama Server
2. 统一不同 LLM 的调用方式
你不需要针对不同模型（GPT、Gemini、Claude）写不同 API：
只需要 OpenAI 风格的 payload。
3. 提供多后端路由、fallback 和负载均衡
例如：
• 首选本地 Ollama，但若挂掉自动切换到 OpenAI
• 将 embedding 请求分散到多个 GPU 节点
• 多 API Key 自动切换
这些 LiteLLM 都支持。
4. 自建“类 OpenAI 服务”
LiteLLM 可以让你自己搭一个：
http://localhost:4000/v1/chat/completions
等于：
• 你自己就是 OpenAI
• 可以接入任何外部或本地模型
LiteLLM 使用
1. 安装 LiteLLM
pip install litellm
pip install 'litellm[proxy]'
2. 启动代理，将 Ollama 封装为 OpenAI API
litellm --model ollama/qwen3-embedding:0.6b
启动后默认暴露
http://localhost:4000/v1/chat/completions
http://localhost:4000/v1/embeddings
验证1
验证2
curl -X POST http://localhost:4000/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer sk-dummy-key" -d '{"input": "test text", "model": "qwen3-embedding:0.6b"}'
{"model":"ollama/qwen3-embedding:0.6b","data":[{"object":"embedding","index":0,"embedding":[-0.032626607,-0.04718259,-0.012909385,-0.07764769,-0.017243367,0.020494336,0.013601924,0.045632806,-0.07786213,0.009070601,0.0066449605,-0.07473664,0.09884253,-0.0120289,-0.06222988,0.06184298,-0.0725233,0.053415477,0.10230555,-0.03260747,-0.017293224,-0.0084129665,-0.03581761,0.13351837,-0.016029868,0.04569618,-0.06392779,0.053953223,0.034195043,-0.025214326,0.00022348277,-0.029752325,-0.06300391,-0.021198025,0.011440582,-0.014686229,-0.0030991829,0.02269575,-0.015596619,0.025833748,-0.024328208,0.036079746,0.014850982,0.016713746,-0.031154487,-0.065592736,0.03154194,0.03257276,0.04791407,0.01860942,-0.019520456,0.007815935,-0.0022338692,0.022617716,-0.005324301,-0.045967944,0.027742606,0.015884312,0.014938865,0.025698539,-0.03921946,0.09372251,-0.006558598,-0.008122129,-0.0030283695,-0.009300921,0.02448351,-0.043731984,-0.13481855,-0.028205048,0.020778868,-0.00405425,-0.03623363,0.021633552,0.00044696862,-0.012933407,-0.033308834,-0.009971476,0.005323327,0.030190341,-0.01743355,0.062128257,0.023755023,-0.00015589697,-0.011969491,0.04809842,0.011429985,-0.0023270582,0.014009929,-0.013220596,-0.0130167585,0.053460836,-0.0309704,0.023170978,0.009875285,-0.029823707,-0.024622424,-0.0011155787,-0.045939308,0.011828468,-0.009233859,-0.026525205,0.024650486,-0.039584227,-0.038674805,0.05581028,-0.071160346,-0.015343709,-0.037775666,-0.023587836,0.04493225,0.0044017094,0.010756107,0.02482149,-0.028296681,-0.03887348,0.0027553057,-0.025629668,-0.0030361188,0.015511015,0.036494546,-0.04785365,0.00055010477,-0.008078041,0.004792498,0.024970999,-0.032102983,-0.013614414,0.028776105,-0.03336054,0.023151165,-0.02478438,-0.037401527,-0.0831715,0.012158645,0.008711268,0.005656051,0.041605253,0.038398396,0.022099143,0.009310389,0.017901905,-0.003372163,-0.02984916,0.015566409,0.018100621,0.0038577102,-0.013624389,0.021733787,-0.035946332,0.014075876,0.008954101,-0.0145281,-0.012985507,-0.00008892349,0.024338393,-0.006873558,-0.024779137,0.014932994,0.02422854,0.019953707,-0.018292883,-0.024485217,-0.022343619,-0.00854652,0.0060430225,-0.05132581,-0.02891697,0.0362885,-0.0071165753,-0.00047108222,0.012760885,0.0067583066,0.010371983,-0.01082039,-0.015063237,-0.03978506,-0.039938226,-0.027284574,-0.0054394174,0.019066727,-0.028152809,-0.013166358,-0.0073102275,0.017402485,-0.008219373,0.03523686,-0.003599994,-0.02246328,-0.00013300263,-0.03090953,-0.043205082,-0.043990586,-0.012077176,-0.010643322,-0.03899131,0.028445436,-0.02997106,-0.02135719,0.0014877928,0.004239334,-0.07422561,0.03053671,0.018447932,-0.022379674,0.005374792,0.04301634,-0.008634935,-0.022366231,-0.015306392,0.020601274,0.024734389,-0.034077384,0.00429294,-0.02002732,0.035747744,-0.008067649,0.017709957,0.017443566,0.017220013,0.045218933,0.004526733,-0.031537473,0.013991209,0.05438411,-0.03481289,0.013039455,-0.0140739335,-0.03516529,0.042347185,-0.009062282,-0.005830926,0.035344925,-0.002221362,0.08372916,-0.018425262,0.016336745,0.0022677209,0.03609259,-0.020100094,-0.013024532,0.015118401,-0.014650689,-0.0041352822,-0.046367176,0.028414916,0.0016237067,0.0015307447,-0.013305592,0.020143444,-0.03970218,-0.06460047,-0.026737811,0.035010505,0.004334833,-0.005978129,-0.005473762,-0.04095557,-0.008331795,0.024143105,0.04152064,-0.04720525,-0.008470502,0.035948463,0.07677461,0.008347462,-0.021463059,0.046810616,-0.028483596,-0.018201763,0.051549885,0.02149343,0.0019038805,-0.025041798,-0.0158813,0.0007537609,0.019974694,0.06837892,0.05036703,0.06935548,-0.013435698,-0.004591738,0.011701604,-0.06660126,0.022352964,0.027173858,0.044638038,0.031489555,-0.045690794,0.012274531,0.014966118,0.014843109,-0.06141054,0.024512075,-0.021520602,-0.013230133,-0.0130254775,0.075040884,0.02483831,0.010204212,0.0003535681,-0.044644386,-0.0133635225,-0.002034496,-0.045935355,0.0048054284,0.0012997395,-0.03977537,0.03240054,-0.04400314,0.030716144,0.016531117,-0.006736844,-0.030641975,0.012706248,-0.021987136,-0.02151764,0.11139711,-0.0114127835,-0.011911694,-0.029264389,0.018864004,-0.041995883,0.0077304784,0.003821275,0.030130727,-0.012936846,0.00028421948,0.021635085,-0.051617797,0.015496302,0.009171116,0.022512116,-0.009177935,0.008350419,0.063435055,-0.009066847,0.0025567703,-0.0075443746,0.0047093974,0.024056355,-0.025160728,0.019159088,0.000070561575,-0.028550267,0.0032737663,-0.078368396,0.04596512,0.025343953,0.030409817,-0.05242445,0.028688649,0.0123399645,0.007895326,-0.04171273,-0.00957569,-0.013085938,0.009248653,-0.0361861,-0.021248043,-0.02800194,-0.10728622,0.0052113673,0.009283722,-0.04429127,-0.02771819,-0.024536816,-0.008435417,0.053151242,0.03849845,-0.031428974,-0.00017730048,-0.021307925,-0.0005476149,0.033995736,0.08230147,0.050586957,-0.027417632,0.018314833,0.029491404,0.040851604,0.011517337,-0.035452057,0.03033822,-0.055938926,-0.004189061,-0.008351891,-0.039090905,-0.02521377,0.0061673066,0.010453673,-0.041307043,0.005660619,-0.028021295,0.040069263,0.029129388,-0.091972366,-0.005480398,-0.0062635555,-0.019887539,0.004006535,-0.00285424,-0.063212804,0.008649224,-0.024517544,-0.028093515,-0.014843129,0.020942522,-0.0024359324,-0.024864849,0.06545675,-0.016256057,0.0032324963,-0.011839713,0.0469201,0.0039894576,0.1263978,-0.020731382,0.02428568,0.016100006,0.002276005,0.044113874,-0.01926966,-0.051847033,-0.039138526,-0.017483212,-0.046121262,-0.041827787,-0.039464936,0.07806275,-0.011396858,-0.067411505,-0.020488096,0.043047965,-0.007854514,-0.009907425,0.09150255,-0.010619751,-0.00084575906,-0.027138874,-0.025682852,0.043993562,0.01803754,-0.04126101,-0.030861838,-0.011177407,0.007616346,0.019447092,0.045689527,0.022759655,0.04255245,-0.027411882,0.00456836,0.014944668,-0.018570218,-0.053016607,-0.003755381,0.008989129,0.02446066,0.027432555,-0.00049787294,0.017284768,0.01083616,-0.01018242,-0.023030175,-0.012628122,-0.016090894,-0.00945091,0.03420952,-0.004679526,-0.00940997,0.05233846,-0.021455273,-0.007825919,0.022946468,0.007983177,-0.04117699,-0.018207991,-0.040607706,0.03679141,-0.010492273,-0.028260851,0.03931189,-0.03398914,0.05503198,-0.054683894,0.02540339,-0.035789736,-0.06988028,0.0047625904,-0.008959152,0.05457937,-0.009224968,-0.057090748,-0.017567148,0.004170673,-0.029513808,-0.026938377,-0.07277286,0.0026025707,-0.008088297,-0.008559252,0.028710073,0.00566963,-0.045421716,0.01728638,0.022351412,-0.03562261,0.057577536,0.026763096,0.020060437,-0.050578874,-0.031154504,0.028207341,-0.027969278,0.0012136606,-0.0070124716,0.042859122,-0.035354283,0.008864329,-0.017908564,0.05229721,0.019056672,0.030146556,0.033598967,0.040245075,0.016694557,-0.015736451,-0.0028101304,-0.005406192,0.0104496125,0.02547434,-0.010537043,0.025137035,0.01908449,-0.0019907346,0.048959177,0.0024705038,-0.055276744,-0.015167754,-0.04585505,-0.010799167,-0.00065173244,-0.034728292,0.03921607,-0.01755213,-0.007979864,0.011929722,0.040924948,0.04465184,0.030821672,-0.0002852722,0.074697904,-0.021361176,0.014629442,-0.0015847867,0.042675804,0.00014287626,0.008291842,-0.012882184,-0.019738251,-0.009384589,-0.013303583,-0.031510968,0.02005657,0.00042516558,0.015813623,-0.012054695,0.03184884,-0.00084282964,0.05006034,0.003993865,0.019187452,0.029896155,0.034363784,-0.04589113,-0.011824018,0.01966721,-0.0142141115,-0.04032144,0.009241516,0.028885357,-0.014557815,-0.03623026,-0.015700225,-0.01446583,-0.016293189,0.013462364,0.005955621,-0.012553529,-0.034342274,-0.018369908,-0.009783602,-0.010132471,0.061206385,-0.006967328,-0.04086951,-0.001843226,0.016094187,-0.005662785,0.007147845,0.063074715,0.014047399,-0.006652334,-0.0029128734,0.012269368,-0.0017195483,0.017286044,0.025764683,0.015559005,-0.01141093,0.009099018,-0.039312776,0.040967144,0.05501801,-0.043883737,-0.041257523,-0.0013265741,0.023772113,-0.06390883,-0.0072328458,0.021076838,-0.022326203,0.022804838,-0.00534553,0.03163781,0.0010720652,-0.0389723,-0.032958962,-0.006402114,-0.037853863,-0.011244876,-0.060338542,0.015602039,0.032492798,-0.021635117,0.012477533,0.000101079924,-0.0019191536,-0.018200576,-0.021859983,-0.0024956535,-0.011865344,0.018001266,-0.010029507,0.03977739,-0.025982482,-0.013731208,0.020106692,-0.040010657,0.077066995,-0.019211913,-0.010502089,0.0031373037,-0.011655559,0.03330677,0.018123824,0.016477266,0.040495016,-0.009242387,-0.034290805,-0.034328282,-0.004831931,0.003046239,-0.029425621,-0.012484602,0.0517286,0.053929437,0.027664386,-0.043910865,0.009282762,-0.037989207,-0.020502297,-0.013095729,0.02489363,-0.049315315,-0.022770882,-0.0072874534,-0.069674484,-0.03955695,-0.021619394,-0.044497557,0.013940519,-0.038578827,0.027170645,0.009693417,-0.016767848,0.030562753,0.045194857,0.0006306597,-0.012834012,-0.014261012,0.031817723,-0.060591232,-0.026820937,0.032202184,-0.0039616534,0.005392474,-0.046867583,-0.022887986,-0.024129955,-0.014255459,-0.030470388,0.02370097,-0.0001349874,-0.026114311,-0.048180446,-0.023065118,-0.045187127,-0.045066666,0.0023001707,0.01577148,0.019111805,0.010025168,-0.010633771,-0.0121434815,-0.015428599,0.017651606,-0.00923716,-0.004700815,-0.013195125,-0.015500792,0.025603194,0.051793244,-0.039723,-0.012007689,0.022203973,0.0142831085,0.009203425,0.0012669723,-0.04594714,-0.029017244,-0.0040731747,0.0016869975,0.017776944,0.005405685,0.01637764,-0.0039687203,0.012756297,0.0056086523,0.018472508,0.003182957,-0.010315152,-0.028642459,-0.0021376244,0.04475617,-0.011345308,-0.03128183,0.040532846,0.025431886,0.06643516,-0.013864246,-0.01253757,0.06759082,-0.039319344,-0.067378424,-0.0056807715,0.013748955,-0.013081686,0.007812031,0.014621929,-0.0009968178,0.009353054,0.0236879,0.028722692,0.003374323,0.016946465,-0.041101325,-0.015079721,-0.054866035,0.0143043175,-0.035223667,0.004579902,0.057666108,0.0057212454,-0.023011854,0.038579047,-0.024823157,0.00095409155,0.015581556,0.025019709,-0.010618241,0.037505876,0.013108707,-0.012252982,0.05701231,0.000045457844,0.01969766,0.06324656,-0.011113176,0.02006983,0.054211143,0.0055452213,0.04359289,-0.014942108,0.026874783,-0.017478202,-0.007419916,0.0077900463,0.020993633,-0.033745937,0.046957333,-0.013268143,-0.048011124,0.03156537,0.019361675,-0.059059374,0.015194878,-0.022194099,-0.015274099,0.018108469,0.077788256,0.021568792,9.579596e-6,-0.005669762,0.055532277,-0.03680087,0.011178877,-0.006712572,0.001854423,-0.01683322,0.023308739,0.061474368,0.03459606,-0.015579488,-0.0044895993,0.040943116,0.02665679,0.005359514,-0.005240668,0.014946543,-0.06358604,0.05049763,0.009302316,0.053926963,0.033264015,-0.011712764,-0.023378756,-0.023612605,0.022480858,0.024053883,-0.013070069,-0.008918332,0.0630357,-0.033990685,0.0063828,0.018065574,0.00085262605,0.019753857,0.015313811,0.016757526,0.048770737,-0.041362178,-0.007914039,-0.039393384,0.019185415,0.058195863,-0.022080205,-0.014518933,0.010355267,-0.006483981,-0.012111662,-0.018715112,-0.043302212,0.0018500043,0.04772622,0.00080424885,0.050821703,0.026649727,-0.013127923,0.05051607,-0.039011844,-0.009966296,0.03359474,-0.05164105,0.023846224,-0.005097954,0.00893234,0.01748592,0.04123259,-0.025907855,0.030176891,-0.0003829574,-0.0032038307,-0.017794035,0.008093152,0.019068778,-0.0018444341,-0.0135180075,-0.014599101,0.03698338,-0.03633942,-0.01722751,0.032806277,-0.049240556,0.020083575,0.015992595,0.01484581,0.011029237,-0.023498507,0.007478974,-0.025379581,0.002906934,-0.047490478,-0.039148618,-0.018974435,0.04553179,-0.011595125,-0.021576138,0.00078977406,-0.012198312,0.025559058,0.017706253,0.0059527806,-0.005864759,-0.00891576,-0.03418869,-0.025373746,0.0119640045,-0.006588551,0.011155844,0.007146667,0.03545212,0.0027550298,-0.038952794,-0.0060924855,-0.0042583025,-0.008721222,0.02257686,0.043618932,-0.019146051,-0.011544839,-0.0004833456,0.07595721,0.00655668,0.010166915,-0.0004174417,-0.022991559,0.020540927,0.013911684,0.057057895,-0.042466428,0.064719625,0.010460334,-0.0014978907,-0.015541963,0.014350639,-0.031728152,0.04718453,0.004211115,0.0073292013,-0.045792866,-0.014702181,0.024118636,-0.012702539,-0.051641393,-0.04732392,0.0019333692,-0.0051101693,-0.0011136855,0.015782787,-0.025968414,0.04519613,0.051002197,0.028576268,-0.00057220994,0.0058263955,0.021255096,0.027054911,-0.029587246,0.0068984544,0.0065420372,0.0074299695,0.01678545,0.04927971,0.011776156,-0.007611446,0.004333081,-0.0065343115,-0.015051315,-0.03537147,0.0009138903,0.039782308,0.03741491,-0.005655874,0.013346599,0.0061192815,0.03607811,-0.019915061,-0.02737344,0.022363055,0.056089915,-0.010807988,-0.010420669,0.011331184,-0.012632062,0.0130780395,-0.031034486,-0.014252887,-0.012749608,-0.03551506,0.016925383,0.0142698325,-0.00930842,0.016141724,0.021930704,-0.01267876,0.026243802,0.010101162,0.03954812,0.028860413,-0.013772259,-0.014065102,-0.0
LiteLLM 能力一览
| 能力 | 说明 |
|---|---|
| OpenAI API 兼容性 | 统一 chat / embeddings / completion |
| Ollama/OpenAI/Gemini 等多后端支持 | 无缝代理 |
| 多模型路由 | 自动选择最便宜或最快 |
| 自动 fallback | 某个模型挂了自动换另一个 |
| 请求限流 | 内置 QPS/RPS 控制 |
| LLM 调用日志 | 自动记录调用、延迟、用量 |
| API Key 池化 | 多密钥轮询 |
| 负载均衡 | 多实例 round-robin |
扫一扫
20万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
