zeroentropy

/

zerank-2-reranker

like 71

Follow

ZeroEntropy 137

Hi,

I can get it to work but the scores make no sense, so probably broken in a way. You can run it by adding command line args --hf-overrides '{"architectures": ["Qwen3ForSequenceClassification"], "classifier_from_token": ["no", "yes"], "is_original_qwen3_reranker": true}' . The reranking scores seem random though. Here's my test case:

curl -s -X POST "http://rtx5070:8000/v1/score" -H "Content-Type: application/json" \
 -d '
   {
      "model": "zeroentropy/zerank-2",
      "text_1": "What are the symptoms of diabetes?",
      "text_2": [
        "Common diabetes symptoms include increased thirst, frequent urination, fatigue, and blurred vision.",
        "The Great Wall of China is over 13000 miles long.",
        "Basketball was invented by James Naismith in 1891.",
        "Type 2 diabetes can often be managed with diet and exercise.",
        "Symptoms may also include slow-healing wounds and tingling in hands or feet."],
      "use_activation": false
  }'

This gives these scores:

Doc Raw Logit Softmax Score Content
0 4.25 0.986 Diabetes symptoms (thirst, urination...)
4 6.51 0.998 More symptoms (wounds, tingling)
3 2.87 0.947 Diabetes management
1 2.74 0.940 Great Wall (irrelevant)
2 2.67 0.936 Basketball (irrelevant)
The raw logits show better differentiation (4.25 vs 2.74 for relevant vs irrelevant), but after softmax they compress to 0.986 vs 0.940. In any case the scores are not great, with irrelevant documents getting half the score that a fully relevant document is getting. I assume this is caused by the way of invocation.