Conversation
These are the details for my current pipeline System Description: Hybrid Search pipeline combining dense embeddings (VoyageAI voyage-4-large 2048-dim) via Alibaba's Zvec database, with sparse lexical retrieval (BM25), fused together and passed through a cross-encoder reranker (VoyageAI Rerank-2.5). System Type: Hybrid / RAG Retriever Open Source: Not open source (rerankers and the embedding models are closed source, vector db and sparse retrieval is open source)
|
@nimit2801 @prakhar7651 The description validation workflow is failing I am unable to find any suitable template in the readme. Do i have to follow any official template for the submission? |
|
hey @shrey2003 Kindly add this in your PR description: https://app.devrev.ai/devrev/works/ISS-269621 |
shrey2003
changed the title
|
Hey Shreya! |
|
Hey @prakhar7651 thanks for your evaluation! |
|
As per the instructions this json contains my results: test_queries_results.json @prakhar7651 @nimit2801 |
|
Hey! |
|
@prakhar7651 Thanks for the evaluation. I have tried improving my script and rerun the results there were few bugs affecting he results which i found out. Can I resubmit after improving? |
|
Yes, you can. Let me know when you're done and tell me which file to evaluate. |
|
test_queries_results_new.json @prakhar7651 this is the corrected file. Please evaluate it. Thanks! |
|
In this commit - |
|
@prakhar7651 Yes I have run the results with a corrected python script I will update the notebook if the results are better that's why I didn't upload it can you check how is the new result performing? |
|
@prakhar7651 can you evaluate this now as today I think is the last day for submission? |
|
@shrey2003 can you please update your code before we release the results for latest submission so we can verify the code and ensure the results are reproducible |
|
@prateekjain2606 I have already pushed run_submission.py file please check |
|
@prakhar7651 Can you evaluate this too I have already added my submission file and code here |
|
Here are your updated scores, The previous scores which we posted for your old submission, those were incorrect (some error on our side), these are the true values for your old submission |
|
Looking at the quality of submissions and eagerness for folks to contribute, we're extending the deadline to April 7th. Evaluations would be still going on. Please keep contributing. |
|
@prakhar7651 Now this seems good! I was pretty much amazed earlier to see it perform so badly as I have almost compared from every benchmark and my tests that voyage has the best reranker and embedding models out their, no one can compete with them there was some issue in my earlier code Will try more combinations and check out if I can improve the scores further |
4 participants
I built a custom hybrid retrieval pipeline using SOTA models rather than the baseline FAISS approach.
System Details:
System Description: Hybrid Search pipeline combining dense embeddings (VoyageAI voyage-4-large 2048-dim) via Alibaba's Zvec database, with sparse lexical retrieval (BM25), fused together and passed through a cross-encoder reranker (VoyageAI Rerank-2.5).
System Type: Hybrid / RAG Retriever
Open Source: Not open source (rerankers and the embedding models are closed source, vector db and sparse retrieval is open source)
Looking forward to seeing the results on the leaderboard!
https://app.devrev.ai/devrev/works/ISS-269621/
ISS-269621