zhiminy (Zhimin Zhao)

reacted to their post with 🤗 about 1 month ago

Post

1937

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best production-level generative-ai open-source toolkits! 🚀

Check it out here: zhiminy/awesome-production-genai-search

posted an update about 1 month ago

Post

1937

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best production-level generative-ai open-source toolkits! 🚀

Check it out here: zhiminy/awesome-production-genai-search

reacted to their post with 🧠➕👍🤯❤️🚀👀🔥🤗 4 months ago

Post

1909

# 🚀 SE Arena: Evaluating Foundation Models for Software Engineering

SE Arena is the first open-source platform for evaluating foundation models in real-world software engineering workflows.

## What makes it unique?

- RepoChat: Automatically injects repository context (issues, commits, PRs) into conversations for more realistic evaluations
- Multi-round interactions: Tests models through iterative workflows, not just single prompts
- Novel metrics: Includes a "model consistency score" that measures model determinism through self-play matches and "conversation efficiency index" that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions.

Try it now: SE-Arena/Software-Engineering-Arena

## Why it matters

Traditional evaluation frameworks don't capture how developers actually use models in their daily work. SE Arena creates a testing environment that mirrors real engineering workflows, helping you choose the right model for your specific software development needs.

From debugging to requirement refinement, see which models truly excel at software engineering tasks!

posted an update 4 months ago

Post

1909

# 🚀 SE Arena: Evaluating Foundation Models for Software Engineering

SE Arena is the first open-source platform for evaluating foundation models in real-world software engineering workflows.

## What makes it unique?

- RepoChat: Automatically injects repository context (issues, commits, PRs) into conversations for more realistic evaluations
- Multi-round interactions: Tests models through iterative workflows, not just single prompts
- Novel metrics: Includes a "model consistency score" that measures model determinism through self-play matches and "conversation efficiency index" that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions.

Try it now: SE-Arena/Software-Engineering-Arena

## Why it matters

Traditional evaluation frameworks don't capture how developers actually use models in their daily work. SE Arena creates a testing environment that mirrors real engineering workflows, helping you choose the right model for your specific software development needs.

From debugging to requirement refinement, see which models truly excel at software engineering tasks!

reacted to their post with 🤗 8 months ago

Post

2273

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best production-level machine-learning open-source toolkits! 🚀

Check it out here: zhiminy/Awesome-Production-Machine-Learning-Search

posted an update 8 months ago

Post

2273

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best production-level machine-learning open-source toolkits! 🚀

Check it out here: zhiminy/Awesome-Production-Machine-Learning-Search

reacted to their post with 😎 about 1 year ago

Post

814

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best-curated lists related to artificial intelligence! 🚀

Check it out here: https://github.com/zhimin-z/awesome-awesome-artificial-intelligence

posted an update about 1 year ago

Post

814

Hey everyone,

We're thrilled to introduce our latest project: a hand-curated list of the best-curated lists related to artificial intelligence! 🚀

Check it out here: https://github.com/zhimin-z/awesome-awesome-artificial-intelligence

reacted to their post with 👀 about 1 year ago

Post

2005

Hey everyone!

Our team just dropped something cool! 🎉 We've published a new paper on arxiv diving into the foundation model leaderboards across different platforms. We've analyzed the content, operational workflows, and common issues of these leaderboards. From this, we came up with two new concepts: Leaderboard Operations (LBOps) and leaderboard smells.

We also put together an awesome list with nearly 300 of the latest leaderboards, development tools, and publishing organizations. You can check it out here: https://github.com/SAILResearch/awesome-foundation-model-leaderboards

If you find it useful or interesting, give us a follow or drop a comment. We'd love to hear your thoughts and get your support! ✨

Link to the paper: https://arxiv.org/abs/2407.04065

posted an update about 1 year ago

Post

2005

Hey everyone!

Our team just dropped something cool! 🎉 We've published a new paper on arxiv diving into the foundation model leaderboards across different platforms. We've analyzed the content, operational workflows, and common issues of these leaderboards. From this, we came up with two new concepts: Leaderboard Operations (LBOps) and leaderboard smells.

We also put together an awesome list with nearly 300 of the latest leaderboards, development tools, and publishing organizations. You can check it out here: https://github.com/SAILResearch/awesome-foundation-model-leaderboards

If you find it useful or interesting, give us a follow or drop a comment. We'd love to hear your thoughts and get your support! ✨

Link to the paper: https://arxiv.org/abs/2407.04065

Zhimin Zhao PRO

AI & ML interests

Recent Activity

Organizations

Zhimin Zhao PRO

AI & ML interests

Recent Activity

Organizations

zhiminy's activity