We had a wonderful time at the Frontier Data Summit, a one-day, invite-only event organized by Snorkel AI. The summit brought together researchers and practitioners working on benchmark research that shapes the AI frontier. We were delighted to participate and present SREGym at the poster session.
A Refreshing Perspective on AI Eval Research
The summit was very well organized, featuring an inspiring keynote, “The Research Era of AI Data,” by Alex Ratner, Co-founder and CEO of Snorkel AI, along with a series of engaging fireside chats. One highlight was “How Benchmarks Are and Aren’t Used in Model Development Today,” featuring Braden Hancock, a mentor to the SREGym team, and David Hall. We thoroughly enjoyed the talks and the thoughtful discussions.
Interestingly, this was Tianyin’s first-ever AI event. Coming from a systems research background, he found the mindsets and perspectives of the AI community both refreshing and surprisingly different. He was also struck by the breadth and diversity of the community.
His personal highlight was the opportunity to reconnect with and learn from several UIUC students and alumni attending the summit. It was wonderful to see familiar connections across such a vibrant research community.
SREGym at the Poster Session
SREGym is very well received at the summit. Throughout the poster session, many attendees stopped by to learn about SREGym and discuss the challenges and opportunities of AI for SRE and its evaluation. The conversations ranged from AI researchers curious about the SRE domain to core AI SRE researchers such as colleagues from ORCA-bench and Datadog.
Jackson did an outstanding job presenting SREGym (according to Tianyin) and engaging with attendees from a wide range of backgrounds. Much of his effort went beyond explaining the project itself: he introduced AI researchers to the SRE domain, helped them understand its unique challenges, and made the case for why SRE is an important frontier for AI research and evaluation. These conversations sparked thoughtful discussions about the broader implications of our work and opened up opportunities for future collaborations.
Some of the most memorable moments came from meeting both current and potential users of SREGym. Colleagues from Datadog made our day by sharing that they had been using SREGym for Datadog's internal agents. Another fun moment came when a colleague tried to recruit Jackson during the event. Tianyin couldn't help but laugh and feel proud.
The Community Behind SREGym
One of SREGym's strengths has always been the community we are building together.
SREGym now has more than 60 contributors, who have helped expand the benchmark with new problems, develop features, and fix bugs. Our Slack community has grown to 280 members, bringing together researchers, engineers, and practitioners who share an interest in advancing AI SRE.
What excites us even more is seeing collaborations grow beyond SREGym itself. Our community has sparked several new projects and research initiatives, some developing independently of the SREGym team.
This is what makes an open research community truly valuable: not just the work we accomplish together, but also the new ideas, collaborations, and opportunities that grow from it.
Looking Ahead: SREGym 2.0
The most frequently asked question at our poster was: "When is SREGym 2.0 coming?"
We were particularly excited by Alex Ratner's vision of Data 2.0, presented in his keynote, which resonated strongly with our own aspirations for SREGym.
One of SREGym's core missions is to help push the boundaries of frontier AI through increasingly challenging and meaningful evaluations. Keeping pace with rapidly advancing AI capabilities is not without challenges. As models become more capable, our benchmarks must evolve to continue challenging them and revealing where meaningful progress is needed. But that's exactly what excites us – there are always new mountains to climb.
Our team has been making great progress on SREGym 2.0 through a new technique we're calling environment scaling, and we hope to release our 2.0 soon. Stay tuned!
Jackson made a great poster:
We thank Snorkel AI again for organizing this great summit and inviting us to be a part of it.