Backend Engineer (Monitoring)
Apollo Research • GBPython
Application deadline: We accept submissions until 16 January 2026. We review applications on a rolling basis and encourage early submissions.
THE OPPORTUNITY
Join our new AGI safety monitoring team and help transform complex AI research into practical tools that reduce risks from AI. As a Backend Engineer, you'll work closely with our CEO, monitoring engineers and Evals team software engineers to build tools that make AI agent safety accessible at scale. We are building tools that monitor AI coding agents for safety and security failures.
You will join a small team and will have significant ability to shape the team & tech, and have the ability to earn responsibility quickly. You will like this opportunity if you care about building tools that genuinely make AI agents safe and thrive in high-paced environments as well as enjoy closely working with researchers.
KEY RESPONSIBILITIES
Infrastructure & Architecture
- Design and implement scalable backend systems capable of processing and analyzing large volumes of AI agent logs in real-time
- Build and maintain data processing pipelines that extract, transform, and store agent trajectory data efficiently
- Architect database schemas and data models optimized for both high-throughput writes and complex analytical queries
- Design for reliability, implementing robust error handling, retry logic, and graceful degradation strategies
- Monitor system performance and optimize bottlenecks to ensure sub-second latency for critical monitoring operations
API Development
- Develop secure, well-documented RESTful APIs that allow users to integrate our monitoring tools into their workflows
- Implement authentication, authorization, and rate limiting to protect users data and ensure fair resource usage
- Build webhook systems and real-time notification services to alert users about critical safety events
- Design API interfaces that are intuitive for developers while remaining flexible for diverse user use cases
- Design and implement integrations with Security Information and Event Management (SIEM) systems, enabling users to stream monitoring alerts and security events into their existing security operations workflows
Data Systems
- Implement efficient storage solutions for both structured data (monitoring results, metadata) and unstructured data (agent logs, code outputs)
- Build data processing systems that can handle everything from streaming real-time monitoring to batch analysis of historical data
- Design and implement caching strategies to optimize frequent queries and reduce infrastructure costs
- Create data retention and archival policies that balance users needs with storage efficiency
Monitoring & Observability
- Build comprehensive logging, metrics, and tracing systems to ensure visibility into system health and performance
- Implement alerting systems that notify the team of infrastructure issues before they impact users
- Create dashboards and tools that help the team understand system behavior and diagnose issues quickly
- Design systems that make debugging production issues straightforward and minimize time-to-resolution
Collaboration & Quality
- Work closely with our researchers to understand their needs and translate research prototypes into production-ready systems
- Collaborate with frontend engineers to design APIs and data structures that enable excellent user experiences
- Participate in code reviews to maintain high standards for code quality, security, and performance
- Document architectural decisions, API specifications, and system behaviors to facilitate knowledge sharing
- Contribute to technical discussions about technology choices, trade-offs, and implementation approaches
JOB REQUIREMENTS
Bonus
We want to emphasize that people who feel they don't fulfill all of these characteristics but think they would be a good fit for the position nonetheless are strongly encouraged to apply. We believe that excellent candidates can come from a variety of backgrounds and are excited to give you opportunities to shine.