In short
Contract software engineers create challenging, real-world coding tasks (bug fixes, features, refactors, optimizations) for AI model evaluation: building reproducible environments, reference solutions, and deterministic verifiers. Requires deep expertise in at least one of Python, Java, Rust, Go, C++, or TypeScript; fully remote, ~15 hrs/week.
About the role
We are looking for experienced Software Engineers to support a project focused on creating challenging coding tasks for AI model evaluation.
You’ll use your software engineering expertise to build realistic coding problems from real repositories, create reproducible environments, and develop reference solutions and objective verifiers.
What You'll Do
You’ll use your software engineering expertise to build realistic coding problems from real repositories, create reproducible environments, and develop reference solutions and objective verifiers.
What You'll Do
- Create challenging coding tasks involving bug fixing, feature development, refactoring, and performance optimization.
- Build reproducible environments for each task.
- Develop deterministic verifiers to evaluate different valid solutions.
- Create reference solutions for assigned problems.
- Work with real-world codebases and engineering scenarios.
- Document technical requirements and solution approaches.
- Identify and flag tasks that are ambiguous or difficult to verify.
- Strong hands-on expertise in at least one of: Python, Java, Rust, Go, C++, or TypeScript
- Strong understanding of algorithms, data structures, and software engineering principles.
- Experience debugging complex systems and resolving software defects.
- Experience with feature implementation, performance optimization, or codebase refactoring.
- Strong technical documentation and communication skills.
- Interest in AI and software engineering challenges; prior AI experience is not required.
- Role: Software Engineer
- Type: Contractor / Project-based
- Location: Fully Remote
- Commitment: Approximately 15 hours/week
- Estimated Task Time: ~5 hours for task creators
- Start: Selected candidates are expected to begin within 24–48 hours of onboarding
Requirements
- Strong hands-on expertise in at least one of: Python, Java, Rust, Go, C++, or TypeScript