SaferAI is seeking a Research Engineer/Scientist with a strong ability to perform technical research & software engineering. Ideal candidates will have experience both with conducting research on large language models and with traditional software engineering work.
As a Research Engineer/Scientist, you will be responsible for the core technical work we conduct to advance quantitative risk modelling, i.e. the ability to quantify the probability of AI risk scenarios and their steps. By breaking down scenarios leading to harm in steps whose probability or quantity can be estimated, we intend to advance our understanding of AI risk and be able to systematically translate empirical indicators such as benchmarks, evaluations and incident reports into real-world risk estimates. The initial area of focus for development of our risk modeling methodology is assessing the impact of LLMs on cyber.
SaferAI is a fast-moving, mission-driven organization advancing and promoting AI risk management to reduce AI risks, in particular extreme risks from advanced AI systems. We’re uniquely positioned at the intersection of technical research work and policy, ensuring that the knowledge we produce has an impact on AI risk management and the governance of LLMs.
As the only NGO member of a G7 OECD taskforce in charge of writing a reporting framework for frontier AI, a founding member of the US AI Safety Institute Consortium and a primary contributor to EU standards and GPAI Code of Practice, we’re responsible for significant contributions to AI risk management of large language models in the policy realm.
Our technical work is key to maintaining and furthering the unique expertise we bring to governments and companies. We released the first AI risk management framework combining traditional risk management and frontier AI safety policies, cited by NVIDIA’s risk assessment work. We co-drafted the G42 risk management policy. We developed the first AI risk management rating system for AI companies’ risk management practices, featured in TIME and Euractiv, informing the decisions of major investors.
Our current core research focus is to develop risk models that enable us to aggregate existing empirical measurements (benchmarks, evaluations, incident reports) and turn them into granular risk quantification. To render this scalable, we’re accelerating the process by utilising LLMs to complement expert Delphi studies and perform predictions.
Your core objective will be to perform research & engineering focused on rendering quantitative AI risk modelling more scalable for AI developers. Your responsibilities will include:
Once our LLM for cyber risk modelling work matures, you will play a key role in our expansion into modeling risks related to misalignment and accidental harms. This will likely involve implementing or reimplementing control setups and evaluations to feed into our risk models in these areas.
We are excited about our team members shaping our medium-term research directions and we are keen to support and enable new research ideas that align with our mission. Here are a few examples of areas we’d like to explore in the medium-run:
To apply for this position, please complete this application form. We will evaluate the candidates on a rolling basis starting now until we fill the role. We hope to fill the role by mid April.
We encourage you to lean towards applying, even if you don’t have all the skills and experience required.
If you have any questions or concerns throughout the application process, please don't hesitate to reach out to us at careers@safer-ai.org. We look forward to reviewing your application.