SERI Summer Research Fellowship

The Stanford Existential Risk Initiative (SERI) supports research aimed at mitigating existential risks through a 10-week paid research fellowship in Berkeley, CA.

Do research to protect our future 

From June 13th to August 19th, fellows will tackle ambitious projects to work towards solutions to the most pressing problems of our time. Fellows will collaborate with a mentor, receive daily support from our team, a $7,500 stipend, accommodations, office space, and travel to and from our location in Berkeley. A remote program will also be held for fellows who cannot attend in-person.

Summer 2021 Project Samples

AI SAFETY
RESEARCH
Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts
Reward learning methods intended for use in multi-agent settings with realistic human actors must account for suboptimal human reasoning and model social dynamics and outcomes which accurately reflect this. Multi-agent inverse reinforce-ment learning (MIRL) can be used to learn rewards from agents in social envi-ronments, but to do so realistically, must break from the conceptually simple andcomputationally tractable formalisms of game
AI SAFETY
RESEARCH
Cheaper language model alignment from human feedback
To better align with human preferences, recent text generation algorithms leverage human feedback on examples of the task at hand. However, high-quality feedback data is prohibitively expensive. We propose a methodology in which meaningful binary comparisons are drawn from noisy user feedback to advicegiven in an online forum.
AI GOVERNANCE
RESEARCH
China's National Team Approach to AI Policy Making
Scientists play an ever crucial role in policy-making about complex policy issues such as artificial intelligence, but the role of Chinese scientists in policy-making is significantly less studied. This article argues that existing frameworks to study the roles of Chinese scientists in policy-making are not yet fit for the purpose because the effect of political regime types on scientists' influence is less considered. The Chinese regime and political environment shape the role of scientists in the primary ways:
AI GOVERNANCE
RESEARCH
Self-Enforcing Treaties Reduce Risk from Technology Races
Certain treaties aim to reduce risk from dangerous technologies, such as nuclear weapons and ozone-depleting chemicals. This paper examines how treaties can reduce risk from future races to develop powerful technologiessuch as artificial intelligence. Technology races risk loss of life or other catastrophes when safe development slowsa project’s progress in the race. This tradeoff gives competitors an incentive to skimp on safety, which increasesthe risk of catastrophe to all competitors. I model a treaty that uses side payments to persuade projects todevelop a technology safely, reducing risk from the race. I find that a treaty only occurs if the race includes manycompetitors, otherwise a treaty cannot be negotiated. In contrast to earlier work, I show that openness abouttechnical capabilities can reduce risk by allowing states to negotiate a treaty.1
BIOSECURITY
RESEARCH
Modelling a Global Outbreak of a hypothetical human-transmissible Highly Pathogenic Avian Influenza H5N1
Highly Pathogenic Avian Influenza H5N1 is an influenza virus that causes disease in humansand other animals, most notably birds. While H5N1 is not efficiently transmissible betweenhumans, it has extremely high virulence, with 58% of cases reported globally from 2003to August 2021 resulting in death. In the past decade, H5N1 has been the subject of
NUCLEAR RISK
RESEARCH
Nutrition in Abrupt Sunlight Reduction Scenarios
Abrupt sunlight reduction scenarios (ASRS) following catastrophic events, such as a nuclear war, a large volcanic eruption or an asteroid strike, could prompt global agricultural collapse. There are low-cost foods that could be made available in an ASRS: resilient foods. Nutritionally adequate combinations of these resilient foods are investigated for different stages of a scenario with an effective response, based on existing technology. While macro- and micronutrient requirements were overall met, some—potentially chronic—deficiencies were identified (e.g., vitamins D, E and K). Resilient sources of micronutrients for mitigating these
Climate Change
RESEARCH
Novel Ecosystems: The New Normal and Management Approaches
Novel ecosystems are human-induced, alternative ecosystem states that com-prise new species assemblages and abiotic conditions. As anthropogenic forces including climate change, human-assisted species invasions, habitat modification, and fragmentation continue gaining traction, it is highly likelythat ecological novelty will soon be the new normal, where we will simplybe living within and around rapidly changing ecosystems.
AI SAFETY
RESEARCH
Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts
Reward learning methods intended for use in multi-agent settings with realistic human actors must account for suboptimal human reasoning and model social dynamics and outcomes which accurately reflect this. Multi-agent inverse reinforce-ment learning (MIRL) can be used to learn rewards from agents in social envi-ronments, but to do so realistically, must break from the conceptually simple andcomputationally tractable formalisms of game
BIOSECURITY
RESEARCH
Biosafety and Biosecurity of Engineered Microbial Consortia
In this work, we identify and analyze several biosafety and biosecurity risks arising due to the synthetic microbial consortia and offer the perspectives on the technological and policy solutions that could reduce the risks due to genetically modified microbial communities. 
AI SAFETY
RESEARCH
Cheaper language model alignment from human feedback
To better align with human preferences, recent text generation algorithms leverage human feedback on examples of the task at hand. However, high-quality feedback data is prohibitively expensive. We propose a methodology in which meaningful binary comparisons are drawn from noisy user feedback to advicegiven in an online forum.
NUCLEAR RISK
RESEARCH
South-Asian Nuclear Risk:
Nationalism, Media and Public Opinion in India
India-Pakistan relations, though always high with tensions, have worsened since their 2019 near-war experience. The Indian Defence Minister Rajnath Singh announced in 2020 that India might revoke its no-first-use policy, which has added to existing anxieties about South-Asian nuclear stability. A potential nuclear war between the two neighbors would lead to the death of 50 to 125 million people, long-term regional humanitarian and economic dislocation, and nuclear-ignited fires would likely trigger global climate change and agricultural damage.
AI GOVERNANCE
RESEARCH
China's National Team Approach to AI Policy Making
Scientists play an ever crucial role in policy-making about complex policy issues such as artificial intelligence, but the role of Chinese scientists in policy-making is significantly less studied. This article argues that existing frameworks to study the roles of Chinese scientists in policy-making are not yet fit for the purpose because the effect of political regime types on scientists' influence is less considered. The Chinese regime and political environment shape the role of scientists in the primary ways:
AI GOVERNANCE
RESEARCH
Self-Enforcing Treaties Reduce Risk from Technology Races
Certain treaties aim to reduce risk from dangerous technologies, such as nuclear weapons and ozone-depleting chemicals. This paper examines how treaties can reduce risk from future races to develop powerful technologiessuch as artificial intelligence. Technology races risk loss of life or other catastrophes when safe development slowsa project’s progress in the race. This tradeoff gives competitors an incentive to skimp on safety, which increasesthe risk of catastrophe to all competitors. I model a treaty that uses side payments to persuade projects todevelop a technology safely, reducing risk from the race. I find that a treaty only occurs if the race includes manycompetitors, otherwise a treaty cannot be negotiated. In contrast to earlier work, I show that openness abouttechnical capabilities can reduce risk by allowing states to negotiate a treaty.1

FAQ

About the Fellowship

Will this program be in-person or remote?

An in-person program will be held in Berkeley, California. We will also be hosting a remote program for fellows who cannot attend in-person. All fellows are encouraged to participate full-time. Part-time fellows may be accepted on a case-by-case basis.

An in-person program will be held in Berkeley, California. We will consider hosting a remote program for fellows who cannot attend in-person based on the strength of remote applicants.

What are the key dates for the fellowship?

Application responses will be released in early May, and expedited evaluations can be requested. The program begins on June 13th, and will culminate in a symposium on August 19th. 

Applications are due on April 17th at 11:59 p.m. PST. Application responses will be released in early May, and expedited evaluations can be requested. The program begins on June 13th, and will culminate in a symposium on August 19th. 

What does a typical week-day look like?

Expect to spend eight hours working on your project each day, which includes meetings with other fellows, members of our staff, and your mentor. Our office space will be open to in-person fellows throughout the day, and lunch will be provided. All fellows are encouraged to take breaks as needed. Fellows can expect a variety of talks and social activities to be offered in the evenings throughout the program.

Expect to spend eight hours working on your project each day, which includes meetings with other fellows, members of our staff, and your mentor. Our office space will be open to in-person fellows throughout the day, and lunch will be provided. All fellows are encouraged to take breaks as needed. Fellows can expect a variety of talks and social activities to be offered in the evenings throughout the program.

What is the expected outcome of the program?

Participants are expected to produce a written deliverable aimed at reducing existential risk, which may take a variety of forms. For example, graduate students might first-author a paper while a less experienced participant may produce a well-written essay, academic poster, or contribute to another individual's paper. During the last week, there will be a symposium where fellows will present their work.

Participants are expected to produce a written deliverable aimed at reducing existential risk, which may take a variety of forms. For example, graduate students might first-author a paper while a less experienced participant may produce a well-written essay, academic poster, or contribute to another individual's paper. During the last week, there will be a symposium where fellows will present their work.

Who is eligible to apply?

We are primarily targeting undergraduate and graduate students, but anyone is welcome to apply. Unfortunately, international applicants are only eligible for the remote program, as we are unable to assist with acquiring visas. We encourage international applicants to still apply as we may host a remote program.

We are primarily targeting undergraduate and graduate students, but anyone is welcome to apply. Unfortunately, international applicants are only eligible for the remote program, as we are unable to assist with acquiring visas. We encourage international applicants to still apply as we may host a remote program.

Will I be paid? Are any additional expenses covered?

All full-time research fellows will receive a stipend of $7,500 for ten weeks of full-time research. All travel expenses to and from Berkeley, accommodations, office space access, and week-day lunches for in-person fellows will also be covered.

All full-time research fellows will receive a stipend of $7,500 for ten weeks of full-time research. All travel expenses to and from Berkeley, accommodations, office space access, and week-day lunches for in-person fellows will also be covered.

All fellows are encouraged to participate full-time. Part-time fellows may be accepted on a case-by-case basis.

Mentors

Who will my mentor be?

Your mentor will be an experienced researcher in your chosen cause area who is working on problems we think are important. In past years, mentors have been professors, post docs, and graduate students at top universities, researchers at non-profit organizations, think tanks, and in government roles, along with other experienced professionals. 

Your mentor will be an experienced researcher in your chosen cause area who is working on problems we think are important. In past years, mentors have been professors, post docs, and graduate students at top universities, researchers at non-profit organizations, think tanks, and in government roles, along with other experienced professionals. 

What should I expect from my mentor?

You can expect to meet with your mentor roughly twice per week for thirty minutes. Your mentor will help to scope and define the research question you are working on, direct the trajectory of your project, and assist with the development of your research skills. We expect most project mentors will not be based in the Bay Area and meetings will be virtual. A member of our team will also be available to provide cause area research support.

You can expect to meet with your mentor roughly twice per week for thirty minutes. Your mentor will help to scope and define the research question you are working on, direct the trajectory of your project, and assist with the development of your research skills. We expect most project mentors will not be based in the Bay Area and meetings will be virtual. A member of our team will also be available to provide cause area research support.

Am I responsible for finding my own mentor?

Our team will work to find research mentors before the program. If you have a specific mentor in mind that you would like to work with, you’ll have an opportunity to include this in the application.

Our team will work to find research mentors before the program. If you have a specific mentor in mind that you would like to work with, you’ll have an opportunity to include this in the application.

Have a different question?

Team