Aligning Technology Development with Public Input: A Conversation with Saffron Huang

As the rise of large language models increasingly alters the ways we create, work, and even think, the need for more public input on the development of these technologies is clear — but technological development containers and policy-making bodies aren’t constructed to democratize the process by which transformative technologies reach the public. Saffron Huang, co-director of the Collective Intelligence Project, hopes to change this by prototyping more effective ways of surfacing and aggregating public input, and of making those insights available to decision makers. Read on for more about alignment assemblies, data commons, and coordinating collective action.

Hannah Scott: Tell us more about who you are and how you came to start the Collective Intelligence Project.

Saffron Huang: I co-founded the Collective Intelligence Project with Divya Siddharth about a year ago. I had just worked at DeepMind for a few years, working on AI research. I’ve kind of hopped between technical areas of research and AI governance for a while. Divya was at Microsoft in the Office of the CTO office, and we were both kind of thinking about similar things from slightly different angles, where I felt that there are a lot of issues around the political economy of AI or the way we structure the development of technology — I felt like those topics were missing from the discourse. Divya had worked on data cooperatives and quadratic funding projects. She wanted to do much bigger-picture things around how these economic mechanisms apply to transformative technologies generally. We both felt that technological development needed more collective input and collective governance. At the moment, we are focusing a lot on AI, but the vision of CIP is about how do we improve our collective intelligence capabilities, our abilities to sort of cooperate and work together towards shared goals?

HS: I think there’s a lot of correspondence between how we at C/Change frame cultural exchange (as a capacity that enables communities to collaborate and coordinate action across borders) and how you frame collective intelligence. Can you expand upon what collective intelligence means in practice? How does it offer an alternative to current technological development practices?

SH: The term “collective intelligence” has been around for a while, and a lot of it looks at how we better surface information from the collective and behave together more intelligently. People who study collective intelligence study things like behavior of bee colonies, or animal behavior more generally. They also study how humans work together and cooperate in teams. And they also think about things like crowd science, so how do we crowd-source data from lots of people in order to better understand, for example, weather patterns. It’s pretty broad, but the way I think about it is really conditioned by my time at DeepMind, where we thought a lot about the meaning of intelligence and what it means to build intelligent artificial agents. We always talked about it as not just knowledge, not just knowing things or being able to absorb and use information, but in terms of intelligent behavior, or whether an agent is behaving towards the goals that they have set out for themselves. I think of collective intelligence as an extension of this, where it’s not just about aggregating data more efficiently from a group of people, but about whether we are behaving intelligently together towards the goals that we have set out for ourselves. With the amount of collective action problems and coordination issues in the world today, I think collective intelligence is really worth thinking about.

There are two pillars structuring CIP’s approach to collective intelligence. The first asks how we surface, aggregate, and understand conflicting values, different preferences and goals? How do we incorporate them together? Then the other pillar is the institutional design pillar, which is concerned with what we do after we understand our goals and values. How do we actually construct organizational structures and institutions so that we can work towards these goals and align ourselves towards them in a consistent and sustainable way?

HS: In thinking about ways teams working on various projects might democratize their process, “participatory design” or “community-based research” are often the only available frameworks teams can reach for. Do these ways of thinking about building go far enough? Where are their weak points in activating collective intelligence?

SH: We’ve been thinking a lot about what meaningful participation in AI looks like. There are frameworks out there, like participatory AI or human-centered design, that often think about how to design AI systems in consultation with users to meet their needs. This is a pretty scoped way of looking at it. There are things like human-in-the-loop machine learning, which is even more scoped, and says, OK, we have this entire structure. Now, we just need humans to supervise or label data in these particular ways so that we can improve our models. I think human-centered design work is really useful. But I think, generally, when it comes to AI, it doesn’t work for all needs. I don’t think all validated decisions are specifically design decisions. Human-centered design doesn’t account for policy questions where you need different kinds of more direct input that is potentially more critical with respect to questions like, should we even be developing these things in this specific way? What makes a given system safe for release at all? At CIP, we’ve been thinking a lot about forms of deliberative or participatory democracy. One way into this is thinking about how you best incorporate expert and public input, especially with a very new technology like AI. Are things like citizens assemblies the best or only way of doing things? Citizens assemblies are super useful, but when you think about the timescale on which things like that operate, and then the timescale on which tech companies are making really consequential decisions, citizens assemblies are really expensive and take a long time. Can we have more lightweight and efficient processes that operate at a different pace layer that can deal with the many smaller decisions that companies like OpenAI have to make?

CIP had a hand in shaping OpenAI’s call for experiments in democratic inputs to AI, which has been really interesting because it takes a tool-oriented approach. The call asks how we might develop new processes that we can integrate into the technology to facilitate democratic input. We’ve also been working with the Digital Minister of Taiwan, Audrey Tang, on doing an alignment assembly in Taiwan at their idea-thon next month. There are a lot of balls in the air, and our ideal is not just to have us run alignment assemblies; we’re too small of a team to do that. I think the best thing is to create a few examples of processes that anybody can run, and then to make that as open as possible and create a system where you can aggregate all the results back in one place. That way, you can say to large corporations or governments, “hey, here’s what all these people in different places have said they want with respect to AI.” I think trying to give that as a protocol to enable people to hold alignment assemblies themselves is very important.

HS: Can you sketch out some more details about what an alignment assembly looks like in practice?

SH: One very bare bones approach we’re taking at the moment starts with a decision that we want to impact, for example, the evaluations that are developed for social impact. This is kind of a subtle question. It’s not about the specific risk we want to mitigate, but what evaluations are we developing so that we even know what the impacts are? We’ve thought about this because there are so few evaluations and metrics for assessing the impact of AI on the world at the moment. So we’ve been thinking a lot about something like an IPCC for AI, where the IPCC is the Intergovernmental Panel on Climate Change, the body for creating and collecting metrics on climate change. Should we have something like that for AI? Should we have something like a CO₂ indicator for AI? What would that actually look like? With AI, it’s much more subjective and normative because it’s more social than the natural sciences. But if we could have a set of metrics, what would that look like? And people have developed metrics; I’ve even developed ways of “red-teaming” large models to understand where they are in terms of privacy leakage and toxicity. But what are other evaluations that we need to be developing to understand what’s going on? Because you first have to understand what’s going on in order to do anything about it.

So once you pick a decision that you want to impact, you can assess whether it is a decision that could benefit from public input. There’s an element of realizing that a few researchers sitting in a room don’t necessarily have all the information they need to understand what risks are most important because technologies like AI are being deployed, they are very widespread, but people don’t actually know all the things other people are doing with it. In that case, you probably want more public input into this decision to be made, and then you start thinking about the right audience. And I think that there are many right audiences, so you just need to start with one. We decided to talk to a demographically representative sample of the US public. We asked them questions via a wiki survey tool, which allows survey participants to actually contribute some of the questions on the survey. It’s much more conversational than a regular survey. It’s also a pretty scalable thing because it’s not a small group of people having extended conversations with each other, but rather lots of people having a kind of asynchronous digital conversation. So we’re using wiki survey tools like All Our Ideas and Polis to essentially have people think about and prioritize risks that they care about and feel like are important.

Going back from the beginning, it’s picking the decision, then picking the audience, picking the question, picking the tool, and then essentially aggregating all that together. We will also run a roundtable on the results of this with engineers, researchers, and some members of the surveyed public. Then we’ll write up the results of the wiki survey and the results of the roundtable, as well as broadcast the roundtable so that people can chime in on a live stream. The alignment assembly is really this multi-step consideration of how you have different scales of input and aggregate them. How do you set up a way for experts and members of the public to talk to each other? And how do you get the ultimate decision-maker to commit to implementing public input? This protocol helps us think about all the different moving parts of aggregating input and what it means to create a meaningful process, but one that’s also reasonably achievable.

HS: What steps might founders, technologists, and artists take toward building alternative “development containers” even as they operate within a “legacy” ecosystem?

SH: In the ecosystem of technology development, the default assumption is that you’re doing a startup and you’re raising venture capital, and that prioritizes a certain kind of technology, a certain kind of business model. You want to scale to as many people as possible, and you want to do this quickly. It’s bad for long-term, hardware-intensive projects. There’s a quote about how we were promised flying cars and got 280 characters. If we want flying cars, we need different development containers in which to build technology. I think there are lots of little interventions, or relatively small interventions, we’ve already been thinking about. One of those possible interventions asks how we might use AI to improve our institutions. Part of this comes from my background in AI — I’ve done things like multi-agent reinforcement learning, which looks at how people work together, or how agents work together under different social structures. I’m interested in whether things like multi-agent reinforcement modeling can help us come up with different ways of organizing ourselves in a sort of A/B test of different organizational designs. We hosted a workshop in Oxford in April on this, and there were a lot of interesting ideas that came out of that. CIP is going to host another workshop soon, but I think the long-term goal is to research lots of different smaller interventions and put together a design for a collectively intelligent corporation, which would have much more collective governance from the beginning and more public goods-oriented funding. A lot of this is very blue sky, a lot of this is quite hard, because it relies on really long-term innovations and mapping up legal frameworks. But we would love for more exchange with other people working on similar things, because it’s such a huge vision. We’re trying to figure out how to tackle some chunk of it that we can bite off.

How do we sustainably maintain [our digital] commons, grow it, and not degrade it when things like language models come into the picture?

HS: You’ve written a lot about the digital commons, and the various ways in which generative AI can both accelerate its deterioration and aid our ability to contribute positively to it. Can you speak a bit more about your thinking around the concept of the digital commons and sort of how you see it transforming as AI-generated content proliferates?

SH: I’ve been thinking about the digital commons in a very LLM-based way, although it’s obviously much broader than the impacts of language models. The digital commons is the online commons of digital resources that we all benefit from and contribute to. The internet is a very public place with a lot of infrastructure and information; it’s an epistemic commons as well. The question is how do we sustainably maintain this commons, grow it, and not degrade it when things like language models come into the picture? We were working a bunch on this question, especially last year. We thought about whether data dividends make sense for things like LLMs, where you pay people out for their data. Thinking about the commons-based nature of the problem reorients the question a little bit — we don’t really know exactly who to pay out to, and lots of people whose data has been taken are anonymous, so it doesn’t make a lot of sense — but then what governance structures would make sense to combat the problem of exploitation and degradation of the commons. To give a little more color to that, LLMs are trained off data scraped from the internet at a very, very large scale. They are generating more content in a way that is faster, and not always high quality; in fact, lots of generated content is pretty low quality. So perhaps these language models are both exploiting and using commons resources, and actually degrading the commons. For example, maybe, Stack Overflow is flooded with AI-generated answers, and a lot of it is incorrect. There is a sci-fi magazine that had closed submissions at one point because it was getting too many AI-generated stories. These things seem relatively minor at the moment, but can we future-proof in terms of thinking about what happens if AI-generated content becomes a much more significant proportion of what we see and read and take to be truth online? I think this relates to the question of how we have healthy online cultural exchange. We want to feel secure and trusting of the things that we see and that we are talking to a person and not an AI, or we are talking to an AI and not a person. These are really difficult verification and trust problems. So how do we take the commons-based nature of this into account when we build governance structures?

HS: What’s on the horizon for the Collective Intelligence Project and how can our readers best engage with the project?

SH: In the spirit of online cultural exchange, one of the things we are trying to build more of is creating a community to talk about these things with and exchange ideas. This is not a solved problem for us, because we’re really new and small, and we get a lot of interest, but we’re actively figuring out how to involve more people in the work. You can always subscribe to our newsletter, or talk to us on Twitter, or reach out to like myself or Divya. The thing on the horizon for us is building out all of these things that I’ve been talking about — building out a really robust pillar around development containers, building alignment assemblies, finishing the initial few reports, engaging with the public. If folks are interested in helping us with that, and are really interested in these ideas, please let us know. It’s about scaling this up and making it more of a live conversation.