Let’s talk about why and what game to play next.
For a few days, I got to use Claude Fable 5, Anthropic’s new model that, for now, has been “export controlled” out of existence by the Trump administration. In those few days, I rewrote my grant (which needed several structural changes, the most significant of which was reducing the main section from 14 to 7 pages). I wrote three articles. And I wrote a book. I also experienced my post-doc’s theoretical chat about his work on polarization and democratic backsliding with Fable 5, which he called the most in-depth and enlightening conversation he had about his work with anyone. (And this was not for the lack of talking to leading figures in his field all the time.) From a social and political science perspective, Claude Fable 5 was true PhD-level intelligence.
When I want to stress-test a new AI infrastructure, I like to give it hard tasks and see how it performs. The most significant leaps, to date, were Claude Code’s Christmas promotion and Claude Fable 5. The ERC grant rewrite was amazing. It gives me a very good starting point, and I will not have to suffer the weeks of excruciating pain to suffer the 50% cuts myself. The first paper I gave was almost done. Intro was written. The case studies were described. Analyses for those case studies were done. It just needed assembly. The model did fine. I had a round of comments I passed to my coauthor but we agreed that we could even give these comments back to Fable, and it would be able to just do it. (Except by the time we got there, Fable was gone. He’s finishing the paper as we speak.) The second paper I gave Fable, was one I wrote for a conference a few years ago at the request of the organizers. There was simply no way that paper was ever going to get written. I had collaborators on this project (two seniors and one junior), but they didn’t move forward with the paper either. So I gave Fable the conference slides (mildly edited to include only what I want in the paper), the replication code, and prompted it to write the paper with the target journal, structure, and style I have in mind. The results were absolutely impressive, but I have to admit, I am not so familiar with all the literature it built on. I will be doing a lot of reading before I move forth with this paper, but if it is as good as it sounds, I am extremely impressed. So I decided to give it a similar conference presentation. I know the literature for this one, but I didn’t really know what to do with the study, so I mostly abandoned it. For a couple of years, it’s been at the back of my mind that I should do something with it. I am a comparativist, and unless it is the US, I am not used to writing a single country case study, even if it is my native Hungary. The result from Fable were practically flawless. Literature was spot on. The theoretical arguments laid out framed the paper better than I could have (maybe not better than my Hungarian comparativist friends could have, but definitely better than I could). The model even pointed out a major flaw in the comparison cases that had completely escaped my attention due to insufficient contextual knowledge, and proposed and implemented a fix. At this point, I seriously contemplated what this means as I handed Fable the most difficult problem I had.
I had a pop-science book in mind for a while. I started extensively outlining it last October. After a very very deep dive into how novelists use AI who do not want to just produce slop, I was determined to try what they do with something. This book was already extensively outlined. The only voice in the book is the narrator. I wrote two chapters entirely myself to establish the narrator’s voice. I even supplemented this with style sheets with examples of how he talks about X type of characters, Y types of characters. The book was outlined with specific chapters and chapter goals. For the first 10 chapters I outlined the main arguments in the chapter, and the case studies to highlight the arguments. My goal was to do all this for every chapter and then use Novel Crafter to write the book. But on a whim I decided to just hand it all to Fable and have it write the first 10 chapters. Then, even with the less extensive outlining, but with the first 10 example chapters in hand, also write the rest. Once again, while I sense that the first 10 chapters are richer, the rest will need more editing and fact-checking, the results were truly great.
All of this now begs many questions (and a severe existential crisis) that have been floating around in my head for a long time.
In a world where one can rewrite a grant, write three papers, and a book in 3 days with AI, what is left of an academic’s job? How do we evaluate performance from this point forward? Is any of this OK? What is OK to do when conducting social science with AI? What is not OK? How far can we push this? (Think, agent as a research assistant.) What can we do without Fable? Is it OK to teach this? Is it even OK to tell anyone about this, to write this blog post? What inequalities does this produce? (The question we so conveniently avoided addressing during COVID, when half of academics wrote a paper a month while the others were sick, depressed, or on caring duties.) What does this mean for the social sciences?
So let’s start.
When I coauthor with more junior scholars, it is usually on an idea I had (I may even have some preliminary analysis), or it is something I see as publishable as an offshoot from their own dissertation research. Sometimes it is papers that come out of joint conversations and brainstorms. I usually never write the first draft. We do a couple of rounds of drafting, feedback, and rewrites. We often do analyses side by side, exploring. I found this to be a good learning experience for junior scholars. Then I usually take one final, extensive pass. They submit. (I hate that part, and it helps the first-time authors get past a serious mental barrier: the submission. Once you have done it once, it’s easy.) I am never the first author (unless the topic is controversial, in which case I need to serve as the shield). Junior colleagues need first-authored publications more than I ever will. (There are deviations from these things, of course. They are rare.)
What I did with papers 2 and 3 above was replace the more junior collaborator with the AI. The process was not at all different in anything but speed. Also, I can’t say I enjoyed the collaboration as much, and, yes, what I did took an opportunity away from a junior scholar. (Not worried about the latter. It was never the ideas that were the scarce resource. I can honestly say, these two papers would NEVER have been written without Fable’s intervention here.)
When Fable was taken from us, I made a list of things I should do in addition to these three papers. There’s a 3-4 strong list I am sitting on in addition, of things I would never have finished. Yesterday, I grabbed one of them and asked the question, can I do this with Claude Opus 4.8 Max effort? The answer is that it is harder. One has to be much more careful with the feedback one gives to Opus 4.8. I couldn’t just write all 17 points I had as feedback, because, not unlike PhD students sometimes, the model literally got overwhelmed. If Fable is PhD level intelligence, I would still consider Opus PhD student level intelligence. I will try this again soon. I will try to structure my feedback better and go a couple of points at a time next time. From what I have already seen, I know this will work and produce a great paper once again. And you don’t even need the $200 Claude Max 20x subscription to do this. The $100 5x Max is just fine for such tasks. (Maybe the $20 one as well, but you will hit limits often, which is frustrating, and these limits will slow you down a lot.)
Is this an acceptable mode of Human-AI collaboration for writing scientific studies? It is my ideas, my data, my (AI-assisted) analyses even. I refuse to put any of it out without engaging with literature it suggested as relevant that I am not familiar with. The arguments are refined with the help of AI. Composition, analytical tweaking, and presentation are handled by the AI, but the work is extensively edited and, most importantly, audited by me. I stand behind everything that is there. Am I just using a tool when I do this or am I doing something I am not supposed to? I don’t know if this is OK.
OK or not, I know many people use these tools very extensively. And we haven’t even started to explore all modalities where AI extensively assists science. I have a project now where the experiments are done within a computer through simulation. I have over a dozen research questions, full research papers sketched that I would need to generate and analyze the data for. The papers are super formulaic, mostly empirical-methods papers without theory. The questions are simple, interesting (for the community that use such simulations) and the results constitute important contributions. I know I could train a Hermes and OpenClaw (AI) agent to do all this for me. All I would need to do is audit and edit the final analyses and papers. Would this be OK? Would this not be OK? Why so? Why not?
In fact, I can think of very formulaic social scientific subfields where important, substantive, and directly useful contributions could be fully automated. If we work out reliable, fully automated pipelines, would there be anything wrong with using (and publishing with) them? Why? Why not? I have been working on this for a couple of years now, and while the Fall of 2024 attempts were cute, every 6 months I get more and more scared. We are going to do something about this in the Fall with my team. And this time we plan to publish the results. It is time. I think we need to get serious about this not only for computer science (a visible AI research fetish) but also for all the other sciences including social science.
I think I know one red line. We can’t just prompt papers into existence without specifics to build on. Those who have tried certainly burned themselves. But I am also sure a lot of garbage makes it through. And future attempts of this will become less and less bad, and harder and harder to catch. But at minimum we need to have a serious and open conversation about what we have all tried, what we all did, before we can understand what this means for our fields and how to move forward in our new reality.
I do have some recommendations for moving forward.
I work in Hungary. The introductions of strong incentives to publish or perish have upped the quality of Hungarian social science where it was implemented. These were important and necessary steps. But they did come at the expense of certain fields of political research like anthropological work. Those who spend months, years in the field can’t publish 3-6 papers a year. Today, those who work in the more quantitative areas will be the first to be booted, if not replaced by AI. We must deal with this and move away from publishing quantity-based evaluations.
Journals are drowning as is. Reviewers figured out during the COVID lockdown that they don’t have to say yes to everything. Quantity-based assessment is not sustainable. (Good on the publishers for starting to purge journals from their portfolios.) Less is more. So we need a system where less is more.
One admittedly ugly way of thinking about scientific contribution is to ask how much we contribute to the training data. If our study adds little new information beyond what the AI already could have derived from its training, we are not really contributing. Anything that produces primary sources and new interpretations is the most useful scientific contribution in this new world.
Second, let’s start rewarding hard work, not just output. There are good models for this. The British REF or promotion reviews assess only four publications per person for a multi-year period. They are (or at least should be) assessed on quality and impact, not on quantity and impact factor. The prestige of journals still matters, but that is fine. Denmark invests extensively in the qualitative assessment of people’s work during hiring, promotion, and grant assessments. These models are forward-looking. Let’s look at how much work and how strong a contribution one has produced. Recently, I was looking at an acquaintance’s job market paper. All I could think was that this girl has worked more on this paper than I have in my entire research career. (She landed an amazing job with multiple offers.) This is what we need to reward. In the age of AI, studies can more easily become bigger, grander. Editors should ask for more substantial contributions. Bosses should ask for less, but better papers.
We also need to reevaluate what else social scientists do. Conversations about public scholarship are not new in political science. Should we reward going into the media or not? Public communication of works (like blogs and podcasts) is rarely rewarded. In an age when academics are often viewed skeptically, maybe we should make it our explicit purpose to communicate with the public, not to mention be of use to them.
I work extensively with LLMs and assess their performance on various scientific tasks: writing, behavior modeling, idea generation, etc. Which model works well for certain tasks and which doesn’t is highly unpredictable, even if, at first glance, we think it shouldn’t be. For coding, you can expect more parameters to yield a better coding model. For writing and modeling human behavior, this is not the case. I already told my team that we should consider creating and publishing benchmarks for such performances as new models come out. We have the infrastructure to do it. It is not an article, but such a tool could be extremely useful for someone who wants to use these LLMs for related things.
We can recycle old conversations about collegiality, teaching, mentorship, and service, all of which are important in academia. I sincerely believe that in the world that is coming in the age of AI, one’s value as a person will be a direct function of how many people’s lives they have made better and by how much. This should be the guiding principle. Let’s also evaluate academics on this metric and help make academia a better place overall.
The sooner we start these conversations, the better academic life will be during this extremely uncomfortable transition. One thing I am sure of. Those who want to eliminate AI from academia and go back to the way things were before will not get their wishes. Head in the sand is no longer a viable strategy. (If the above did not demonstrate this effectively, nothing will.) And let’s try to move forward without making each other’s lives even more miserable along the way, fighting about this, and gatekeeping. The current academic publishing game is super boring as it is. It is not fun. I am not sad to see it go. But let’s make sure the next game is one that creates a freer, more productive, and more humane academia.
