(Updated October 17, 2023 – still not yet.)
I invested a decent amount of time over the past weeks trying to figure out if there’s a good workflow to follow on AI-assisted article discovery, systematization, and literature review writing for social science academic work.
TL;DR: Nope!
I did this by writing a piece where I was not so knowledgeable about the two dependent variables. I was quite familiar with the 4 independent ones. I needed to review the literature on 8 sets of relationships and the dependent variable in general.
Most LLM tools just make shit up and the verification process to try to figure out what’s real and what’s not and if the real citations say what the AI says they say takes longer (and is substantially more frustrating) than my old-fashioned discovery approach, you can read about here.
There are OK tools for discovery. But they often fail to find the most relevant pieces. But I do recommend you check these out:
- elicit.org (the new beta is not yet fully functional – but it may be worth buying credits now as it will be more expensive later, I suspect)
- scite.ai (worth subscribing probably – I did and paid out of pocket, though I didn’t pay full price as I was able to find a coupon.)
- consensus.app (another one that pays attention to not making stuff up. Good free tier for now – added a month after the others)
Not LLM based, but another useful literature review tool I came across in the process was this: researchrabbitapp.com (And I understand this is free.) Check also: connectedpapers.com
No tool I found was useful in rewriting (messy) notes on relevant articles into a clean literature review draft. It either added stuff it should not have, it got the message, flow, logic, or relationships tested wrong. It failed to recover all the citations from links and notes. I tried ChatGPT, Claude, Llama 2 (of which I run several versions on my own computer), scite, elicit, Bing and Google’s beta search tools. None of them worked very well. Grammarly (with subscription) along with a citation manager and some good old-fashioned writing was quicker and more useful in turning messy notes, and bullet points into good writing. (Well, I wouldn’t say good writing but clear and adequately bland writing for academia.)
This said, I had luck with more organized notes summarization, summarization of PDFs I uploaded which it allows for (including articles I was wondering if I should read fully) using claude.ai. Unfortunately you need to be in the UK or US to use it (or have a VPN that makes it think you are in the UK or the US). Out of all the tools, I liked Claude’s writing the most when I told it to summarize my notes on the literature in the style of a concise social science literature review.
This note will likely be outdated in 3-6 months (even with this one-month update). The day this changes is the day that LLMs solve the hallucination problem and can, effectively cross-reference links with text written or the day Google Scholar (or one of the others) implements a good LLM. This is not far away at all. I honestly thought we were there anyway, hence the experiment. But we’re not.