I invested a decent amount of time over the past weeks trying to figure out if there’s a good workflow to follow on AI-assisted article discovery, systematization, and literature review writing for social science academic work.
I did this by writing a piece where I was not so knowledgeable about the two dependent variables. I was quite familiar with the 4 independent ones. I needed to review the literature on 8 sets of relationships and the dependent variable in general.
Most LLM tools just make shit up and the verification process to try to figure out what’s real and what’s not and if the real citations say what the AI says they say takes longer (and is substantially more frustrating) than my old-fashioned discovery approach, you can read about here.
There are OK tools for discovery. But they often fail to find the most relevant pieces. But I do recommend you check these out:
- elicit.org (the new beta is not yet fully functional – but may be worth buying credits now as it will be more expensive later I suspect)
- scite.ai (worth subscribing probably – I did and paid put of pocket, though I didn’t pay full price as I was able to find a coupon.)
Not LLM based, but another useful literature review tool I came across in the process was this: researchrabbitapp.com (And I understand this is free.)
No tool I found was useful in rewriting (messy) notes on relevant articles into a clean literature review draft. It either added stuff it should not have, it got the message, flow, logic, or relationships tested wrong. It failed to recover all the citations from links and notes. I tried ChatGPT, Claude, Llama 2 (of which I run several versions on my own computer), scite, elicit, Bing and Google’s beta search tools. None of them worked very well. Grammarly (with subscription) along with a citation manager and some good old-fashioned writing was quicker and more useful in turning messy notes, and bullet points into good writing. (Well, I wouldn’t say good writing but clear and adequately bland writing for academia.)
This note will likely be outdated in 3-6 months. The day this changes is the day that LLMs solve the hallucination problem and can, effectively cross-reference links with text written or the day Google Scholar (or one of the others) implements a good LLM. This is not far away at all. I honestly thought we were there anyway, hence the experiment. But we’re not.