More on AI Literature Reviews in the Social Sciences.

Recently I needed to throw together an overview of the literature for a specific topic. The application was, surprisingly, non-scientific. People who work in the actual world of politics asked me what the literature says about a certain topic. This gave me another opportunity to test AI tools and their capabilities, especially since the topic was not something I was neck-deep in. Simply speaking, I would have had to do a literature discovery from scratch using something like my classic literature review technique if it wasn’t for AI.

If you haven’t read my old piece on AI literature reviews, it mostly still stands. Start there. Not here.

To make a long story short, after trying many things that worked very poorly, (lots of inaccuracies and hallucinations), I finally cracked it and found a fairly good way of doing this. The downside is that it is incredibly work-intensive. Maybe not as work-intensive as doing the classic literature review from scratch, but it really pushed the boundaries of what is possible with AI today and what is not. All in all, I had a lot of fun doing it once I got passed the excruciatingly long and painful preparatory phase.

I tried all the tools in my old piece and then some to try to automate this literature review process. Unfortunately, even scite_ is becoming less and less capable of social science research as they put up more and more guardrails. Its writing is shittier and shittier. Article discovery is, unfortunately, quite poor as (at least without tweaking) it does not find the most relevant literature. It’s still the best thing for discovery but what it writes up is inadequate. (It is good enough for the natural sciences, but not for social science.) I also tried more creative approaches but all of my attempts lead to the AI model just making shit up.

But I recently heard on a podcast, one of the three early April 2024 episodes of The Ezra Klein Show that current models can keep around a book’s worth of information “in mind” while chatting. And since I now have a Claude AI subscription I figured I should put this to the test. I grabbed a whole year of a relevant topical journal. I figured I should try to chat with the model about the articles published in the journal that year. (Yes. Download one at a time. Merge the PDFs and upload.) But I was told it was too much. So I did this for a half year. It worked OK, but it was very slow. Claude 3 told me I could only ask a few questions before it cut me off. Clearly, I hit the limits well before it was useful. And what I really wanted to do is to chat about 6-10 years of issues.

So I tried ripping the PDFs into plain text but I ended up hitting similar limits very quickly. And it didn’t seem to work too well. Looking at what came out of the pdf-to-text auto-generation, I am not shocked the AI didn’t know what to do with that.

Here’s what worked. I just decided to make article discovery mostly “manually.” Fortunately, here the task was either to find everything in the top journals about topic X or glance through everything in the X topical journals that could be used for purpose A. So this was done manually with searches within the journal’s webpages augmented with some specialized Google Scholar work. For the topical journals, I read the titles of entire issues. I also grabbed a few things that the journals recommended along the way in other journals they publish. (In fact, this was done already for months.) To add, I used scite_, I used Preplexity.

After all this, if an article looked like it could be relevant, I took its citation, abstract, and discussion/conclusion and manually pasted it into a text file. I separated articles by #####. This took forever. (In 2024, why did I have to manually copy and paste 60 article abstracts, I have no idea. I guess I do have an idea. Moving on…) But once I had a document with around 60 articles (around 82k words), I was able to upload this to Claude AI and have a good discussion about what is in there. (Pro tip. Never leave a DOI out of a citation if you ever do this. Ask me how I know.)

Yes, every prompt was super slow and I was hitting limits in about 6-8 prompts until it cut me off for a few hours. And this was with the paid Claude AI subscription. So I had to be thoughtful about what I asked. I rolled up my sleeve and used all my prompt engineering (and even more prompt engineering) skills not to waste a single prompt. (If the latter link was helpful to you, sign up for their newsletter. I am not sure I was supposed to give this to you without you signing up. There are many more cool pages where this came from. And if you are at all AI curious, the newsletter is a great quick read every day. ) And sometimes I had to wait until the next prompt… for hours. But it worked. To be sure, first I asked for a bibliography. (That was correct.) And I even got correct quotes when I asked it to go step by step and first identify the main points it will make, then find good quotes in the uploaded txt file to support those claims, and then write up the lit review. Prompts for a full lit review with a few subheadings to guide yielded something much shorter than I wanted (despite size specifications). Going section by section did the trick.

After going through and checking everything (themes the articles were cited for, quotes, cites, bibliography), I can honestly say this worked very well, especially for the narrow purpose. The only issue I had was made-up DOI links when the document uploaded left the DOI link out. The literature discovery was practically manual labor, and the prep work for the uploaded book-length document was mind-numbing, but the results are very good, and overall, it was a fifth of the time to do it this way even including double-triple checking everything.

If it wasn’t so excruciatingly slow, and if I wasn’t running up against limits, it would be amazing to just have a conversation about the pieces with the AI. Have a discussion with the papers or paper authors. In fact, I am now convinced journals should train their models (GPTs?) simply with their own text and allow subscribers to have conversations about the content.

Can AI write literature reviews for social science articles? Nope. Not yet.

(Updated October 17, 2023 – still not yet.)

I invested a decent amount of time over the past weeks trying to figure out if there’s a good workflow to follow on AI-assisted article discovery, systematization, and literature review writing for social science academic work.

TL;DR: Nope!

I did this by writing a piece where I was not so knowledgeable about the two dependent variables. I was quite familiar with the 4 independent ones. I needed to review the literature on 8 sets of relationships and the dependent variable in general.

Most LLM tools just make shit up and the verification process to try to figure out what’s real and what’s not and if the real citations say what the AI says they say takes longer (and is substantially more frustrating) than my old-fashioned discovery approach, you can read about here.

There are OK tools for discovery. But they often fail to find the most relevant pieces. But I do recommend you check these out:

  • (the new beta is not yet fully functional – but it may be worth buying credits now as it will be more expensive later, I suspect)
  • (worth subscribing probably – I did and paid out of pocket, though I didn’t pay full price as I was able to find a coupon.)
  • (another one that pays attention to not making stuff up. Good free tier for now – added a month after the others)

Not LLM based, but another useful literature review tool I came across in the process was this: (And I understand this is free.) Check also:

No tool I found was useful in rewriting (messy) notes on relevant articles into a clean literature review draft. It either added stuff it should not have, it got the message, flow, logic, or relationships tested wrong. It failed to recover all the citations from links and notes. I tried ChatGPT, Claude, Llama 2 (of which I run several versions on my own computer), scite, elicit, Bing and Google’s beta search tools. None of them worked very well. Grammarly (with subscription) along with a citation manager and some good old-fashioned writing was quicker and more useful in turning messy notes, and bullet points into good writing. (Well, I wouldn’t say good writing but clear and adequately bland writing for academia.)

This said, I had luck with more organized notes summarization, summarization of PDFs I uploaded which it allows for (including articles I was wondering if I should read fully) using Unfortunately you need to be in the UK or US to use it (or have a VPN that makes it think you are in the UK or the US). Out of all the tools, I liked Claude’s writing the most when I told it to summarize my notes on the literature in the style of a concise social science literature review. 

This note will likely be outdated in 3-6 months (even with this one-month update). The day this changes is the day that LLMs solve the hallucination problem and can, effectively cross-reference links with text written or the day Google Scholar (or one of the others) implements a good LLM. This is not far away at all. I honestly thought we were there anyway, hence the experiment. But we’re not.

Twitter is a good tool to promote yourself and your work. Don’t screw it up from the get go.

Recently with the Guardian projects, Team Populism data releases and etc. I received quite a bit of twitter traffic. Maybe I shouldn’t do this but I am one of those people who wants to follow everyone back. So I do, except…

So, young scholars. If you want to use Twitter to promote your work or yourself, I have three small pieces of advice for you.

1. Fill out your twitter profile. Add your academic level (PhD Candidate, Post-Doc for example), your institution, maybe where you graduated from. Maybe link your website. Say what you are working on academically. And then add anything personal if you want. If it is not clear from this summary, like when it is empty, filled with BS or all personal stuff, that you are a fellow scholar, your academic peers will not follow you, or follow you back, and your ability to promote yourself and your work will be limited.

2. Pin a post that is representative of your work. Make it a tweet about your most recent achievement. You have a recent publication? I am sure you tweeted about it. (And if not, do it now.) Pin that tweet. You have multiple pieces to promote, put together one tweet about the multiple pieces. I actually have three. Here it is:

In 2019: w/@b_castanho @cmbosancianu Multilevel Structural Equation Modeling w/Hawkins Contemporary US Populism in Comparative Perspective & The Ideational Approach to Populism also Carlin & Rovira Kaltwasser

But don’t feel like you need to have several things published before you can pin something. This tweet doesn’t need to be so much. Maybe you are not there yet, maybe you haven’t published anything so far. But you wrote something for sure. Maybe something you did for a class? Maybe you have something under review or getting ready to send something. It is best if such a thing is related to your research topic you want to pursue. Tweet about it! (Link it if you are brave enough.) Offer a summary of the premise and the finding. (Good exercise summing these up w/ 280 characters.) And then pin it. And as your academic work evolves, pin new things as you go.

3. Write in English. (Full stop.) It is OK to have tweets of more personal nature (though some advise against these altogether) but if I can’t read your profile without machine translation (which is still annoying), I will not follow you. Unfortunately, because of this you also want to limit what you retweet as well. If it is not in English, you may retweet it, but it is best to do it with the function where you write about a tweet. Tell everyone, in English, what that retweet is about.

…so receiving this higher traffic I was quite amazed how many people simply don’t have their profile filled out. I could not tell if the person is an enthusiast who I will not care about, a bot, a troll, or a fellow scholar who is clueless. Needless to say, you know who wasn’t followed back despite my total openness policy for following everyone back on Twitter if I believe they may some something I may care about. Don’t be that person.

How to write a survey?

OK, so you figured out you need to conduct a survey. What’s next?

Everyone’s gut reaction is, let’s start writing questions. But if you want to do it right this is exactly what you should not do. In fact, resist all urges to even think about what questions to ask. To write a better survey, the first question you need to answer is:

1. what am I interested in finding out?

It is very important that you do not phrase the answer to this in a survey question. You never really cared how people respond to any question anyway, right? You care about what is behind those responses. Let’s say, you want to know if a customer likes a product. Maybe you want to know if they are likely to refer you, as a service provider, to others. Ask yourself, what are the relevant demographic information needed for your study. Make an exhaustive list of what you want to find out.

Now you write questions? No! You still should not. Rather, ask yourself…

2. what will you do with this information?

You have to think about this question in two different ways, two different sequential steps.

a. How will I analyze this data. Develop an analytical strategy. Will I look at (or present) a histogram? Do I want to see any association between two of the constructs defined in point 1? Are older or younger people more likely to refer my services to others? Men or women? Compile an exhaustive list of such questions you have, that you want to get out of the data. Once you asked your questions, come up with an analytical strategy. Do I just want some descriptive information (like a cross-tabulation) all the way to needing to develop an instrumental variable regression model to causally ascertain the relationship between a key independent and the dependent variable? For the latter you may also need to find three or four plausible instruments. Have an analytical strategy in mind. It could be as simple as calculating the mean, it could be a simple inferential statistic like a correlation or a two sample t-test or it could be something complex. Just make sure you have a preliminary analytical strategy.

b. Know what you will do with the information. Devise action strategies. If I see that young people do not refer my service to friends, I will develop a marketing strategy that will nudge young people to do this. Maybe you are not the person taking action on the survey, then devise a recommendation strategy. If you work for a client (even if it is in-house) push them hard on devising this strategy before you start the survey. The better they know what they want to do with the data, the better chance you have a writing a useful survey for them.

To aid steps 1 and 2 (which you may need to go back and forth on a bit) it is a good idea to draw things out. It is OK to go back and forth between your current and previous steps, but don’t go beyond that.

3. Take the constructs you identified as crucial and figure out how best to operationalize them. Chances are, this is the stage where the constructs (if you are following these steps closely, that are drawn up in a web of relationships of interests) are going to start turning into survey questions. It is OK to ask multiple survey questions trying to tap one construct. Beware (and if need be, modify) the analytical strategy developed in step 2 as you start operationalizing. Maybe you thought you will look at a correlation, but it turns out a simple yes-no question is the best way to ask about something. Then you will only have a dichotomy and not a continuous variable as called for by a correlation. So, you may need to adjust your analytical strategy. At this stage, don’t go back to step 1 anymore.

Follow conventional rules of questionnaire design. Make sure you are asking questions (not just throwing words at the respondent. “Gender: ” should be “What is your gender”? The survey process is a conversation. Don’t break the basic conversational rules. Make sure the response categories you offer are unique, mutually exclusive and they answer the question you ask. Label all your response categories and no need to throw numbers you will use in the analysis at people. Unless you are some survey researcher or quantitative social scientist (which you probably are, or slowly becoming if you got this far), it is wholly unnatural to map a conversation on to some numeric space, so don’t make people do it. Also, don’t even bother them with your numeric mapping. And – very important – make sure the response categories actually answer the question. If the question starts with “how many”, the answer is never “strongly agree” or “disagree”.

Remember that bipolar scales should be no wider than 7 points (11 for experts – but good luck labeling all of them…) and unipolar scales no wider than 5 points (7 for experts). Don’t let your respondent just run through tables of questions with the same responses. They will lose attention. Better if you write question specific response categories.

Write at a grade level that is around 3-5 years lower than the lower end of your population. Don’t use big words, homonyms, heteronyms or jargon that may not be understood by the respondent. Ask one question at a time (the words and + or are usually red flags in survey questions). You can offer a don’t know option, just remember, it encourages people to not engage with the survey, not to think about the survey. (And men, on average are less likely to admit not knowing anything anyway, so even if your goal is to find out of people don’t know something, just know that your results will be biased no matter what, so why bother.) This is hardly an exhaustive list. But there are a million more pointers and also great survey question writing tutorials online. Read through a few. What I see less of is tutorials demonstrating this broader process that looks beyond the question writing and, IMHO, is absolutely necessary to acquire quality responses that you can effectively use.

Finally, please remember that most people hate surveys. This process ensures that you only ask what is necessary and what you know you need and know what to do with. The longer a survey is, the worse the data quality will be. Off the bat, fewer people will take a survey that seems long to them (and it is a good idea to tell people anyway how long the survey will take to ensure they have enough time to do it when they do end up taking it – some people may never take a survey as they don’t know if they will have time to do so, unless you give them a ballpark estimate of how long). People’s attention spans are more and more limited in today’s day and age. After 10 minutes, you can forget about them paying much attention which will be at the expense of data quality. This process ensures no unnecessary questions are asked.

When you start a survey by writing questions, you become fond of those questions and more likely to ask them (or hesitate on cutting them later). This is why it is especially important to first know what research question you need answered and only then, start designing the survey questions that will help you do it.

Of course, there will always be that stakeholder who comes and says, we also should ask question XYZ… and sometimes they have good ideas with obvious implications. But most of the time, this is not the case. The best weapon against such a proposal is the demonstration of the thoughtful development as described above. You come back to them demonstrating how and why all questions are in this survey and asking them, now with this in mind, why do you want to ask that too? They will either improve your design on the spot or back down. Either way, it is a win-win.

The Productivity Trap


There is a trap that is very easy to fall into in academia. You are constantly under pressure to do more. Deadlines are coming up, the work you have to get through is simply unmanageable. So you get stressed, easily distracted, mentally tired. I keep running for food (which especially sucks on a diet). Have way too much coffee with hopes to jump-start my brain. Even now I am writing this blog post instead of doing what I need to.

When this happens to me, I simply cannot get things done. Trivial tasks, as doing edits to a draft I have already marked up with paper (well, iPad) and pencil. This should be easy.

And the next thing I know, half the day is gone and I made a dozen edits that would normally take me 20 minutes with a fresh mind. Would it not have been better to just take the darn day off and get it all done in 20 minutes the next day with a fresh mind? Yeah, it would have.

Don’t be like me. Take time off! Recognize the trap and do not fall into it. If you do this right, it will make you more productive and you will get more done, on average.

On procrastination, or how internet corporatism messes with our brains

Another guest post, this time by my friend, colleague and trusted teaching assistant for Scopes and Methods: Stanislas Richard.

Procrastination is not a new phenomenon – Leonardo Da Vinci is famous for having left unfinished a fair number of his paintings. However, his procrastination was not entirely a problem neither for him nor his creativity. If Leonardo decided to postpone the completion of the Adoration of the Magi and to work on The Last Supper instead, it is more likely he did so because he felt the final work would not meet his expectation (perfectionism is often the main cause of procrastinating behaviour), rather than him just being lazy.  Neither Andrea del Verrocchio, Leonardo’s early master, nor Ludovico Sforza, his first unfortunate patron, could complain that his procrastination was unproductive

Delaying the completion of a task was not a problem for the Renaissance man’s career, and this, in a nutshell, is the difference between Leonardo and me (the only one, really). When I feel bad because I wasted my day on Facebook, my complaint is not that I yet again failed to start this overdue paper, it is rather that I did not do anything at all. Not starting this paper would have been all right if I used that time to read this collection of essay on Neuroscience and Political Theory which is gathering dust on my shelf since I bought it and that I promise myself to read every time I see it.

This “unproductivity” is I think a new feature of contemporary procrastination. It is also why I believe that most of us suffer from something else than putting off an unpleasant task, our problem is rather an incapacity to focus our attention on any task that we could perform instead, any task at all.

So what happened?

I am not the first person in my family to write a PhD – I am kind of a walking proof of this appalling feature of modern society that is social reproduction. Incidentally, my grandfather has a Ph.D. in History, and my mother has a Ph.D. in International Relations.

I was not there when my granddad wrote his dissertation in communist Poland. But I can imagine the picture – he walks in his office, sits at his desk, and puts it in order while lighting a cigarette (he was a heavy smoker). Let us have a look at his desk. I bet it resembled something like this.

What’s lying there is simple, his books and the notes he took in the archives, his typewriter, perhaps an ashtray and the latest Trybuna Ludu. In other words, nothing that could distract him for what he intended to do. He has literally no choice but to get to work. Sure, he could read a couple of lines in the newspaper, but at some point, he will have finished it, and will be required to start writing, since there is nothing else to do. The human brain hates emptiness, it needs to be busy with something. Grandpa could not do anything else than diving into his work. He could procrastinate and write poetry instead (I have no idea if he did, I need to ask Grandma), but that would not change the fact that he would have something to show for it at the end of his day.

Now let us have a look at how my day sometimes looks like. I get inside the lab, sit down in front of my computer, full of good intention and ready to write a new Critique of Pure Reason and … I go on Facebook – just to check, after that, I swear, I start writing. But I see a friend of mine who wants to catch up. He will know I read his message, so not to appear rude, I will write a lengthy paragraph to reassure him that my life is overall as uninteresting as his. And, since I am at it, I will go check the Instagram of that cute girl I like. And now, Levi asks me to write a guest post for his blog, so I will go check on YouTube some videos about procrastination. A couple of clicks latter, I am watching a debate about transubstantiation in byzantine theology. I look at my watch. It is time for lunch, and the only thing that looks like the Critique of Pure Reason is the original version on pdf that I downloaded six months ago but still have not read (guess what happened every time I tried).

Does this story sound familiar? What’s the problem? Why cannot I be like Grandpa?

Well, there is one difference between him and myself, my desk looks like this:

And on the desk, thrones the father of all procrastinators, the matrix of time wasting, the vortex that devours countless hours of the little time I have in my earthy life – the internet.

The internet is a wonderful thing. The problem is that you have everything, every content and distraction you want, one click away. And it never ends – it is not like Grandpa’s Trybuna Ludu. So, is the solution to boost your productivity to unplug your router cable?  Well, not really. You cannot work without the internet after all – most of the articles and paper you write are now digitalized – and it is a good thing, it does save a lot of time, and it is simply convenient. To get the literature he needed, Grandpa had to travel a lot between libraries and archives, and it is just easier and faster to download everything. So, perhaps, it would be just a matter of self-discipline? That can be part of the solution, but it will not do, simply because the internet is literally designed to keep you away from your work, and to drag your attention away from this paper you wanted to start writing two months ago. This is the fundamental problem that we all struggle with. The internet is fundamentally user unfriendly as a work tool, since the range of what it can do is too broad.

Imagine my Grandpa typing his dissertation. He gets bored for a minute – it happens. Well, he does not have much to do, does he?  He can light a cigarette or fetch some tea, but quickly he will get back to work. Our situation is different. When we get bored, we have the entire cultural legacy of human history at the tip of our fingers. At the first sign of boredom, we can immediately switch to the new season of Game of Thrones or Erika Lust’s films. It is surprising that any of us ends up working at all. We should be in awe in front of the self-discipline we display every time we write a single sentence.

The internet has also a problem in the way it is supposed to function. Internet, as we all know it when we get our bills, is a private sphere. Everything there is designed to make money, and the more you spend on a given site, the more its owner gets in ad revenues. The more we scroll down on our Facebook feeds, the more Facebook stock shares increase in value. There are fortunes being made on us mindlessly browsing social media – some people have a financial interest in preventing us from being productive. Most internet sites, social media and applications are designed with the goal to make us stick to them with methods that sometimes border neurological manipulation. If you want to know more about this? Check for yourself.

This is pretty insane when you think about it – and I hope it is not a surprise for anyone. The invasion of private life by corporate interests has become so rampant that even your free time, your very procrastination, is now monetised. This is just the tip of the iceberg – I do not have the space here to talk about shadow work, or the neurological consequences of the aggressive advertisement campaign that the Internet allows, all of which are often responsible for you not being able to finish this overdue paper.

The worst part of the story is that we are our own enemies in this. Humans are feeble, weak creatures, and yet the internet is designed as if we were automatons, made of steel, endowed with an iron willpower, and without, needs, urges, or lazy days – the fantasy of every self-respecting capitalist if you think about it. If this reminds you of the strategy used by the tobacco industry that adjust nicotine dosages to render cigarettes even more addictive, you are right – first render a product essential, second,  manipulate the users so they cannot quit. This business model has a name now, it is called the attention economy, and its devastating effects on the human brain, on public health, and even on democracy starts being criticized in the Silicon Valley itself. Read about it HERE.

Our entire economic system is built around the key notion of productivity and yet it does everything to sabotage it at the same time by messing with natural, human fallibility. We actually cannot even procrastinate properly anymore – like Leonardo Da Vinci did.

But before you decide to give up on civilization and start training in an anarcho-primitivist terrorist camp in Amazonia to overthrow capitalism, I have some good news. There are solutions.

Solution 1: Install attention nannies.

Human willpower is a limited resource, and the best way not to sin is not to be tempted. There are applications that exist which allow you to focus on your work simply by blocking time wasting sites. I am not exaggerating when I say they saved me hundreds of hours.  Here is a list of them:

  • Stayfocusd is an app that allows you to block or reduce the available amount of time you can spend on a select list of sites. My favourite feature is that if you want to give yourself more time to browse them, you need to copy, by hand, a text on procrastination, and why it is bad.
  • Leechblock functions like stayfocusd but it also counts the number of hours you spend on time wasting site overall. No time to practice a musical instrument or going to the gym? Well, look how much you spent on Instagram today…
  • KillFBfeed (for Firefox) or Kill News Feed (for Chrome) and there are others that make your Facebook feed appear blank. Simple, yet very effective if you want to keep Facebook messenger open. (Levi: I installed this immediately after Stan mentioned it in class. I am better for it. I kind of missed that there was an election in Germany, but I digress…). Newsfeed Eradicator works the same, but instead of leaving your newsfeed blank it puts some awesome quotes about freedom, discipline and work that some clever people once said or wrote. And if you feel smarter, or just like insulting yourself, you can even put your own.
  • Rescue Time provides you with a picture of your day as given by the list of sites you browsed that day (and how much time you spent on them). Allow me to tell you that when I tried, it did not look pretty. The premium version is not free, but the free service still allows you to plan your day most effectively.

Solution 2: Plan your “Social media time”.

Whether we like it or not, it is difficult to live without emails, Facebook, tweets, and social media in general. I do not like RSS feeds, and Facebook provides me access to a lot of content that I am happy to read. But I do not want to do it all the time. I check my Facebook only twice a day, for instance. When people used to write letters (yeah, people used to do that), they were devoting an hour or two every morning to it. Let’s do the same, but for social medias. From 9 to 10, I read the articles, papers, and interesting stuff that appeared on my news feed. Afterwards, I switch it off (with Killfbfeed).

Solution 3: Make some non-digital space.

Have a non-digital space on your desk. This sounds silly, but think about it, most of us, if we wanted to work on a paper without using our computers, would not even have a flat space to put our sheets of paper on. Our desks have become spaces planned around our electronic devices. Have a place on it for you to work without switching them on. If you need to put your keyboard away to read and annotate a printed work, it means your computer takes too much space, and you cannot start working without it if you do not have space for analogue work.

Solution 4: Learn how to switch off your laptop/phone and have hobbies.

A lot of us just go on the internet because they do not have anything else to do. This is bad because if we get the practice of visiting time-wasting sites in our free time, we might as well do the same during our working time (and we waste time anyway). I remember that once I had to live two weeks without an Internet connection, so I could not just sit at my desk and spend my time in front of my screen. The amount of time that was liberated blew my mind. It is like someone opened a tap from which free hours flowed that I could devote to reading, going on a stroll, or to play music. This is probably the most difficult (and I still do not do it), but let’s learn how to switch off our devices. You want to read the news? Take the newspaper. You want to listen to music while cooking? Switch on the radio and leave your laptop in the office. You want to prepare a draft for a paper? Do it in a notebook, not on Microsoft Word. Want to read an article to be up to date with the literature. Print it (Levi: poor trees) and have a look at it on the couch. You come back home after a long day? Sit in front of your piano instead of your Facebook. Reducing the amount of time, you spend on a screen is the best way to avoid procrastination – and to banish the corporate parasite from our lives and brains.

How to do a literature review

from PhDComics

Before we go into this topic, lets just take a moment and recognize how spoiled we are. Nobody under 40 (and that includes me for a little while longer) had to ever look though paper based library catalogues to find an article, never had to take multiple paper copies to a conference to exchange with colleagues and never had to stand in the paper room at a conference trying to find the new works of the people they were interested in reading only to find that every copy is already gone. (In fact, most of us don’t even know what a paper room at a conference is.)

The Internet democratized science and while some barriers to accessing papers still exist, titles, author lists, abstracts, at minimum, are readily available at your fingertips. For the rest, we should thank our library staffs who not only work tirelessly to help us by tearing down those barriers, their jobs over the past couple decades transformed completely. They adopted and we reap the benefits.

Back in the day, you couldn’t do this, but today you have incredible tools at your disposal to do a literature review. So how do you proceed?

Usually you have a starting point. You have a paper, a book that inspired you. If you don’t have that starting point, find one. Dig up the wikipedia article on the subject you are interested in, (if there isn’t one, make one) and see if they cite a scholarly piece? Or just do a Google Scholar search (that is – and remember this website as it will come up again.)

(1) Find a starting point. (Sometimes, you have multiple starting points, that is fine as well.)

(2) Read your starting point(s). Make note of every (and I mean EVERY) relevant piece they cite.

(3) Take a look at the list of pieces you complied from the citations. When looking at a paper (and see also the How to Read post), first, you should always read the title. Sometimes the title will already tell you if this is something you should be interested in (you should figure this out from just the citation and without looking up the paper, the abstract). Always take note of the author, check the outlet it is published in. (Assess the quality of the outlet, this may require a google search or two.) If you still feel interested, read the abstract. At this point you should be able to decide with almost certainty if the paper (or book, I may overuse paper or article here, but it applies to anything that could be cited) is worth reading on or not. (Note that sometimes things are, in fact, worth reading but you don’t necessarily need to end up citing all of it. Maybe it will be a better fit for a future study. You won’t end up citing everything you read.)

This strategy of looking who people cite takes you back in time from your starting point. But, today, there is another path. This one takes you in the other direction temporally. (Not all the way to the future, but counting from the publication date of a key paper, yes.)

(4) With Google Scholar you can look at who cites a certain article. You probably still want to start with the ones you selected in Point 1. Type (or copy and paste) the title (precisely) into Google Scholar. Usually the first hit will be the paper you searched for (and if not, it is usually not much lower). Here, among other things, you will see the number of citations that paper received, to date. Click the number, and voilà, you have the list of papers that cite this study. Especially if you are looking at an important book or article, there could be many items on this list. Google orders them by the number of times they have been cited (which is a reasonable proxy of importance, especially for studies that have been out for a few years). Sometimes the task of looking through all these seems daunting. You don’t have the reference of any context as before (like in what context a paper was cited). At the same time you will find that many of these articles will not be relevant for you. (Sometimes it is surprising how many contexts a study could be cited in.) Remember the reading strategy. Read the titles on this list first (and most of the time your search will end here). If you feel you have to filter more, stick to the pieces with the highest number of citations and augment this list with pieces published in the past year or two . (Yes, you can filter by year.) These pieces may be important but did not have time to accumulate too many citations yet).

(5) And do it over and over and over and over. Once you identify and read an important article, save it, read it, (it is also a good idea to take notes on it), identify new cites, check who cites this article.

Once you are at the point where you are going in circles, you are seeing the same pieces come up over and over with almost no new information coming your way anymore, you are done.

One important thing to keep in mind is when doing a literature review, you need the literature that most closely connects with your work, your points, your arguments. If you go too broad, the work will certainly be unmanageable. Sure, pay homage to the most important publications in the general field, but you do not need to cite (or read) everything that vaguely relates to what you do. Knowing what is too distant and what is close enough to consider is kind of an art form, but with practice you will pick this up quick. What helps in this exercise the most is if you know what you are doing when you start.

Also, I am not going to lie, this is a work intensive, time-consuming and brain wrecking activity. When you start on it, the work will seem absolutely unmanageable. Work systematically and, I promise, you will get to the end. At the same time it is also a very fulfilling process. During these steps you are becoming a total expert in a field. Maybe it won’t feel like it at the time because you cannot process so much information at once, but after the initial steps the information, with time and thinking, it will skink in. Focus on the key points that most capture your attention. Write about them during the process as part of your reading notes, annotated bibliographies. Revisit your notes after you took some rest. Go take a day off after a multi-day, intensive, literature review. Go out in nature, clear your head to kick-start your thinking again.

I have gone through this process a few times. The most notable was when I changed MS thesis topics for my Survey Research and Methodology Masters Degree (the second one and already after I received my PhD in Political Science). I knew exactly what I wanted to do. Use a twin studies approach on survey response style. I found an article I liked on measuring extreme response style in an everyday survey. But I was much more uncertain about the measurement of acquiescence and I needed a solid literature review on how these two phenomena are conceptualized and/or measured.

This literature dates back the 1920s, though it is less broad than some other topics in the social sciences. Cycling through the above process took me 4-5 days. I went to my favorite coffee shop (at the time) and just worked 10-14 hour days. In the end I collected, read, took notes on 60 articles. I probably considered (read the titles of, or maybe started looking at the abstract of) over 1000 and fully read the abstract of hundreds. Depending on the topic, your experience with academic reading, your reading speed, your personal discipline, your interest in the activity, your mileage may vary greatly.

Interestingly, I think, this was the closest I ever came to experiencing Flow. I am a horrible programmer, (I do quite a bit of it, but I passionately hate it) so it is quite unlikely that I will ever get there with programming. But this process was just flying. I read about flow only at a later time, but it immediately made sense specifically because of this literature review experience. So there are certainly synergies with Flow productivity. Consider training your brain in that as well and you may be surprised how efficient you can be during this process.

Best of luck.

Enjoy the Budapest Music Scene

I am a firm believer that life does not stop when grad school work begins. (If for you it did, you are doing it wrong. Your grad school productivity is a function of how much rest and off time your brain has. Give it the time it needs to get things done efficiently, in less time. More on this in another post.) So, sparked by a conversation with one of you, students, on the departmental trip, I asked two of the pros, Juraj Medzihorský and Natalia A. Peral who recently left our PhD program to write up some tips and tricks for you. (Many thanks for the contribution.) This is our first guest post. There should be more coming. If you think you have something to offer to students, talk to me about posting it here.

The (mostly) classical music scene of Budapest (by Juraj Medzihorský)

One of the many great features of being a student in Budapest is the chance to see a lot of great music for really cheap. There are two principal venues for classical music Müpa Budapest (Palace of Arts) and the Liszt Academy.

Müpa is where the CEU graduation ceremony usually takes place, so as a CEU student you’ll get there eventually. There’s no reason to wait for your graduation to visit Müpa.

With a student card, such as the CEU ID, you can buy student tickets for 500 HUF. (Levi: Lets hope the Hungarian government did not mess this up as well allowing only the government issued student IDs. I would not put it past them.) These tickets are sold on the spot before the concert. Usually, the tickets are for the uppermost balcony. However, in case there are still unsold seats minutes before the concert, these are sold as students tickets as well. You can guess your chances of getting a good seat by checking which seats are already sold on Müpa’s website. A note of warning: for the majority of concerts it is easy to get a student ticket, and often with a pretty good seat. However, for high-profile performers and orchestras–think Daniel Barenboim or the Berlin Philharmonic–there will be a cue for the student tickets already hours in advance, and many will still not get a ticket. Just the same, you might get lucky as your humble correspondent did on an occasion or two, and end up paying 500 HUF for a 10 000 HUF seat

You can see the list of performances at

As you can see, Müpa’s program is by no means limited to classical music. Jazz and world music is also on feature there regularly.  On some days there are two concerts in parallel–this is possible because Müpa has two concert halls. One of the biggest attractions regularly appearing in Müpa is the Budapest Festival Orchestra. BFO is often ranked among the top 10 world leading orchestras, and it has a distinct character.

Perhaps the easiest way to get to Müpa, especially if you are going from the CEU campus, is to take the tram number 2.

Finally, a note on the dress code. While most concertgoers at Müpa dress up a bit, no one is going to stare at you for wearing smart casual.

The Liszt Academy is a different type of venue, with a central role in Hungarian musical education. It is also placed in the center of Budapest, about 20 minutes walk from CEU. Alas, the Academy does not offer student tickets of the same type as Müpa. However, performances there are often priced reasonably (think 1-2 thousand Forints), and quite a few of them are for free. That being said, tickets at the Academy often sell out days if not weeks before the performance. So plan ahead. You can find the program here.

Just like at Müpa, it includes also jazz and occasionally other genres as well. And just like at Müpa, you can do no wrong attending in smart casual.

For Opera Lovers (by Natalia A. Peral)

Do you love Opera? Budapest is your place!

In Hungary, the costs of the tickets are a bargain for the quality of the Opera you will see. You must know that the Opera season runs from September to May every year, June being mostly a month with a program for kids, specific events and one or two Gala nights. By early March the Opera also launches the season tickets so keep this mind if you are willing to get those!

As for the season 2017/2018 the Opera house will be closed and a shorter program will take place in Erkel Teather.  The Opera building is under renovation so you will have to wait until 2018/2019 season to really enjoy the full spirit of Opera in Budapest.

Do you want to know how to access to cheap tickets, even free performances? Here are some tips.

  • The cheapest seats are not necessarily the cheapest option.  Although there is a fixed price you can see online (my favorite website is hereor at the cashier in the Opera house, sometimes you can also access to even cheaper tickets through the following options
    • Get the ticket just few minutes before the Opera starts. I have done that many times paying 1000 HUF for a ticket that normally costs 17,000 HUF. Yes, it is possible and you need to be lucky! Generally, in every Opera day there are some empty seats of the seasonal tickets (people goes on holidays, get sick etc.!). People with seasonal tickets donate them to the Opera house so for the house to be able to re-sell them and collect extra money.
    • Ask for rehearsals and morning Opera performances: there used to be some at 11am, and rehearsals timing also varies. Just ask for these options at the cashier at the Opera house. I know, you have to be a true opera lover to find any charm in an Opera at 11am! Yet, tickets are very cheap, I recall paying 400 HUF, and even nothing at all!
    • Know a dancer, a staff member, a singer!  They are not only the best people to hang out with but also sometimes the best and only option to get access to performances that are easily sold out.  This is a win-win option, you not only will learn a lot and turn into a great fan of your talented friend but also you will be able to access to tickets for a very decent price, or to gain access to tickets that are not available otherwise!
  • The most expensive seats at best prices are lately more difficult to find since Erkel Theater opened its doors again. However, you can still try some options:
    • Ask for a last minute season ticket or last minute specific seat that you know is expensive due to its better location.
    • Check the various cultural events that take place in Budapest: for example, in September each year the official buildings are opened to the public and the Opera house sometimes distribute tickets on first come first serve basis for any performance on that specific day. This might change this year as only Erkel Theater is running the Opera performances!
    • Buy Season Tickets as early as possible. Know that most of them are sold out on the very first day they are announced, that happens on the same day that the Opera house announces its program for the year. This requires that you stay in tune with the cultural life in Budapest, or the Opera website! Remember that is generally in March when this takes place.

The dress code in Hungary is extremely free so don’t let the prejudices related to dress code and the Opera of the 19 century traditions deter you. However, keep in mind that the holders of season tickets in Hungary are generally traditional families and well experienced Opera lovers. So if you plan to get a seat that is part of the season tickets then, be aware that some might make you feel uncomfortable with their gaze – if you are just reaching the opera with your sport clothes, for example. Traditional season tickets holders are very protective of Opera traditions, everywhere, not only in Hungary! Nevertheless, except for the specific gala nights with specific dress code (generally thematic as well), you are free to reach the Opera with any outfit you want.

Whether is your first time at the Opera or not, there are two main performances I will always recommend in Budapest “La Boheme” and “La Traviata”. Performance, music and scenography are of an impeccable taste and level! This is just a personal viewpoint. I still truly believe that the best performances I have seen in the world of those two pieces are from my beloved Opera in Budapest!

Nevertheless, the Opera in Budapest is of high quality so keep exploring anything there is in the program until you build your own favorites!

If this is indeed your first time at an Opera performance, keep in mind that no necessarily you will get the flavor of the whole art-form in one opera day.  I particularly do not enjoy much Wagner but I do love Verdi and Puccini. So, I was very lucky that my first time at an Opera house was to see Tosca. My advice, your first time deserves the company of an Opera lover, so get someone next to you with the right passion to make of this an unforgettable memory!

As a disclaimer, keep also in mind that if this is your first time at an Opera performance, do not prejudge it through your experience at the Erkel Teather this year. Opera performances at the Opera house are sublime!  However, that might not necessarily be the case at Erkel Theater. I still resist returning there after it ruined the acoustic of “Aida”. I can tolerate the ugly building but I cannot tolerate the bad acoustic!  I haven’t returned since then, 2 years ago, maybe this has been fixed already!

Again, this is just a personal taste, do not let it deter you! Try it out! The quality of the Opera in Budapest is supreme and the cheapest in the world! No kidding!


Back your stuff up

from PhD Comics

If you will only keep one advice posted on this blog, follow this one. It is the one that, most likely, you will not need. But the ones that end up needing it, its absence can be catastrophic.

BACK YOUR STUFF UP! Yes. Back up your work device.

Every year I watch how one MA student loses their work. Sometimes it is to hard drive failure (as hard drives fail, even the lovely non-moving electronic ones), sometimes it is to theft, fortunately I have not seen fire or other disaster loss but it could happen, the worst I have seen (and I mean seen as it happened right in front of me) was a tray of café lattes landing on a computer (which never worked again). Out of the ~50-80 students we have every year, it happens to one. Do not let this be you.

In fact, I follow a backup strategy that is multi-tiered. I have a backup at home, I have one at the office and I use cloud service to back up everything I do in between backups (ie. all the time I am connected). And I do not think anything less is acceptable. Once I had redundant copies of a few hundred pictures. I had to swap hard drives in a computer so I figured it is fine to just delete one, I will back up again as soon as I restart. One copy I deleted, the other had a hard drive failure right there and then. I was an idiot. Never again. But even if you do not tinker with your computers, fire, water damage, theft in your apartment can happen (at the same time) to both the backup and the computer. You cannot afford to have just one backup sitting next to your computer thinking you are safe from drive failure.

So what do I do?

I use an external hard drive I plug into my computer at the office. (And I do not do this enough as I do not go to the office that often.) I have Apple Time Capsule at home backing me up whenever I am connected to my Wi-Fi at home. (By the way, this is not a good way to back up. It is a comfortable, but sub-optimal solution if you ask me. Time Machine on macs is not the best. Try to get a temporary replacement computer with less storage than your lost computer originally had and you will see why. You will have a hard time restoring to it. I only dare to do this because, at this point in my career, I can walk into a store and buy a top of the line mac if I absolutely need it. It is a calculated risk.) And I have multiple cloud back up services running. I work in folders that are auto-synced when I am online. And I make sure I am connected at all times. For photos, I back up to Google Photo. My Android phone is backed up to Google and my iPad is backed up with Apple. (All you need to do on these devices is turn backing up on.)

About cloud back up, I would simply suggest you use it. You only really need one of these. I use multiple, but that is my choice. I started with Dropbox where I acquired some extra free space in the early days and I still like it the best. It never let me down. The app is reliable. It always syncs. I know it is not very secure, and I do not care. I just make sure not to put any sensitive data in an unencrypted folder. (You should too, BTW.) Most colleagues like Dropbox for sharing files so it is here to stay (at least for me). I understand you get some free storage and I think they have some education set up as well that gives extra to students. Another obvious choice for CEU students is OneDrive as the Office 365 subscription you get with your email account comes with a big chunk of free storage. If you are new to cloud back up (and you are a student at CEU), just use this. It has nice Microsoft Office integration and now that CEU is going all Microsoft platform, I also have no choice but to use it. It works nicely with both Mac and Windows. There is the third giant, Google Drive. I actually pay for this as my Gmail needs the extra storage anyways. I also like Google Docs and some colleagues prefer this to Dropbox for collaboration. I really just use it sparingly but I use it and it works.

If you are from the more paranoid walks of life (and it is totally understandable – no judgement) you could use some secure encrypted service. I use one such service for things that could be more sensitive (prone to identity theft). I also encrypt my whole computer. So, again, no judgement for me. There are several such safe, encrypted solutions (and if you are so paranoid as to not even trust these services, I may be judging a little for sure). I use Spider Oak. I have a folder that is with the more sensitive things like tax paperwork, banking papers, etc. That is really all I sync here. And I would not be Hungarian if I did not mention Tresorit as a similar secure alternative (based in Hungary among other places). But these services, look like, will cost you. I got a few GB from Spider Oak for free back in the day. If they still have that option, they are hiding it well. There may be other solutions.

So, in sum, BACK YOUR STUFF UP. Buy an external hard drive (can even be a pen drive) and back up your work. Keep a backup at home and one in your locker/office/etc. And use some kind of cloud storage. Every year I watch disaster hit and how people are able to cope with the disaster 100% depends on this one thing they did or did not do. So, no sympathy and no excuses. I will not care what kind of coffee plantation, thief, fire or dog ate your homework, thesis, etc. and neither should any my colleagues. You have been warned.

How to read


Dear Scholars

One of the most important misperceptions about reading in academia is that it is like any other reading, except harder, less fun and therefore it takes longer. No wonder people freak out when they are faced with a graduate student work load of several hundred pages each week.

When faced with this, fear not. There are some great techniques to make your academic reading more efficient.

Lets think about what academic reading is all about. It is about finding out what other scholars have done in a certain area of our interest (or sometimes the interest of our professors who force it on us – oh well). A novel you probably will enjoy reading front to back; with a murder story you usually don’t want to flip to the end finding out who did it. This is not the case in academia. You want to know what was done, how it was done and what you should take away from the scholarship at hand. And there are a few techniques to help you get there and get there more efficiently than “just” reading. More experienced researchers do this intuitively, but there is absolutely no reason why anyone should not just follow the instructions laid out here and act like a more experienced researcher.

When you grab an academic article here’s how you should approach it.

1. Read the title. Very often the title will already tell you that you can stop reading, and if it did not, the next step certainly will. (I guess, if someone assigned the text for you to read, you have no choice but to read it. You still probably want to follow the advice laid out here.) If the title is good, it will effectively communicate what was done in the study. (People like to get cute with their title. It is a bad idea unless you have the luxury of space to get cute – then it is just cute. But more on this in a future blog entry on how to write.)

2. Read the abstract. A good abstract will tell you mostly everything you are interested in. What was done, why, what is the result and what you should take away from this. (Once again, some people don’t like to give away the end of the study, but that is strategically a bad decision.)

3. Look at the tables and figures (if there are any). A good table and/or figure is self-contained and self-explanatory without consulting the text. Sometimes it takes some practice looking at tables and figures and figuring out how to read them, but very quickly it becomes second nature, especially if the figures are good. These excerpts can help get detailed information on what was done, what the author would like to communicate from the study.

4. Read the introduction and the conclusion. In addition to the most important results from tables and figures (especially in empirical studies) why the study was done, why it was interesting to conduct and what you should take away from the results is explained in these two sections.

(Some people prefer to do 4 first and then 3. Figure out what works best for you and use that strategy.)

5. Now, armed with all the most important information, you are ready to just read the whole article. And here’s the trick. It will go much smoother and much faster. You already know what you are reading, why you are reading it, what the findings were. You will not have to stop and think what is going on. You most likely will never have to re-read sections to make sure you are getting it right. In fact, because you got the most important things, now you are just looking for at the details that may be interesting or important. You will be much quicker, your brain can be at ease that you will not miss anything major and therefore you will be at ease reading faster. Cognitively, this faster reading will still allow you to process all the information better than the resource intensive, cognitively more taxing, very careful reading you would do if you were just reading a study front to back.

+1 Take notes as you are reading. Do not highlight. Highlighting is a lazy person’s way of taking notes. But there is a huge problem with it. When you highlight something, you are basically telling your brain, “this is important, so I will have to get back to this”. At the same time you are also telling your brain, “for now, you can ignore this” making your brain less able to process the information you just highlighted. Sure, you could come back and read it again, but how efficient is that? First, you read cognitively ignoring the most important points. And then you have to read again? When you take notes, always rephrase key points in your own words. You make your brain process the information better when you reformulate the points in a way to become more consumable for yourself. It is important that you do not just copy as that is also cognitively lazy (like highlighting). In the end, you have a record of what you found important, memory cues in your words for things your brain already processed so if you want to refresh the material, such notes work very well. But most of the time you will not even need a refresher, unless it is for an exam at a much later date (and such exams are rare in grad school).

I would certainly not propose that you skip any of these steps (including and especially the full read), though in a literature review, earlier points may lead you to conclude the information presented is irrelevant to you. Then it is fine to stop. (Not so much if you are reading because someone above you made you read a piece. Then it is best to really read it, but the techniques laid out here will still help you read the material faster and process it better.)

Final notes: the technique was developed for articles but it works well for books as well. Try it!  To produce this post, I drew heavily from Oxford University Academic good practice – a practical guide (yes, Oxford has this in their “academic guidelines” – but sadly half of that document is about plagiarism…) and conversations with Darren Schreiber.