On how to research

One of the best courses I ever took when I was at library school was a class on how to evaluate academic papers without needing to be an expert in the subject matter. It’s an invaluable skill even when doing research within a subject you know fairly well. Things like checking the name of the author, looking at the sources used in the bibliography, and examining how any numbers were being generated. 

While you don’t need to put on your researcher hat (I’m picturing something like Taemin’s top hat in the Sherlock MV) just to enjoy a song and dance but if you are interested in digging below the surface here are some things to keep in mind while navigating the overwhelming sea of information out there.

1. Who is the author and where is it published?

Everybody has biases, that’s just part of being human. It’s impossible for anybody--myself included--to be perfectly objective all the time. But that said, when you read a glowing review of an album, it’s worth taking a look to see who wrote it. I’m a known fan of Johnny’s & Associates group A.B.C-Z, so I’m probably going to rate their new single higher than somebody who doesn’t follow their career. It doesn’t mean that I’m wrong or that the other person is wrong but, like I said in Episode 31, it’s natural to prefer hearing somebody you like sing poorly, to hearing somebody you don’t care about sing well.

For example, take a glowing review in PopMatters for BTS’s Be, when you Google the author, you can easily find her Medium, which contains an article about how much BTS’s messages of self-care mean to her. Again, it doesn’t mean that the review is worthless but it does mean it should be read with some degree of skepticism because the author is a fan. 

There are also many different kinds of publications out there. Somebody writing for a publication that depends on access to idols for interviews and awards performances (as well as depending on fans for clicks and engagement and so on) is generally not going to be encouraged to give a negative review or say anything remotely critical and certainly will be encouraged to copy-paste favorable press releases. In other words, access journalism is rampant at the kinds of outlets that do those idols playing with puppies videos or special song showcases. 

For example, here’s a 30 million view video from Billboard featuring Blackpink being adorable and here’s Billboard talking up Blackpink’s “Ice Cream”. To put it in perspective the average video on the Billboard youtube channel barely cracks 10k views, maybe they’ll get a million if it’s a big, big star (maybe 2-3 million if it’s BTS). There’s a strong incentive for that outlet to keep those fans happy and engaged with the content.  Again, it doesn’t mean that idols playing with puppies videos are evil or you shouldn’t watch them but it does mean that you probably shouldn’t take the outlet all that seriously when they say a song is the hit of the summer or a group is the next big thing. 

When you’re looking at academic writing and serious criticism, things get a little more complex. What is an academic author’s background? Are you reading a paper from somebody in a sociology department who has no background in the arts attempting to critique music? Is somebody from the Korean Studies department writing authoritatively about Japanese or Chinese culture? And check to see if the author is trying to sell you something on the topic--a book, a ticket to a conference, her expertise as a media source, etc. These things don’t necessarily disqualify the contents of the piece but they are big flags that it’s almost certainly not objective.

2. What are the sources being used? 

Something to be very aware of in the world of Asian pop culture news and criticism in English, is that it is very easy for journalists to coast along on “narrative”. What I mean is that it’s easy to pick up on the arc of the story happening on social media and just… go with it. You see this all the time in sports and politics writing too. It’s great for journalists on a tight deadline because it involves no actual reporting or subject knowledge or analysis. Just read through a bunch of other people’s takes online and then synthesize them into your own. Is everybody saying variations of the same thing? That doesn’t mean it’s true. It may just be the “narrative” at work.

Did K-pop fans do anything by hijacking various political hashtags or is it just the “narrative”? Was there any real on-the-ground reporting done or did the journalists merely compile a bunch of tweets and crib from other people’s posts?

When you’re dealing with Asian popular culture, language and cultural expertise are really important. The biggest rule I have is that you should never trust anybody who doesn’t use domestic sources or who is cagey on their language ability. Reading an article on Asian popular culture that references only English language sources? Huge red flag. They’re going to be building off of secondhand analysis, filtered through biases passed along like a huge game of telephone. 

Say you wanted to write about Japan’s all female Takarazuka Revue. If you limit yourself to English-language sources, then you’re going to be relying on fan translations from people like myself who self-select our material and a small number of articles written from a Queer Studies perspective, which is not representative at all of how the Revue is seen or experienced in Japan. (Source: the Japanese language guidebooks I have.) Okay, so maybe you supplement by reading theater articles about Broadway--but again that’s not what Japanese audiences would be familiar with. You’re assuming a stage show in Japan is identical to one in America except for the language and then that assumption is going to be baked into the analysis and passed down to anybody using your paper as a source. 

This is the problem I see all the time with writers putting the roots of K-pop in the Jackson Five (or Backstreet Boys) rather than in Sobangcha. 

3. What do the numbers actually say?

Say that somebody showed you two charts showing the relative popularity of two idol groups and told you one was true and one was false. How do you evaluate the charts for yourself?

Here are some important questions to ask: 

Where are these charts being pulled from?

What were they intended to demonstrate in the original source?

How was “popularity” being measured in each chart?

Who was the population being measured in each chart?

“Popularity” is not something where you can pull out a tape measure and give an objective number and say okay so BigBang has 165 cm of popularity today.

Fandoms have figured out how to do things online like bulk purchase mp3s, mass vote, and zombie stream playlists in order to mimic general popularity so relying exclusively on those metrics like those is problematic and--at best--can only measure the strength of a particular fandom against other fandoms they are in competition with. These things are not a measure of broad popularity.

You also have to look at who the population being measured is. Is this a self-selected and global group of Kpop fans voting in an Internet poll? High school girls in Korea responding to a magazine survey? Asking these two populations about who is popular will give you two very different results.

I’ve talked about the changing metrics that go into what makes a Billboard “number one” many times over so I won’t do it here again but I will emphasize (again) that it’s meaningless to compare a number one hit under the new rules to one of years past. When “Gangnam Style” reached number two on the Hot 100, YouTube views were not factored into “hotness”. Now they are. Apples and oranges. 

So, can “popularity” even be measured accurately when fandoms have become so adept at manipulation themselves? Japan still has the Nikkei power rankings which uses tried-and-tested survey methods to ask actual humans whether they know an artist and if they feel positively towards that artist. Public opinion says Arashi has been the most popular musical act in Japan for nine years running. 

For anybody interested in deep diving into how charts and numbers can be manipulated to make you believe they say something that they don’t, I highly recommend John Allen Paulos’s book Innumeracy but the quick and dirty version I’ll give you here is this: Look at the scale of the charts. 

A drop from 100 to 95 will look huge if your chart only shows 90-100 but will look tiny if your chart shows 0-100. Are you looking at a chart that spans years or merely weeks or days? Is the timeline cherry picked to be flattering? You can play around with different stock charts in Google to see how it works. 

Pick a view of one day and say, “Look it’s rising.”

Pick 6 months and say, “Wow it’s really down.”

Pick a year and say, “Oh, actually it’s just back to about the same as it was a year ago.” 

The numbers are the same but check the scale of the chart on the left hand side and see how differently they appear. Don’t be fooled by these visual tricks.

4. Don’t just believe what anybody tells you, even me.

Try to keep both a skeptical eye and an open mind. Just because what somebody is saying is unpopular or is something that you don’t want to hear doesn’t mean it’s not true. The reverse, of course, is that just because you personally find an idol group or style of music or dancing distasteful doesn’t mean that the people who like it are lying or bots or whatever. And remember that what is common knowledge on stan twitter or in your fandom circles is not necessarily true or even well known outside those circles. 

I’ve told this story many times before on my podcast but I keep coming back to it because I think it demonstrates how skewed our views are in online fandom circles. I will never forget riding the subway in Seoul with a Korean friend of mine and having her turn to ask me who some idol was in a birthday signboard. “Who’s this? Do you know him?” she asked. It was BTS’s Jimin, who had topped the idol popularity rankings for months and months at that point. Now, my friend and I had just spent hours over tea discussing the latest show business scandal with rigged voting and gossiping about well known celebrities (to include the idol group Winner). She is not unfamiliar with Korean entertainment or idols. That she didn’t know the idol who had the highest brand ranking of all the idols should tell you everything you need to know about what those charts are measuring. It’s not broad mainstream popularity. 

If anybody has specific questions about how to research or evaluate sources please feel free to get in touch. Media literacy is not something that is taught in school and I know it can be overwhelming to develop the skill but it will help you sort out a lot of bullshit once you do start to pick up on the tells and red flags that appear.