Instructions for completing the fact checking work

Each sentence of an article needs to be classified in 4 separate ways. It is important to remember to:

1. complete each classification for each sentence, and

2. to evaluate each sentence by itself and not in a global context.

There is one important exception to rule #2 above: If a sentence contains pronounces such as "they" or "it", you should use the context of other sentences to resolve these pronounces to their correct referent when classifying.

Type

Each claim can be classified into 8 possible types, and each claim can possibly have more than 1 type.

Events: Any claim about a factual event that happened in the past or is happening in the present (but not a prediction about the future). The important characteristic of these events is that they can in principle be verified by an external source.

Positive examples:

1. In 1987, North Korean secret agents bombed a South Korean airplane.

2. The United States promised to build two nuclear reactors in North Korea in exchange for a halt of the North's weapons development program.

3. The United States and North Korea are still technically in a state of war.

4. It is not known if Kim Jong Un eats hamburgers.

5. The summit between Kim Jong Un and Trump was cancelled.

Regulations: Any claim involving either national or international laws/procedures/treaties/agreements/procedures/contract/other formal agreement.

Positive examples:

1. The UN has passed severe sanctions against North Korea.

2. Trump instituted a "travel ban" that makes it illegal for Americans to visit North Korea.

3. North Korea did not violate the 1994 Agreed Framework.

Negative examples:

1. Distrusting North Korean negotiators is a good rule of thumb.

2. Asian cultures bow as a sign of respect. (Bowing is a custom, but it is not required by law.)

Quantity: Any claim involving a quantity or amount, whether the quantity is specified in absolute or relative terms.

Positive examples:

1. The CIA estimates that North Korea has at least 20 nuclear weapons.

2. The Yongbyon nuclear reactor can generate up to 10 kilograms of plutonium per year.

3. North Korea has more nuclear weapons than Israel.

4. There are no nuclear weapons in South Korea.

5. The number of nuclear weapons in North Korea is not known.

Negative examples:

1. In the year 2020, North Korea and the United States will have signed a peace treaty.

Prediction: Any claim that makes a prediction about the future, or considers counterfactual or conditional statements.

Positive examples:

1. Experts believe that North Korea will never give up its nuclear weapons in negotiations.

2. If the United States invades North Korea, then the North Koreans will launch nuclear weapons against South Korea.

3. The United States would have won the Korean War if China had not intervened.

4. By the year 2020, North Korea will have 100 nuclear weapons.

Personal: Any claim that is about a subjective personal experience, and therefore cannot in principle be verified by any external source. (Verifying the claim would require direct access to a person's inner thoughts and feelings.)

Positive examples:

1. I felt scared when I first learned about nuclear weapons.

2. My friend visited North Korea as a tourist and loved it. (This is not a claim from a first person perspective, but is a claim about a shared subjective experience.)

Negative examples:

1. In December of 1994, I met with Kim Il Sung. (This is a claim made from a first person perspective, but it is not about a subjective experience and is therefore in principle verifiable with outside information.)

Normative: Any claim that makes an evaluative assumption about what would be better/worse, or what someone should/should not do.

Positive examples:

1. The world would be a better place if President Trump invaded North Korea.

2. The United States should spend more effort on negotiating with North Korea.

3. I wish China were less involved in the US-North Korean negotiations.

4. The Chinese/American negotiations over North Korea have worsened relations between the US and South Korea.

5. American military bases in Korea ensure the security of the South Korean people. (The implication is that these military bases are a good thing.)

Negative examples:

1. Launching a preemptive strike against North Korean nuclear facilities would prevent North Korea from developing more advanced nuclear weapons but may escalate into a full war. (This is a neutral statement that describes the pros and cons of an action, but does not indicate that the speaker has a preference about whether the event takes place.)

Other: A claim is being made, but it does not fit into any of the other categories.

Positive examples:

1. North Korean missiles can reach the United States. (This is a claim about the capability of something.)

2. Academics believe that the military will launch an airstrike without fully considering the consequences of their actions. (This is a claim about the behaviour of a large group of people.)

None: If no claim is being made, then do not check any of the boxes.

Citations

An important goal of this project is to identify what types of sources articles about North Korea use. I have ranked

3rd person source (named):

Positive examples:

1. According to Mike Pomepeo, "tensions between North Korea and the United States are at an all time high."

2. The US Secretary of State reported that they are investigating recent military maneuvers by North Koreans.

3. Washington Post journalist Carolyn Hax writes that the US is preparing for a military intervention.

3rd person source (anonymous):

Positive examples:

1. An unnamed government official states that Trump and Kim had a good meeting.

2. North Korean defectors report that the country is near collapse.

3. Experts believe that North Korea will never give up its nuclear weapons in negotiations.

4. China wants to be involved in negotiations between the US and North Korea, according to a source close to Xi Jinping.

1st person source:

Positive examples:

1. On my trip to North Korea, I observed that their economy is doing well.

2. In my discussions with North Korean diplomats, I have learned that they are very meticulous workers. (This is not referring to a particular discussion with a North Korean diplomat, but all discussions broadly.)

Negative examples:

1. In my discussions with an anonymous North Korean diplomat, I learned that North Korea hopes to make a deal with the United States. (This should be a 3rd person anonymous source because it is referring to a particular person and event)

News source:

Positive examples:

1. Bloomberg News reports that North Korea has executed a spy.

2. North Korean state media claim President Trump is evil.

Negative examples:

1. Washington Post journalist Carolyn Hax writes that the US is preparing for a military intervention. (This does reference a news source, but it also references a person by name, and that takes priority.)

Other source: Any reference that does not fall into one of the above categories. For example, references to organizations.

Positive examples:

1. According to the CIA, North Korea tortures and executes Christians.

Negative examples:

1. Christian groups claim that Christians are persecuted in North Korea. (This is not a citation because it does not reference specific Christian groups. This should fall into the no citation category.)

No citation:

Any time a claim was made that does not fit into one of the other categories.

Claim Stance

At the top of the excel file is a reference claim that all of the articles are somehow related to. Each sentence of the article should be compared to the reference frame to see if it supports the claim, refutes the claim, discusses the claim, or does not discuss the claim.

A statement "supports"/"refutes" the reference claim if it provides evidence that "many people" would consider to make the claim stronger/weaker, and does not need to directly address the claim. For example, if the reference claim is: "The United States should attack North Korea" then the statement "airstrikes at key nuclear facilities would halt the North Korean weapons development program" would support the reference claim; and the statement "a war on the Korean peninsula would result in millions of civilian casualties" would refute the claim. You specifically should not consider whether the argument is good or bad, only whether an argument could be made.

Any statement that discusses the reference claim but does not clearly provide evidence supporting or refuting the claim should be labeled discuss. In particular, a claim should be labeled as discuss if it provides no evidence (e.g. "President Trump has ordered the Secretary of Defense to create plans for an invasion of North Korea") or provides contradictory evidence (e.g. "The CIA has recommended a military strike against North Korea, but the Navy has recommended against military action").

Bias

A statement is "biased" if tries to provoke an emotional responses in the reader by including non-factual information. Facts that provoke an emotional response (or selective reporting of facts) do not count as bias.

pro-North Korea:

Positive examples:

1. North Korea is a socialist paradise.

Negative examples:

1. There is no homelessness in North Korea.

anti-North Korea:

Positive examples:

1. Kim Jong Un is a dictator.

2. North Korea is a terrorist country.

Negative examples:

1. All media in North Korea is controlled by the state.

2. When North Korean secret agents destroyed a South Korean airplane in 1987, the event was widely condemned as a terrorist incident.