Discord
@iarpa/ace-2011-2015
Dataset
3
0
Public

Good Judgment Project ACE Data 2011-2015

This listing contains forecasting data collected by @IARPA (part of @ODNI) during the Aggregative Contingent Estimation—ACE—program.

This listing contains forecasting data collected by @IARPA (part of @ODNI) during the Aggregative Contingent Estimation—ACE—program.

This data set includes millions of forecasts made by participants over the four years of the ACE forecasting competition and led to many noteworthy findings regarding the accuracy of human judgment about geopolitical events, including the finding that averaging the most historically accurate forecasters outperforms individual forecasters.

I. Summary

This file contains documentation for understanding the prediction market data used by the Good Judgment Project.

II. Background of Prediction Markets

Year 2: The prediction market in year 2 was a continuous double auction (CDA) market run by Lumenogic. The first group (“lum1”) did not receive training, and the second group (“lum2”) did receive training. All participants worked independently.

Year 3: There were several prediction market conditions in year 3. The first market (“lum1”) consisted entirely of US citizens, which was not the case with the other Lumenogic markets (“lum2a” and “lum2”). Participants in the “lum1” and “lum2a” markets worked independently. The “lum2a” market operated from the beginning of the third year until 2014-01-06. At that point, participants were placed in teams. This “lum2” market of teams ran from 2014-01-07 until the end of the third year. These first three markets were CDA markets run by Lumenogic. The fourth prediction market was run by Inkling. This was a logarithmic market scoring rule (LMSR) market with participants working independently.

Year 4: All five prediction markets used in year 4 were LMSR markets run by Inkling. The control market was similar to the Inkling market in year 3. Participants did not receive training and worked independently of one another. The other four prediction markets used batch auctions on the first day of an IFP to determine the price. The “batch.notrain” market was similar to the control market (e.g., no training) except for this batch auction at the beginning of IFPs. Top participants from year 3 who opted for prediction markets were placed in the “supers” market. The similar condition for non-supers participants was the “batch.train” market. Participants in the “teams” market were placed on teams.

III. Description of Folders and Files

Transactions and Batch Orders: This folder contains prediction market files for transactions and batch orders. The suffixes on the file names were described in the previous section. There are additional files unique to year 4, specifically batch auctions (e.g., pm_batch_orders.supers.yr4.csv). The year 4 Inkling control market does not have a file for batch orders.

IV. Description of Lumenogic Data

The columns in the data are fairly self-explanatory, but a few notes are in order with the transactions files.

  • Usernames have been removed from the original file.

  • Some of the operation types and column names were renamed from year 2 to year 3.

  • For non-conditional, binary questions, participants trade on only one answer option.

  • OpType, refers to the operation type. Participants could have submitted orders, cancelled orders, or created agents.

  • The type of trade is constructed by the “buy” and “long” columns:

    • Regular buy: buy = TRUE, long = TRUE
    • Regular sell: buy = FALSE, long = TRUE
    • Short sell: buy = TRUE, long = FALSE
    • Buy back: buy = FALSE, long = FALSE
  • Operations associated with agents include the following:

    • AgentCreated/agentEnabled: generated when the user goes from manual trading to agent trading. The 'price' column contains the initial agent forecast.
    • AgentFcst: generated when the user modifies an existing agent's forecast, it is also used to restart an agent whose fuse has blown. The 'price' column contains the modified agent forecast.
    • AgentDestroyed / agentDisabled: generated when the user switches back to manual trading, or when the user has retrieved (in the form of cash) all the cash and holdings from the agent, or when the market closes and the agent is automatically terminated.
    • AgentCashXfer, AgentHldgXfer: When an agent is created, these variables record cash transfers and share/holdings transfers, respectively from the 'manual' account to the agent account.
    • AgentFuseBlown: generated when the agent's fuse is blown because the market trading price crosses the agent's forecast.
      • AgentConfig: Agent configurations, which appear in year 3. This was specified in the the following columns: Low.Fuse, Max.Bid, Min.Ask, High.Fuse, Min.Qty, Divest.Only.
  • The with.MM column refers to trades conducted with the market-maker. This was typically used in the beginning of an IFP to get the market started.

  • The experience column is a user’s self-reported rating on a scale of 1 to 5 of their level of expertise on the IFP.

  • Note that in year 2, the matching.Order.ID column repeats values for trades. This format was admittedly not intuitive and was changed in year 3.

  • When submitting orders, participants in year 3 were asked about their beliefs on the true probability of the event, which is the “Tru.Belief” column.

V. Description of Inkling Data

Most of the columns in the data are fairly straightforward, but a few things should be mentioned.

  • Usernames and comments have been removed from the original file.
  • Some of the column names changed from year 3 to year 4.
  • For non-conditional, binary questions, participants trade on only one answer option.
  • Recall that in year 4, four of the five prediction markets initialized IFPs with a batch auction.
  • Negative values in the “spent” column correspond to sells. In other words, the participant had gained money from the trade.
  • When submitting orders, participants were also asked about their beliefs on the true probability of the event, which is the “probability estimate” column.

The ACE program was run by team Good Judgment is now available for use by the public and the research community.

Originally released on the Harvard Dataverse on April 28, 2017.

Using this data

Individual Differences Data (all-individual-differences.csv)

This file contains information about the individual differences data (all-individual-differences.csv), including how the variables are named. Generally, columns are named like this:

  • scale _ item _ modifier _ year

Some basic naming conventions used include:

  • score - total score for scale or test
  • cor - 1 if correct, 0 if not
  • rev - indicates column has been reversed
  • yr1 - data from year 1 of GJP
  • yr12 - data from years 1 and/or 2

For yes/no questions, 1 = yes and 0 = no.

For brevity, some columns are listed below without year tags. You can assume that coding is the same for all year tags unless otherwise specified.

Identifiers

  • user_id__ - Unique User ID
  • userid_yr4__ - Unique User ID for year 4. Some users were reassigned user ids in year 4.

Demographics

  • gender - gender. 0 = Male, 1 = Female
  • age - age from years 1-4, binned
  • ed_yr12 - education from years 1-2.
    • 0 = No Bachelor's
    • 1 = Bachelor's
    • 2 = Master's
    • 3 = Doctorate
  • ed_yr3_ - "What is your highest level of education?" Coding same for yr4
    • 1 = Not a high school graduate
    • 2 = High school graduate or GED recipient
    • 3 = Associate degree (for example: AA, AS)
    • 4 = Bachelor's degree (for example: BA, AB, BS)
    • 5 = Master's degree
    • 6 = Professional degree or professional doctorate (for example, MD, DVM, JD, PsyD)
    • 7 = Other Doctorate degree (for example: PhD or EdD)
  • citizen_ - 1 = US citizen, 0 = non-citizen. Coding same for yr4
  • workhours - "How many hours do you work at your job in a typical week?".
    • 1 = 0-10 hours
    • 2 = 11-20 hours
    • 3 = 21-30 hours
    • 4 = 31-40 hours
    • 5 = 41-50 hours
    • 6 = 51-60 hours
    • 7 = more than 60 hours
  • tradeexp - "What is your level of experience with market trading?".
    • 1 = No experience
    • 2 = Novice
    • 3 = Intermediate
    • 4 = Advanced
    • 5 = Expert
  • tradeenjoy - "If you have had experience, do you like buying and selling shares in stock or prediction markets?".
    • 0 = cannot say
    • 1 = Dislike Extremely
    • 2 = Dislike
    • 3 = Neither dislike nor like
    • 4 = Like
    • 5 = Like Extremely
  • newshrs - "How much time per week do you spend reading about international news and politics? Please estimate the number of hours (you can use decimals if you like)."

Raven's Advanced Progressive Matrices - Short Form (APM-SF)

  • raven_score - Number of correct items (max. 12)
  • raven_adjusted_score - In a few cases, some matrix images did not load. This adjusted score reflects the percentage correct on all visible items.
  • raven_1 - Response to Raven's item 1
  • raven_1_cor - 1 if item 1 response was correct, 0 if incorrect
  • raven_1_time - time (in seconds) of page submit for Raven's item 1 in year 3 survey

Extended Cognitive Reflection Test (by Jon Baron).

  • crt_score_yr12 - Score on all CRT items (out of 7) for years 1-2.
  • crt_score_yr3 - Score on all CRT items (out of 18) for year 3.
  • crt_score_yr4 - Score on all CRT items (out of 18) for year 4.
  • crt_orig_score - Score on 3 original (Frederick, 2005) CRT items.
  • crt_imp_score - Number of intuitive/impulse responses on 3 original CRT items.
  • crt_seenbefore - "Have you seen the three questions immediately above before?" (about original Frederick items) 1 = yes, 0 = no.
  • crt_1_cor - 1 if response to CRT item 1 is correct, otherwise 0
  • crt_1_time - time (seconds) of page submit for item 1

ECRT Items for Years 1 and 2

  • crt_1_yr12__ - CRT Q1 response. "A bat and a ball cost $1.10 The bat costs $1 dollar more than the ball. How much does the ball cost?" (5 or .05)

  • crt_2_yr12__ - "If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?" (5)

  • crt_3_yr12__ - "In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?" (47)

  • crt_4_yr12__ - "All flowers have petals. Roses have petals. If these two statements are true can we conclude that roses are flowers?" yes=1, no=2. (no)

  • crt_5_yr12__ - "Sally is making sun tea. Every hour, the concentration of the tea doubles. If it takes 6 hours for the tea to be ready, how long would it take for the tea to reach half of the final concentration? (Enter the number of hours as a numeral)" (5)

  • crt_6_yr12__ - "In a box, no green things are round, and all round things are large. What can we conclude?" 1="No green things are large." 2="Some green things are not large." 3="We cannot conclude anything about green and large things." (3)

  • crt_7_yr12__ - "All things that have a motor need oil. Automobiles need oil. If these two statements are true, can we conclude from them that automobiles have a motor?" 1=yes, 2=no. (no)

ECRT Items for Years 3 and 4

  • crt_1_yr3 - Response to CRT item 1, "A bat and a ball cost $1.10 in total. The bat costs a dollar more than the ball. How many cents does the ball cost? (Do not use any decimals or any symbols or letters.)" (5)
  • crt_2_yr3 - "If it takes 5 machines 5 minutes to make 5 widgets, how many minutes would it take 100 machines to make 100 widgets? (Enter the number of minutes)" (5)
  • crt_3_yr3 - "In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how many days would it take for the patch to cover half of the lake? (Enter the number of days)" (47)
  • crt_4_yr3 - "All cats are furry. Rabbits are furry. If these two statements are true, can we conclude from them that rabbits are cats?" (no / 2)
  • crt_5_yr3 - "All mammals walk. Whales are mammals. If these two statements are true, can we conclude from them that whales walk?" (yes / 1)
  • crt_6_yr3 - "All fish are swimmers. Some Olympic athletes are swimmers. If these two statements are true, can we conclude from them that some Olympic athletes are fish?" (no / 2)
  • crt_7_yr3 - "All flowers have petals. Roses have petals. If these two statements are true, can we conclude from them that roses are flowers?" (no / 2)
  • crt_8_yr3 - "All things that have a motor need oil. Automobiles need oil. If these two statements are true, can we conclude from them that automobiles have a motor?" (no / 2)
  • crt_9_yr3 - "All living things need water. Roses need water. If these two statements are true, can we conclude from them that roses are living things?" (no / 2)
  • crt_10_yr3 - "All vehicles have wheels. Boats are vehicles. If these two statements are true, can we conclude from them that boats have wheels?" (yes / 1)
  • crt_11_yr3 - "All things that are smoked are good for the health. Cigarettes are smoked. If these two statements are true, can we conclude from them that cigarettes are good for the health?" (yes / 1)
  • crt_12_yr3 - "All bears are ferocious. Some stuffed animals are bears. If these two statements are true, can we conclude from them that some stuffed animals are ferocious?" (yes / 1)
  • crt_13_yr3 - "All wives are married. Some women are married. If these two statements are true, can we conclude from them that some women are wives?" (no / 2)
  • crt_14_yr3 - "If animals need vitamin Q, can we conclude that oysters need vitamin Q?" (yes / 1)
  • crt_15_yr3 - "If oxygen in the air is poisonous to animals, can we conclude that oxygen in the air is poisonous to dogs?" (yes / 1)
  • crt_16_yr3 - "If it takes 2 nurses 2 minutes to measure the blood of 2 patients, how many minutes would it take 200 nurses to measure the blood of 200 patients? (Enter the number of minutes)" (2)
  • crt_17_yr3 - "Soup and salad cost $5.50 in total. The soup costs a dollar more than the salad. How much does the salad cost (in dollars, without a dollar sign)?" (2.25)
  • crt_18_yr3 - "Sally is making sun tea. Every hour, the concentration of the tea doubles. If it takes 6 hours for the tea to be ready, how long would it take for the tea to reach half of the final concentration? (Enter the number of hours)" (5)

Numeracy

  • numeracy_score_yr12__ - sum of correct answers to Numeracy questions (0-3)
  • numeracy_1_yr12__ - Numeracy Q1 response. "The chance of getting a viral infection is .0005. Out of 10,000 people, about how many of them are expected to get infected? (5)
  • numeracy_2_yr12__ - Numeracy Q2 response. "Imagine you are in a class, and your chances of being asked a question are 1% during the first week and double each week thereafter (i.e. a 2% chance in Week 2, a 4% chance in Week 3, an 8% chance in Week 4). What are the chances you will be asked a question in Week 7? (Enter a number between 0 and 100 to indicate the percentage) (64)
  • numeracy_3_yr12__ - Numeracy Q3 response. "Imagine an area where 1 out of every 10,000 doctors is infected with a virus. Individuals differ in the risk of infections. In a group of high risk individuals, 20 out of every 100 people are also infected. A test for the virus gives a positive result in 99% of those who are infected and in 1% of those who are not infected. A randomly selected doctor and a randomly selected individual from the high risk group test positive for the disease. Who is more likely to actually have the disease?" (High Risk Individuals)
  • numeracy_1_cor_yr12__ - 1 if response to Numeracy item 1 is correct, otherwise 0

Berlin Numeracy (yr3 & yr4)

This test is administered and scored in an adaptive format. Each item is intended to be a median split of those viewing it. If one correctly answers item 1, they view item 2b (else 2a). If one correctly answers 2b, their score is 4/4 (else they get item 3). If one correctly answers item 3, their score is 4/4 (else 3/4). If one correctly answers item 2a, their score is 2/4 (else 1/4).

  • numeracy_score_yr3 - score (out of 4) on Berlin Numeracy in year 3.
  • numeracy_1_yr3 - Response to Berlin Numeracy item 1, "Out of 1,000 people in a small town 500 are members of a choir. Out of these 500 members in the choir 100 are men. Out of the 500 inhabitants that are not in the choir 300 are men. What is the probability that a randomly drawn man is a member of the choir? Please indicate the probability as a decimal between 0 and 1 (0=0%, .5=50%, 1=100%)" (0.25)
  • numeracy_2a_yr3 - "Imagine we are throwing a five-sided die 50 times. On average, out of these 50 throws how many times would this five-sided die show an odd number (1, 3, or 5)?" (30)
  • numeracy_2b_yr3 - "Imagine we are throwing a loaded die (6 sides). The probability that the die shows a 6 is twice as high as the probability of each of the other numbers. On average, out of 70 throws how many times would the die show the number 6?" (20)
  • numeracy_3_yr3 - "In a forest 20% of mushrooms are red, 50% brown and 30% white. A red mushroom is poisonous with a probability of 20%. A mushroom that is not red is poisonous with a probability of 5%. What is the probability that a poisonous mushroom in the forest is red? Please indicate the probability as a decimal between 0 and 1." (0.5)

Shipley-2 Abstraction

This test was given to all forecasters in the year 3 survey. It comprises 25 items and asks for one character per text box to complete the sequence (of numbers, words, or letters). Responses were timed in blocks of 5 items.

  • shipa_score_yr3 - Score (out of 25) on Shipley-2 Abstraction
  • shipa_1_1_yr3 - text entered in item 1 (first number), box 1 (second number)
  • shipa_1_cor_yr3 - 1 if answer to item 1 is correct
  • shipa_time_1_yr3 - Timestamp (in seconds) of page submit for first block of 5 items (1-5).
  • shipa_time_6_yr3 - Timestamp (in seconds) of page submit for second block of 5 items (6-10). Pattern continues

Shipley-2 Vocabulary

This test was given to all forecasters in the year 3 survey. It comprises 40 multiple choice items where participants select the most similar word to the target word. Responses were timed in blocks of 5 items.

  • shipv_score_yr3 - Score (out of 40) on Shipley-2 Vocabulary.
  • shipv_1_yr3 - Response to item 1 (multiple choice, 1-4).
  • shipv_time_1_yr3 - Timestamp (in seconds) of page submit for first block of 5 items (1-5).
  • shipv_time_6_yr3 - Timestamp (in seconds) of page submit for second block of 5 items (6-10). Pattern continues

Political Knowledge - True/False

  • pk_score - Number of correct items

  • pk_numincor - Number of incorrect answers

  • pk_sumconf - Sum of confidence points (.5,.6,.7,.8,.9, 1)

  • pk_sumconfcor - Sum of confidence points for correct answers

  • pk_sumconfincor - Sum of confidence points for incorrect answers

  • pk_calib - Calibration score

  • pk_discrim - Discrimination score

  • pk_1_cor - 1 if item 1 correct, 0 if incorrect

  • pk_1_conf confidence (.5,.6,.7,.8,.9, 1) judgment for item 1

  • pk_1_time - time (in seconds) of page submit for answer to item 1

  • pk_1_conf_time- time (in seconds) of page submit for confidence judgment for item 1

  • pk_score_china_yr3 - China (general) cluster score, out of 5. Includes items 1, 2, 3, 28, 29.

  • pk_score_iran_yr3 - Iran cluster score, out of 4. Includes items 4, 5, 6, 7.

  • pk_score_mideast_yr3 - Middle East cluster score, out of 9. Includes items 15, 16, 17, 18, 19, 20, 21, 22, 55.

  • pk_score_sasia_yr3 - South Asia cluster score, out of 8. Includes items 41, 42, 43, 44, 45, 46, 47, 38.

  • pk_score_eurozone_yr3 - Eurozone cluster score, out of 2. Includes items 12, 48.

  • pk_score_russia_yr3 - Russia cluster score, out of 2. Includes items 39, 40.

  • pk_score_globgov_yr3 - Global Governance cluster score, out of 3. Includes items 24, 34, 49.

  • pk_score_latinam_yr3 - Latin America cluster score, out of 3. Includes items 36, 37, 52.

  • pk_score_nkorea_yr3 - North Korea cluster score, out of 3. Includes items 8, 9, 10.

  • pk_score_africa_yr3 - Sub-Saharan Africa cluster score, out of 4. Includes items 31, 32, 33, 35.

  • pk_score_globecon_yr3 - Global Economy cluster score, out of 9. Includes items 11, 13, 23, 25, 30, 53, 27, 4, 14.

Year 1 Items and Answers (35 items)

  1. The leadership of Saudi Arabia, like that of Iran, embraces the Shiite branch of Islam. [F.]

  2. Pakistan has a majority Hindu population. [F]

  3. The Japanese economy has registered GDP growth rates between eight and 12% for the last 15 years. [F]

  4. The International Monetary Fund has had to lend Greece many billions of dollars in the last year to prevent a debt default. [T]

  5. The United Kingdom never adopted the euro as its currency. [T]

  6. Azerbaijan and Armenia have formally settled their border dispute. [F]

  7. Turkey currently has a military-dominated government. [F]

  8. As of July, 2011, Japan has the second-largest economy (measured by GDP) in the world. [F]

  9. Israel has yet to sign a formal peace treaty with an Arab nation. [F]

  10. There have historically been deep divisions in Iraq between the Sunni and Shiite populations. [T]

  11. Vietnam and China have yet to resolve their disagreements over oil and mineral rights in the South China Sea. [T]

  12. Canada elected a majority conservative government in 2011. [T]

  13. The country providing the most natural gas to the European Union is Russia? [T]

  14. The population of Russia exceeds 300 million. [F]

  15. The Brazilian economy has registered GDP growth rates between -3% and +3% over the last 10 years. [F]

  16. The United States is a signatory to the Nuclear Non-proliferation Treaty. [T]

  17. Norway is a major oil-producing country. [T]

  18. As of July, 2011, the price of a barrel of Brent crude oil has never exceeded $300. [T]

  19. United States runs a large trade surplus vis-a-vis China. [F]

  20. There have historically been deep divisions in Nigeria between the Christian south and the Muslim north. [T]

  21. Myanmar has a military dominated government. [T]

  22. The International Monetary Fund has had to lend France many billions of dollars in the last year to prevent a debt default. [F]

  23. Iceland is not a signatory to the treaty known as NAFTA (North American Free Trade Agreement). [T]

  24. Zimbabwe is currently the last remaining African country to be under the control of a white-minority government. [F]

  25. The Middle East Quartet promoting the peace process includes the United States, the United Kingdom, France, and the United Nations? [F]

  26. Russia is one of the so-called BRICs. [T]

  27. NASDAQ began as the world's first electronic stock market. [T]

  28. Vietnam is a member state in the Shanghai Cooperation Organization? [F]

  29. Hamas is a Palestinian political-military movement that currently controls Gaza. [T]

  30. Nelson Mandela is the President of South Africa. [F]

  31. The International Atomic Energy Commission has concluded that Iran is not on the technological path toward developing nuclear weapons. [F]

  32. Pakistan is a nuclear power. [T]

  33. The World Trade Organization views agricultural subsidies as a major problem in moving forward with trade liberalization. [T]

  34. The United Kingdom currently has a majority Conservative Party government. [F]

  35. There has been virtually zero economic growth in India over the last 10 years. [F]

Year 2 Items and Answers (50 items)

  1. The current agreements underpinning the World Trade Organization (WTO) are the outcome of the Uruguay Round negotiations that revised the GATT. [T]

  2. Germany is a permanent member of the UN Security Council. [F]

  3. ASEAN successfully managed the Cambodia-Thailand border dispute in 2011, avoiding bloodshed. [F]

  4. Nahdlatul Ulama or NU, Indonesia's largest traditionalist Islamic organization, supports an Islamic state. [F]

  5. Switzerland is a member of the European Union. [F]

  6. Kenya moved forces into Jubaland in Southern Somalia in October 2011. [T]

  7. Jose Eduardo dos Santos recently took power in Angola promising a break from the past rule of the People's Movement for the Liberation of Angola (also known as the MPLA). [F]

  8. Japan and South Korea dispute sovereignty over the Dokdo islands. [T]

  9. Since democratization, the Indonesian economy has grown faster than it did during the Suharto dictatorship. [F]

  10. Colombia borders five countries (Ecuador, Panama, Venezuela, Brazil, and Peru), but the epicenter of its long-standing armed conflict is in the periphery bordering Ecuador and Venezuela. [T]

  11. Goodluck Jonathan's victory in the 2011 Nigeria election was a significant change because he was the first person from the North to win in ten years. [F]

  12. The European Union accounts for 75% of the Foreign Direct Investment in Russia. [T]

  13. Spain has evolved over the last 50 years from one of the most centralized to one of the most decentralized states in Europe. [T]

  14. The International Court of Justice found in 2010 that Kosovo's declaration of independence violated international law. [F]

  15. In the dispute over Kashmir, Pakistan controls Jammu and India controls Azad Kashmir. [F]

  16. The International Atomic Energy Commission has concluded that Iran is not on the technological path toward developing nuclear weapons. [F]

  17. Pakistan is a nuclear power. [T]

  18. The World Trade Organization views agricultural subsidies as a major problem in moving forward with trade liberalization. [T]

  19. Vietnam is a member state in the Shanghai Cooperation Organization. [F]

  20. Hamas is a Palestinian political-military movement that currently controls Gaza. [T]

  21. The United Progressive Alliance in India is built around the BJP (Bharatiya Janata Party). [F]

  22. Nelson Mandela is the President of South Africa. [F]

  23. Zimbabwe is currently the last remaining African country to be under the control of a white-minority government. [F]

  24. NASDAQ began as the world's first electronic stock market. [T]

  25. Iceland is not a signatory to the treaty known as NAFTA (North American Free Trade Agreement). [T]

  26. The International Monetary Fund has had to lend Finland many billions of dollars in the last year to prevent a debt default. [F]

  27. Laos is a communist single-party socialist republic. [T]

  28. The United States is a signatory to the Nuclear Non-proliferation Treaty. [T]

  29. There have historically been deep divisions in Nigeria between the Christian south and the Muslim north. [T]

  30. Russia is one of the so-called BRICs. [T]

  31. Norway is a major oil-producing country. [T]

  32. As of August 2012, the price of a barrel of Brent crude oil has never exceeded $200. [T]

  33. The country providing the most natural gas to the European Union is Russia. [T]

  34. The population of Russia exceeds 300 million. [F]

  35. The Brazilian economy has registered GDP growth rates between -3% and +3% over the last 10 years. [F]

  36. The Three Gorges Dam, a massive construction project begun in 1994, is located on the Yangzi River. [T]

  37. Vietnam and China have yet to resolve their disagreements over oil and mineral rights in the South China Sea. [T]

  38. There have historically been deep divisions in Iraq between the Sunni and Shiite populations. [T]

  39. The dominant ethnicity/race in China is the Han. [T]

  40. Canada elected a majority conservative government in 2011. [T]

  41. Turkey currently has a military-dominated government. [F]

  42. The United Kingdom never adopted the Euro as its currency. [T]

  43. Azerbaijan and Armenia have formally settled their border dispute. [F]

  44. As of August 2012, Japan has the second-largest economy (measured by GDP) in the world. [F]

  45. The leadership of Saudi Arabia, like that of Iran, embraces the Shiite branch of Islam. [F]

  46. Pakistan has a majority Hindu population. [F]

  47. Israel has yet to sign a formal peace treaty with an Arab nation. [F]

  48. India's liberalization reforms now allow for 100% Foreign Direct Investment (FDI) stake in ventures. [T]

  49. The GINI coefficient measures the rate of economic expansion. [F]

  50. Anatoly Sobchak was an early adversary of Vladimir Putin. [F]

Year 3 Items and Answers (55 items)

  1. Japan, China and South Korea dispute sovereignty over the Senkaku Islands. (False)

  2. Xi Jinping is the Premier of the People's Republic of China. (False)

  3. There are reportedly about 311 million members of the Chinese Communist Party. (False)

  4. Iran has ended its territorial dispute with the United Arab Emirates concerning three islands in the Persian Gulf. (False)

  5. Mahmoud Ahmadinejad has been the Chair of the Non-Aligned Movement since 2012. (True)

  6. The nuclear power plant at Bushehr, Iran that opened in September 2011 was built with mostly North Korean assistance. (False)

  7. Supreme Leader in Iran, the Vilayat-e Faqih, is selected by the Assembly of Experts. (True)

  8. North Korea has successfully launched a satellite into orbit. (True)

  9. North Korea is a party to the Nuclear Non-Proliferation Treaty. (False)

  10. Kaesong was established to facilitate better ties between North Korea and South Korea through economic relations. (True)

  11. Since 2008, the International Monetary Fund has loaned funds to Angola, Tunisia, and Albania to combat the effects of the global economic crisis. (True)

  12. The United Kingdom is the only European Union member state that has not adopted the Euro as its currency. (False)

  13. No country currently uses the gold standard as the basis of its monetary system. (True)

  14. In January 2013, Japan announced measures that were expected to devalue its currency, creating concern of a possible currency war. (True)

  15. Egypt was the first country to experience regime change during the Arab Spring. (False)

  16. Naftali Bennet has agreed Israel should withdraw from most of the West Bank and recognize a Palestinian State as part of the peace process. (False)

  17. Salman bin Abdulaziz is the Crown Prince in Saudi Arabia. (True)

  18. 30 women now sit on the Shura Council, which advises Saudi Arabia's king. (True)

  19. Nouri al-Maliki is the secretary general of the Islamic Dawa Party. (True)

  20. Muqtada al-Sadr is well known for advocating the division of Iraq into sectarian mini-states. (False)

  21. Yair Lapid advocates that Orthodox Jews in Israel should receive more favorable social support and continued exemptions from the military. (False)

  22. Thousands of people converged in the streets of Tunisia for the funeral of assassinated opposition politician Chokri Belaid. (True)

  23. Trade between India and the member states of ASEAN (the Association of Southeast Asian Nations) grew at over 22 percent annually during the 2005-2011 period. (True)

  24. Japan has agreed to drop tariffs on key agricultural products in exchange for being admitted to the Trans Pacific Partnership talks. (False)

  25. Singapore receives the best possible credit rating (AAA) from Standard & Poor's. (True)

  26. In 2012, China imported more oil than did the United States. (True)

  27. Both China and Mexico are among the top 10 oil-producing countries in the world. (True)

  28. China has the world's largest economy as determined by GDP. (False)

  29. China's nominal GDP growth rate was above 15% in 2011. (False)

  30. Brazil recorded the lowest GDP growth rate in South America during 2011. (True)

  31. According to The World Bank, in 2010 less than 50% of people in Nigeria lived on less than $1.25 per day. (False)

  32. President Francois Boizize was ousted out of office of the President of Central African Republican in March 2013 by the Seleka rebel alliance. (True)

  33. The Tutsi in Rwanda have traditionally been the poor underclass in the country. (False)

  34. The UN Security Council voted unanimously in March 2013 to end part of a 21-year-old arms embargo on Somalia. (True)

  35. Ghana was the first country in sub-Saharan Africa to gain independence from the United Kingdom's colonial rule. (True)

  36. Cuba helped to organize negotiations between the government of Colombia and FARC (Spanish acronym), the Revolutionary Armed Forces of Colombia. (True)

  37. The German government has explicitly supported political changes initiated by Bolivia's President Evo Morales. (True)

  38. The Istanbul Process refers to region-led dialogues and meetings started in late 2011 that focus on encouraging security, political, and economic cooperation among

Turkey and its neighbors. (False)

  1. The European Union accounts for 75% of the Foreign Direct Investment in Russia. (True)

  2. Anatoly Sobchak was an early adversary of Vladimir Putin. (False)

  3. Pakistan has the world's largest population of Muslims. (False)

  4. The Afghanistan-Pakistan Transit Trade Agreement allows India to export goods to Afghanistan but does not allow Afghanistan to export goods to India. (False)

  5. India's liberalization reforms now allow for 100% Foreign Direct Investment stake in ventures. (True)

  6. The United Progressive Alliance in India is built around the BJP (Bharatiya Janata Party). (False)

  7. The Pakistan People's Party has widespread support in Sindh. (True)

  8. Syed Abul Ala Maududi is widely seen as the founder of the Jamaat-E-Islami. (True)

  9. In the dispute over Kashmir, Pakistan controls Jammu and India controls Azad Kashmir. (False)

  10. In 2010, the Italian political parties Futuro e Liberta (Future and Freedom) and Il Popolo Della Liberta (The People of Freedom) merged to form a united conservative party. (False)

  11. Germany is a signatory to the Treaty on the Non-Proliferation of Nuclear Weapons. (False)

  12. The Moro Islamic Liberation Front and the government of the Philippines signed a peace deal in 2012. (True)

  13. The Liberal Democratic Party has controlled the Japanese government for the majority of the years since World War II. (True)

  14. Jose Ramon Machado Ventura is the head of the Cuban Communist Party. (False)

  15. If Chinese economic growth were to slow dramatically, the Australian dollar would most likely increase in value. (False)

  16. The Conservative Party currently controls the government of Canada. (True0

  17. Iraq has a majority Shia population. (True)

Political Knowledge - Multiple Choice

This test comprises 15 multiple choice questions asked in years 3 and 4.

  • pkmc_score - Score (out of 15) on year 3 multiple choice political knowledge test
  • pkmc_1 - Response (1-4) to multiple choice item 1
  • pkmc_1_time - time (in seconds) of answer to item 1

Items and Answers

  1. Was the North Atlantic Treaty Organization primarily formed as

    1. A military security alliance (correct)
    2. A tariff-free trade zone
    3. An economic development union
    4. A Maritime & Fishing Agreement
  2. Who is the Secretary General of the United Nations?

    1. Javier Solana
    2. George Mitchell
    3. Ban Ki-moon (correct)
    4. Dominique de Villepin
  3. What country did Kosovo seek independence from?

    1. Russia
    2. Serbia (correct)
    3. Turkey
    4. Greece
  4. The name of the organization founded by the Rome Statute that is often referred to as the ICC is

    1. The International Computing Convention
    2. The Italian Commerce Corporation
    3. The International Criminal Court (correct)
    4. The International Commercial Court
  5. In which of the following countries does the majority of the population follow Shi?۪a Islam?

    1. Saudi Arabia
    2. Egypt
    3. Pakistan
    4. Iran (correct)
  6. Which of the following countries is NOT a permanent member of the UN Security Council?

    1. United Kingdom
    2. Germany (correct)
    3. China
    4. Russian Federation
  7. Who is the Chief Justice of the Supreme Court?

    1. Antonin Scalia
    2. John Roberts Jr. (correct)
    3. Ruth Bader Ginsberg
    4. Stephen Breyer
  8. To overturn a presidential veto what percentage of the Congress needs to vote to do so?

    1. 51% of the Senate
    2. 60% of the House
    3. 66% of the Senate and House (correct)
    4. 75% of the Senate and House
  9. Who is the Speaker of the House of Representatives?

    1. John Boehner (correct)
    2. Eric Cantor
    3. Mitch McConnell
    4. Harry Reid
  10. What sort of vote is needed to ratify a treaty that is signed by the president of the United States?

    1. 66% of the House
    2. 66% of the Senate (correct)
    3. 66% of the House and Senate
    4. 75% of the House and Senate
  11. How many Federal Reserve Districts are there in the Federal Reserve system?

    1. 6
    2. 12 (correct)
    3. 20
    4. 50
  12. Which of the following positions does not require Senate confirmation?

    1. The Secretary of Commerce
    2. The Chairman of the Federal Reserve Board
    3. The National Security Advisor (correct)
    4. The US Permanent Representative to the United Nations
  13. Which one of the following North African nations has not experienced a revolutionary overthrow of its government in the last five years?

    1. Egypt
    2. Tunisia
    3. Libya
    4. Algeria (correct)
  14. The current president of France (Hollande) is linked to which political party?

    1. The Socialists (correct)
    2. The Gaullists
    3. The National Front
    4. United Europe
  15. OPEC stands for:

    1. Organization for Pacific Economic Cooperation
    2. Organization of Petroleum Exporting Countries (correct)
    3. Organization for Palestinian Energy Conservation
    4. Organization for Preventing Ecological Collapse

Cluster Political Knowledge

This multiple choice test is intended to measure political knowledge in 4 clusters: China, global economy, Iran, and Russia.

  • cpk_score: total score out of 32
  • cpk_score_china: total score on China questions, out of 8
  • cpk_score_globecon: total score on Global economy questions, out of 8
  • cpk_score_iran: total score on Iran questions, out of 8
  • cpk_score_russia: total score on Russia questions, out of 8
  • cpk_1: choice for item 1. 1:a, 2:b, 3:c, 4:d
  • cpk_correct_1: 1 if item 1 correct
  • cpktime_1: timestamp (in seconds) of page submit on item 1
  • cpkconf_1: "What is the probability that your correct?", item 1. Ranges from 25 to 100

Items and Answers

China

  1. Who is the President and General Secretary of the Communist Party of China? 1. Hu Jintao 2. Li Keqiang 3. Xi Jinping (correct) 4. Li Yuanchao
  2. China has been increasing the size of its navy. How many aircraft carriers does it now have available? 1. One (correct) 2. Five 3. Ten 4. Fifteen
  3. China's GDP, according to World Bank data through 2012, is: 1. The largest in the world 2. The second largest in the world (correct) 3. The third largest in the world 4. The fourth largest in the world
  4. In 2013, there were political tensions between China and Japan. In that year, how did total trade in goods and services between Japan and China fare? 1. Trade declined slightly (< 1%) 2. Trade declined noticeably (> 4%)(correct) 3. Trade increased slightly (< 1%) 4. Trade increased noticeably (> 4%)
  5. From 1921-1940 Taiwan was under the control of: 1. China 2. Japan (correct) 3. Great Britain 4. France
  6. Which of the following countries is not a member of the Shanghai Cooperation Organization? 1. China 2. Philippines (correct) 3. Russia 4. Uzbekistan
  7. In the past few years, Vietnamese nationalism has lead Hanoi to: 1. Cooperate with China to jointly control islands claimed by Japan 2. Stay out of the Association of Southeast Asian Nations (ASEAN) 3. Confront China over islands claimed by both China and Vietnam (correct) 4. Refuse to normalize diplomatic relations with the United States
  8. Xi Jinping has recently announced that China?s policy in response to provocation can best be thought of as: 1. Accommodative regionalism 2. Reactive assertiveness (correct) 3. Peaceful harmonization 4. Communist solidarity

Global Economy

  1. Who is the president of the World Bank? 1. Jim Yong Kim (correct) 2. Christine Lagarde 3. Janet Yellen 4. Ban Ki Moon
  2. OECD stands for
    1. Organization for European Common Defense,
    2. Organization for Ecological Conservation and Development,
    3. Organization for Economic Cooperation and Development (correct)
    4. Organization for Energy Conservation and Distribution
  3. The rate of inflation in the Euro area during 2012-2014 was:
    1. Less than 2% (correct) 2. More than 2% but less than 4% 3. More than 4% but less than 6% 4. More than 6% [NOTE: This should have asked about 2012-13 or 2013-14, or specified that we meant an average rate of inflation over both years. The average rate was less than 2%, so (a) is technically correct. However, the rate was above 2% for most of 2012 and below 2% for most of 2013.]
  4. Where does the International Monetary Fund get its money?
    1. From the commercial banks that are members
    2. From taxes levied on international trade
    3. From the quotas assigned to its member countries (correct)
    4. From the United Nations Development Program
  5. The average Brent Crude oil price from 2012-2014 has been:
    1. Less than $75 per barrel
    2. More than $75 but less than $125 per barrel
    3. More than $125 but less than $175 per barrel (correct)
    4. More than $175 per barrel
  6. Which of the following is NOT a principle of the World Trade Organization?
    1. Countries should enjoy most favored nation (MFN) status
    2. Imported and locally produced goods should be treated equally once in the market,
    3. Freer trade should be achieved through negotiation
    4. Customs duties and tariffs cannot be charged on imported goods (correct)
  7. From 2009-2013, which of the following countries had the smallest growth in GDP?
    1. Brazil
    2. China
    3. India
    4. Russia (correct)
  8. What does FATCA stand for?
    1. Federally Administered Tax Compliance Act
    2. Foreign Agency for Treasuries & Commodities Administration
    3. Federal Agency for Trade & Commodities Administration
    4. Foreign Account Tax Compliance Act (correct)

Iran

  1. Who is the supreme leader of Iran?
    1. Ayatollah Hashemi Rafsanjani
    2. Ayatollah Ali Khamenei (correct)
    3. Ayatollah Abu al-Qasim al-Khoei
    4. Ayatollah Mahmoud Taleghani
  2. Iranian President Hassan Rouhani spent the twenty years before becoming president in which sector of Iranian life?
    1. Banking and commerce
    2. Oil and petro-chemicals
    3. Religious and clerical
    4. Law and diplomacy (correct)
  3. The Basij in Iran is generally seen as:
    1. Supportive of the regime (correct)
    2. Opposing the regime,
    3. A remnant of the Shah's wealthy class,
    4. An upper-middle class movement on university campuses
  4. Which of the following countries has the largest population?
    1. Saudi Arabia
    2. Syria
    3. Iran (correct)
    4. Iraq
  5. Iran's Shahab-3 missile is a virtual copy of:
    1. A U.S. Aim-9 Sidewinder
    2. A French Exocet
    3. A North Korean Rodong (correct)
    4. A Russian SS-18 Satan
  6. The nuclear plant at Bushehr, Iran, that opened in September 2011 was built with the help of mostly:
    1. North Korea
    2. Pakistan
    3. China
    4. Russia (correct)
  7. Which country is not part of the P5+1 that has negotiated the interim nuclear deal with Iran?
    1. Italy (correct)
    2. China,
    3. France,
    4. Germany
  8. Iran provides military training, weapons, and financial support for what Shia militant group based in Lebanon?
    1. Hezbollah (correct)
    2. Hamas
    3. Al-Qaeda in the Arabian Peninsula
    4. Asbat an-Ansar

Russia

  1. In Russia, which is the party that is most closely associated with President Vladimir Putin?
    1. The Communist Party of Russia
    2. The United Russia Party (correct)
    3. The Russian United Democratic Party
    4. The Political Party LDPR
  2. When Kosovo declared independence from Serbia in 2008, Russia:
    1. Supported a common United Nations response
    2. Supported the Serbian government in Belgrade (correct)
    3. Supported the Kosovo government in Pristina
    4. Supported the NATO policy toward Serbia
  3. Which of the following countries has the most nuclear warheads?
    1. China
    2. India
    3. Pakistan
    4. Russia (correct)
  4. The Molotov-Ribbentrop Pact led to the division of which of the following countries?
    1. Poland
    2. Germany (correct)
    3. Korea
    4. India
  5. How much of the natural gas consumed by Germany is supplied by Russia?
    1. Less than 25%
    2. More than 25% but less than 50% (correct)
    3. More than 50% but less than 75%
    4. More than 75%
  6. The New START treaty signed in 2010 limits:
    1. Carbon emissions
    2. Immigration to European Union states
    3. Nuclear warheads and delivery vehicles (correct)
    4. The international movement of money between banks
  7. Under President Putin, Russia has:
    1. Not allowed Gazprom to cooperate with China?s CNPC,
    2. Held joint military exercises with China in the East China Sea (correct)
    3. Prevented Central Asian states from opening any oil pipelines to China
    4. Stopped military sales and shared energy exploration with Vietnam
  8. Which country is not a member of the Eurasian Economic Union?
    1. Russia,
    2. Belarus,
    3. Moldova (correct)
    4. Kazakhstan?

General Political Knowledge

This 15-item multiple choice tests includes a general set of questions about world politics asked in Year 4.

  • gpk_score: total score out of 15
  • gpk_1: choice for item 1. 1:a, 2:b, 3:c, 4:d
  • gpk_correct_1: 1 if item 1 correct
  • gpktime_1: timestamp (in seconds) of page submit on item 1
  1. Timor-Leste
  • gpkconf_1: "What is the probability that your correct?", item 1. Ranges from 25 to 100

Items and Answers

  1. The newest member state of the United Nations is: 1. Montenegro 2. Kosovo 3. South Sudan (correct)

  2. The current secretary of the United States Treasury is: 1. Timothy Geithner 2. Jacob Lew (correct) 3. Ben Bernanke 4. Henry Paulson

  3. OSCE stands for: 1. Office of the Security Council in Europe, 2. Organization for Security and Conflict Education, 3. Office for Security and Co-operation in Economics, 4. Organization for Security and Cooperation in Europe (correct)

  4. Brazil?s current president, Dilma Rousseff, is associated with which party? 1. Brazilian Social Democracy Party (PSDB) 2. Workers Party (PT) (correct) 3. Democrats Party (DEMS) 4. Brazilian Democratic Movement Party (PMDB)

  5. Which country will assume the chairmanship of the Arctic Council in 2015? 1. Canada 2. United States (correct) 3. Sweden 4. Russia

  6. What do the Geneva Conventions regulate?. 1. Global financial flows 2. Global carbon emissions 3. International humanitarian law during wartime (correct) 4. Convention on the law of the seas

  7. Which country has the world?s largest Muslim population? 1. Iraq 2. Saudi Arabia 3. India 4. Indonesia (correct)

  8. Who is the incoming president of Ukraine? 1. Petro Poroshenko (correct) 2. Viktor Yanukovich, 3. Oleksandr Turchynov, 4. Yulia Tymoshenko

  9. According to the United States constitution, who has the power to declare war? 1. President 2. Congress (correct) 3. National Security Council 4. Joint Chiefs of Staff

  10. Who is the U.S. National Security Adviser?

    1. Samantha Power
    2. Condoleezza Rice
    3. Susan Rice (correct)
    4. Harry Reid
  11. The African Union is composed of how many member states?

    1. 12
    2. 25
    3. 54 (correct)
    4. 70
  12. Who is the Prime Minister of Japan?

    1. Yoshihiko Noda
    2. Shinzo Abe (correct)
    3. Emperor Akihito
    4. Naoto Kan
  13. Which will allow the value of a currency to fluctuate according to the foreign exchange market?

    1. fixed exchange rate
    2. floating exchange rate (correct)
    3. linked exchange rate
    4. futures exchange rate
  14. Which country is currently seeking NATO membership?

    1. Turkey
    2. Georgia (correct)
    3. Afghanistan
    4. Spain
  15. Which accord determines a nations exclusive economic zones around its coastal borders?

    1. The Lisbon Treaty

    2. United Nations Convention on the Rights of Persons with Disabilities

    3. The Rome Treaty

    4. United Nations Convention on the Law of the Sea (correct)?

Motivation

Time contribution question: "If the forecasting tournament seems fun, how much time per week could you see yourself contributing 4 months from now?"

  • timecontrib_least_yr3 - "What is the least amount of time you see yourself contributing in a typical week?" (text entry, minutes)
  • timecontrib_avg_yr3 - "What is the average amount of time you see yourself contributing in a typical week?" (text entry, minutes)
  • timecontrib_most_yr3 - "What is the mostamount of time you see yourself contributing in a typical week?"

Motivation question: Why did you choose to participate in the forecasting tournament? Likert scale from 1 = Completely Disagree to 7 = Completely Agree.

  • motiv_improv_yr3 - "Because I want to improve my judgment skills"
  • motiv_interest_yr3 - "Because I am interested in world affairs"
  • motiv_compet_yr3 - "Because I like to participate in competitions"
  • motiv_top_yr3 - "Because I want to be among the top forecasters"
  • motiv_protectus_yr3 - "Because accurate forecasts can help protect the United States.
  • motiv_protectworld_yr3 - "Because accurate forecasts can help protect the world.

Fox - Hedgehog

  • foxhedge_single - "In a famous essay, Isaiah Berlin classified thinkers as hedgehogs and foxes: The hedgehog knows one big thing and tries to explain as much as possible using that theory or conceptual framework. The fox knows many small things and is content to improvise explanations on a case-by-case basis. When it comes to making predictions, would you describe yourself as more of a hedgehog or more of a fox?"
    • 1 = Very Much More Fox-Like
    • 2 = Somewhat More Fox-Like
    • 3 = Equally Fox and Hedgehog
    • 4 = Somewhat More Hedgehog-Like
    • 5 = Very Much More Hedgehog-Like
  • foxhedge_score_yr12 - average of Fox Hedgehog Scale items. Higher scores = hedgehog
  • foxhedge_1_yr12 - response for Hedgehog Fox scale, item 1. Lower scores indicate endorsement of Fox items, while higher scores indicate endorsement of Hedgehog. "I try to ground my thinking in robust and principles about how human beings behave and how society functions."
  • foxhedge_2_yr12 - "When I ground my thinking in sound principles, I find that seemingly unpredictable phenomena often become predictable consequences of deeper principles."
  • foxhedge_3_yr12 - "I often step outside my preferred framework and look at events from other perspectives."
  • foxhedge_3_rev_yr12 - Fox Hedgehog scale item 3, reversed
  • foxhedge_4_yr12 - "I see lots of exceptions and uncertainties in human affairs that make prediction extremely difficult."
  • foxhedge_4_rev_yr12 - Fox Hedgehog scale item 4, reversed
  • foxhedge_5_yr12 - "I do my best to cherry-pick good ideas wherever I can them?whether from the left, right or center."
  • foxhedge_5_rev_yr12 - Fox Hedgehog scale item 5, reversed
  • foxhedge_6_yr12 - "I see the need for a unifying framework that imposes order on an otherwise unmanageably complex world."
  • foxhedge_7_yr12 - "People don't need to know me all that well to be able to predict my opinions on new issues."
  • foxhedge_8_yr12 - "I avoid rushing to judgment in new situations."
  • foxhedge_8_rev_yr12 - Fox Hedgehog scale item 8, reversed
  • foxhedge_9_yr12 - "I can make more accurate forecasts when I pay less attention to the immediate headlines and more attention to basic principles proven by history to be true."
  • foxhedge_10_yr12 - "It is amazing how often history repeats itself."

Actively Open Minded Thinking

Items 1-7 were asked each year. Items 8 and 9 were added in year 3, and scores adjusted to reflect the average of all 9 items (or 7 for forecasters who dropped out before season 3)

  • aomt_score - Average of 9 AOMT items
  • aomt_1 - "Allowing oneself to be convinced by an opposing argument is a sign of good character."
  • aomt_2 - "People should take into consideration evidence that goes against their beliefs."
  • aomt_3 - "People should revise their beliefs in response to new information or evidence."
  • aomt_4 - "Changing your mind is a sign of weakness."
  • aomt_4_rev - item 4 reversed
  • aomt_5 - "Intuition is the best guide in making decisions."
  • aomt_5_rev - item 5 reversed
  • aomt_6 - "It is important to persevere in your beliefs even when evidence is brought to bear against them."
  • aomt_6_rev - item 6 reversed
  • aomt_7 - "One should disregard evidence that conflicts with one's established beliefs."
  • aomt_7_rev - item 7 reversed
  • aomt_8 - "People should search actively for reasons why their beliefs might be wrong." (New in year 3)
  • aomt_9 - "It is more useful to pay attention to those who disagree with us than to pay attention to those who agree." (New in year 3)

Cultural Worldview Scale (Dan Kahan)

This scale comprises 12 items rated for agreement using a 7-point Likert scale where 1 = Completely Disagree and 7 = Completely Agree. It is organized by 2 orthogonal factors (higher scored indicate named pole): Individualist (opposite pole Communitarian) and Hierarchy (Egalitarian). There are 3 reversed items per factor (6 total)

  • cultwv_ind_score - Average score on individualist items.
  • cultwv_hier_score - Average score on hierarchy items.
  • cultwv_ind_1 - Response to item 1, individualism factor, "The government interferes far too much in our everyday lives."
  • cultwv_ind_2 - "Sometimes government needs to make laws that keep people from hurting themselves." Needs reversed.
  • cultwv_ind_2_rev - Reversed response to item 2
  • cultwv_ind_3 - "It's not the government's business to try to protect people from themselves."
  • cultwv_ind_4 - "The government should stop telling people how to live their lives."
  • cultwv_ind_5 - "The government should do more to advance society?۪s goals, even if that means limiting the freedom and choices of individuals." Needs reversed.
  • cultwv_ind_5_rev - Reversed response to item 5.
  • cultwv_ind_6 - "Government should put limits on the choices individuals can make so they don?۪t get in the way of what?۪s good for society." Needs reversed.
  • cultwv_ind_6_rev - Reversed response to item 6.
  • cultwv_hier_7 - Item 7, hierarchy factor, "We have gone too far in pushing equal rights in this country."
  • cultwv_hier_8 - "Our society would be better off if the distribution of wealth was more equal." Needs reversed.
  • cultwv_hier_8_rev - Reversed response to item 8.
  • cultwv_hier_9 - "We need to dramatically reduce inequalities between the rich and the poor, whites and people of color, and men and women." Needs reversed
  • cultwv_hier_9_rev - Reversed response to item 9.
  • cultwv_hier_10 - "Discrimination against minorities is still a very serious problem in our society." Needs reversed.
  • cultwv_hier_10_rev - Reversed response to item 10.
  • cultwv_hier_11 - "It seems like blacks, women, homosexuals, and other groups don't want equal rights, they want special rights just for them."
  • cultwv_hier_12 - "Society as a whole has become too soft and feminine."

Social Value Orientation (SVO) Slider

This 6-item task asks participants to select from 9 possible distributions of hypothetical resources between oneself and another person. It was given in year 3.

  • svo_1_yr3 - Response to SVO Slider 1 (naming pattern is the same for other sliders)
  • svo_1_self_yr3 - amount allocated to self for item 1
  • svo_1_other_yr3 - amount allocated to other for item 1
  • svo_self_yr3 - mean amount allocated to self for all 6 items
  • svo_other_yr3 - mean amount allocated to other for all 6 items
  • svo_angle - SVO angle; arctangent of [ (mean allocated to other - 50) / (mean allocated to self - 50) ], then converted from radians to degrees
  • svo_category - Interpretation of SVO angle:
    • altruism = SVO > 51.15
    • prosociality = 22.45 < SVO < 51.15
    • individualism = -12.04 < SVO < 22.45
    • Competitiveness = SVO < -12.04

Grit

Items are rated for agreement on a 1-7 Likert scale where 1 = Completely Disagree and 7 = Completely Agree.

  • grit_1_yr1 - "Setbacks don't discourage me."

  • grit_2_yr1 - "I often set a goal but later choose to pursue a different one."

  • grit_2_rev_yr1 - GRIT item 2, reversed

  • grit_3_yr1 - "I finish whatever I begin."

  • grit_score_yr1 - average of GRIT items

  • grit_score_yr3 - Score on short Grit scale (average of items)

  • grit_1_yr3 - "I often set a goal but later choose to pursue a different one." Needs reversed. Note that it's the same item as grit_2_yr1

  • grit_1_rev_yr3 - item 1 reversed (same item as grit_2_rev_yr1)

  • grit_2_yr3 - "I have been obsessed with a certain idea or project for a short time but later lost interest." Needs reversed.

  • grit_2_rev_yr3 - item 2 reversed

  • grit_3_yr3 - "I have difficulty maintaining my focus on projects that take more than a few months to complete." Needs reversed.

  • grit_3_rev_yr3 - item 3 reversed

  • grit_4_yr3 - "New ideas and projects sometimes distract me from previous ones." Needs reversed.

  • grit_4_rev_yr3 - item 4 reversed

  • grit_5_yr3 - "I finish whatever I begin." Note that it's the same item as grit_3_yr1.

  • grit_6_yr3 - "Setbacks don't discourage me." Note that it's the same item as grit_1_yr1.

  • grit_7_yr3 - "I am diligent."

  • grit_8_yr3 - "I am a hard worker."

Extraversion

This scale comprises 8 items from the BFI-44, rated for agreement on a 1-7 Likert scale where 1 = Completely Disagree and 7 = Completely Agree.

  • extr_score_yr3 - score on extraversion scale (average of items)
  • extr_1_yr3 - Response to item 1, "I am someone who is talkative."
  • extr_2_yr3 - "I am someone who is reserved." Needs reversed.
  • extr_2_rev_yr3 - item 2 reversed
  • extr_3_yr3 - "I am someone who is full of energy."
  • extr_4_yr3 - "I am someone who generates a lot of enthusiasm."
  • extr_5_yr3 - "I am someone who tends to be quiet." Needs reversed.
  • extr_5_rev_yr3 - item 5 reversed
  • extr_6_yr3 - "I am someone who has an assertive personality."
  • extr_7_yr3 - "I am someone who is sometimes shy, inhibited." Needs reversed.
  • extr_7_rev_yr3 - item 7 reversed
  • extr_8_yr3 - "I am someone who is outgoing, sociable."

Concientiousness

Items are rated for agreement on a 1-7 Likert scale where 1 = Completely Disagree and 7 = Completely Agree.

  • consc_score_yr12 - average score for Conscientiousness
  • consc_1_yr12_ - response to Conscientiousness Q1. "I do a thorough job."(1-7)
  • consc_2_yr12_ - "I can be somewhat careless."
  • consc_2_rev_yr12__ - reversed response to Conscientiousness Q2
  • consc_3_yr12__ - "I am a reliable worker."
  • consc_4_yr12__ - "I tend to be disorganized."
  • consc_4_rev_yr12__ - Conscientiousness Q4 reversed
  • consc_5_yr12__ - "I tend to be lazy."
  • consc_5_rev_yr12__ - Conscientiousness Q5 reversed
  • consc_6_yr12__ - "I persevere until the task is finished."
  • consc_7_yr12 - "I do things efficiently."
  • consc_8_yr12 - "I make plans and follow through with them."
  • consc_9_yr12 - "I am easily distracted."
  • consc_9_rev_yr12 - Conscientiousness Q9 reversed

Openness to Experience

Items are rated for agreement on a 1-7 Likert scale where 1 = Completely Disagree and 7 = Completely Agree.

  • open_score_yr12 - average of Openness to Experience items
  • open_1_yr12 - Openness to Experience scale, response to item 1. "I like to reflect on and play with ideas."
  • open_2_yr12 - "I prefer work that is routine."
  • open_2_rev_yr12 - Openness to Experience, item 2 reversed
  • open_3_yr12 - "I value artistic and aesthetic experiences."
  • open_4_yr12 - "I am inventive."
  • open_5_yr12 - "I have an active imagination."
  • open_6_yr12 - "I am an ingenious, deep thinker."
  • open_7_yr12 - "I am curious about many different things."
  • open_8_yr12 - "I am original and come up with new ideas."

Need for Cognition

This scale comprises 18 items rated for agreement from 1 = Completely Disagree to 7 = Completely Agree

  • nfcog_score_yr12 - average of Need for Cognition items
  • nfcog_1_yr12 - response to item 1 of Need for Cognition. "I prefer complex to simple problems."
  • nfcog_2_yr12 - "I like the responsibility of handling a situation that requires a lot of thinking."
  • nfcog_3_yr12 - "Thinking is not my idea of fun."
  • nfcog_3_rev_yr12 - Need for Cognition Q3 reversed
  • nfcog_4_yr12 - "I would rather do something that requires little thought than something that is sure to challenge my thinking abilities."
  • nfcog_4_rev_yr12 - Need for Cognition Q4 reversed
  • nfcog_5_yr12 - "I try to anticipate and avoid situations where there is a likely chance I will have to think in depth about something."
  • nfcog_5_rev_yr12 - Need for Cognition Q5 reversed
  • nfcog_6_yr12 - "I find satisfaction in deliberating hard and or long hours."
  • nfcog_7_yr12 - "I only think as hard as I have to."
  • nfcog_7_rev_yr12 - Need for Cognition Q7 reversed
  • nfcog_8_yr12 - "I prefer to think about small daily projects rather than long term ones."
  • nfcog_8_rev_yr12 - Need for Cognition Q8 reversed
  • nfcog_9_yr12 - "I like tasks that require little thought once I've learned them."
  • nfcog_9_rev_yr12 - Need for Cognition Q9 reversed
  • nfcog_10_yr12 - "The idea of relying on thought to make my way to the top appeals to me."
  • nfcog_11_yr12 - "I really enjoy a task that involves coming up with new solutions to problems."
  • nfcog_12_yr12 - "Learning new ways to think doesn't excite me very much."
  • nfcog_12_rev_yr12 - Need for Cognition Q12 reversed
  • nfcog_13_yr12 - "I prefer my life to be filled with puzzles that I must solve."
  • nfcog_14_yr12 - "The notion of thinking abstractly is appealing to me."
  • nfcog_15_yr12 - "I would prefer a task that is intellectual, difficult, and important to one that is somewhat important but does not require much thought."
  • nfcog_16_yr12 - "I feel relief rather than satisfaction after completing a task that requires a lot of mental effort."
  • nfcog_16_rev_yr12 - Need for Cognition Q16 reversed
  • nfcog_17_yr12 - "It's enough for me that something gets the job done; I don't care how or why it works."
  • nfcog_17_rev_yr12 - Need for Cognition Q17 reversed
  • nfcog_18_yr12 - "I usually end up deliberating about issues even when they do not affect me personally."

Need for Closure

This scale comprises 11 items rated for agreement from 1 = Completely Agree to 7 = Completely Disagree.

  • nfclo_1_yr1 - Need for Closure, item 1. "Even after I have made up my mind about something, I am always eager to consider a different opinion."
  • nfclo_1_rev_yr1 - Need for Closure item 1 reversed
  • nfclo_2_yr1 - "I can usually see how both parties in a conflict can believe they are right."
  • nfclo_2_rev_yr1 - Need for Closure item 2 reversed
  • nfclo_3_yr1 - "It is annoying to listen to someone who cannot seem to make up his or her mind."
  • nfclo_4_yr1 - "I prefer interacting with people whose opinions are very different from my own."
  • nfclo_4_rev_yr1 - Need for Closure item 4 reversed
  • nfclo_5_yr1 - "An expert who does not come up with a definite answer probably does not know much."
  • nfclo_6_yr1 - "I dislike uncertainty."
  • nfclo_7_yr1 - "When faced with a problem, I usually see the best solution very rapidly."
  • nfclo_8_yr1 - "I find it easy to change my mind in response to unexpected events."
  • nfclo_8_rev_yr1 - Need for Closure item 8 reversed
  • nfclo_9_yr1 - "I am skeptical that there are absolute scientific truths."
  • nfclo_9_rev_yr1 - Need for Closure item 9 reversed
  • nfclo_10_yr1 - "I prefer tried and proven approaches."
  • nfclo_11_yr1 - "I enjoy thinking about why I believe the things I do."
  • nfclo_11_rev_yr1 - Need for Closure item 11 reversed
  • nfclo_score_yr1 - sum of need for closure items

Working Memory

This set of questions asks participants to solve a multi-step math problem and memorize a word in under ten seconds. After viewing all 6 items, participants are asked to recall the 6 words.

  • memory_both_score: total score out of 12; 1 point earned if math question and word recall are both correct for a given item
  • memory_math_score: total score out of 6 on math questions
  • memory_word_score: total score out of 6 on word recall
  • memory_1_math: response to math item 1 (1=yes, 2=no)
  • memory_math_correct_1: 1 if correctly answered math item 1
  • memory_word_correct_1: 1 if correctly recalled item 1 word
  • memory_1_word: recall of item 1 word_
  • memory_1_math: IS (5 X 3) + 4 = 17? (2 / no)
  • memory_2_math: IS (6 X 2) - 3 = 8? (2 / no)
  • memory_3_math: IS (4 X 4) - 4 = 12? (1 / yes)
  • memory_4_math: IS (3 X 7) + 6 = 27? (1 / yes)
  • memory_5_math: IS (4 X 8) - 2 = 31? (2 / no)
  • memory_6_math: IS (9 X 2) + 6 = 24? (1 / yes)
  • memory_1_word: BOOK
  • memory_2_word: HOUSE
  • memory_3_word: JACKET
  • memory_4_word: CAT
  • memory_5_word: PEN
  • memory_6_word: WATER

Self-Rated Math Ability

  • fractions_yr12_ - self-rated ability "Working with fractions" (1-7)
  • percentages_yr12_ - self-rated ability "Working with percentages" (1-7)
  • tips_yr12_ - self-rated ability "Calculating a 15% tip" (1-7)
  • discounts_yr12_ - self-rated ability "Figuring out how much a shirt will cost if it is 25% off" (1-7)

Forecasting Mindset

  • fcpractice_yr12_ - "Forecasting skills depend mostly on practice, training and careful analysis of the underlying causes of previous failures and successes." (1-7, Completely Disagree:Completely Agree)
  • fcability_yr12_ - "Forecasting skills depend mostly on innate ability that people either have or don?t have." (1-7, Completely Disagree:Completely Agree)
  • fcluck_yr12_ - "Forecasting skills depend mostly on luck (coincidences in which hunches and preconceptions just happen to correspond to reality for a stretch of time)." (1-7, Completely Disagree:Completely Agree)
  • fcmotivation_yr12_ - "I am committed to devoting a great deal of time and effort to improving my forecasting skills even if initial results are discouraging." (1-7, Completely Disagree:Completely Agree)

Political Philosophies

Items are rated for agreement on a 1-7 Likert scale where 1 = Completely Disagree and 7 = Completely Agree.

  • realism_yr12 - "World politics remains a jungle in which (to quote Thucydides) the strong do what they will and the weak accept what they must."(1-7)
  • institut_yr12 - "International institutions increasingly constrain the conduct of nation-states."
  • ecodoom_yr12 - "Economic and population growth are stretching nature to its breaking point."
  • boom_yr12 - "Just when humanity seems to be stretching resources to their limits, humans are ingenious at inventing cost-effective technological fixes that permit economic growth to continue."
  • realism_2_yr12 - "The rise of China to superpower status will inevitably entail sharp conflicts with the United States."
  • boom_2_yr12 - "The rise of radical Islam will be short lived, and pragmatic forces will prevail in contested areas."
  • antiecodoom_yr12 - "I doubt that global climate change modelers know as much about climate trends as they claim."
  • antiinstitut_yr12 - "European monetary integration should be scaled back sharply."
  • lib_1_yr12 - "On political and economic issues, I am more liberal than conservative."
  • lib_2_yr12 - "Government should routinely intervene in the economy to achieve fairer outcomes."
  • conserv_yr12 - "Free markets function well with minimal government intervention."
  • boom_score_yr12 - average of boom and boom2
  • lib_score_yr12 - average of lib.resp1 and lib.resp2
  • swing_yr12 - "I would rather be wrong in an interesting way than right in an uninteresting way."
  • libcon: "Generally speaking, I think of myself as 1: Staunch Liberal to 7: Staunch Conservative"?

Maximizing vs. Satisficing

  • max_satis_1_yr2 - Maximization scale, satisficing item 1. "All decisions have pros and cons."
  • max_search_1_yr2 - Maximization scale, alternative search item 1. "I cannot come to a decision unless I have carefully considered all my options."
  • max_diff_1_yr2 - Maximization scale, decision difficulty item 1. "I do not agonize over decisions."
  • max_search_2_yr2 - "I take the time to consider all alternatives before making a decision."
  • max_diff_2_yr2 - "I find myself often faced with difficult decisions."
  • max_satis_2_yr2 - "I cannot possibly know everything before making a decision."
  • max_search_3_yr2 - "I just won't make a decision until I am comfortable with the process."
  • max_diff_3_yr2 - "I often change my mind several times before making a decision."
  • max_satis_score_yr2 - average of satisficing items
  • max_diff_score_yr2 - average of decision difficulty items
  • max_search_score_yr2 - average of alternative search items

Analytical / Intuitive (Nygren and White)

This scale comprises 10 items organized by 2 factors (analytical and intuitive thinking) and rated for agreement from 1 = Completely Disagree and 7 = Completely Agree.

  • analytic_1_yr2 - analytical/intuitive cognitive style, analytical item 1. "I feel that if I plan my decisions carefully I will make good decisions."
  • analytic_2_yr2 - "Before I make a decision, I like to figure out the most efficient way of studying it."
  • analytic_3_yr2 - "I'm very rational when it comes to evaluating risky options."
  • analytic_4_yr2 - "In making decisions I try to examine the importance of the good and bad points of each alternative."
  • analytic_5_yr2 - "I generally rely on careful reasoning in making up my mind."
  • intuit_1_yr2 - analytical/intuitive cognitive style, intuitive item 1. "I can get a good feeling for most decision situations very quickly."
  • intuit_2_yr2 - "I think that relying on one?s gut feelings is a sound decision making principle."
  • intuit_3_yr2 - "My first reaction to a decision situation usually turns out to be the best one."
  • intuit_4_yr2 - "I make my best choices when I have to make quick, instinctive decisions."
  • intuit_5_yr2 - "I let my intuition play a big part in most decisions I make."
  • analytic_score_yr2 - average of analytical cognitive style items
  • intuit_score_yr2 - average of intuitive cognitive style items

Financial Risk Decisions, Perception, Preferences

"Please indicate the likelihood that you would engage in the described activity or behavior if you were to find yourself in that situation." 1=extremely unlikely, 7=extremely likely.

  • risk_dec_1_yr2 - Financial Risk scale, decision item 1. "Investing 10% of your annual income in a moderate growth mutual fund."
  • risk_dec_2_yr2 - "Betting a day?s income at a high-stake poker game."
  • risk_dec_3_yr2 - "Investing 5% of your annual income in a very speculative stock."
  • risk_dec_4_yr2 - "Betting a day's income on the outcome of a sporting event."
  • risk_dec_5_yr2 - "Investing 10% of your annual income in a new business venture."

"People often see some risk in situations that contain uncertainty about what the outcome or consequences will be and for which there is the possibility of negative consequences. However, riskiness is a very personal and intuitive notion, and we are interested in your gut level assessment of risk. For each of the following statements, please indicate how risky you perceive each situation. Provide a rating on a scale from 1 to 7 (where 1 =Not at all Risky, 2 = Slightly Risky, 3 = Somewhat Risky, 4 = Moderately Risky, 5 = Risky, 6= Very Risky, 7 = Extremely Risky)."

  • risk_percept_1_yr2 - "Investing 10% of your annual income in a moderate growth mutual fund."
  • risk_percept_2_yr2 - "Betting a day?s income at a high-stake poker game."
  • risk_percept_3_yr2 - "Investing 5% of your annual income in a very speculative stock."
  • risk_percept_4_yr2 - "Betting a day's income on the outcome of a sporting event."
  • risk_percept_5_yr2 - "Investing 10% of your annual income in a new business venture."

###Financial Literacy

  • finance_score: total score out of 5
  • finance_1: response to item 1
  • finance_correct_1: 1 if correctly answered item 1 Items (correct answers in bold)

Items and Answers

  1. Suppose you have $100 in a savings account earning 2 percent interest a year. After five years, how much would you have? 1. More than $102 (correct) 2. Exactly $102 3. Less than $102 4. Don't know
  2. Imagine that the interest rate on your savings account is 1 percent a year and inflation is 2 percent a year. After one year, would the money in the account buy more than it does today, exactly the same or less than today? 1. More 2. Same 3. Less (correct) 4. Don't know
  3. If interest rates rise, what will typically happen to bond prices? Rise, fall, stay the same, or is there no relationship? 1. Rise 2. Fall (correct) 3. Stay the same 4. No relationship 5. Don't know
  4. True or false: A 15-year mortgage typically requires higher monthly payments than a 30-year mortgage but the total interest payment over the life of the loan will be less. 1. True (correct) 2. False 3. Don't know
  5. True or false: Buying a single company's stock usually provides a safer return than a stock mutual fund. 1. True 2. False (correct) 3. Don't know?

P-Beauty Contests

These game-theoretic items ask participants to estimate a number that is a function of the set of all participants' estimates. Each item has a Nash equilibrium (either 0 or 100) from which an error score was calculated.

Items and Answers

  1. "Choose a number between 0 and 100 that is as close as possible to 2 times the average (mean) of all numbers chosen by other participants for this question." Nash equilibrium is 100.
  2. "Choose a number between 0 and 100 that is as close as possible to 2/3 times the average (mean) of all numbers chosen by other participants for this question." Nash equilibrium is 0.
  3. "Choose a number between 0 and 100 that is as close as possible to 4/3 times the average (mean) of all numbers chosen by other participants for this question." Nash equilibrium is 100.
  4. "Choose a number between 0 and 100 that is as close as possible to 0.7 times the average (mean) of all numbers chosen by other participants for this question." Nash equilibrium is 0.

Helping

These items were suggested by Shefali Patil; From Grant and Sumanth (2009) Response scale is 1=Strongly Disagree to 7=Strongly Agree

  • helping_score: average (scale from 1 to 7) of 5 questions
  • helping_1: response to item 1

Items

  1. I get energized by working on tasks that have the potential to benefit others.
  2. It is important to me to have the opportunity to use my abilities to benefit others.
  3. I prefer to work on tasks that allow me to have a positive impact on others.
  4. I do my best when I'm working on a task that contributes to the well-being of others.
  5. I like to work on tasks that have the potential to benefit others.?

Organizational Commitment (Meyer and Allen, 1984)

  • orgcommit_score: average of items (1 to 7 scale, strongly disagree to strongly agree)
  • orgcommit_1: response to item 1

Items

  1. I really feel as if GJP's problems are my own.
  2. I feel a strong sense of belonging to GJP.
  3. I feel emotionally attached to GJP.
  4. I feel like a part of the family at GJP.
  5. GJP has a great deal of personal meaning for me.?

Intrinsic, Extrinsic, Prosocial Motivation - Rioux and Penner (2001) and Tierney, Farmer, and Graen (1999).

  • motivation.intrinsic.score: intrinsic motivation score as average of items 1, 4, 6.
  • motivation.prosocial.score: prosocial motivation score as average of items 2, 5, 7, 8.
  • motivation.extrinsic.score: extrinsic motivation score. Same as item 3.

Items

Why are you motivated to participate in this forecasting tournament?

  1. Because I enjoy forecasting itself.
  2. Because I find forecasting engaging.
  3. Because it's fun.
  4. Because I care about benefiting others through forecasting.
  5. Because I want to help others through forecasting.
  6. Because I want to have positive impact on others.
  7. Because it is important to me to do good for others through forecasting.
  8. Because I need to earn money.