Open Reasoning Tasks

Author

Contributors

Published

July 27, 2024

1 Tasks List

This is a collection of open reasoning tasks for language models.

Task Description Tags
Analogical Problem Solving This task involves using analogies to solve problems by applying solutions from one domain to another similar situation. It evaluates the model’s ability to recognize… Problem Solving, Analogical Reasoning, Creative Thinking, Interdisciplinary Application, Systems Thinking, Synthetic
Analogy Creation This task involves creating analogies that effectively compare two different concepts, situations, or objects to highlight similarities in their relationships or structures.… Creative Thinking, Abstract Reasoning, Conceptual Mapping, Communication, Explanatory Skills, Synthetic
Analyzing Cultural Differences This task involves comparing and contrasting different cultural practices, beliefs, or norms to identify similarities, differences, and potential reasons for these… Cultural Analysis, Comparative Studies, Social Norms, Cross-Cultural Communication, Global Awareness, Synthetic
Analyzing Decision-Making Processes This task involves examining and evaluating the steps, factors, and reasoning behind decision-making processes. It assesses the model’s ability to understand complex… Critical Thinking, Decision Analysis, Problem Solving, Strategic Planning, Risk Assessment, Synthetic
Analyzing Decision Trees This task involves interpreting and evaluating decision trees to understand the logic behind decision-making processes. It assesses the model’s ability to follow branching… Decision Making, Logical Reasoning, Tree Structures, Conditional Logic, Problem Diagnosis, Synthetic
Analyzing Historical Counterfactuals This task involves considering alternative outcomes of historical events by changing key factors or decisions. It evaluates the model’s ability to understand complex… Historical Analysis, Counterfactual Thinking, Cause and Effect, Critical Thinking, Scenario Planning, Synthetic
Analyzing Rhetorical Strategies This task involves identifying and evaluating the rhetorical strategies used in a given text or speech. It assesses the model’s ability to recognize persuasive techniques… Rhetoric, Persuasion, Communication Analysis, Critical Thinking, Language, Synthetic
Assessing Risk and Uncertainty This task involves evaluating scenarios with incomplete information to determine potential risks and uncertainties. It assesses the model’s ability to identify possible… Risk Assessment, Decision Making, Probability, Scenario Analysis, Critical Thinking, Synthetic
Bias Detection This task involves identifying and explaining biases present in given scenarios, statements, or data. It evaluates the model’s ability to recognize various types of biases… Critical Thinking, Bias Identification, Data Interpretation, Logical Reasoning, Research Methodology
Bias Mitigation Given a statement and the bias that led to it, this task involves constructing a statement as close to the ground truth as possible. It evaluates the model’s ability to… Critical Thinking, Bias Correction, Logical Reasoning, Data Interpretation, Objective Analysis
Calculating Probabilities This task involves calculating probabilities for various scenarios, including simple and compound events. It evaluates the model’s ability to apply probability theory, use… Mathematics, Probability Theory, Statistical Reasoning, Problem Solving, Quantitative Analysis
Categorizing Information into Hierarchies This task involves organizing information into hierarchical structures based on relationships, properties, or other logical criteria. It evaluates the model’s ability to… Classification, Hierarchical Thinking, Conceptual Organization, Pattern Recognition, Logical Structuring, Synthetic
Causal Chain Analysis This task involves identifying and analyzing a sequence of events or factors that lead to a specific outcome. It evaluates the model’s ability to understand cause-and-effect… Cause and Effect, Systems Thinking, Environmental Science, Economics, Complex Systems Analysis, Synthetic
Completing Analogies This task involves completing analogies in the form “A is to B as C is to ?” to evaluate the model’s ability to recognize relationships between pairs of words and apply them… Language, Reasoning, Analogies, Vocabulary, Relationships
Constructing Valid Arguments This task involves creating logically sound arguments to support a given conclusion or claim. It evaluates the model’s ability to use premises, apply logical reasoning, and… Logic, Critical Thinking, Argumentation, Reasoning, Premise-Conclusion Relationships, Synthetic
Counterfactual Analysis Counterfactual analysis involves examining hypothetical scenarios that are contrary to what actually happened. This task requires the model to consider alternative outcomes… Critical Thinking, Historical Analysis, Cause and Effect, Hypothetical Scenarios, Complex Systems, Synthetic
Critical Factor Identification in Theory of Mind Understand that similar agents may act differently, or different agents may act similarly, based on a third factor. This task evaluates the model’s ability to identify these… Theory of Mind, Behavioral Analysis, Critical Thinking, Comparative Psychology, Social Cognition
Critiquing Argument Structures This task involves analyzing and evaluating the structure, logic, and effectiveness of arguments. It assesses the model’s ability to identify strengths and weaknesses in… Critical Thinking, Logical Analysis, Argumentation, Fallacy Identification, Reasoning Skills, Synthetic
Curry’s Paradox (semantic) This task involves examining a specific form of semantic paradox known as Curry’s Paradox. It tests the model’s ability to reason about self-reference, logical implication… Logic, Paradoxes, Self-Reference, Semantic Analysis, Critical Thinking, Philosophical Reasoning
Deciphering Ambiguous Instructions This task involves interpreting and clarifying instructions that are unclear, incomplete, or potentially contradictory. It evaluates the model’s ability to identify… Language Interpretation, Critical Thinking, Clarification, Problem Solving, Instruction Analysis, Synthetic
Deconstructing Complex Systems This task involves breaking down complex systems, processes, or phenomena into their constituent parts and explaining how these parts interact to create the overall system.… Systems Analysis, Cause and Effect, Component Interaction, Process Understanding, Analytical Thinking, Synthetic
Deconstructing Metaphors This task involves analyzing and explaining the meaning behind metaphors. It evaluates the model’s ability to understand figurative language, interpret symbolic… Figurative Language, Literary Analysis, Symbolic Interpretation, Critical Thinking, Communication Skills, Synthetic
Deducing Motives from Actions This task involves analyzing the actions of individuals or groups and inferring their underlying motivations or intentions. It evaluates the model’s ability to understand… Psychological Analysis, Behavioral Interpretation, Critical Thinking, Motivation Theory, Context Consideration, Synthetic
Deducing Rules from Examples This task involves analyzing a set of examples to infer the underlying rule or pattern that governs them. It evaluates the model’s ability to recognize patterns, generalize… Pattern Recognition, Logical Reasoning, Inductive Reasoning, Linguistic Analysis, Mathematical Thinking, Synthetic
Deductive Logic Puzzles This task involves solving deductive logic puzzles to evaluate the model’s ability to use given information, make logical inferences, and arrive at a correct conclusion. Logic, Deductive Reasoning, Problem Solving, Critical Thinking, Inference
Describing Spatial Relationships This task involves accurately describing the relative positions and orientations of objects in space. It evaluates the model’s ability to understand and communicate spatial… Spatial Reasoning, Descriptive Skills, Object Orientation, Visual-Spatial Awareness, Communication
Detecting Sarcasm and Irony This task involves identifying and explaining instances of sarcasm or irony in given statements or scenarios. It evaluates the model’s ability to recognize subtle linguistic… Language Comprehension, Context Analysis, Figurative Language, Social Intelligence, Linguistic Nuance, Synthetic
Determining Alternative Outcomes This task involves analyzing historical events or decisions and reasoning about possible alternative outcomes if key factors had been different. It evaluates the model’s… Historical Analysis, Counterfactual Thinking, Cause and Effect, Critical Thinking, Scenario Planning
Distinguishing Correlation from Causation This task involves analyzing given scenarios or statistical relationships to identify cases where correlation does not imply causation, especially in unintuitive or… Critical Thinking, Statistics, Data Analysis, Logical Fallacies, Causal Reasoning
Distinguishing Fact from Opinion This task involves differentiating between factual statements and opinions in given texts or scenarios. It evaluates the model’s ability to recognize objective, verifiable… Critical Thinking, Information Literacy, Objectivity, Media Analysis, Reasoning, Synthetic
Equation Derivation This task involves deriving mathematical equations from given information or scenarios to evaluate the model’s ability to translate word problems into mathematical… Mathematics, Algebra, Word Problems, Equation Formulation, Problem Solving
Estimating Duration This task involves estimating the time required for various activities or processes. It evaluates the model’s understanding of time scales and its ability to make reasonable… Time Estimation, Temporal Reasoning, Process Understanding, Comparative Analysis, General Knowledge
Ethical Dilemma Resolution This task involves analyzing complex ethical scenarios, weighing conflicting moral principles, and proposing reasoned solutions. It evaluates the model’s ability to consider… Ethics, Critical Thinking, Decision Making, Cultural Sensitivity, Environmental Ethics, Moral Philosophy, Synthetic
Evaluating Analogies for Accuracy This task involves assessing given analogies for their accuracy and appropriateness. It evaluates the model’s ability to critically analyze relationships between concepts… Critical Thinking, Analogy Analysis, Conceptual Relationships, Logical Reasoning, Metaphor Evaluation, Synthetic
Evaluating Competing Theories This task involves analyzing and comparing multiple theories or explanations for a phenomenon, assessing their strengths and weaknesses, and determining which theory is best… Critical Thinking, Evidence Evaluation, Comparative Analysis, Scientific Reasoning, Theory Assessment, Synthetic
Evaluating Policy Implications This task involves analyzing proposed policies or decisions and predicting their potential consequences across various domains such as economics, society, environment, and… Policy Analysis, Systems Thinking, Predictive Reasoning, Stakeholder Analysis, Unintended Consequences, Socioeconomic Impact Assessment, Synthetic
Evaluating Source Credibility This task involves assessing the reliability and trustworthiness of various information sources. It evaluates the model’s ability to consider factors such as expertise… Critical Thinking, Information Literacy, Source Evaluation, Research Skills, Media Literacy, Synthetic
Fermi Estimation Fermi estimation, named after physicist Enrico Fermi, involves making educated guesses to estimate quantities that are difficult or impossible to measure directly. This task… Estimation, Problem Decomposition, Quantitative Reasoning, Order of Magnitude, Logical Thinking, Synthetic
First-Order False Belief This task involves identifying why a misinformed agent may behave contrary to reality due to inaccurate beliefs. It evaluates the model’s theory of mind ability and… Theory of Mind, False Beliefs, Cognitive Psychology, Reasoning, Social Cognition
First-Order Ignorance This task involves identifying why an agent may lack knowledge or awareness of certain facts or events. It evaluates the model’s understanding of ignorance and its impact on… Theory of Mind, Ignorance, Cognitive Psychology, Decision Making, Social Cognition
Forecasting Technological Impacts This task involves predicting and analyzing the potential effects of emerging or hypothetical technologies on society, economy, and daily life. It evaluates the model’s… Futurism, Technology Assessment, Scenario Planning, Trend Analysis, Societal Impact, Synthetic
Generating Creative Solutions This task involves developing innovative and unique solutions to given problems or challenges. It evaluates the model’s ability to think outside the box, combine ideas in… Innovation, Problem-solving, Lateral Thinking, Idea Generation, Unconventional Approaches, Synthetic
Higher Order False Belief This task involves handling a complex chain of agent’s beliefs about the knowledge (and accuracy of the knowledge) of other agents - to ultimately predict the behavior of an… Theory of Mind, Complex Reasoning, Social Cognition, False Beliefs, Interpersonal Dynamics
Hypothesis Formation This task involves generating plausible hypotheses to explain observed phenomena or solve problems. It evaluates the model’s ability to apply scientific thinking, create… Scientific Method, Critical Thinking, Problem Solving, Analytical Skills, Ecological Reasoning, Synthetic
Identifying Anachronisms This task involves recognizing elements that are out of place in a given historical context. It evaluates the model’s knowledge of historical periods and ability to detect… Historical Knowledge, Temporal Reasoning, Anachronism Detection, Critical Thinking, Context Analysis
Identifying Cause and Effect Relationships This task involves analyzing given scenarios or statements to identify and explain the cause and effect relationships present. It evaluates the model’s ability to understand… Critical Thinking, Analysis, Causal Relationships, Logic, Reasoning
Identifying Cognitive Biases This task involves recognizing and explaining various cognitive biases in given scenarios or decision-making processes. It evaluates the model’s ability to understand how… Cognitive Psychology, Critical Thinking, Decision Making, Behavioral Economics, Psychological Biases, Synthetic
Identifying Hallucination-Prone Questions This task involves recognizing questions that are likely to lead to its hallucination, or questions it simply doesn’t know the answer to. It evaluates the model’s ability to… Hallucination Prevention, Question Analysis, Knowledge Boundaries, Information Reliability, Self-awareness
Identifying Implicit Biases in Language This task involves recognizing subtle, often unintentional biases embedded in language use. It evaluates the model’s ability to detect underlying assumptions, stereotypes… Language Analysis, Bias Detection, Critical Thinking, Social Awareness, Equality and Inclusion, Synthetic
Identifying Logical Fallacies This task involves identifying and explaining common logical fallacies in given arguments or statements to evaluate the model’s ability to recognize flawed reasoning. Logic, Critical Thinking, Argumentation, Fallacies, Reasoning
Identifying Logical Inconsistencies This task involves detecting and explaining logical inconsistencies or contradictions within a given statement, argument, or scenario. It evaluates the model’s ability to… Logic, Critical Thinking, Contradiction Detection, Argument Analysis, Reasoning, Synthetic
Identifying Relationships This task involves identifying the relationship between pairs of words or concepts to evaluate the model’s ability to recognize various types of connections and articulate… Language, Conceptual Relationships, Critical Thinking, Vocabulary, Analysis
Identifying Unstated Assumptions This task involves recognizing and articulating implicit assumptions that underlie statements, arguments, or scenarios. It evaluates the model’s ability to think critically… Critical Thinking, Logical Analysis, Argument Evaluation, Implicit Reasoning, Decision-Making, Synthetic
Inference Drawing from Incomplete Data This task involves making logical deductions or inferences based on limited or incomplete information. It tests the model’s ability to use available data, apply reasoning… Critical Thinking, Logical Reasoning, Data Analysis, Hypothesis Formation, Scientific Inference, Synthetic
Inferring Emotional States This task involves analyzing given scenarios or descriptions of behavior to infer the emotional states of individuals. It evaluates the model’s ability to understand and… Emotional Intelligence, Behavioral Analysis, Social Cognition, Psychological Interpretation, Non-verbal Communication, Synthetic
Inferring Motivations from Actions This task involves analyzing described actions or behaviors to deduce the underlying motivations, intentions, or goals of the individuals involved. It evaluates the model’s… Psychology, Behavioral Analysis, Social Dynamics, Critical Thinking, Empathy, Synthetic
Interpreting Ambiguous Statements This task involves analyzing statements that have multiple possible interpretations and identifying the different ways they can be understood. It evaluates the model’s… Language Analysis, Semantic Interpretation, Context Consideration, Linguistic Ambiguity, Critical Thinking, Synthetic
Interpreting and Creating Timelines This task involves reading or creating timelines to represent a series of events or processes. It evaluates the model’s ability to visualize and interpret temporal data. Timeline Creation, Historical Analysis, Data Visualization, Temporal Reasoning, Business Development
Interpreting Body Language Cues This task involves analyzing and interpreting non-verbal communication signals, including facial expressions, postures, gestures, and micro-expressions. It evaluates the… Non-verbal Communication, Social Intelligence, Cultural Context, Emotional Intelligence, Behavioral Analysis, Synthetic
Interpreting Legal Language and Precedents This task involves analyzing and interpreting legal texts, statutes, or case law to understand their implications and applications. It evaluates the model’s ability to… Legal Interpretation, Constitutional Law, Case Law Analysis, Legal Precedents, Criminal Procedure, Synthetic
Interpreting Nonverbal Communication This task involves analyzing and interpreting nonverbal cues in human communication, such as body language, facial expressions, and gestures. It evaluates the model’s… Social Intelligence, Body Language, Communication Skills, Emotional Intelligence, Behavioral Analysis, Synthetic
Interpreting Statistical Data This task involves analyzing and interpreting statistical data presented in various formats (e.g., tables, graphs, or text descriptions). It evaluates the model’s ability to… Statistics, Data Analysis, Trend Identification, Critical Thinking, Quantitative Reasoning
Lateral Thinking Puzzles This task involves solving lateral thinking puzzles to evaluate the model’s ability to think creatively, consider unconventional scenarios, and ask relevant questions to… Creative Thinking, Problem Solving, Lateral Thinking, Word Play, Unconventional Scenarios
Mathematical Word Problems This task involves presenting the model with mathematical word problems to assess its ability to interpret, set up, and solve real-world scenarios using mathematical concepts. Mathematics, Problem Solving, Arithmetic, Word Problems, Applied Mathematics
Mental Rotation Tasks This task involves mentally rotating objects or shapes and predicting their appearance from different angles. It evaluates the model’s ability to manipulate spatial… Spatial Reasoning, Mental Imagery, Geometric Transformation, Visual-Spatial Skills, Cognitive Processing
Moral Reasoning in Everyday Situations This task involves analyzing everyday scenarios that present moral dilemmas and reasoning through the ethical implications of different actions. It evaluates the model’s… Ethical Reasoning, Moral Dilemmas, Decision Making, Interpersonal Relationships, Conflict Resolution, Synthetic
Multiturn Latex Generation This task involves multiturn conversations to generate and edit a latex document from broad descriptions. Following are the properties of multiturn latex generation task: LaTeX Generation, Document Structure, Measurement, Equation Formulation, Chain of Thoughts
Narrative Gap Filling This task involves filling in missing information or events in a narrative to create a coherent story. It evaluates the model’s ability to understand context, make logical… Creative Writing, Logical Reasoning, Storytelling, Context Understanding, Inference, Synthetic
Parsing Complex Sentences This task involves breaking down complex sentences into their constituent parts, identifying grammatical structures, and explaining the relationships between different… Grammar Analysis, Syntax, Linguistic Complexity, Sentence Structure, Clause Identification, Synthetic
Pattern Recognition in Spatial Arrangements This task involves identifying patterns or rules in the spatial arrangement of objects or shapes. It evaluates the model’s ability to recognize spatial regularities and… Pattern Recognition, Spatial Reasoning, Sequence Completion, Logical Thinking, Visual-Spatial Skills
Perspective-Taking in Social Scenarios This task involves analyzing social situations from different viewpoints to understand the motivations, emotions, and potential reactions of various parties involved. It… Empathy, Social Intelligence, Conflict Resolution, Emotional Intelligence, Multiple Perspectives, Synthetic
Predicting Market Trends This task involves analyzing various economic indicators, historical data, and current events to forecast potential future market trends. It evaluates the model’s ability to… Economic Analysis, Market Forecasting, Trend Prediction, Data Synthesis, Strategic Planning, Synthetic
Predicting Outcomes Based on Scenarios This task involves analyzing given scenarios and predicting potential outcomes based on the information provided. It evaluates the model’s ability to apply logical… Critical Thinking, Scenario Analysis, Prediction, Logical Reasoning, Problem Solving
Prioritizing Conflicting Goals This task involves analyzing and resolving situations where multiple objectives are in conflict, requiring trade-offs and strategic decision-making. It evaluates the model’s… Decision Making, Strategic Planning, Resource Allocation, Goal Setting, Trade-off Analysis, Synthetic
Proof Verification This task involves verifying mathematical proofs to evaluate the model’s ability to understand logical arguments, identify correct steps in a proof, and spot errors or gaps… Mathematics, Logic, Proof Verification, Number Theory, Reasoning
Recognizing Emotional Subtext This task involves identifying and interpreting the underlying emotional content or implications in a given text that are not explicitly stated. It evaluates the model’s… Emotional Intelligence, Communication Analysis, Subtext Interpretation, Social Cognition, Language Nuance, Synthetic
Recognizing Patterns in Behavior This task involves identifying recurring patterns or trends in human or animal behavior based on given scenarios or data. It evaluates the model’s ability to recognize… Behavioral Analysis, Pattern Recognition, Data Interpretation, Predictive Reasoning, Consumer Behavior, Synthetic
Recognizing Patterns in Sequences This task involves identifying and extending patterns in numerical, alphabetical, or symbolic sequences. It evaluates the model’s ability to recognize underlying rules and… Pattern Recognition, Sequence Analysis, Logical Reasoning, Mathematical Thinking, Problem Solving, Synthetic
Reconciling Conflicting Information This task involves analyzing and resolving contradictory information from multiple sources. It evaluates the model’s ability to critically assess different pieces of… Critical Thinking, Information Analysis, Problem Solving, Decision Making, Evidence Evaluation, Synthetic
Reverse Engineering Processes This task involves analyzing the end result of a process and working backwards to determine the steps or components that led to that outcome. It evaluates the model’s… Analytical Thinking, Process Analysis, Logical Reasoning, Systems Thinking, Problem Solving, Synthetic
Risk Assessment in Decision-Making This task involves evaluating potential risks and benefits associated with different courses of action in a given scenario. It assesses the model’s ability to identify… Decision Making, Risk Analysis, Critical Thinking, Scenario Planning, Cost-Benefit Analysis, Synthetic
Second-Order False Belief This task involves understanding that an agent may hold a false belief about another agent’s belief, leading to misinterpretations of actions or intentions. It additionally… Theory of Mind, False Beliefs, Higher-Order Reasoning, Social Cognition, Cognitive Psychology
Sequencing Events This task involves arranging a set of events in chronological order. It evaluates the model’s ability to understand temporal relationships and logical sequences. Temporal Reasoning, Logical Sequencing, Historical Knowledge, Process Understanding, Chronological Ordering
Solving Riddles and Word Puzzles This task involves deciphering and solving various types of riddles and word puzzles. It evaluates the model’s ability to think creatively, interpret figurative language… Problem Solving, Lateral Thinking, Language Skills, Deductive Reasoning, Creative Thinking, Synthetic
Solving Word Problems with Multiple Variables This task involves interpreting word problems, identifying relevant variables, and constructing equations to solve complex scenarios. It evaluates the model’s ability to… Mathematics, Algebra, Word Problems, System of Equations, Problem Solving, Synthetic
Spatial Problem-Solving This task involves using spatial reasoning to solve practical problems or puzzles. It evaluates the model’s ability to apply spatial concepts to real-world scenarios and… Problem Solving, Spatial Reasoning, Practical Application, Geometric Thinking, Logical Deduction
Stack-Based Reasoning This task exercises the ability to prioritize tasks into a stack so that an implementing worker may pop items from the top of the stack to complete a task. The required… self verification of tests, stack based reasoning, FORTH-style stacks, First In Last Out, FILO, reverse thinking, functional programming, tail recursion
Syllogism Reasoning This task involves providing a series of syllogisms to the model to evaluate its logical reasoning capabilities. Logic, Deductive Reasoning, Syllogisms, Critical Thinking
Towers of Hanoi Solve the classic Towers of Hanoi puzzle. Given a number of disks and three pegs, move all disks from the first peg to the last peg following these rules: 1) Only one disk… Problem Solving, Recursion, Mathematical Puzzle, Algorithm, Game Theory
Trait Attribution in Behavioral Scenarios This task involves analyzing described behaviors or actions and inferring personality traits or characteristics that might explain those behaviors. It evaluates the model’s… Psychology, Personality Assessment, Behavioral Analysis, Social Cognition, Character Inference, Synthetic
Trend Analysis and Forecasting This task involves examining historical data or patterns to identify trends and make predictions about future outcomes. It requires the ability to recognize patterns… Data Analysis, Pattern Recognition, Forecasting, Critical Thinking, Quantitative Reasoning, Contextual Analysis, Synthetic
Truth Table Completion This task involves completing truth tables for given logical expressions to evaluate the model’s understanding of Boolean logic and its ability to determine the truth value… Logic, Boolean Algebra, Truth Tables, Logical Operators, Propositional Logic
Understanding Time-Based Relationships This task involves analyzing and explaining relationships between events based on their timing. It evaluates the model’s ability to understand concepts like causality… Causal Reasoning, Temporal Analysis, Event Sequencing, Cause and Effect, Systems Thinking
Understanding Time Zones and Global Time Differences This task involves calculating time differences across various time zones and understanding how global time works. It evaluates the model’s ability to work with time zone… Time Zone Conversion, Global Time, International Date Line, Travel Time Calculation, Temporal Reasoning
Unraveling Paradoxes This task involves analyzing and explaining apparent contradictions or logical puzzles known as paradoxes. It evaluates the model’s ability to think critically, identify… Critical Thinking, Logic, Philosophy, Conceptual Analysis, Problem Solving, Synthetic
No matching items