How recognizing and overcoming the major tasks can get us closer to success

Argument mining in law is the automatic extraction of arguments or units of reasoning from legal documents. Writing software that automatically mines legal argument is proving extremely difficult. Even lawyers can find it difficult to identify, classify and extract legal arguments. But the overall task becomes a bit more manageable if we recognize and overcome 7 major challenges to success.
Defining a Unit of Argument
Argument mining in law has a technical definition. It is the automatic extraction of units of argument or reasoning from natural-language legal documents, with the goal of providing structured data for computational models of argument and for reasoning engines. (See Atkinson et al., "Toward Artificial Argumentation" AI Magazine (2017); Habernal and Gurevych, "Argumentation Mining in User-Generated Web Discourse" Computational Linguistics (2017); Lawrence and Reed, "Argument Mining: A Survey" Computational Linguistics (2020).)
What is an "argument"? An argument consists of argument components and logical relations among those components.
A unit of argument contains at least two propositions as components – one conclusion and one or more premises. A "proposition" is a complete thought that is capable of being true or false, typically expressible by a simple declarative sentence. For example, the English sentence "the child kicked the ball" expresses the proposition that the child kicked the ball. But other English sentences can express the same proposition (e.g., "the ball was kicked by the child"). And sentences in French and Spanish can express that same proposition. So, we can think of a proposition as the logical meaning expressed by a simple sentence in a natural language. Logic studies arguments made up of propositions, regardless of how those propositions are expressed in a natural language.
We can attach additional labels argument components. For example, Stephen Toulmin’s influential model calls the conclusion a "claim," and it divides premises into "data" and "warrant." (Stephen E. Toulmin, The Uses of Argument: Updated Edition (Cambridge University Press 2003).) The data are defined as "the facts we appeal to as a foundation for the claim." (Id., 90.) In legal proceedings, the admitted evidence plays the role of such data. For example, part of the evidence might be that "the witness testified that the event occurred as described." Toulmin describes warrants as "general, hypothetical statements, which can act as bridges, and authorize the sort of step to which our particular argument commits us." (Id., 91.) In legal arguments, warrants in this sense could be common-sense generalizations, scientific methods, or inference rules established by law. For example, a common-sense warrant might be, "if the witness testified that the event occurred as described, then it probably did." The conclusion of the argument might be that the event actually did occur as described.
Argument units can also differ in the logical relations that link their propositional components together. (See Lawrence and Reed (2020), p. 777.) A premise is connected to the conclusion by either a deductive or probabilistic inferential relation. Moreover, a premise itself might be logically complex, consisting of multiple propositions linked together by logical connectives, such as conjunction and disjunction.
The Mining Process
An argument unit, therefore, consists of a conclusion inferred from a premise, or from a set of logically connected premises. The objectives of argument mining in law are:
· to identify within a legal document the sentences that are related to the argument;
· to extract the propositional components of the argument from the sentences;
· to extract the argument’s logical relations from the words and phrases;
· to accurately construct the complete unit of argument; and
· to connect units of argument into coherent lines of reasoning ("argumentation").
The process of "mining" usually occurs in layers. On the surface, a legal document is a sequence of sentences (usually grouped into paragraphs, which may be grouped into larger document sections). Sentences (or clauses, and sometimes phrases, within sentences) express propositions. Some words or phrases within those sentences usually express the logical relations at work (e.g., "and", "or", "therefore", "unless"). Argument mining requires connecting the appropriate propositions together, using the appropriate logical relations, to reflect the arguments being made. It also requires relating whole arguments to each other (e.g., supporting or opposing arguments).
The mining operation must "dig out" all these structures and "bring them to the surface" – make them visible. And it must do so correctly, without distorting the meaning of the original document. The graphic image at the beginning of this article suggests the layered nature of this mining operation.
7 Challenges to Automated Argument Extraction
Automated mining requires creating software that can perform these tasks well, with little or no assistance by humans. From my long experience with legal reasoning, logic, and automated extraction processes, I break this overall process down into 7 major tasks, with each task presenting its challenges. If we can successfully deal with each one, we have gone a long way toward automating argument mining for law. In this discussion, I will focus on extracting arguments from fact-finding decisions – that is, from legal documents written to announce how a trier of fact has applied the legal rules to evidence to reach conclusions about facts. Such decisions are normally issued by trial courts or by administrative tribunals.
1. Mining Computable Systems of Legal Rules
The first challenge to argument mining from legal decisions is capturing the governing legal rules in a computable form. Legal rules identify the issues to be proved, and they also structure the proof process. Lawyers keep complex sets of such legal rules in their minds, and they use those rules constantly in making arguments. Where do we find such legal rules? Can machines extract them, and then form them into systems? How should we represent and store such systems of rules in computers? I have found that we can represent legal rules using a special form of inference tree (a "rule tree"), which can then function as Toulmin warrants in arguments. The details of how software might extract and formulate rule trees must remain for another story. But as I explain in another post, we know that argument mining in law requires capturing legal rules in a computable format, so they can be used in argumentation.
2. Mapping Conclusions of Fact to Legal Issues
The legal rules tell us the issues to be proved, but the court or tribunal must decide which rules are satisfied in a particular case. The second challenge to argument mining is identifying a tribunal’s conclusions on the legal issues presented, including mapping each conclusion to its appropriate legal issue (which rule the conclusion is "about"). A party’s lawyer interprets the evidence and uses the legal rules, with the goal of persuading the trier of fact to reach conclusions in favor of the lawyer’s client. Decisions written by the decision maker usually explain what the legal rules are, the arguments of the parties, the relevant evidence, the decision maker’s reasoning about the evidence, and the trier of fact’s conclusions (called "findings of fact"). A major task in argument mining from a case’s written decision is identifying those sentences (or parts of sentences) that announce the findings of fact of the tribunal. These "finding sentences" tell us which arguments the tribunal accepted, and which arguments the tribunal rejected.
3. Determining the Evidence that is "Relevant"
But which evidence in a case is a premise for which conclusion or finding? The third challenge to argument mining is identifying the evidence that is relevant to any particular conclusion of fact. In law, evidence is "relevant" if it makes a conclusion more or less probable than it would be without the evidence. (See, e.g., Federal Rule of Evidence 401.) Can we create a general algorithm for classifying evidence as "relevant" in this legal sense? Conducting accurate argument mining from reported decisions in similar cases can help us answer such questions. Perhaps we can inductively evolve methods for linking types of evidence to types of findings.
4. Reasoning from Evidence and Rules to Findings
Supposing that we can mine legal rules, findings of fact, and relevant evidence, then what is the reasoning that connects them all together? The fourth challenge is identifying the reasoning of the decision maker. As discussed above, the evidence supplies the Toulmin "data" for arguments, and legal rules supply some of the inference "warrants." But most intermediate inferences (paths of reasoning from evidence to conclusion) are not specified by the legal rules. The trier of fact draws upon common sense, or statistical theory, or science, or some other source. Lawyers on all sides of an issue might argue alternative paths for drawing an inference. However, we want to identify the tribunal’s own reasoning, as distinct from the arguments of the parties. For a start, we can identify any sentences from reported decisions that explicitly state such reasoning. But what if the reasoning is only implicit? We may be able to supply such implicit reasoning, if we develop an informative taxonomy of typical argument patterns or schemes.
5. Identifying Argument Patterns
The fifth challenge to argument mining in law is formulating an adequate set of recurring argument patterns. A typical pattern or scheme can help supply implicit inferences. In addition, we need categories or types for classifying the units of argument that we find in decisions. Categories or types will allow us to calculate rates of success or failure for different arguments in different circumstances. Past decisions can provide an empirical basis for developing such a type system, if we can inductively generate a useful classification system. We must develop such a system for classifying units of argument, and label a sufficient amount of data, before we can hope to automate the process of detecting instances of argument types.
6. Creating Enough Accurate Semantic Data
At each step in the mining process, we want to develop predictive models that will help us do the classification. The sixth challenge to argument mining in law is creating enough semantic data that are sufficiently accurate, so that we can develop such models. Semantic data consist of portions of text labeled or classified with respect to their meaning or significance. In the specific domain of law, legal-semantic data capture the legal meaning or significance. Creating a sufficient quantity of good data can be both difficult and expensive. Machine-learning algorithms can create models that will help with data generation, but they cannot eliminate the need for first creating enough accurate data. Fortunately, empirical work suggests that legal language is so regular that relatively small amounts of semantic data can adequately train machine-learning algorithms.
7. Developing Machine-Learning Predictive Models
But what does "adequately train" mean here? Can Machine Learning ever do successful argument mining in law? The seventh challenge to argument mining in law is developing machine-learning models that can automatically create semantic data with sufficient accuracy. What counts as "sufficient accuracy" depends upon the intended use case, the linguistic characteristics of the text of the legal document, and the abstractness of the semantic data we need. Currently, we can create enough semantic data to train models that add significant value for lawyers and judges when they perform various tasks. These predictive models are helping to automate many of the subtasks involved in meeting these 7 challenges. But fully automatic extraction of whole arguments in general may be a difficult problem for a long time.
Conclusion
We can decompose the problem of argument mining from legal decisions into a series of tasks, each of which presents challenges. Overall, the strategy consists of:
· identifying those types of sentences that are likely to contain different types of logical information;
· extracting the desired logical information from those sentences; and
· using argument patterns to formulate the argument units that the text contains.
As we dig deeper from the linguistic surface of the legal document, down to the word level of selected sentences, we use the linguistic features of the text to label it with more abstract logical concepts. Argument mining is therefore as paradoxical as it is difficult, particularly in law. But identifying the series of challenges, and making progress with each task, gives us a strategy for success on the overall problem.