Although the prompt injection attack seems ineffective at this point, Spader warned that prompt injection “was not among the dangers we contemplated” when courts were first grappling with AI scrambling justice systems. In Connecticut, like many other court systems, the focus so far has been on policing AI outputs that damage trust in courts, like hallucinated citations or fabricated quotes, not inputs like prompt injections.
Courts will most likely need to draft rules around prompt injection, too, Spader suggested, since Elliott’s case shows the technology and its misuses are rapidly advancing. If not, attorneys may find their own clients using prompt injections to manipulate court filings without their knowledge, Spader warned.
To Spader, there is a lesson to be learned from Elliott’s failed prompt injection attacks that he thinks “reaches well beyond this case.”
Elliott seemingly turned to prompt injection after using AI to build his case as a pro se litigant without a legal expert to assist in drafting his arguments. Such use is widespread among pro se litigants these days, Spader acknowledged, but those inexperienced in the courtroom are seemingly using chatbots in a way that hurts their cases, he suggested.
What frequently happens, Spader explained, is that pro se litigants build their argument backward, asking the chatbot to help them advocate only for their position, without ever asking the chatbot for the actual truth or to advance opposing arguments. This is “a genuine hazard of the technology, and one that judges now see often,” Spader said, as chatbot sycophancy then entrenches litigants in their arguments despite any ruling to the contrary. In Elliott’s case, defending his arguments fiercely meant turning to prompt injection to try to force the court to agree with him.
“An argument prompted only to agree with its author is, in the end, dishonest even with its author,” Spader said. “Those using these tools must ask them to test a position as readily as to advance it.”