A recent test of OpenAI's ChatGPT's deep research feature has raised concerns about the tool's ability to provide comprehensive and accurate information on complex legal topics. Senior tech editor Adi Robertson put the feature to the test by asking it to summarize recent court rulings related to Section 230 of the Communications Decency Act, a law often referred to as the "26 words that created the internet." While ChatGPT was able to accurately select and summarize a set of recent court rulings, it missed broader points and ignored a full year's worth of legal decisions that have significantly impacted the status of the law.

The deep research feature, available on ChatGPT's $200 per month Pro tier, is designed to produce complex and sophisticated reports on specific topics. In this case, Robertson asked ChatGPT to compile a list of federal court and Supreme Court rulings from the last five years related to Section 230 and summarize any significant developments in how judges have interpreted the law. The resulting report was dense and formatted with helpful headers, but it lacked context and omitted important recent developments.

Legal expert Eric Goldman, who reviewed the results, noted that while ChatGPT's report was largely accurate, it ignored context and broader dynamics that are essential to understanding the law. Goldman pointed out that the report failed to include any cases from 2024, a year that saw significant developments in Section 230 jurisprudence, including a Third Circuit ruling against granting the law's protections to TikTok. This omission fundamentally changes the story the report tells, making it seem like the law remains a "robust shield" when in fact it is facing significant challenges.

The issue is not unique to this test, as other users have reported similar omissions when using the deep research feature. The problem seems to be that ChatGPT is not designed to identify important questions that aren't being asked, a key aspect of human expertise. While the tool is impressive from a technological standpoint, it falls short of providing a comprehensive understanding of complex legal topics.

The implications of this are significant, as ChatGPT's deep research feature has the potential to be a powerful tool for legal research. However, its limitations mean that it cannot be relied upon to provide a complete picture of a given topic. As Robertson notes, the feature may be more useful for obscure legal topics with less human coverage, but it is not yet a replacement for human expertise and judgment.

In conclusion, while ChatGPT's deep research feature shows promise, it is clear that it still has significant limitations. The tool's inability to provide context and identify important questions that aren't being asked means that it cannot be relied upon to provide a comprehensive understanding of complex legal topics. As the technology continues to evolve, it will be important to carefully evaluate its limitations and ensure that it is used in a way that complements human expertise, rather than replacing it.