Investigative Research: How to Use Data About U.S. Terrorism Cases

Or, steal our data, please!

A well-organized and maintained database of public information on a story of continuing interest creates an archive of information that can generate stories and insights for years to come.

Here’s one of the best examples from my time at The Intercept.

https://trial-and-terror.theintercept.com

The Intercept first published this database on April 20, 2017, and it was last updated on June 14, 2023.

Trevor Aaronson and I continue to update the database whenever we are alerted to new terrorism cases, to court decisions on the cases, and on movements and releases of defendants convicted of terrorism in the control of the federal Bureau of Prisons.

The most recent updates are not currently reflected on The Intercept’s Trial And Terror project page, which only displays data through June 2023. We are working to get the most current information online soon. In our internal database, as of December 15, 2023, we have 1,003 people charged since September 11, 2001, and 602 released from prison after serving their sentences. Among those persons charged are dozens who were never brought to court and are designated as “fugitives,” including people who traveled overseas to join ISIS and al-Shabaab, the offshoot of al-Qaeda in Somalia.

Trial and Terror: investigation stories

Trevor Aaronson reported and wrote the stories and I reported and researched in the documents and data. Our project won the Online News Association’s The University of Florida Award for Investigative Data Journalism in 2017. This Substack post is adapted from the presentation we gave at the awards event at the University of Florida in Gainesville.

“A year-long investigation by The Intercept reveals that more than half of the almost 800 people prosecuted on international terrorism-related charges since 9/11 have been released, often with no provision for supervision or ongoing surveillance. That’s not a reason to be alarmed, however, it’s a reason to be skeptical of what these prosecutions achieved in the first place. The FBI spends more than $3 billion a year on counterterrorism efforts, and every new terrorism prosecution provides new justification for that expense.”

Findings from the Trial and Terror database

The landing page for Trial and Terror offers top navigation features for sorting and filtering the data by types of cases, defendants, and status of prosecution.

If you click on the EXPLORE THE DATABASE button at the bottom of the page, you can find even more detail, including the names of the defendants.

Thanks to the database features implemented by Akil Harris, Moiz Syed and Westley Hennigh-Palermo, you can see that 27% of the defendants were charged in New York state and none in South Dakota.

As result, we could confidently generalize about U.S. terrorism prosecutions:

Learn About Each Case

You can also see a brief profile of each of the cases, machine-generated from the data we entered in the database. Example:

If you don’t recall Mr. Kotey, he was one of a UK quartet of ISIS terrorists who were seen in gruesome videos as they beheaded American hostages. The New York Times reported on his sentencing in 2022. British Terrorist Receives Life Sentence for Role in Americans’ Deaths

Kotey was sentenced to life in prison which he is serving in the Bureau of Prisons Supermax prison in Colorado.

Use Our Data!

This database, which can be downloaded as a CSV file, is licensed under Creative Commons for noncommercial uses with appropriate attribution. If you publish this database, in part or whole, you must credit Trevor Aaronson and Margot Williams.

When you open the .csv file in Excel or another spreadsheet program, it looks like this:

About This Database

This database of terrorism prosecutions and sentencing information was created using public records including three lists of prosecutions from the U.S. Department of Justice (from 2010, 2014, and 2015), court files available through the federal judiciary’s case management system, DOJ press releases, and inmate data from the Bureau of Prisons. For each defendant in the database, U.S. criminal code data related to charges has been categorized according to this legend.

Trevor Aaronson created the first iteration of this database as part of a project funded by the Investigative Reporting Program at the University of California, Berkeley. Mother Jones magazine published that data in 2011, along with accompanying articles, in a package that is still available online.

Beginning in 2016, Aaronson and Margot Williams collaborated to update and expand the database, with a new emphasis to include Bureau of Prisons data because so many post-9/11 terrorism defendants had been released. The cases include any prosecutions after September 11, 2001, that the U.S. government labeled as international terrorism-related. The Intercept first published this database on April 20, 2017, and it was last updated on June 14, 2023.

Who Did This?

Of course all the effort to get the project online, and to make the data usable to readers, was the work of many people, credited here:
Database compiled by Trevor Aaronson and Margot Williams
Data Visualization: Moiz Syed
Developers: Akil Harris, Westley Hennigh-Palermo
Editors: Sharon Weinberger, Andrea Jones, Roger Hodge, Eseosa Olumhense
Additional Research: Alleen Brown, Malak Habbak, John Thomason
Art Direction: Philipp Hubert, Soohee Cho
Product Manager: LJ Ruell
Research Director: Lynn Dombek
Editor-in-Chief: Betsy Reed

Thanks to all!

How We Built Trial and Terror

Type of Database

Trial and Terror’s database is hybrid of an existing database and a built database. It expands considerably on previously released lists of DOJ terrorism defendants by joining with original research and Bureau of Prisons data.

Sources

DOJ terrorism defendant lists, DOJ National Security Division press releases, U.S. District Court records, Bureau of Prisons inmate data

Tools

Research: ECF, RECAP, Docket Alarm
Analysis: R, Excel, Google Sheets, OpenRefine
Scrape: We built a tool to scrape BOP data daily and automatically update inmate information
Website: Javascript framework React with graphQL
Visualizations: D3, with Tachyons CSS library
Distribution: GitHub
Maintenance: Custom CMS allows researchers to update and add new defendants in real time.

Where did we find the list of people charged with terrorism?

We started with this data from the Department of Justice National Security Division. It is obviously a spreadsheet, but the DOJ made it into a PDF. To work with the data, we had to turn it back into a spreadsheet. As you can see, the text and numbers are hard to read and garbled. When we ran it through Optical Character Recognition (OCR) software, like Adobe Acrobat, the text and numbers were garbled, too, so we actually had to type it into Microsoft Excel or Google Sheets. This added significant time to the project, as it took months for me to do, along with my daily research in my role as Research Editor for Investigations at The Intercept.

How to follow cases in federal court

We used PACER (Public Access to Court Electronic Records) for searching dockets and downloading court filings. You need a (free) account to search for records. Then there are fees for printing results and downloading documents. Why? These are public federal court records and we have already paid our government taxes!

A Solution to the (Absurd) Expense for Federal Court Records

Courtlistener and RECAP are projects of the Free Law Project that provide free access to federal court dockets and documents that other RECAP users have downloaded. Use Courtlistener to look up cases and view dockets and download documents. If you add the RECAP extension to your Chrome browser, dockets and documents you have had to pay for will be added to the database so that they will be free to other searchers.

Bureau of Prisons data

Use the Bureau of Prisons Inmate Locator to find federal inmates incarcerated since 1982 and whether and when they were released. Investigative researcher note: if a federal prisoner doesn’t turn up here, there’s a mystery to unravel.

News and updates from Department of Justice

DOJ & National Security Division Press Releases

Back End to Custom Content Management System(CMS)

This application was created by Akil Harris. We use it to enter the information about each defendant charged with terrorism in federal courts. You can see, the information doesn’t magically appear. Trevor and I have to put it in there.

Filtered by “Released”

Scraping BOP for Updates

Akil wrote a program that finds the daily updates from the BOP database, and Trevor and I receive these updates in email.

Download the Database From GitHub

Our data is made publicly available on GitHub. This is where you can download the .csv file. The README file tells you more.

Stories that have used our data findings include:

Ideas For Using Our Data

Context for a Terrorism Prosecution in Your Area

The next time DOJ announces a terrorism-related arrest, the defendant may be from your city. Trial and Terror’s data will allow you give context: how many international terrorism prosecutions since 9/11 have there been in your district, how many involved stings, or how many involved the use of the controversial material support law.

Released Terrorists in Your Community

When DOJ brings terrorism charges, the cases get a lot of publicity. What’s gone largely unnoticed are the hundreds of people convicted on terrorism-related charges who have been released. These people could be your neighbors. Our data can help you determine if you have released “terrorists” in your community.

Roll Your Own Terrorism Data Project

Since our data is released under Creative Commons licensing, you can take our data and expand on it by populating new fields that would give your project an original angle or you can create a new interactive project or visualization.

We’ll Even Help You Steal Our Data

Are you pursuing journalism or academic research on this topic? DM us on Twitter at @trevoraaronson and @MargotWilliams if you want help in using the Trial and Terror data for your project.