The Cost of Cleaning

We’ve frequently mentioned that people who work on data projects tell us that frequently, 80% of their projects are consumed by data preparation and cleaning, so it is interesting to get this data point from Kaggle: (2) How long is a typical project? When working with a top 0.5% data scientist, projects take just eight … Read more The Cost of Cleaning

Burglary Rhythm Maps

A Rhythm Map is a heat map that displays time in the X and Y dimensions. They are an excellent way to visualize repeating patterns in time, such as how crimes occur by hour and data of week. Here we look at some interesting patterns in burglaries in the City of San Diego. First, here is … Read more Burglary Rhythm Maps

SANDAG / ARJIS Crime File Archive

For the last 5 months, SANDAG has been publishing their crime incident data to the web. The file they publish only stores the last 180 days, and it is a bit hard to find, so we’re archiving the files to our data repository .

Day/Time Crime Heatmaps

Here is an interactive data application that explores how crime incidents vary over the day of week and time of day. In the checkbox below, select one or more crime types, and the heatmap will show the relative intensity of those crime types over day of week and time of day. There many interesting patterns … Read more Day/Time Crime Heatmaps

Why is Open Data Hard?

Last year, the U.S. Department of Housing and Urban Development commissioned a report to study the feasibility of creating a nationwide database of parcel information. This is a difficult task because the parcels are usually maintained by the counties, and the US has about 3200 counties. The resulting report is remarkably thorough, including a description of … Read more Why is Open Data Hard?

Q&A On Open Data

The Voice of San Diego is running a Q&A regarding Open Data, which just happens to involve an interview with the Director of the Library.