Inquiry
According to the open data portal, the Multi-Agency permits dataset curates data on DOB Jobs permits and DOHMH permits. This dataset is reported to be updated daily, and indeed, the DOB Jobs Permits data appears to be up to date with geography listed for most recent entries. However, when filtering down to the DOHMH data, only about half of the entries have any geography associated with them, and almost none of the entries for food establishment permits that expire after 2016 have any geography associated with them. In addition to this, there are many entries across the dataset that include issuance, expiration, or status dates listed as 01/01/1900.
After looking into this issue, we have come to understand that this is likely due to the way that entries are getting updated in the dataset. If there is already a row for the permit holder in the dataset, whenever the permit is updated, a new row is created in the dataset, but only the permit holder, the permit type, and the updated fields are added to the row. All of the other fields that do not change after the update are left blank, or in the case of dates, listed as 01/01/1990. For example, if you filter the dataset to the all of the permits for Harriet’s Kitchen Inc., you would see that there are 9 entries - one for each time the food establishment permit was issued. However, only the first entry has the establishment’s permit number, issuance date, address, BBL, BIN, coordinates, and permit holder’s name. For every other entry these fields are blank or listed as 01/01/1900. For these other entries, only the permit expiration date and permit status date field gets updated.
We looked into cleaning this on our own, but unfortunately we won’t be able to. We had considered downloading the data, grouping the rows with a common food establishment permit holder name, and filling in the missing fields on newer entries with the data contained in the first entry. However, individual chain stores are often all listed under the same permit name and can have thousands of entries with missing geographic information and unique permit numbers. For instance, Domino’s Pizza LLC has over a thousand entries in dataset, and less than half have a unique license number or geographic information. In this case, we can’t know which Domino’s the row is referring to, and so we do not know, which geographic information and license numbers should be added to which entries.
At BetaNYC, in collaboration with Manhattan Borough President Gale Brewer's office, we are trying to map where food establishments with active permits are located in the city. We are doing so because we've received requests from community boards and the Manhattan Borough President's staff to make this data more accessible. This is why it will be important to us to be able to access license numbers and expiration dates on the most recent food establishment entries.
Assigned Agencys Response
The agency is in the process of preparing the data for release. The expectation is to have the dataset on the Open Data portal by 09/30/2018. Please check back then.