About This Talk
When you have 60,000 tables and views across hundreds of schemata sitting on top of a 3 petabyte storage footprint, automation and access privileges are key if you want to expose these data through a RESTful API. Wharton Research Data Services (WRDS) leverages Python, Django, Django REST Framework, and PostgreSQL to accomplish this. We are continuing to open source the tools we have built to make this possible!
Databases stay relevant by continuing to reinvent themselves to serve new technologies further up the stack. The latest buzzwords further up that stack are APIs and microservices, so prevalent that it is hard to see a tech advertisement that doesn’t mention them. While related to “the cloud” and “big data”, whatever the heck those terms actually mean, APIs and microservices have a slightly less annoying marketing schtick and more concrete relations to relational databases.
But what do these relations look like in practice? In this talk, I will show how our team at The Wharton School has evolved from providing financial data exclusively in SAS data formats to a robust backend powered by PostgreSQL, which allows financial research to happen in many ecosystems: still available in SAS, but also R, Python, Perl, Matlab, Julia, and more. A case study will be shown that uses Django REST Framework to build an “API through introspection.” This case study will show how WRDS built a Django web site and RESTful web service by introspecting financial data stored in a PostgreSQL database cluster.
The models for the ORM, serializers for the RESTful API, views for presenting the data to a user, filters for refining queries, URL routing, web browsable interface, user token authorization, and permissions, are all handled by introspecting various features of the PostgreSQL database information schema, all with Python. Many of these components have been open-sourced, such as the ability to export an endpoint as a spreadsheet file (XLSX), and a generic automated API builder, called Automagic REST.
Tim joined Wharton Research Data Services in 2008, after working for local companies and starting a few of his own. Having started programming at age six, he is currently an organizer of the Philadelphia Python Users Group, DjangoCon US, and a member of the Python Software Foundation and Django Software Foundation. Tim is an avid hockey fan, was the first person to sell real world items for virtual microcurrency long before Bitcoin, and enjoys beating his head against brick walls (as demonstrated by his passion for coding, cycling in Philadelphia, and never-ending quest to get his cat to behave). A life-long Philadelphia resident, he graduated from the University of Pennsylvania in 1996 with an individualized major, “Computer Aided Information Acquisition”.