The ability to write Python code instead of SQL can speed up web application development, especially at the beginning of a project. The potential development speed boost comes from not having to switch from Python code into writing declarative paradigm SQL statements. While some software developers may not mind switching back and forth between languages, it's typically easier to knock out a prototype or start a web application using a single programming language.
ORMs also make it theoretically possible to switch an application between various relational databases. For example, a developer could use SQLite for local development and MySQL in production. A production application could be switched from MySQL to PostgreSQL with minimal code modifications.
In practice however, it's best to use the same database for local development as is used in production. Otherwise unexpected errors could hit in production that were not seen in a local development environment. Also, it's rare that a project would switch from one database in production to another one unless there was a pressing reason.
Python ORM libraries are not required for accessing relational databases. In fact, the low-level access is typically provided by another library called a database connector, such as psycopg (for PostgreSQL) or MySQL-python (for MySQL). Take a look at the table below which shows how ORMs can work with different web frameworks and connectors and relational databases.
The above table shows for example that SQLAlchemy can work with varying web frameworks and database connectors. Developers can also use ORMs without a web framework, such as when creating a data analysis tool or a batch script without a user interface.
There are numerous downsides of ORMs, including
The phrase "impedance mismatch" is commonly used in conjunction with ORMs. Impedance mismatch is a catch-all term for the difficulties that occur when moving data between relational tables and application objects. The gist is that the way a developer uses objects is different from how data is stored and joined in relational tables.
This article on ORM impedance mismatch does a solid job of explaing what the concept is at a high level and provides diagrams to visualize why the problem occurs.
One of the concerns that's associated with any higher-level abstraction or framework is potential for reduced performance. With ORMs, the performance hit comes from the translation of application code into a corresponding SQL statement which may not be tuned properly.
ORMs are also often easy to try but difficult to master. For example, a
beginner using Django might not know about the
select_related()
function
and how it can improve some queries' foreign key relationship performance.
There are dozens of performance tips and tricks for every ORM. It's possible
that investing time in learning those quirks may be better spent just
learning SQL and how to write stored procedures.
There's a lot of hand-waving "may or may not" and "potential for" in this section. In large projects ORMs are good enough for roughly 80-90% of use cases but in 10-20% of a project's database interactions there can be major performance improvements by having a knowledgeable database administrator write tuned SQL statements to replace the ORM's generated SQL code.
The code for working with an application's data has to live somewhere. Before ORMs were common, database stored procedures were used to encapsulate the database logic. With an ORM, the data manipulation code instead lives within the application's Python codebase. The addition of data handling logic in the codebase generally isn't an issue with a sound application design, but it does increase the total amount of Python code instead of splitting code between the application and the database stored procedures.
There are numerous ORM implementations written in Python, including
There are other ORMs, such as Canonical's Storm, but most of them do not appear to currently be under active development. Learn more about the major active ORMs below.
The Django web framework comes with its own built-in object-relational mapping module, generally referred to as "the Django ORM" or "Django's ORM".
Django's ORM works well for simple and medium-complexity database operations. However, there are often complaints that the ORM makes complex queries much more complicated than writing straight SQL or using SQLAlchemy.
It's technically possible to drop down to SQL but it ties the queries to a specific database implementation. The ORM is coupled closely with Django so replacing the default ORM with SQLAlchemy is currently a hack workaround. Note though that some of the Django core committers believe it is only a matter of time before the default ORM is replaced with SQLAlchemy. It will be a large effort to get that working though so it's likely to come in Django 1.9 or later.
Since the majority of Django projects are tied to the default ORM, it's best to read up on advanced use cases and tools for doing your best work within the existing framework.
SQLAlchemy is a well-regarded Python ORM because it gets the abstraction level "just right" and seems to make complex database queries easier to write than the Django ORM in most cases. SQLAlchemy is typically used with Flask as the database ORM via the Flask-SQLAlchemy extension.
Peewee is a Python ORM written to be "simpler, smaller and more hackable" than SQLAlchemy. The analogy used by the core Peewee author is that Peewee is to SQLAlchemy as SQLite is to PostgreSQL. An ORM does not have to work for every exhaustive use case in order to be useful.
Pony ORM is another Python ORM with a slight twist in its licensing model. The project is multi-licensed. Pony is free for use on open source projects but has a commercial license that is required for commercial projects. The license is a one-time payment and does not necessitate a recurring fee.
SQLObject is an ORM that has been under active open source development since before 2003.
Schema migrations, for example when you need to add a new column to an existing table in your database, are not technically part of ORMs. However, since ORMs typically lead to a hands-off approach to the database (at the developers peril in many cases), libraries to perform schema migrations often go hand-in-hand with Python ORM usage on web application projects.
Database schema migrations are a complex topic and deserve their own page. For now, we'll lump schema migration resources under ORM links below.
There's also a detailed overview of what ORMs are on another page of the website.
This example GitHub project implements the same Flask application with several different ORMs: SQLAlchemy, Peewee, MongoEngine, stdnet and PonyORM.
Martin Fowler addresses the ORM hate in an essay about how ORMs are often misused but that they do provide benefits to developers.
If you're confused about the difference between a connector, such as MySQL-python and an ORM like SQLAlchemy, read this StackOverflow answer on the topic.
Django models, encapsulation and data integrity is a detailed article by Tom Christie on encapsulating Django models for data integrity.
Django Debug Toolbar is a powerful Django ORM database query inspection tool. Highly recommended during development to ensure you're writing reasonable query code. Django Silk is another inspection tool and has capabilities to do more than just SQL inspection.
Making a specific Django app faster is a Django performance blog post with some tips on measuring performance and optimizing based on the measured results.
Why I Hate the Django ORM is Alex Gaynor's overview of the bad designs decisions, some of which he made, while building the Django ORM.
Going Beyond Django ORM with Postgres is specific to using PostgreSQL with Django.
Migrating a Django app from MySQL to PostgreSQL is a quick look at how to move from MySQL to PostgreSQL. However, my guess is that any Django app that's been running for awhile on one relational database will require a lot more work to port over to another backend even with the power of the ORM.
Django Model Descriptors discusses and shows how to incorporate business logic into Django models to reduce complexity from the views and make the code easier to reuse across separate views.
Supporting both Django 1.7 and South explains the difficulty of supporting Django 1.7 and maintaining South migrations for Django 1.6 then goes into how it can be done.
Adding basic search to your Django site shows how to write generic queries that'll allow you to provide site search via the Django ORM without relying on another tool like ElasticSearch. This is great for small sites before you scale them up with a more robust search engine.
How to use Django's Proxy Models is a solid post on a Django ORM concept that doesn't frequently get a lot of love or explanation.
Tightening Django Admin Logins shows you how to log authentication failures, create an IP addresses white list and combine fail2ban with the authentication failures list.
Django Migrations - a Primer takes you through the new migrations system integrated in the Django core as of Django 1.7, looking specifically at a solid workflow that you can use for creating and applying migrations.
Django 1.7: Database Migrations Done Right explains why South was not directly integrated into Django, how migrations are built and shows how backwards migrations work.
Squashing and optimizing migrations in Django shows a simple example with code for how to use the migrations integrated into Django 1.7.
Sorting querysets with NULLs in Django shows what to do if you're struggling with the common issue of sorting columns that contain NULL values.
If you're interested in the differences between SQLAlchemy and the Django ORM I recommend reading SQLAlchemy and You by Armin Ronacher.
There is an entire chapter in the Architecture of Open Source Applications book on SQLAlchemy . The content is detailed and well worth reading to understand what's happening under the covers.
SQLAlchemy vs Other ORMs provides a detailed comparison of SQLAlchemy against alternatives.
Most Flask developers use SQLAlchemy as an ORM to relational databases. If you're unfamiliar with SQLAlchemy questions will often come up such as what's the difference between flush and commit? that are important to understand as you build out your app.
Managing database connections with Peewee explains the connection pool and ExecutionContext of the ORM.
An encrypted command-line diary with Python is an awesome walkthrough explaining how to use SQLite, SQLCipher and Peewee to create an encrypted file with your contents, diary or otherwise.
The official Peewee quickstart documentation along with the example Twitter clone app will walk you through the ins and outs of your first couple Peewee-powered projects.
Shortcomings in the Django ORM and a look at Peewee from the author of the Peewee ORM explains how some of the design decisions made in Peewee were in reaction to parts of the Django ORM that didn't work so well in practice.
How to make a Flask blog in one hour or less is a well written tutorial that uses the Peewee ORM instead of SQLAlchemy for the blog back end.
Why you should give Pony ORM a chance explains some of the benefits of Pony ORM that make it worth trying out.
An intro to Pony ORM shows the basics of how to use the library, such as creating databases and manipulating data.
The Pony ORM author explains on a Stack Overflow answer how Pony ORM works behind the scenes. Worth a read whether or not you're using the ORM just to find out how some of the magic coding works.
This post on Object-Relational Mapping with SQLObject explains the concept behind ORMs and shows the Python code for how they can be used.
Ian Bicking presented on SQLObject back in 2004 with a talk on SQLObject and Database Programming in Python.
PostgreSQL, often written as "Postgres" and pronounced "Poss-gres", is an open source relational database implementation frequently used by Python applications as a backed for data storage and retrieval.
PostgreSQL is the default database choice for many Python developers, including the Django team when testing the Django ORM. PostgreSQL is often viewed as more feature robust and stable when compared to MySQL, SQLServer and Oracle. All of those databases are reasonable choices. However, because PostgreSQL tends to be used by Python developers the drivers and example code for using the database tend to be better documented and contain fewer bugs for typical usage scenarios. If you try to use an Oracle database with Django, you'll see there is far less example code for that setup compared to PostgreSQL backend setups.
PostgreSQL's open source license allows developers to operate one or more databases without licensing cost in their applications. The open source license operating model is much less expensive compared to Oracle or other proprietary databases, especially as replication and sharding become necessary at large scale. In addition, because so many people ranging from independent developers to multinational organizations use PostgreSQL, it's often easier to find developers with PostgreSQL experience than other relational databases.
The PostgreSQL core team also releases frequent updates that greatly enhance the database's capabilities. For example, in the PostgreSQL 9.4 release the jsonb type was added to enhance JavaScript Object Notation (JSON) storage capabilities so that in many cases a separate NoSQL database is not required in an application's architecture.
To work with relational databases in Python you need to use a database driver, which is also referred to as a database connector. The most common driver library for working with PostgreSQL is psycopg2. There is a list of all drivers on the PostgreSQL wiki, including several libraries that are no longer maintained. If you're working with the asyncio Python stdlib module you should also take a look at the aiopg library which wraps psycopg2's asychronouos features together.
To abstract the connection between tables and objects, many Python developers use an object-relational mapper (ORM) with to turn relational data from PostgreSQL into objects that can be used in their Python application. For example, while PostgreSQL provides a relational database and psycopg is the common database connector, there are many ORMs that can be used with varying web frameworks, as shown in the table below.
Learn more about Python ORMs on that dedicated topic page.
If you're on Linux it's easy to get PostgreSQL installed using a package manager. However, once the database is installed and running your responsibility is just beginning. Before you go live with a production application, make sure to:
pg_hba.conf
file When possible have someone qualified do a PostgreSQL security audit to identify the biggest risks to your database. Small applications and bootstrapped companies often cannot afford a full audit in the beginning but as an application grows over time it becomes a bigger target.
The data stored in your database is the lifeblood of your application. If you have ever accidentally dropped a production database or been the victim of malicious activity such as SQL injection attacks, you'll know it's far easier to recover when a bit of work has been performed beforehand on backups, replication and security measures.
Many quickstarts and tutorials exist specifically for Django, Flask and other web application frameworks. The ones below are some of the best walkthroughs I've read.
Setting up PostgreSQL with Python 3 and psycopg on Ubuntu 16.04 provides instructions for getting a fresh Ubuntu install working with PostgreSQL and Python 3.
This post on using PostgreSQL with Django or Flask is a great quickstart guide for either framework.
This article explains how and why PostgreSQL can handle full text searching for many use cases. If you're going down this route, read this blog post that explains how one developer implemented PostgreSQL full text search with SQLAlchemy.
django-postgres-copy is a tool for bulk loading data into a PostgreSQL database based on Django models. Say hello to our new open-source software for loading bulk data into PostgreSQL is an introduction to using the tool in your own projects.
How to speed up tests in Django and PostgreSQL explains some hacks for making your schema migration-backed run quicker.
Full Text Search in Django using Database Back-Ends provides code for both PostgreSQL and MySQL for adding simple full text search into your application.
Records is a wrapper around the psycopg2 driver that allows easy access to direct SQL access. It's worth a look if you prefer writing SQL over using an ORM like SQLAlchemy.
PostgreSQL tutorials not specific to Python are also really helpful for properly handling your data.
PostgreSQL: The Nice Bits is a good overview slideshow of why PostgreSQL is a great relational database.
PostgreSQL Weekly is a weekly newsletter of PostgreSQL content from around the web.
Braintree wrote about their experiences scaling PostgreSQL. The post is an inside look at the evolution of Braintree's usage of the database.
This post estimates the costs of a PostgreSQL connection.
There is no such thing as total security but this IBM article covers hardening a PostgreSQL database.
Handling growth with Postgres provides 5 specific tips from Instagram's engineering team on how to scale the design of your PostgreSQL database.
Inserting And Using A New Record In Postgres shows some SQL equivalents to what many developers just do in their ORM of choice.
Following a Select Statement Through Postgres Internals provides a fascinating look into the internal workings of PostgreSQL during a query.
If you're just getting started with PostgreSQL here are 10 beginner tasks you should know how to execute.
The title's a bit presumptuous but here's a useful list of 7 PostgreSQL data migration hacks you should be using, but aren't.
awesome-postgres is a list of code libraries, tutorials and newsletters focused specifically on PostgreSQL.
While you can use a graphical interface for working with PostgreSQL, it's best to spend some time getting comfortable with the command-line interface.
Backing up databases is important because data loss can and does happen. This article explains how to back up a PostgreSQL database hosted on an Amazon Web Services EC2 instance if managing your own database on a cloud server is your preferred setup.
How to fix undead PostgreSQL queries shows a bit of a hack for when what to do when you can't kill certain PostgreSQL queries.
Is bi-directional replication (BDR) in PostgreSQL transactional? explores a relatively obscure topic with the final result that BDR is similar to data stores with eventual consistency rather than consistency as a requirement.
PostgreSQL-metrics is a tool built by Spotify's engineers that extracts and outputs metrics from an existing PostgreSQL database. There's also a way to extend the tools to pull custom metrics as well.
Creating a Document-Store Hybrid in Postgres 9.5 explains how to store and query JSON data, similar to how NoSQL data stores operate.
PostgreSQL Indexes: First Principles is a detailed look at what indexes are, what they are good for and how to use them in PostgreSQL.
This slideshow on high availability for web applications has a good overview of various database setups common in production web applications.
Monitoring one or more PostgreSQL instances and trying to performance tune them is a rare skillset. Here are some resources to get you started if you have to handle these issues in your applications.
This guide to PostgreSQL monitoring is handy for knowing what to measure and how to do it.
Craig Kerstiens wrote a detailed post about understanding PostgreSQL performance.
This article on performance tuning PostgreSQL covers how to find slow queries, tune indexes and modify your queries to run faster.
What PostgreSQL tells you about its performance explains how to gather general performance metrics and provides the exact queries you should run to get them. The article also covers performance monitoring and how to analyze trigger functions.
MySQL is an open source relational database implementation for storing and retrieving data.
MySQL is a viable open source database implementation for Python web applications. MySQL has a slightly easier initial learning curve than PostgreSQL. However, PostgreSQL's design is often preferred by Python web developers, especially when data migrations are run as an application evolves.
Accessing MySQL from a Python application requires a database driver (also called a "connector"). While it is possible to write a driver as part of your application, in practice most developers use an existing open source driver.
There was a major issue with MySQL drivers since the introduction of Python 3. One of the most popular libraries called MySQLdb did not work in its existing form with Python 3 and there were no plans to update it. Therefore a fork of MySQLdb named mysqlclient added Python 3 compatibility.
The mysqlclient fork was good in that existing MySQLdb users could drop mysqlclient into existing projects that were upgrading to Python 3. However, the fork often causes confusion when searching for which Python driver to use with MySQL. Many developer simply decide to use PostgreSQL because there is better support for Python drivers in the PostgreSQL community.
With that driver support context in mind, it's absolutely possible to build a Python 3 web application with MySQL as a backend. Here is a list of drivers along with whether it supports Python 2, 3 or both.
mysqlclient is a fork of MySQLdb that supports Python 2 and 3.
MySQL Connector is Oracle's "official" (Oracle currently owns MySQL) Python connector. The driver supports Python 2 and 3, just make sure to check the version guide for what releases work with which Python versions.
MySQLdb supports Python 2 and was frequently used by Python web applications before the mass migration to Python 3 began.
PyMySQL is a pure Python (no C low-level code) implementation that attempts to be a drop-in replacement for MySQLdb. However, some MySQL APIs are not supported by the driver so whether or not your application can use this connector will depend on what you're building.
The database is deployed in production at some of the highest trafficked sites such as Twitter, Facebook and many others major organizations. However, since MySQL AB, the company that developed MySQL, was purchased by Sun Microsystems (which was in turn purchased by Oracle), there have been major defections away from the database by Wikipedia and Google. MySQL remains a viable database option but I always recommend new Python developers learn PostgreSQL if they do not already know MySQL.
Python 3.4.0 with MySQL database and Python 3 and MySQL provide context for the commonly asked question about which database MySQL driver to use with Python 3.
Terrible Choices: MySQL is a blog post about specific deficiencies in MySQL's implementation that hinder its usage with Django's ORM.
Graph Data From MySQL Database in Python is an interesting study with code of how to pull data out of MySQL and graph the data with Plotly.
MySQL Python tutorial uses the MySQLdb driver to connect to a MySQL server instance and shows some examples for inserting and querying data.
28 Beginner's Tutorials for Learning about MySQL Databases is a curated collection on various introductory MySQL topics.
This tutorial shows how to install MySQL on Ubuntu.
Pinterest open sourced many of their MySQL tools to manage instances of the database.
Bye Bye MySQL & MongoDB, Guten Tag PostgreSQL goes into details for why the company Userlike migrated from their MySQL database setup to PostgreSQL.
Growing up with MySQL is a story about how one company went through dramatic growth and had to keep up with it by quickly scaling their MySQL database.
Monitoring MySQL metrics is the first of a three part series, with the other parts on collecting metrics and monitoring & collecting specifically with the DataDog tool. The series explains what metrics you should be collecting and monitoring in your production database along with the purpose for why those metrics are important.
SQLite is an open source relational database included with the Python standard library as of Python 2.5. The pysqlite database driver is also included with the standard library so that no further external dependencies are required to access a SQLite database from within Python applications.
sqlite3 - embedded relational database is an extensive tutorial showing many of the common create, read, update and delete operations a developer would want to do with SQLite.
A simple step-by-step SQLite tutorial walks through creating databases as well as inserting, updating, querying and deleting data.
My list of SQLite resources is a nice roundup of useful tools to use with SQLite and tutorials for learning more about the database.
Extending SQLite with Python uses the Peewee object-relational mapper (ORM) to implement virtual tables and aggregates on top of SQLite.
Using SQLite with Flask explains how Flask code can directly query a SQLite database without an ORM.
SQLite Browser is an open source graphical user interface for working with SQLite.
The official sqlite3 module in the Python stdlib docs contains a bunch of scenarios with code for how to use the database from a Python application.
Using the SQLite JSON1 and FTS5 Extensions with Python shows how to compile SQLite 3.9.0+ with json1 and fts5 (full-text search) support to use these new features.
SQLite with a fine-toothed comb digs into the internals of SQLite and shows some bugs found (and since fixed) while the author was researching the SQLite source code.
Application programming interfaces (APIs) provide machine-readable data transfer and signaling between applications.
HTML, CSS and JavaScript create human-readable webpages. However, those webpages are not easily consumable by other machines.
Numerous scraping programs and libraries exist to rip data out of HTML but it's simpler to consume data through APIs. For example, if you want the content of a news article it's easier to get the content through an API than to scrap the text out of the HTML.
There are several key concepts that get thrown around in the APIs world. It's best to understand these ideas first before diving into the API literature.
Representation State Transfer (REST)
Webhooks
JavaScript Object Notation (JSON) and Extensible Markup Language (XML)
Endpoints
A webhook is a user-defined HTTP callback to a URL that executes when a system condition is met. The call alerts the second system via a POST or GET request and often passes data as well.
Webhooks are important because they enable two-way communication initiation for APIs. Webhook flexibility comes in from their definition by the API user instead of the API itself.
For example, in the Twilio API when a text message is sent to a Twilio phone number Twilio sends an HTTP POST request webhook to the URL specified by the user. The URL is defined in a text box on the number's page on Twilio as shown below.
Zapier has an APIs 101 free guide for what APIs are, why they are valuable and how to use them properly.
What RESTful actually means does a fantastic job of laying out the REST principles in plain language terms while giving some history on how they came to be.

Simplicity and Utility, or, Why SOAP Lost provides context for why JSON-based web services are more common today than SOAP which was popular in the early 2000s.
API tools for every occasion provides a list of 10 tools that are really helpful when working with APIs that are new in 2015.
Learn the API concepts of machine-to-machine communication with JSON and XML, endpoints and webhooks.
Integrate an API such as Twilio or Stripe into your web application. Read the API integration section for more information.
Use a framework to create an API for your own application. Learn about web API frameworks on the API creation page.
Expose your web application's API so other applications can consume data you want to share.
The majority of production Python web applications rely on several externally hosted application programming interfaces (APIs). APIs are also commonly referred to as third party services or external platforms. Examples include Twilio for messaging and voice services, Stripe for payment processing and Disqus for embedded webpage comments.
There are many articles about proper API design but best practices for integrating APIs is less commonly written about. However, this subject continuously grows in importance because APIs provide critical functionality across many implementation areas.
Runscope is a service specifically designed for APIs that assists developers with automated testing and traffic inspection.
Apiary provides a blueprint for creating APIs so they are easier to test and generate clean documentation.
Some developers prefer to use Requests instead of an API's helper library. In that case check out this tutorial on using requests to access web APIs.
Product Hunt lists many commonly used commercial and free web APIs to show "there's an API for everything".
There's a list of all government web APIs at 18F's API-All-the-X list. The list is updated whenever a new API comes online.
If you use Requests check out this handy guide on gracefully handling HTTP errors with Python.
John Sheehan's "Zen and the Art of API Maintenance" slides are relevant for API integration.
This post on "API Driven Development" by Randall Degges explains how using APIs in your application cuts down on the amount of code you have to write and maintain so you can launch your application faster.
Safe Sex with Third Party APIs is a funny high level overview of what you should do to protect your application when relying on third party services.
Retries in Requests is a nice tutorial for easily re-executing failed HTTP requests with the Requests library.
My DjangoCon 2013 talk dove into "Making Django Play Nice With Third Party Services."
If you're looking for a fun project that uses two web APIs within a Django application, try out this tutorial to Build your own Pokédex with Django, MMS and PokéAPI.
vcr.py is a way to capture and replay HTTP requests with mocks. It's extremely useful for testing API integrations.
Caching external API requests is a good post on how to potentially limit the number of HTTP calls required when accessing an external web API via the Requests library.
Pick an API known for top notch documentation. Here's a list of ten APIs that are a good starting point for beginners.
Read the API documentation for your chosen API. Figure out a simple use case for how your application could be improved by using that API.
Before you start writing any code, play around with the API through the commandline with cURL or in the browser with Postman. This exercise will help you get a better understanding of API authentication and the data required for requests and responses.
Evaluate whether to use a helper library or work with Requests. Helper libraries are usually easier to get started with while Requests gives you more control over the HTTP calls.
Move your API calls into a task queue so they do not block the HTTP request-response cycle for your web application.
Creating and exposing APIs allows your web application to interact with other applications through machine-to-machine communication.
Django REST framework and Tastypie are the two most widely used API frameworks to use with Django. The edge currently goes to Django REST framework based on rough community sentiment. Django REST framework continues to knock out great releases after the 3.0 release mark when Tom Christie ran a successful Kickstarter campaign.
Flask-RESTful is widely used for creating web APIs with Flask. It was originally open sourced and explained in a blog post by Twilio then moved into its own GitHub organization so engineers from outside the company could be core contributors.
Flask API is another common library for exposing APIs from Flask web applications.
Sandman is a widely used tool to automatically generate a RESTful API service from a legacy database without writing a line of code (though it's easily extensible through code).
Cornice is a REST framework for Pyramid.
Restless is a lightweight API framework that aims to be framework agnostic. The general concept is that you can use the same API code for Django, Flask, Bottle, Pyramid or any other WSGI framework with minimal porting effort.
Eve is a Python REST framework built with Flask, MongoDB and Redis. The framework's primary author Nicola Iarocci gave a great talk at EuroPython 2014 that introduced the main features of the framework.
Falcon is a fast and lightweight framework well suited to create RESTful APIs.
Hug built on-top of Falcon and Python3 with an aim to make developing Python driven APIs as simple as possible, but no simpler. Hug leverages Python3 annotations to automatically validate and convert incoming and outgoing API parameters.
Pycnic is a JSON-API-only framework designed with REST in mind.
Building, running and maintaining APIs requires as much effort as building, running and maintaining a web application. API testing frameworks are the equivalent of browser testing in the web application world.
Runscope is an API testing SaaS application that can test both your own APIs and external APIs that your application relies upon.
API Science is focused on deep API testing, including multi-step API calls and monitoring of external APIs.
SmartBear has several API monitoring and testing tools for APIs.
An API is only as good as its documentation is a strongly held mantra in the web API world because so many APIs have poor documentation that prevents ease-of-use. If an API is not well documented then developers who have options to use something else will just skip it.
Adventures in running a free, public API is a quick story of a developer's geolocation API being abused and his lack of resources for preventing further abuse. Eventually he had to shut down the free plan and only provide a paid plan in addition to allowing others to host the open source code. Fraud and malware prevention are difficult problems so keep an eye on server utilization and endpoint calls growth to separate legitimate from illegitimate traffic.
API Doc JS allows a developer to embed markup in their documentation that will generate a site based on the endpoints available in the API.
10 Reasons Why Developers Hate Your API (And what to do about it) goes through the top difficulties and annoyances developers face when working with APIs and how you can avoid your API falling into the same traps.
Versioning of RESTful APIs is a difficult and contentious topic in the web API community. This two-part series covers various ways to version your API and how to architect a version-less API.
NARWHL is a practical API design site for developers confused about what is appropriate for RESTful APIs.
18F's API standards explains the details behind their design decisions on creating modern RESTful APIs.
Design a beautiful REST API reviews common design decisions regarding endpoints, versioning, errors and pagination. There is also a source material YouTube video where this blog post derives its recommendations from.
Move Fast, Don't Break Your API are slides and a detailed blog post from Amber Feng at Stripe about building an API, separating layers of responsibility, hiding backwards compatibility and a whole slew of other great advice for developers and API designers.
Self-descriptive, isn't. Don't assume anything. is an appeal that metadata makes a difference in whether APIs are descriptive or not.
Designing the Artsy API has their recommendations list for building an API based on their recent experiences.
Some REST Best Practices is a high level summary of rules to follow while creating your API.
Hacker News had a discussion on what's the best way to write an API spec? that provides a few different viewpoints on this topic.
Apigee's Web API Design ebook is free and contains a wealth of practical advice for what design decisions to make for your web API.
1-to-1 Relationships and Subresources in REST APIs tells the story of design decisions that were made during an API's creation and why those choices were made.
How many status codes does your API need? gives an answer from a Dropbox API developer as to their decision making process.
This API Design Guide is based on Heroku's best practices for the platform's API.
Choosing an API framework for Django by PyDanny contains questions and insight into what makes a good API framework and which one you should currently choose for Django.
Create a REST API in Minutes with Pyramid and Ramses is a thorough tutorial from start to finish that uses the Pyramid web framework along with Ramses, a library that uses YAML files to generate a RESTful API.
RESTful web services with Python is an interesting overview of the Python API frameworks space.
Implementing a RESTful Web API with Python & Flask is a good walkthrough for coding a Flask app that provides standard web API functionality such as proper HTTP responses, authentication and logging.
REST Hooks is an open source Python project that makes it easier to implement subscription-based "REST hooks". These REST hooks are similar to webhooks, but provide a different mechanism for subscribing to updates via a REST interface. Both REST hooks and webhooks are far more efficient than polling for updates and notifications.
Serialization is common for transforming objects into web API JSON results. One company found the serialization performance of Django REST framework was lacking so they created Serpy and wrote a blog post with the results of its performance.
Building better API docs shows how Square used Swagger with React to create more helpful docs.
This multi-part series on getting started with Django REST framework and AngularJS (part 1) along with its second part do a good job of showing how a RESTful API can serve as the backend for a client front end built with a JavaScript MVC framework.
If you're looking for a working example of a Django REST framework project, check out the PokeAPI, open sourced under the BSD license.
Pick an API framework appropriate for your web framework. For Django I recommend Django REST framework and for Flask I recommend Flask-RESTful.
Begin by building out a simple use case for the API. Generally the use case will either involve data that users want in a machine-readable format or a backend for alternative clients such as an iOS or Android mobile app.
Add an authentication mechanism through OAuth or a token scheme.
Add rate limiting to the API if data usage volume could be a performance issue. Also add basic metrics so you can determine how often the API is being accessed and whether it is performing properly.
Provide ample documentation and a walkthrough for how the API can be accessed and used.
Figure out other use cases and expand based on what you learned with the initial API use case.
Deployment involves packaging up your web application and putting it in a production environment that can run the app.
Your web application must live somewhere other than your own desktop or laptop. A production environment is the canonical version of your current application and its associated data.
Python web application deployments are comprised of many pieces that need to be individually configured. Here is a map that visually depicts how each deployment topic relates to each other. Click the image to pull up a PDF version.
There are four options for deploying and hosting a web application:
The first three options are similar. The deployer needs to provision one or more servers with a Linux distribution. System packages, a web server, WSGI server, database and the Python environment are then installed. Finally the application can be pulled from source and installed in the environment.
Note that there are other ways of installing a Python web application through system-specific package management systems. We won't cover those in this guide as they are considered advanced deployment techniques.
If you need a step-by-step guide to deploying a Python web application, I wrote a whole book on exactly this topic called The Full Stack Python Guide to Deployments that you'll find super helpful.
Deploying Python web applications is an episode of the great Talk Python to Me podcast series where I discuss deploying web applications based on a fairly traditional virtual private server, Nginx and Green Unicorn stack.
Thoughts on web application deployment walks through stages of deployment with source control, planning, continuous deployment and monitoring the results.
Deploying Software is a long must-read for understanding how to deploy software properly.
Practical continuous deployment defines delivery versus deployment and walks through a continuous deployment workflow.
In this free video by Neal Ford, he talks about engineering practices for continuous delivery. He explains the difference between continuous integration, continuous deployment and continuous delivery. Highly recommended for an overview of deployment concepts and as an introduction to the other videos on those subjects in that series.
Continuous deployment at Instagram is the story of how their deployment process evolved over time from a large Fabric script to continous deployments. Along the way they encountered issues with code reviews, test failures, canary builds and rollbacks. It's a great read that sheds some light on how Python deployments can be done well at large scale.
Stack Overflow's guide on how they do deployment is an awesome in-depth read covering topics ranging from git branching to database migrations.
If you're using Flask this detailed post on deploying it to Ubuntu is a great way to familiarize yourself with the deployment process.
If you're tight on time look at the platform-as-a-service (PaaS) options. You can deploy a low traffic project web app for free or low cost. You won't have to worry about setting up the operating system and web server compared to going the traditional server route. In theory you should be able to get your application live on the web sooner with PaaS hosting.
Traditional server options are your best bet for learning how the entire Python web stack works. You'll often save money with a virtual private server instead of a platform-as-a-service as you scale up.
Read about servers, operating systems, web servers and WSGI servers to get a broad picture of what components need to be set up to run a Python web application.
Servers are the physical infrastructure to run all the layers of software so your web application can respond to requests from clients such as web browsers.
Your web application must live somewhere other than your own desktop or laptop. Servers should ideally be accessible 24 hours a day, 7 days a week, with no unplanned downtime. The servers that host your web application for actual users (as opposed to test users) are known as production servers. Production servers hold real data (again as opposed to test data) and must be secure against unauthorized access.
The term bare metal refers to purchasing the actual hardware and hooking it up to the Internet either through a business-class internet service provider (ISP) or co-locating the server with other servers. A "business-class" ISP is necessary because most residential Internet service agreements explicitly prohibit running web servers on their networks. You may be able to get away with low traffic volume but if your site serves a lot of traffic it will alert an ISP's filters.
The bare metal option offers the most control over the server configuration, usually has the highest performance for the price, but also is the most expensive upfront option and the highest ongoing maintenance. With bare metal servers the ongoing operating cost is the electricity the server(s) use as well as handling repairs when server components malfunction. You're taking on manual labor working with hardware as well as the rest of the software stack.
Buy actual hardware from a vendor either pre-built or as a collection of components that you assemble yourself. You can also buy pre-configured servers from Dell or HP. Those servers tend to be in smaller case form factors (called "blades") but are correspondingly more expensive than putting off-the-shelf components together yourself in a standard computer case.
Virtual private servers (VPSs) are slices of hardware on top of a larger bare metal server. Virtualization software such as Xen and VMWare allow providers such as Linode and prgmr (as well as a many others) to provide fractions of a full server that appear as their own instances. For example, a server with an 8-core Xeon processor and 16 gigabytes of memory can be sliced into 8 pieces with the equivalent of 1-core and 2 gigabytes of memory.
The primary disadvantage of virtualized servers is that there is resource overhead in the virtualization process. In addition, physical constraints such as heavy I/O operations by a single virtualized instance on persistent storage can cause performance bottlenecks for other virtualized instances on the shared server. Choosing virtualized server hosting should be based on your needs for urgency of service ticket requests and the frequency you require for ongoing maintenance such as persistent storage backups.
Choosing a low cost VPS reviews the factors that you should weigh when deciding on hosting providers.
How to set up your Linode for maximum awesomeness shows how to work with a VPS once you've got the server up and running.
CPU Load Averages explains how to measure CPU load and what to do about it.
Infrastructure-as-a-service (IaaS) overlaps with virtualized servers because the resources are often presented in the same way. The difference between virtualized servers and IaaS is the granularity of the billing cycle. IaaS generally encourages a finer granularity based on minutes or hours of server usage instead of on monthly billing cycles.
IaaS can be used in combination with virtualized servers to provide dynamic upscaling for heavy traffic. When traffic is low then virtualized servers can solely be used. This combination of resources reduces cost at the expense of greater complexity in the dynamically scaled infrastructure.
The most common IaaS platforms are Amazon Web Services and Rackspace Cloud.
The disadvantage to IaaS platforms is the lock-in if you have to write custom code to deploy, dynamically scale, and generally understand your infrastructure. Every platform has its quirks. For example, Amazon's standard Elastic Block Store storage infrastructure has at least an order of magnitude worse I/O throughput than working with your local disk. Your application's database queries may work great locally but then when you deploy the performance is inadequate. Amazon has higher throughput EBS instances but you will pay correspondingly more for them. EBS throughput is just one of many quirks you need to understand before committing to an IaaS platform.
5 common server setups for your web application is a great introduction to how hosting can be arranged.
Apache Libcloud is a Python library that provides a unified API for many cloud service providers.
Amazon Web Services has official documentation for running Python web applications.
boto is an extensive and well-tested Python library for working with Amazon Web Services.
Poseidon is a Python commandline interface for managing Digital Ocean droplets (servers).
Sign up for a hosting provider. I recommend getting a Linode VPS to set up your initial infrastructure and deploy your web application there. Digital Ocean and prgrmr are other VPS options. You can change hosting providers later after the deployment process is automated.
Provision your first server. It will be ready but in a shutdown state while awaiting your instructions.
Move to the operating systems section to learn how to load Ubuntu 14.04 LTS as a base OS for Python web applications.
A platform-as-a-service (PaaS) provides infrastructure and a software layer on which a web application is deployed. Running your web application from a PaaS removes the need to know as much about the underlying servers, operating system, web server, and often the WSGI server.
Note: If you are not interested in deploying to a PaaS you can move ahead to the WSGI servers section.
The PaaS layer defines how the application accesses resources such as computing time, files, and external services. The PaaS provides a higher-level abstraction for working with computing resources than deploying an application to a server or IaaS.
A PaaS makes deployment and operations easier because it forces the developer
to conform applications to the PaaS architecture. For example, Heroku looks
for Python's requirements.txt
file in the base directory of the repository
during deployment because that is the file's de facto community standard
location.
If you go the PaaS route, you can skip configuring an operating system and web server prebaked into PaaS offerings. PaaS offerings generally start at the WSGI server layer.
Although PaaS offerings simplify setting up and maintaining the servers, operating system, and web server, developers still have responsibilities for other layers of their web stack.
While it's useful to know the operating system that underpins your PaaS, for example Heroku uses Ubuntu 10.04, you will not have to know as much about securing the operating system and server level. However, web applications deployed to a PaaS are just as vulnerable to security breaches at the application level as a standard LAMP stack. It's still your responsibility to ensure the web application framework and your app itself is up to date and secured. See the security section for further information.
PaaS bakeoff: Comparing Stackato, OpenShift, Dotcloud and Heroku for Django hosting and deployment by Nate Aune.
Deploying Django by Randall Degges is another great free resource about Heroku.
AWS in Plain English shows what current Amazon Web Services individual services are currently called and what they could've been called to be more clear to users.
5 AWS mistakes you should avoid explains how common beginner practices such as manually managing infrastructure, not using scaling groups and underutilizing instances can create problems you'd be better off avoiding altogether.
Heroku's
Python deployment documentation
provides clear examples for how to work with virtualenv, pip and
requirements.txt
to get a applications deployed to their platform.
Miguel Grinberg's Flask tutorial contains an entire post on deploying Flask applications to Heroku.
This series on DevOps Django by Randall Degges is great reading for using the Heroku service:
Deploying a Django App to AWS Elastic Beanstalk is a fantastic post that shows how to deploy to Amazon Web Service's own PaaS.
Deploy your hack in 3 steps: Intro to AWS and Elastic Beanstalk shows how to deploy a simple Ruby Sinatra app, but the steps are generally applicable to Python web apps as well.
Are you wondering what it will cost to deploy a reasonable sized production app on a platform-as-a-service like Heroku? Check out Cushion's transparent costs list where they include their expenses from using a PaaS as well as other services.
The beginner's guide to scaling to 11 million users on AWS is a useful list of services you'll need to look at as you grow an application from 10 to 100 to 1000 to 500,000 and beyond to millions of users.
How to Separate Your AWS Production and Development Accounts is a basic post on keeping developer sandbox accounts separate from production AWS environments.
How much is Spotify Paying Google Cloud? provides some insight into how Spotify runs all of their infrastructure on Google Cloud and posits what they may be paying to run their service.
Review the potential Python platform-as-a-service options listed above.
Sign up for a PaaS account at the provider that appears to best fit your application needs. Heroku is the PaaS option recommended for starters due to their detailed documentation and walkthroughs available on the web. However, the other options are also viable since their purpose is to make deploying applications as easy as possible.
Check if there are any PaaS-specific configuration files needed for your app to run properly on the PaaS after it is deployed.
Deploy your app to the PaaS.
Sync your application's configuration with the database.
Set up a content delivery network for your application's static content unless your PaaS provider already handles this deployment step for you.
Check if the application's functionality is working and tweak as necessary.
An operating system runs on the server or virtual server and controls access to computing resources. The operating system also includes a way to install programs necessary for running your Python web application.
An operating system makes many of the computing tasks we take for granted easy. For example, the operating system enables writing to files, communicating over a network and running multiple programs at once. Otherwise you'd need to control the CPU, memory, network, graphics card, and many other components with your own low-level implementation.
Without using an existing operating system like Linux, Mac OS X or Windows, you'd be forced to write a new operating system as part of your web application. It would be impossible to write features for your Python web application because you'd be too busy hunting down a memory leak in your assembly code, if you even were able to get that far.
Fortunately, the open source community provides Linux to the Python world as a rock solid free operating system for running our applications.
The only recommended operating system for production Python web stack deployments is Linux. There are several Linux distributions commonly used for running production servers. Ubuntu Long Term Support (LTS) releases, Red Hat Enterprise Linux, and CentOS are all viable options.
Mac OS X is fine for development activities. Windows and Mac OS X are not appropriate for production deployments unless there is a major reason why you must use them in lieu of Linux.
Ubuntu is a Linux distribution packaged by the Canonical Ltd company. Ubuntu uses the Debian distribution as a base for packages, including the aptitude package manager. For desktop versions of Ubuntu, GNOME (until the 11.04 release) or Unity (11.10 through current) is bundled with the distribution to provide a user interface.
Ubuntu Long Term Support (LTS) releases are the recommended versions to use for deployments. LTS versions receive five years of post-release updates from Canonical. Every two years, Canonical creates a new LTS release, which allows for an easy upgrade path as well as flexibility in skipping every other LTS release if necessary. As of April 2016, 16.04 Xenial Xerus is the latest Ubuntu LTS release. Xenial Xerus includes Python 3.5 as its default Python version, which is a major update compared with 2.7 in Ubuntu 14.04 LTS.
There are several Aptitude packages found on Linux servers running a Python stack. These packages are:
python-dev for header files and static library for Python
python-virtualenv for creating and managing Python virtualenvs to isolate library dependencies
Red Hat Enterprise Linux (RHEL) and Community ENTerprise Operating System (CentOS) are the same distribution. The primary difference between the two is that CentOS is an open source, liberally licensed free derivative of RHEL.
RHEL and CentOS use a different package manager and command-line interface from Debian-based Linux distributions: RPM Package Manager (RPM) and the Yellowdog Updater, Modified (YUM). RPM has a specific .rpm file format to handle the packaging and installation of libraries and applications. YUM provides a command-line interface for interacting with the RPM system.
What is a Linux distribution and how do I choose the right one?
Lifehacker's guide to choosing a Linux distro.
The Ops School curriculum is a comprehensive resource for learning about Linux fundamentals and how to perform the work that system administrators typically handle.
Since Linux is your go-to production operating system, it's important to get comfortable with the Unix/Linux commands and philosophy. Study up on this introduction to Unix tutorial to become more familiar with the operating system.
First 5 Minutes on a Server shows the first several security steps that should be done manually or automatically on any server you stand up.
Digital Ocean has a detailed walkthrough for setting up Python web applications on Ubuntu.
linux-internals is a series of posts about how Linux works under the covers, starting from the low level booting process.
Choose either a Debian-based Linux distribution such as Ubuntu or a Fedora-based distribution like CentOS.
Harden the security through a few basic steps. Install basic security packages such as fail2ban and unattended-upgrades. Create a new user account with sudo privileges and disable root logins. Disable password-only logins and use a public-private keypair instead. Read more about hardening systems in the resources listed below.
Install Python-specific packages to prepare the environment for running a Python application. Which packages you'll need to install depends on the distribution you've selected.
Read up on web servers as installing one will be the next step in the deployment process.
Web servers respond to Hypertext Transfer Protocol (HTTP) requests from clients and send back a response containing a status code and often content such as HTML, XML or JSON as well.
Web servers are the ying to the web client's yang. The server and client speak the standardized language of the World Wide Web. This standard language is why an old Mozilla Netscape browser can still talk to a modern Apache or Nginx web server, even if it cannot properly render the page design like a modern web browser can.
The basic language of the Web with the request and response cycle from client to server then server back to client remains the same as it was when the Web was invented by Tim Berners-Lee at CERN in 1989. Modern browsers and web servers have simply extended the language of the Web to incorporate new standards.
The conceptual web server idea can be implemented in various ways. The following web server implementations each have varying features, extensions and configurations.
The Apache HTTP Server has been the most commonly deployed web server on the Internet for 20+ years.
Nginx is the second most commonly used server for the top 100,000 websites and often serves as a reverse proxy for Python WSGI servers.
Caddy is a newcomer to the web server scene and is focused on serving the HTTP/2 protocol with HTTPS.
A client that sends a request to a web server is usually a browser such as Internet Explorer, Firefox, or Chrome, but it can also be a
Web servers process requests from the above clients. The result of the web server's processing is a response code and commonly a content response. Some status codes, such as 204 (No content) and 403 (Forbidden), do not have content responses.
In a simple case, the client will request a static asset such as a picture or JavaScript file. The file sits on the file system in a location the web server is authorized to access and the web server sends the file to the client with a 200 status code. If the client already requested the file and the file has not changed, the web server will pass back a 304 "Not modified" response indicating the client already has the latest version of that file.
A web server sends files to a web browser based on the web browser's request. In the first request, the browser accessed the "www.fullstackpython.com" address and the server responded with the index.html HTML-formatted file. That HTML file contained references to other files, such as style.css and script.js that the browser then requested from the server.
Sending static assets (such as CSS and JavaScript files) can eat up a large amount of bandwidth which is why using a Content Delivery Network (CDN) to serve static assets is important when possible.
A reference with the full list of HTTP status codes is provided by W3C.
If you're looking to learn about web servers by building one, here's part one, part two and part three of a great tutorial that shows how to code a web server in Python.
rwasa is a newly released web server written in Assembly with no external dependencies that tuned to be faster than Nginx. The benchmarks are worth taking a look at to see if this server could fit your needs if you need the fastest performance trading off for as of yet untested web server.
Choose a web server. Nginx is often recommended although Apache is also a great choice.
Create an SSL certificate. For testing use a self-signed certificate and for a production app buy one from Digicert. Configure the web server to serve traffic over SSL. You'll need SSL for serving only HTTPS traffic and preventing security issues that occur with unencrypted user input.
Configure the web server to serve up static files such as CSS, JavaScript and images.
Once you set up the WSGI server you'll need to configure the web server as a pass through for dynamic content.
A Web Server Gateway Interface (WSGI) server implements the web server side of the WSGI interface for running Python web applications.
A traditional web server does not understand or have any way to run Python applications. In the late 1990s, a developer named Grisha Trubetskoy came up with an Apache module called mod_python to execute arbitrary Python code. For several years in the late 1990s and early 2000s, Apache configured with mod_python ran most Python web applications.
However, mod_python wasn't a standard specification. It was just an implementation that allowed Python code to run on a server. As mod_python's development stalled and security vulnerabilities were discovered there was recognition by the community that a consistent way to execute Python code for web applications was needed.
Therefore the Python community came up with WSGI as a standard interface that modules and containers could implement. WSGI is now the accepted approach for running Python web applications.
As shown in the above diagram, a WSGI server simply invokes a callable object on the WSGI application as defined by the PEP 3333 standard.
Why use WSGI and not just point a web server directly at an application?
WSGI gives you flexibility. Application developers can swap out web stack components for others. For example, a developer can switch from Green Unicorn to uWSGI without modifying the application or framework that implements WSGI. From PEP 3333:
The availability and widespread use of such an API in web servers for Python [...] would separate choice of framework from choice of web server, freeing users to choose a pairing that suits them, while freeing framework and server developers to focus on their preferred area of specialization.
WSGI servers promote scaling. Serving thousands of requests for dynamic content at once is the domain of WSGI servers, not frameworks. WSGI servers handle processing requests from the web server and deciding how to communicate those requests to an application framework's process. The segregation of responsibilities is important for efficiently scaling web traffic.
WSGI is by design a simple standard interface for running Python code. As a web developer you won't need to know much more than
what WSGI stands for (Web Server Gateway Inteface)
that a WSGI container is a separate running process that runs on a different port than your web server
your web server is configured to pass requests to the WSGI container which runs your web application, then pass the response (in the form of HTML) back to the requester
If you're using a standard web framework such as Django, Flask, or Bottle, or almost any other current Python framework, you don't need to worry about how frameworks implement the application side of the WSGI standard. Likewise, if you're using a standard WSGI container such as Green Unicorn, uWSGI, mod_wsgi, or gevent, you can get them running without worrying about how they implement the WSGI standard.
However, knowing the WSGI standard and how these frameworks and containers implement WSGI should be on your learning checklist though as you become a more experienced Python web developer.
The WSGI standard v1.0 is specified in PEP 0333. As of September 2010, WSGI v1.0 is superseded by PEP 3333, which defines the v1.0.1 WSGI standard. If you're working with Python 2.x and you're compliant with PEP 0333, then you're also compliant with 3333. The newer version is simply an update for Python 3 and has instructions for how unicode should be handled.
wsgiref in Python 2.x and wsgiref in Python 3.x are the reference implementations of the WSGI specification built into Python's standard library so it can be used to build WSGI servers and applications.
A web server's configuration specifies what requests should be passed to the WSGI server to process. Once a request is processed and generated by the WSGI server, the response is passed back through the web server and onto the browser.
For example, this Nginx web server's configuration specifies that Nginx should handle static assets (such as images, JavaScript, and CSS files) under the /static directory and pass all other requests to the WSGI server running on port 8000:
# this specifies that there is a WSGI server running on port 8000
upstream app_server_djangoapp {
server localhost:8000 fail_timeout=0;
}
# Nginx is set up to run on the standard HTTP port and listen for requests
server {
listen 80;
# nginx should serve up static files and never send to the WSGI server
location /static {
autoindex on;
alias /srv/www/assets;
}
# requests that do not fall under /static are passed on to the WSGI
# server that was specified above running on port 8000
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://app_server_djangoapp;
break;
}
}
}
Note that the above code is a simplified version of a production-ready Nginx configuration. For real SSL and non-SSL templates, take a look at the Underwear web server templates on GitHub.
There is a comprehensive list of WSGI servers on the WSGI Read the Docs page. The following are WSGI servers based on community recommendations.
Green Unicorn is a pre-fork worker model based server ported from the Ruby Unicorn project.
uWSGI is gaining steam as a highly-performant WSGI server implementation.
mod_wsgi is an Apache module implementing the WSGI specification.
CherryPy is a pure Python web server that also functions as a WSGI server.
PEP 0333 WSGI v1.0 and PEP 3333 WSGI v1.0.1 specifications.
This basics of WSGI post contains a simple example of how a WSGI-compatible application works.
A comparison of web servers for Python web apps is a good read to understand basic information about various WSGI server implementations.
A thorough and informative post for LAMP-stack hosting choices is presented in the "complete single server Django stack tutorial."
The Python community made a long effort to transition from mod_python to the WSGI standard. That transition period is now complete and an implementation of WSGI should always be used instead mod_python.
Nicholas Piël wrote an interesting benchmark blog post of Python WSGI servers. Note that the post is a few years old. Benchmarks should be considered for their specific tested scenarios and not quickly extrapolated as general "this server is faster than this other server" results.
How to Deploy Python WSGI Applications with CherryPy answers why CherryPy is a simple combination web and WSGI server along with how to use it.
Another Digital Ocean walkthrough goes into How to Deploy Python WSGI Apps Using Gunicorn HTTP Server Behind Nginx.
The uWSGI Swiss Army Knife shows how uWSGI can potentially be used for more than just running the Python web application - it can also serve static files and handle caching in a deployment.
Understand that WSGI is a standard Python specification for applications and servers to implement.
Pick a WSGI server based on available documentation and tutorials. Green Unicorn is a good one to start with since it's been around for awhile.
Add the WSGI server to your server deployment.
Configure the web server to pass requests to the WSGI server for appropriate URL patterns.
Test that the WSGI server responds to local requests but not direct requests outside your infrastructure. The web server should be the pass through for requests to and responses from the WSGI server.
Source control, also known as version control, stores software code files with a detailed history of every modification made to those files.
Version control systems allow developers to modify code without worrying about permanently screwing something up. Unwanted changes can be easily rolled back to previous working versions of the code.
Source control also makes team software development easier. One developer can combine her code modifications with other developers' code through diff views that show line-by-line changes then merge the appropriate code into the main code branch.
Version control is a necessity on all software projects regardless of development time, codebase size or the programming language used. Every project should immediately begin by using a version control system such as Git or Mercurial.
Pulling code during a deployment is a potential way source control systems fit into the deployment process.
Note that some developers recommend deployment pipelines package the source code to deploy it and never have a production environment touch a source control system directly. However, for small scale deployments it's often easiest to pull from source code when you're getting started instead of figuring out how to wrap the Python code in a system installation package.
Numerous source control systems have been created over the past several decades. In the past, proprietary source control software offered features tailored to large development teams and specific project workflows. However, open source systems are now used for version control on the largest and most complicated software projects in existence. There's no reason why your project should use anything other than an open source version control system in today's Python development world. The two primary choices are:
Git is a free and open source distributed version control system.
Mercurial is similar to Git, also a free and open source distributed version control system.
Git and Mercurial can be downloaded and run on your own server. However, it's easy and cheap to get started with a hosted version control service. You can transition away from the service at a later time by moving your repositories if your needs change. A couple of recommended hosted version control services are:
GitLab has both a self-hosted version of its open source software as well as their hosted version with pricing for businesses that need additional hosting support.
GitHub provides free open source repositories and paid private repositories for Git.
BitBucket also has free Git and Mercurial repositories for open projects, but adds private repositories for up to five users. Users pay for hosting private repositories with more than five users.
Staging Servers, Source Control & Deploy Workflows, And Other Stuff Nobody Teaches You is a comprehensive overview by Patrick McKenzie of why you need source control.
Version control best practices is a good write up of how to work with version control systems. The post is part of an ongoing deployment guide written by the folks at Rainforest.
This lighthearted guide to the ten astonishments in version control history is a fun way to learn how systems developed over the past several decades.
A visual guide to version control is a detailed article with real-life examples for why version control is necessary in software development.
An introduction to version control shows the basic concepts behind version control systems.
What Is Version Control? Why Is It Important For Due Diligence? explains the benefits and necessity of version control systems.
About version control reviews the basics of distributed version control systems.
Pro Git is a free open source book that walks through all aspects of using the version control system.
Git in Six Hundred Words is a clear and concise essay explaining the fundamental concepts of Git.
A Hacker's Guide to Git covers the basics as well as more advanced Git commands while explaining each step along the way.
Think like a Git is another introduction that focuses more on the graph theory and conceptual ideas behind Git to help the reader understand what's happening as they use Git commands.
A practical git introduction is exactly what the title says it is. This is a well written guide with plenty of code snippets to get you up to speed with Git.
Git from the inside out demonstrates how Git's graph-based data structure produces certain behavior through example Git commands. This is a highly recommended read after you've grasped the basics and are looking to go deeper with Git.
git ready has a nice collection of blog posts based on beginner, intermediate and advanced Git use cases.
git-flow details a Git branching model for small teams.
GitHub Flow builds on git-flow, goes over some of the issues that arise with it and presents a few solutions to those problems.
Git Workflows That Work is a helpful post with diagrams to show how teams can create a Git workflow that will help their development process.
"Our Git Workflow" by Braintree goes over how this payments company uses Git for development and merging source code.
Code Sleuthing with Git shows how to review past changes when a deployment goes wrong to figure out what the heck happened.
Pick a version control system. Git is recommended because on the web there are a significant number of tutorials to help both new and advanced users.
Learn basic use cases for version control such as committing changes, rolling back to earlier file versions and searching for when lines of code were modified during development history.
Ensure your source code is backed up in a central repository. A central repository is critical not only if your local development version is corrupted but also for the deployment process.
Integrate source control into your deployment process in three ways. First, pull the project source code from version control during deployments. Second, kick off deployments when code is modified by using webhooks or polling on the repository. Third, ensure you can roll back to a previous version if a code deployment goes wrong.

Python web applications are built upon the work done by thousands of open source programmers. Application dependencies include not only web frameworks but also libraries for scraping, parsing, processing, analyzing, visualizing, and many other tasks. Python's ecosystem facilitates discovery, retrieval and installation so applications are easier for developers to create.
Python libraries are stored in a central location known as the Python Package Index (PyPi). PyPi contains search functionality with results weighted by usage and relevance based on keyword terms.
Besides PyPi there are numerous resources that list common or "must-have" libraries. Ultimately the decision for which application dependencies are necessary for your project is up to you and the functionality you're looking to build. However, it's useful to browse through these lists in case you come across a library to solve a problem by reusing the code instead of writing it all yourself. A few of the best collections of Python libraries are
Python.org's useful modules which groups modules into categories.
GitHub Explore Trending repositories shows the open source Python projects trending today, this week, and this month.
This list of 20 Python libraries you can’t live without is a wide-ranging collection from data analysis to testing tools.
Wikipedia actually has an extensive page dedicated to Python libraries grouped by categories.
Dependencies are installed separately from system-level packages to prevent library version conflicts. The most common isolation method is virtualenv. Each virtualenv is its own copy of the Python interpreter and dependencies in the site-packages directory. To use a virtualenv it must first be created with the virtualenv command and then activated.
The virtualenv stores dependencies in an isolated environment. The web application then relies only on that virtualenv instance which has a separate copy of the Python interpreter and site-packages directory. A high level of how a server configured with virtualenv can look is shown in the picture below.
The recommended way to install Python library dependencies is with the pip command when a virtualenv is activated.
Pip and virtualenv work together and have complementary responsibilities. Pip downloads and installs application dependencies from the central PyPi repository.
The pip convention for specifying application dependencies is with a
requirements.txt
file. When you build a Python web application you
should include requirements.txt
in the base directory of your project.
Python projects' dependencies for a web application should be specified with pegged dependencies like the following:
django==1.6
bpython==0.12
django-braces==0.2.1
django-model-utils==1.1.0
logutils==0.3.3
South==0.7.6
requests==1.2.0
stripe==1.9.1
dj-database-url==0.2.1
django-oauth2-provider==0.2.4
djangorestframework==2.3.1
Pegged dependencies with precise version numbers or Git tags are important because otherwise the latest version of a dependency will be used. While it may sound good to stay up to date, there's no telling if your application actually works with the latest versions of all dependencies. Developers should deliberately upgrade and test to make sure there were no backwards-incompatible modifications in newer dependency library versions.
There is another type of dependency specification for Python libraries known as setup.py. Setup.py is a standard for distributing and installing Python libraries. If you're building a Python library, such as requests or underwear you must include setup.py so a dependency manager can correctly install both the library as well as additional dependencies for the library. There's still quite a bit of confusion in the Python community over the difference between requirements.txt and setup.py, so read this well written post for further clarification.
Jon Chu wrote a great introduction on virtualenv and pip basics.
A non-magical introduction to virtualenv and pip breaks down what problems these tools solve and how to use them.
Tools of the modern Python hacker contains detailed explanations of virtualenv, Fabric, and pip.
Occasionally arguments about using Python's dependency manager versus one of Linux's dependency managers comes up. This provides one perspective on that debate.
Open source trust scaling is a good piece for the Python community (and other communities) that is based on the left-pad NPM situation that broke many dependent packages in the Node.JS community.
This Stack Overflow question details how to set up a virtual environment for Python development.
Another Stack Overflow page answers how to set environment variables when using virtualenv.
Tips for using pip + virtualenv + virtualenvwrapper shows how to use shell aliases and postactivate virtualenvwrapper hooks to make life easier when using these tools.
Major speed improvements were made in pip 7 over previous versions. Read this article about the differences and be sure to upgrade.
How to submit a package to PyPI is a short and sweet introduction that'll help you quickly get your first package on PyPI.
Autoenv is a tool for activating
environment variables stored in a .env
file in your projects' home
directories. Environment variables aren't managed by virtualenv and although
virtualenvwrapper has some hooks for handling them, it's often easiest to
use a shell script or .env
file to set them in a development environment.
Pipreqs searches through a project for
dependencies based on imports. It then generates a requirements.txt
file
based on the libraries necessary to run those dependencies. Note though that
while this could come in handy with a legacy project, the version numbers
for those libraries will not be generated with the output.
Ensure the libraries your web application depends on are all captured in a requirement.txt file with pegged versions.
An easy way to capture currently installed dependencies is with the
pip freeze
command.
Create a fresh virtualenv and install the dependencies from your
requirements.txt
file by using the pip install -r requirements.txt
command.
Check that your application runs properly with the fresh virtualenv and
only the installed dependencies from the requirements.txt
file.
Some content on a website does not change and therefore should be served up either directly through the web server or a content delivery network (CDN). Examples include JavaScript, image, and CSS files.
Static content can be either assets created as part of your development process such as images on your landing page or user-generated content. The Django framework calls these two categories assets and media.
A content delivery network (CDN) is a third party that stores and serves static files. Amazon CloudFront, Akamai, and Rackspace Cloud Files are examples of CDNs. The purpose of a CDN is to remove the load of static file requests from web servers that are handling dynamic web content. For example, if you have an nginx server that handles both static files and acts as a front for a Green Unicorn WSGI server on a 512 megabyte virtual private server, the nginx server will run into resource constraints under heavy traffic. A CDN can remove the need to serve static assets from that nginx server so it can purely act as a pass through for requests to the Green Unicorn WSGI server.
CDNs send content responses from data centers with the closest proximity to the requester.
The super stupid idiot's guide to getting started with Django, Pipeline, and S3 shows how to host static content on S3 and use those files with Django.
Crushing, caching and CDN deployment in Django shows how to use django-compressor and a CDN to scale static and media file serving.
Uploading with Django and Amazon S3 walks through each step in getting buckets set up so you can upload files to them via Django.
django-storages is a Django library for managing static and media files on services such as Amazon S3 and other content delivery networks.
RevSys has a nice article on a range of important static file optimizations such as setting cache headers, optimizing JavaScript and reducing the size of images.
Twelve folks with significant experience working on and with CDNs provide their perspectives in this piece: CDN experts on CDNs.
Identify a content delivery network to offload serving static content files from your local web server. I recommend using Amazon S3 with CloudFront as it's easy to set up and will scale to high bandwidth demands.
Update your web application deployment process so updated static files are uploaded to the CDN.
Move static content serving from the www subdomain to a static (or similarly named) subdomain so browsers will load static content in parallel to www HTTP requests.
Task queues manage background work that must be executed outside the usual HTTP request-response cycle.
Tasks are handled asynchronously either because they are not initiated by an HTTP request or because they are long-running jobs that would dramatically reduce the performance of an HTTP response.
For example, a web application could poll the GitHub API every 10 minutes to collect the names of the top 100 starred repositories. A task queue would handle invoking code to call the GitHub API, process the results and store them in a persistent database for later use.
Another example is when a database query would take too long during the HTTP request-response cycle. The query could be performed in the background on a fixed interval with the results stored in the database. When an HTTP request comes in that needs those results a query would simply fetch the precalculated result instead of re-executing the longer query. This precalculation scenario is a form of caching enabled by task queues.
Other types of jobs for task queues include
spreading out large numbers of independent database inserts over time instead of inserting everything at once
aggregating collected data values on a fixed interval, such as every 15 minutes
scheduling periodic jobs such as batch processes
The defacto standard Python task queue is Celery. The other task queue projects that arise tend to come from the perspective that Celery is overly complicated for simple use cases. My recommendation is to put the effort into Celery's reasonable learning curve as it is worth the time it takes to understand how to use the project.
The Celery distributed task queue is the most commonly used Python library for handling asynchronous tasks and scheduling.
The RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. RQ is backed by Redis and is designed to have a low barrier to entry. The intro post contains information on design decisions and how to use RQ.
Taskmaster is a lightweight simple distributed queue for handling large volumes of one-off tasks.
Huey is a simple task queue that uses Redis on the backend but otherwise does not depend on other libraries. The project was previously known as Invoker and the author changed the name.
Huey is a Redis-based task queue that aims to provide a simple, yet flexible framework for executing tasks. Huey supports task scheduling, crontab-like repeating tasks, result storage and automatic retry in the event of failure.
Task queue third party services aim to solve the complexity issues that arise when scaling out a large deployment of distributed task queues.
Iron.io is a distributed messaging service platform that works with many types of task queues such as Celery. It also is built to work with other IaaS and PaaS environments such as Amazon Web Services and Heroku.
Amazon Simple Queue Service (SQS) is a set of five APIs for creating, sending, receiving, modifying and deleting messages.
CloudAMQP is at its core managed servers with RabbitMQ installed and configured. This service is an option if you are using RabbitMQ and do not want to maintain RabbitMQ installations on your own servers.
Take a look at the code in this open source Flask application and this Django application for examples of how to use and deploy Celery with a Redis broker to send text messages with these frameworks.
flask-celery-example is a simple Flask application with Celery as a task queue and Redis as the broker.
Getting Started Scheduling Tasks with Celery is a detailed walkthrough for setting up Celery with Django (although Celery can also be used without a problem with other frameworks).
International Space Station notifications with Python and Redis Queue (RQ) shows how to combine the RQ task queue library with Flask to send text message notifications every time a condition is met - in this blog post's case that the ISS is currently flying over your location on Earth.
Evaluating persistent, replicated message queues is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and Kafka's designs and performance.
Queues.io is a collection of task queue systems with short summaries for each one. The task queues are not all compatible with Python but ones that work with it are tagged with the "Python" keyword.
Why Task Queues is a presentation for what task queues are and why they are needed.
Flask by Example Implementing a Redis Task Queue provides a detailed walkthrough of setting up workers to use RQ with Redis.
How to use Celery with RabbitMQ is a detailed walkthrough for using these tools on an Ubuntu VPS.
Heroku has a clear walkthrough for using RQ for background tasks.
Introducing Celery for Python+Django provides an introduction to the Celery task queue.
Celery - Best Practices explains things you should not do with Celery and shows some underused features for making task queues easier to work with.
The "Django in Production" series by Rob Golding contains a post specifically on Background Tasks.
Asynchronous Processing in Web Applications Part One and Part Two are great reads for understanding the difference between a task queue and why you shouldn't use your database as one.
Celery in Production on the Caktus Group blog contains good practices from their experience using Celery with RabbitMQ, monitoring tools and other aspects not often discussed in existing documentation.
A 4 Minute Intro to Celery is a short introductory task queue screencast.
Heroku wrote about how to secure Celery when tasks are otherwise sent over unencrypted networks.
Miguel Grinberg wrote a nice post on using the task queue Celery with Flask. He gives an overview of Celery followed by specific code to set up the task queue and integrate it with Flask.
3 Gotchas for Working with Celery are things to keep in mind when you're new to the Celery task queue implementation.
Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis is a video along with code that shows how to set up Celery with Redis as the broker in a Django application.
Setting up an asynchronous task queue for Django using Celery and Redis is a straightforward tutorial for setting up the Celery task queue for Django web applications using the Redis broker on the back end.
Background jobs with Django and Celery shows the code and a simple explanation of how to use Celery with Django.
Asynchronous Tasks With Django and Celery shows how to integrate Celery with Django and create Periodic Tasks.
Three quick tips from two years with Celery provides some solid advice on retry delays, the -Ofair flag and global task timeouts for Celery.
Pick a slow function in your project that is called during an HTTP request.
Determine if you can precompute the results on a fixed interval instead of during the HTTP request. If so, create a separate function you can call from elsewhere then store the precomputed value in the database.
Read the Celery documentation and the links in the resources section below to understand how the project works.
Install a message broker such as RabbitMQ or Redis and then add Celery to your project. Configure Celery to work with the installed message broker.
Use Celery to invoke the function from step one on a regular basis.
Have the HTTP request function use the precomputed value instead of the slow running code it originally relied upon.
Configuration management involves modifying servers from an existing state to a desired state and automating how an application is deployed.
Numerous tools exist to modify server state in a controlled way, including Puppet, Chef, SaltStack, and Ansible. Puppet and Chef are written in Ruby, while SaltStack and Ansible are written in Python.
Configuration management tools such as Chef, Puppet, Ansible, and SaltStack are not useful for performing ad hoc tasks that require interactive responses. Fabric and Invoke are used for interactive operations, such as querying the database from the Django manage.py shell.
Moving away from Puppet: SaltStack or Ansible? is an openly biased but detailed post on why to choose SaltStack over Ansible in certain situations.
Ansible vs. Shell Scripts provides some perspective on why a configuration management tool is better than old venerable shell scripts.
Ansible vs. Chef is a comparsion of Ansible with the Chef configuration management tool.
This post on Ansible and Salt: A Detailed Comparison shows the differences between these two Python-powered tools.
Ansible is an open source configuration management and application deployment tool built in Python.
An Ansible tutorial is a fantastically detailed introduction to using Ansible to set up servers.
Ansible Text Message Notifications with Twilio SMS is my blog post with a detailed example for using the Twilio module in core Ansible 1.6+.
Python for Configuration Management with Ansible slides from PyCon UK 2013
An introduction to Ansible is a tutorial on the basics of getting started with the tool.
Multi-factor SSH authentication with Ansible and Duo Security
Shippable + Ansible + Docker + Loggly for awesome deployments is a well written detailed post about using Docker and Ansible together with a few other pieces.
Idempotence, convergence, and other silly fancy words we often use
Learn about configuration management in the context of deployment automation and infrastructure-as-code.
Pick a configuration management tool and stick with it. My recommendation is Ansible because it is by far the easiest tool to learn and use.
Read your configuration management tool's documentation and, when necessary, the source code.
Automate the configuration management and deployment for your project. Note that this is by far the most time consuming step in this checklist but will pay dividends every time you deploy your project.
Hook the automated deployment tool into your existing deployment process.
Continuous integration automates the building, testing and deploying of applications. Software projects, whether created by a single individual or entire teams, typically use continuous integration as a hub to ensure important steps such as unit testing are automated rather than manual processes.
When continuous integration (CI) is established as a step in a software project's development process it can dramatically reduce deployment times by minimizing steps that require human intervention. The only minor downside to using CI is that it takes some initial time by a developer to set up and then there is some ongoing maintainence if a project is broken into multiple parts, such as going from a monolith architecture to microservices.
Another major advantage with CI is that testing can be an automated step in the deployment process. Broken deployments can be prevented by running a comprehensive test suite of unit and integration tests when developers check in code to a source code repository. Any bugs accidentally introduced during a check-in that are caught by the test suite are reported and prevent the deployment from proceeding.
The automated testing on checked in source code can be thought of like the bumper guards in bowling that prevent code quality from going too far off track. CI combined with unit and integration tests check that any code modifications do not break existing tests to ensure the software works as intended.
The following picture represents a high level perspective on how continuous integration and deployment can work.
In the above diagram, when new code is committed to a source repository there is a hook that notifies the continuous integration server that new code needs to be built (the continuous integration server could also poll the source code repository if a notification is not possible).
The continuous integration server pulls the code to build and test it. If all tests pass, the continuous integration server begins the deployment process. The new code is pulled down to the server where the deployment is taking place. Finally the deployment process is completed via restarting services and related deployment activities.
There are many other ways a continuous integration server and its deployments can be structured. The above was just one example of a relatively simple set up.
There are a variety of free and open source continuous integration servers that are configurable based on a project's needs.
Note that many of these servers are not written in Python but work just fine for Python applications. Polyglot organizations (ones that use more than a single language and ecosystem) often use a single CI server for all of their projects regardless of the programming language the application was written in.
Jenkins is a common CI server for building and deploying to test and production servers. Jenkins source code is on GitHub.
Go CD is a CI server by ThoughtWorks that was designed with best practices for the build and test & release cycles in mind. Go CD source code is on GitHub.
Strider is a CI server written in node.js. Strider source code is on GitHub.
BuildBot is a continuous integration framework with a set of components for creating your own CI server. It's written in Python and intended for development teams that want more control over their build and deployment pipeline. BuildBot source code is on GitHub.
TeamCity is JetBrains' closed source CI server that requires a license to use.
Assembling a continuous integration service for a Django project on Jenkins shows how to set up a Ubuntu instance with a Jenkins server that'll build a Django project.
My book on deploying Python web applications walks through every step of setting up a Jenkins project with a WSGI application to enable continuous delivery. Take a look if you're not grokking all of the steps provided in these other blog posts.
Setting up Jenkins as a continuous integration server for Django is another solid tutorial that also shows how to send email notifications as part of the build process.
If you're running into difficulty adding an SSH key to your Jenkins system account so you can connect to another server or Git repository this blog post on connecting Jenkins with Git to get the steps to solve that problem.
Running Jenkins in Docker Containers is a short tutorial showing how to use the official Jenkins container on the Docker hub.
Securing Jenkins is the landing page for Jenkins security. If you're deploying your own instance, you'll need to lock it down against unauthorized users.
Can we use Jenkins for that? looks at how one team uses Jenkins for more than typical continuous integration situations - they also use it as an administrative interface, cron jobs, data analytics pipelines and long-running scripts.
What is continuous integration? is a classic detailed article by Martin Fowler on the concepts behind CI and how to implement it.
Continuous Deployment For Practical People is not specific to Python but a great read on what it entails.
Continuous Integration & Delivery - Illustrated uses well done drawings to show how continuous integration and delivery works for testing and managing data.
Diving into continuous integration as a newbie is a retrospective on learning CI from a Rackspace intern on how she learned the subject.
StackShare's Continuous Integration tag lists a slew of hosted CI services roughly ranked by user upvotes.
Good practices for continuous integration includes advice on checking in code, commit tests and reverting to previous revisions.
Deploying to AWS using Ansible, Docker and Teamcity is an example walking through one potential way to use the Teamcity CI server for automated deployments.
Why Continuous Integration Is Important is a high-level overview of how CI can build trust both among developers and between developers and non-technical people in an organization. The post also discusses tasks related to setting up reliable CI such as test environments, integration testing and visibility into the CI results.
Logging saves output such as errors, warnings and event information to persistent storage for debugging purposes.
Runtime exceptions that prevent code from running are important to log to investigate and fix the source of the problems. Informational and debugging logging also helps to understand how the application is performing even if code is working as intended.
Logging is often grouped into several categories:
Logging errors that occur while a web framework is running is crucial to understanding how your application is performing.
When you are running your application on several servers, it is helpful to have a monitoring tool called a "logging aggregator". You can configure your application to forward your system and application logs to one location that provides tools for viewing, searching, and monitoring logging events across your cluster.
Another advantage of log aggregation tools is they allow you to set up custom alerts and alarms so you can get notified when error rates breach a certain threshold.
Sentry started as a Django-only exception handling service but now has separate logging clients to cover almost all major languages and frameworks. It still works really well for Python-powered web applications and is often used in conjunction with other monitoring tools. Raven is open source Python client for Sentry.
Graylog2 provides a central server for log aggregation as well as a GUI for browsing and searching through log events. There are libraries for most major languages, including python. Saves data in Elasticache.
Logstash Similar to Graylog2, logstash offers features to programmatically configure log data workflows.
Scribe A project written by Facebook to aggregate logs. It's designed to run on multiple servers and scale with the rest of your cluster. Uses the Thrift messaging format so it can be used with any language.
Loggly is a third party cloud based application that aggregates logs. They have instructions for every major language, including python. It includes email alerting on custom searches.
Splunk offers third party cloud and self hosted solutions for event aggregation. It excels at searching and data mining any text based data.
Papertrail is similar to both Loggly and Splunk and provides integration with S3 for long term storage.
Raygun logs errors and provides immediate notification when issues arise.
Scalyr provides log aggregation, dashboards, alerts and search in a user interface on top of standard logs.
There is a hosted version of Sentry in case you do not have the time to set up the open source project yourself.
This intro to logging presents the Python logging module and how to use it.
Logging as Storytelling is a multi-part series working the analogy that logs should read like a story so you can better understand what's taking place in your web application. Part 2 describes actions and part 3 talks about types.
A Brief Digression About Logging is a short post that gets Python logging up and running quickly.
Taking the pain out of Python logging shows a logging set up with uWSGI.
Django's 1.3 release brought unified logging into project configurations. This post shows how to set up logging in a project's settings.py file. Caktus Group also has a nice tutorial on central logging with graypy and Graylog2.
Django Logging Configuration: How the Default Settings Interfere with Yours explains a problem with the default Django logging configuration and what to do about in your project.
Exceptional Logging of Exceptions in Python shows how to log errors more accurately to pinpoint the problem instead of receiving generic exceptions in your logs.
Read how to integrate logging into your web application framework.
Ensure errors and anomalous results are logged. While these logs can be stored in monitoring solutions, it's best to have your own log storage location to debug issues as they arise to complement other monitoring systems.
Integrate logging for system events you may need to use for debugging purposes later. For example, you may want to know the return values on functions when they are above a certain threshold.
Monitoring tools capture, analyze and display information for a web application's execution. Every application has issues arise throughout all levels of the web stack. Monitoring tools provide transparency so developers and operations teams can respond and fix problems.
Capturing and analyzing data about your production environment is critical to proactively deal with stability, performance, and errors in a web application.
Monitoring and logging are very similar in their purpose of helping to diagnose issues with an application and aid the debugging process. One way to think about the difference is that logging happens based on explicit events while monitoring is a passive background collection of data.
For example, when an error occurs, that event is explicitly logged through code in an exception handler. Meanwhile, a monitoring agent instruments the code and gathers data not only about the logged exception but also the performance of the functions.
This distinction between logging and monitoring is vague and not necessarily the only way to look at it. Pragmatically, both are useful for maintaining a production web application.
There are several important resources to monitor on the operating system and network level of a web stack.
Application level monitoring encompasses several aspects. The amount of time and resources dedicated to each aspect will vary based on whether an application is read-heavy, write-heavy, or subject to rapid swings in traffic.
statsd is a node.js network daemon that listens for metrics and aggregates them for transfer into another service such as Graphite.
Graphite stores time-series data and displays them in graphs through a Django web application.
Bucky measures the performance of a web application from end user's browsers and sends that data back to the server for collection.
Sensu is an open source monitoring framework written in Ruby but applicable to any programming language web application.
Graph Explorer by Vimeo is a Graphite-based dashboard with added features and a slick design.
PacketBeat sniffs protocol packets. Elasticsearch then allows developers to search the collected data and visualize what's happening inside their web application using the Kibana user interface.
Munin is a client plugin-based monitoring system that sends monitoring traffic to the Munin node where the data can be analyzed and visualized. Note this project is written in Perl so Perl 5 must be installed on the node collecting the data.
New Relic. Application and database monitoring as well as plug ins for capturing and analyzing additional data about tools in your stack.
CopperEgg is lower-level monitoring on server and infrastructure. It's popular with DevOps shops that are making changes to their production environments and want immediate feedback on the results of those modifications.
Status.io focuses on uptime and response metrics transparency for web applications.
StatusPage.io (yes, there's both a Status and StatusPage.io) provides easy set up status pages for monitoring application up time.
PagerDuty alerts a designated person or group if there are stability, performance, or uptime issues with an application.
App Enlight provides performance, exception and error monitoring and is currently specific to Python web applications.
Opbeat Built for django. Opbeat combines performance metrics, release tracking, and error logging into a single simple service.
Practical Guide to StatsD/Graphite Monitoring is a detailed guide with code examples for monitoring infrastructure.
Bit.ly describes the "10 Things They Forgot to Monitor" beyond the standard metrics such as disk & memory usage.
5 years of metrics and monitoring is a great presentation highlighting that visualization so humans can understand measurements is a hard problem. Line graphs are often not the best solution and they are overused.
The Collector Highlight Series has an article on StatsD that explains how to install it and how it works.
This survey on monitoring tools has some nice data and graphs on what developers and operations folks use in their environments.
Ryan Frantz wrote a nice post on Solving Monitoring with a new definition of what monitoring means based on today's complex systems and how the practice should evolve going forward.
Review the software-as-a-service and open source monitoring tools below. Third party services tend to be easier to set up and host the data for you. Open source projects give you more control but you'll need to have additional servers ready for the monitoring.
My recommendation is to install New Relic's free option with the trial period to see how it works with your app. It'll give you a good idea of the capabilities for application-level monitoring tools.
As your app scales take a look at setting up one of the the open source monitoring projects such as StatsD with Graphite. The combination of those two projects will give you fine-grained control over the system metrics you're collecting and visualizing.
Web analytics involves collecting, processing, visualizing web data to enable critical thinking about how users interact with a web application.
User clients, especially web browsers, generate significant data while users read and interact with webpages. The data provides insight into how visitors use the site and why they stay or leave. The key concept to analytics is learning about your users so you can improve your web application to better suit their needs.
It's easy to get overwhelmed at both the number of analytics services and the numerous types of data points collected. Focus on just a handful of metrics when you're just starting out. As your application scales and you understand more about your users add additional analytics services to gain further insight into their behavior with advanced visualizations such as heatmaps and action funnels.
If your application is selling a product or service you can ultimately build a user funnel (often called "sales funnel" prior to a user becoming a customer) to better understand why people buy or don't buy what you're selling. With a funnel you can visualize drop-off points where visitors leave your application before taking some action, such as purchasing your service.
Piwik is a web analytics platform you can host yourself. Piwik is a solid choice if you cannot use Google Analytics or want to customize your own web analytics platform.
Open Web Analytics is another self-hosted platform that integrates through a JavaScript snippet that tracks users' interactions with the webpage.
Google Analytics is a widely used free analytics tool for website traffic.
Clicky provides real-time analytics comparable to Google Analytics' real-time dashboard.
MixPanel's analytics platform focuses on mobile and sales funnel metrics. A developer builds what data points need to be collected into the server side or client side code. MixPanel captures that data and provides metrics and visualizations based on the data.
KISSmetrics' analytics provides context for who is visiting a website and what actions they are taking while on the site.
Heap is a recently founded analytics service with a free introductory tier to get started.
CrazyEgg is tool for understanding a user's focus while using a website based on heatmaps generated from mouse movements.
Building an Analytics App with Flask is a detailed walkthrough for collecting and analyzing webpage analytics with your own Flask app.
Pandas and Google Analytics shows how to use pandas for data analysis with Google Analytics' API to perform calculations not available in the tool itself.
Build your own Google Analytics Dashboard in Excel show how to extract your Google Analytics data via their web API and Python helper library so it can be used in other tools such as Excel.
This beginner's guide to math and stats behind web analytics provides some context for understanding and reasoning about web traffic.
An Analytics Primer for Developers by Mozilla explains what to track, choosing an analytics platform and how to serve up the analytics JavaScript asynchronously.
This post provides context for determining if a given metric is "vanity" or actionable.
Add Google Analytics or Piwik to your application. Both are free and while Piwik is not as powerful as Google Analytics you can self-host the application which is the only option in many environments.
Think critically about the factors that will make your application successful. These factors will vary based on whether it's an internal enterprise app, an e-commerce site or an information-based application.
Add metrics generated from your web traffic based on the factors that drive your application's success. You can add these metrics with either some custom code or with a hosted web analytics service.
Continuously reevaluate whether the metrics you've chosen are still the appropriate ones defining your application's success. Improve and refine the metrics generated by the web analytics as necessary.
Docker is an open source infrastructure management platform for running and deploying software. The Docker platform is constantly evolving so an exact definition is currently a moving target.
Docker can package up applications along with their necessary operating system dependencies for easier deployment across environments. In the long run it has the potential to be the abstraction layer that easily manages containers running on top of any type of server, regardless of whether that server is on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere.
This Docker image contains a Flask application configured to run with uWSGI and Nginx. You can also see the image on Docker hub.
minimal-docker-python-setup contains an image with Nginx, uWSGI, Redis and Flask.
What is Docker and When to Use It clearly delineates what Docker is and what it isn't. This is a good article for when you're first wrapping your head around Docker conceptually.
Andrew Baker presented a fantastic tutorial at PyOhio on beginner and advanced Docker usage. Andrew also wrote the article what containers can do for you and created the O'Reilly Introduction to Docker video that's well worth buying.
Docker curriculum is a detailed tutorial created by a developer to show the exact steps for deploying an application that relies on Elasticsearch.
How to install Docker and get started provides a walkthrough for Ubuntu 13.04 for installing and beginning to use Docker for development.
It Really is the Future discusses Docker and containers in the context of whether it's all just a bunch of hype or if this is a real trend for infrastructure automation. This is a great read to set the context for why these tools are important.
Docker Jumpstart is a comprehensive introduction to what Docker is and how to get started with using the tool.
If you want to quickly use Docker on Mac OS X, check out these concise instructions for setting up your Docker workflow on OS X in 60 seconds.

Docker in Practice - A Guide for Engineers is an explanation of the concepts and philosophy by the authors of the new Manning Docker book in early access format.
Eight Docker Development Patterns shares lessons learned and explains how to work with the containers so you get more use out of them during development.
The marriage of Ansible and Docker is a detailed look at how Docker and Ansible complement each other as deployment tools.
Building Docker containers from scratch is a short tutorial for creating a Docker container with a specific configuration.
10 things to avoid in Docker containers provides a lot of "don'ts" that you'll want to consider before bumping up against the limitations of how containers should be used.
Hosting Python WSGI applications using Docker shows how to use Docker in WSGI application deployments specifically using mod_wsgi.
How to Containerize Python Web Applications is an extensive tutorial that uses a Flask application and deploys it using a Docker container.
The Docker is awesome miniseries explains how to get a Django + AngularJS application running under Docker. Part 2 continues the tutorial.
Docker in Action - Fitter, Happier, More Productive is a killer tutorial that shows how to combine Docker with CircleCI to continuously deploy a Flask application.
Deploying Django Applications in Docker explains some of the concepts behind using Docker for Python deployments and shows how to specifically use it for deploying Django.
A Docker primer – from zero to a running Django app provides specific commands and expected output for running Django apps with Docker and Vagrant.
Using Docker and Docker Compose to replace virtualenv is a tutorial for using Docker instead of virtualenv for dependency isolation.
Lincoln Loop wrote up a closer look at Docker from the perspective of Python developers handling deployments.
Curious how pip and Docker can be used together? Read this article on Efficient management Python projects dependencies with Docker.
Python virtual environments and Docker goes into detail on whether virtual environments should be used with Docker and how system packages can generally be a safer route to go.
Caching can reduce the load on servers by storing the results of common operations and serving the precomputed answers to clients.
For example, instead of retrieving data from database tables that rarely change, you can store the values in-memory. Retrieving values from an in-memory location is far faster than retrieving them from a database (which stores them on a persistent disk like a hard drive.) When the cached values change the system can invalidate the cache and re-retrieve the updated values for future requests.
A cache can be created for multiple layers of the stack.
memcached is a common in-memory caching system.
Redis is a key-value in-memory data store that can easily be configured for caching with libraries such as django-redis-cache.
"Caching: Varnish or Nginx?" reviews some considerations such as SSL and SPDY support when choosing reverse proxy Nginx or Varnish.
Caching is Hard, Draw me a Picture has diagrams of how web request caching layers work. The post is relevant reading even though the author is describing his Microsoft code as the impetus for writing the content.
While caching is a useful technique in many situations, it's important to also note that there are downsides to caching that many developers fail to take into consideration.
Analyze your web application for the slowest parts. It's likely there are complex database queries that can be precomputed and stored in an in-memory data store.
Leverage your existing in-memory data store already used for session data to cache the results of those complex database queries. A task queue can often be used to precompute the results on a regular basis and save them in the data store.
Incorporate a cache invalidation scheme so the precomputed results remain accurate when served up to the user.
Microservices are an application architecture style where independent, self-contained programs with a single purpose each can communicate with each other over a network. Typically, these microservices are able to be deployed independently because they have strong separation of responsibilities via a well-defined specification with significant backwards compatibility to avoid sudden dependency breakage.
Microservices follow in a long trend of software architecture patterns that become all the rage. Previously, CORBA and (mostly XML-based) service-oriented architectures (SOA) were the hip buzzword among ivory tower architects.
However, microservices have more substance because they are typically based on RESTful APIs that are far easier for actual software developers to use compared with the previous complicated XML-based schemas thrown around by enterprise software companies. In addition, successful applications begin with a monolith-first approach using a single, shared application codebase and deployment. Only after the application proves its usefulness is it then broken down into microservice components to ease further development and deployment. This approach is called the "monolith-first" or "MonolithFirst" pattern.
Martin Fowler's microservices article is one of the best in-depth explanations for what microservices are and why to consider them as an architectural pattern.
Why microservices? presents some of the advantages, such as the dramatically increased number of deployments per day, that a well-done microservices architecture can provide in the right situation. Many organizational environments won't allow this level of flexibility but if yours is one that will, it's worth considering these points.
On monoliths and microservices provides some advice on using microservices in a fairly early stage of a software project's lifecycle.
Developing a RESTful microservice in Python is a good story of how an aging Java project was replaced with a microservice built with Python and Flask.
Microservices: The essential practices first goes over what a monolith application looks like then dives into what operations you need to support potential microservices. For example, you really need to have continuous integration and deployment already set up. This is a good high-level overview of the topics many developers aren't aware of when they embark on converting a monolith to microservices.
Using Nginx to Load Balance Microservices explains how an Nginx instance can use configuration values from etcd updated by confd as the values are modified. This setup can be useful for load balancing microservices as the backend services are brought up and taken down.
How Microservices have changed and why they matter is a high level overview of the topic with some quotes from various developers around the industry.
The State of Microservices Today provides some general trends and broad data showing the increasing popularity of microservices heading into 2016. This is more of an overview of the term than a tutorial but useful context for both developers and non-developers.
bla bla microservices bla bla is a transcript for a killer talk on microservices that breaks down the important first principles of distributed systems, including asynchronous communication, isolation, autonomicity, single responsibility, exclusive state, and mobility. The slides along with the accompanying text go into how reality gets messy and how to embrace the constraints inherent in distributed systems.
DevOps is the combination of application development and operations, which minimizes or eliminates the disconnect between software developers who build applications and systems administrators who keep infrastructure running.
When the Agile methodology is properly used to develop software a new bottleneck often appears during the deployment and operations phases. New updates and fixes are produced so fast in each sprint that infrastructure teams can be overwhelmed with deployments and push back on the pace of delivery. To allievate some of these issues, application developers are asked to work closely with operations folks to automate the delivery from development to production.
DevOps vs. Platform Engineering considers DevOps an ad hoc approach to developing software while building a platform is a strict contract. I see this as "DevOps is a process", while a "platform is code". Running code is better than any organizational process.
DevOps: Python tools to get started is a presentation slideshow that explains that while DevOps is a culture, it can be supported by tools such as Fabric, Jenkins, BuildBot and Git which when used properly can enable continuous software delivery.
Why are we racing to DevOps? is a very high level summary of the benefits of DevOps to IT organizations. It's not specific to Python and doesn't dive into the details, but it's a decent start for figuring out why IT organizations consider DevOps the hot new topic after adopting an Agile development methodology.
Nginx is the second most common web server among the top 100,000 websites. Nginx also functions well as a reverse proxy to handle requests and pass back responses for Python WSGI servers or even other web servers such as Apache.
Let's be clear about these two "competing" servers: they are both fantastic open source projects and either will serve your web application deployment well. In fact, many of the top global web applications use both servers in their deployments to function in many steps throughout the HTTP request-response cycle.
I personally use Nginx more frequently than Apache because Nginx's configuration feel easier to write, with less boilerplate than alternatives.
There's also a bit of laziness in the usage: Nginx works well, it never causes me problems. So I stick with my battle-tested Ansible configuration management files that set up Nginx with HTTPS and SSL/TLS certificates
Nginx's default configuration after a standard installation through a system package manager or compiling from source is a good base for security. However, setting up ciphers and redirects can be confusing the first few times you try it. It's a really good idea to read some of these tutorials to make sure you are avoiding the most common security errors that plague HTTP(S) configurations.
Hacker News broke our site – how Nginx and PageSpeed fixed the problem is primarily about optimizing Nginx's configuration for more efficient SSL connections. The post also covers configuration management with Ansible as well as the Pagespeed module that Google released for both Nginx and the Apache HTTP Server.
Secure Web Deployment with Let's Encrypt and Nginx is a detailed walkthrough for setting up HTTPS under Ubuntu 14.04 with Nginx.
How To Secure Nginx on Ubuntu 14.04 explains SSL configurations and IP address blacklisting then provides several other tutorials for more advanced security modules.
Strong SSL Security on Nginx shows how to mitigate high profile SSL attacks like Logjam, Heartbleed and FREAK.
The Nginx chapter in the Architecture of Open Source Applications book has a great chapter devoted to why Nginx is built to scale a certain way and lessons learned along the development journey.
Inside Nginx: How we designed for performance and scale is a blog post from the developers behind Nginx on why they believe their architecture model is more performant and scalable than other approaches used to build web servers.
Test-driving web server configuration is a good story for how to iteratively apply configuration changes, such as routing traffic to Piwik for web analytics, reverse proxying to backend application servers and terminately TLS connections appropriately. It is impressive to read a well-written softare development article like this from a government agency, although UK's Government Digital Service as well as USA's 18F and US Digital Service foster a far more credible culture than most typical agencies.
Nginx for Developers: An Introduction provides the first steps to getting an initial Nginx configuration up and running.
A faster Web server: ripping out Apache for Nginx explains how Nginx can be used instead of Apache in some cases for better performance.
Nginx vs Apache: Our view is a first-party perspective written by the developers behind Nginx as to the differences between the web servers.
Rate Limiting with Nginx covers how to mitigate against brute force password guessing attempts using Nginx rate limits.
Nginx with dynamic upstreams is an important note for setting up your upstream WSGI server(s) if you're using Nginx as a reverse proxy with hostnames that change.
Nginx Caching shows how
to set up Nginx for caching HTTP requests, which is often done by Varnish
but can also be handled by Nginx with the proxy_cache
and related
directives.
Nginx web server tutorials are oldies but goodies on setting up previous versions of Nginx.