Getting a native connection from the ORM

The SQLAlchemy ORM provides a powerful abstraction from the database allowing operations to be performed on objects and queries to be constructed based on object attributes rather than dealing with attribute-to-field correspondence.  But there are still some operations for which you need to talk directly to the underlying database. 
In 'normal' mode SQLAlchemy maintains a connection pool and releases connections from the pool to the application as needed, tracks them, and tries to keep everything tidy.  When the need arises for a 'native' DBAPI connection [for this example I'm using PostgreSQL] it is possible to explicitly check a connection out from the pool - after which it is yours to deal with, to track the isolation, close, etc..
Assuming the database connection has already been created and bound to the session factory with something like:

from sqlalchemy import create_engine, Session
...
engine = create_engine( orm_dsn, **{ 'echo': orm_logging } )
Session.configure( bind=engine )

 - then sessions can be taken from the connection pool simply by calling "db = Session( )".  When the "db.close( )" is performed that session goes back into the pool.
If a connection is to be used outside of all the mechanics that SQLAlchemy provides it can be checked out, as mentioned before, using a rather verbose call:

conn = Session( ).connection( ).connection.checkout( ).connection

Now "conn" is a 'native' DBAPI connection.  You can perform low level operations such as setting the isolation level and creating cursors:

conn.set_isolation_level( psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT )
curs = conn.cursor( )
curs.execute( .... )

This is not a frequent need, but for very database backend specific it is the simplest approach [otherwise you can extend SQLAlchemy...]. One use case is using PostreSQL's asyncronous notify channels to capture database events; for this purpose an application needs to select on the DBAPI connection, there's no need for an ORM in the mix when you are just capturing events.

Setting a course for UTC

Timezones and daylight savings times are confusing; it is much more complicated that offset-from-UTC.  There are times that occur more that once a year [yep, it hurts] as well as times that are between two valid times but never happen.  It probably requires a Tardis to understand why anyone would want it to work this way.  But, sadly, it does work this way. 
If you stick to the rules, you can safely manage times... so long as those times are all localized.   Naive times, times that are not localized, are the enemy.
Unfortunately there is a lot of code out there, even in important and widely used modules, that uses nieve datetimes.  If you try to use a virtuously localized datetime object with those modules you will likely encounter the dreaded "Cannot compare naive and localized values".
One hack is to make sure the time is localized to the system's timezone, then make it naive, call the module's function, and then re-localize the result (again). Tedious and very prone to error.  The one real problem with this hack is that on most systems the Python process has not @*^$&*@* clue what time zone it is in.  Don't believe me? Try it:

>>> import time
>>> time.tzname
('EST', 'EDT')

Eh, that's a tuple.  And while "EST" is a time zone "EDT" is not a timezone.  Yes, I can determine that I am in daylight savings time locally using time.daylight; but I can't localize a datetime to a daylight timezone because daylight is an attribute of a timezone, not a timezone itself.  That is true regardless of what time.tzname says.  And the "EST" doesn't have daylight savings time, "US/Eastern" does.  "EST" is "US/Eastern" when not it daylight savings time. Gnarly.
But I want to use datetime obejcts reliably and safely with modules that require naive datetime objects....  The answer is to make the timezone known!  I cannot reliably get it from the system but I can make it what I want, and what I want is UTC!  Then my naive datetime objects do not have to be concerned with daylight savings time.  I can just localize them to UTC and subsequently convert them to whatever timezone the user needs to see.  This is accomplished using a combination of the os and time modules.  Early on in my Python application I move myself to UTC.  Here is an example that demonstrates the ugliness of naive times in an unknown timezone, and the beauty of the process being in UTC.

from datetime import datetime
import pytz, time, os

print( 'NOW: {0}'.format( datetime.now( ) ) )
print( 'UTCNOW: {0}'.format(datetime.utcnow( ) ) )
# What timezone is local?  Problem is, most of the time we just do not know.
print( 'LOCALIZEDNOW: {0}'.format( pytz.timezone( 'UTC' ).localize( datetime.now( ) ) ) )
print( 'LOCALIZEDUTC: {0}'.format( pytz.timezone( 'UTC' ).localize( datetime.utcnow( ) ) ) )

#Change to UTC
os.environ[ 'TZ' ] = 'UTC'
time.tzset( )

print( 'NOW: {0}'.format( datetime.now( ) ) )
print( 'UTCNOW: {0}'.format( datetime.utcnow( ) ) )
print( 'LOCALIZEDNOW: {0}'.format( pytz.timezone( 'UTC' ).localize( datetime.now( ) ) ) )
print( 'LOCALIZEDUTC: {0}'.format( pytz.timezone( 'UTC' ).localize( datetime.utcnow( ) ) ) )

And the output:

NOW: 2012-10-26 07:03:31.285486
UTCNOW: 2012-10-26 11:03:31.285570
LOCALIZEDNOW: 2012-10-26 07:03:31.285632+00:00
LOCALIZEDUTC: 2012-10-26 11:03:31.285705+00:00
NOW: 2012-10-26 11:03:31.285787
UTCNOW: 2012-10-26 11:03:31.285812
LOCALIZEDNOW: 2012-10-26 11:03:31.285848+00:00
LOCALIZEDUTC: 2012-10-26 11:03:31.285875+00:00

Now the danger of somehow getting a naive datetime into the mix is completely avoided - I can always safely localize a naive time to UTC.

All those SQLite databases...

Many current application use the SQLite database for tracking information; this includes F-Spot, Banshee, Hamster, Evolution, and others.  Even the WebKit component uses SQLite [you might be surprised to discover ~/.local/share/webkit/databases].  It is wonderfully efficient that there is one common local data storage technique all these applications can use,  especially since it is one that is managable using a universally known dialect [SQL]. But there is a dark-side to SQLite.  Much like old Dbase databases it needs to be vacuumed.  And how reliably are all those applications providing their little databases with the required affection?  Also, do you trust those lazy developers to have dealt with the condition of a corrupted database?   If an application hangs, or is slow, or doesn't open... maybe that little database is corrupted?

Aside: As a system administrator for almost two decades I do not trust developers. They still put error messages in applications like "File not found!".  Argh!

On the other hand SQLite provides a handy means of performing an integrity check on databases - the "PRAGMA integrity_check" command.  I've watched a few of these little databases and discovered that (a) they aren't often all that little, and (b) manually performing a VACUUM may dramatically reduce their on-disk size.  Both these facts indicate that developers are lazy and should not be trusted.

Note: in at least one of these cases the application has subsequently been improved. Developers do respond rather quickly when offered a blend of compliments spiced with bug reports.  No, I'm not going to name offending applications as that is too easily used as fodder by nattering nabobs.  And even the laziest Open Source developer is working harder than their proprietary brothers.

In light of this situation my solution is a hack - a Python script [download] that crawls around looking for SQLite databases.  First the script attempts to open the database in exclusive mode, then it performs an integrity check, and if that succeeds it performs a vacuum operation.  Currently it looks for databases in "~/.local/share" [where it will find databases managed by application appropriately following the XDG specification], "~/.cache", "~/.pki", "~/.rcc", and "~/.config".
Download the script and run it. Worst thing that happens is that it accomplishes nothing.  On the other hand it might recover some disk space, improve application performance, or reveal a busted database.

Finding Address Coordinates using Python, SOAP, & the Bing Maps API

Bing maps provides a SOAP API that can be easily accessed via the Python suds module.  Using the API it is trivial to retrieve the coordinates of a postal address.  The only requirement is to acquire a Bing API application key; this process is free, quick, and simple.

import sys, urllib2, suds

if __name__ == '__main__':  
    url = 'http://dev.virtualearth.net/webservices/v1/geocodeservice/geocodeservice.svc?wsdl'
    
    client = suds.client.Client(url)
    client.set_options(port='BasicHttpBinding_IGeocodeService')
    request = client.factory.create('GeocodeRequest')

    credentials = client.factory.create('ns0:Credentials')
    credentials.ApplicationId = 'YOUR-APPLICATION-KEY'
    request.Credentials = credentials

    #Address
    address = client.factory.create('ns0:Address')
    address.AddressLine = "535 Shirley St. NE"
    address.AdminDistrict = "Michigan"
    address.Locality = "Grand Rapids"      
    address.CountryRegion = "United States"
    request.Address = address

    try:        
        response = client.service.Geocode(request)    
    except suds.client.WebFault, e:        
        print "ERROR!"        
        print(e)
        sys.exit(1)

    locations = response['Results']['GeocodeResult'][0]['Locations']['GeocodeLocation']
    for location in locations:        
        print(location)

If you need to make the request via an HTTP proxy server expand the line client = suds.client.Client(url) to:

    proxy = urllib2.ProxyHandler({'http': 'http://YOUR-PROXY-SERVER:3128'})\
    transport = suds.transport.http.HttpTransport()
    transport.urlopener = urllib2.build_opener(proxy)
    client = suds.client.Client(url, transport=transport)

The results will be Bing API GeocodeLocation objects that have an Longitude and Latitude properties.  Note that you may receive multiple coordinates for an address as there are multiple mechanism for locating an address; the method corresponding to the coordinates is a string in the CalculationMethod property of the GeocodeLocation objects.

Python Curses In Action, even on AIX.

So, what if you have an old COBOL application that you want to integrate with some web services or a website? You'd need a "green-screen" application that can take that COBOL applications temporary output file, send it up to the web service, wait for a response, replace the temporary file, and then return control to the COBOL application (or whatever "green screen" application comes next). It sounds simpler than it is in practice: what if the web service takes awhile to complete [or has human involvement on the other end?!] or the web service isn't available right at the moment you try to send your request? For a real production environment your "green screen" application would need to deal with all of those things.
And you need that "green screen" application to work on both LINUX and AIX!

Fortunately pware provides Python 2.6 for AIX, including curses! The curses library and respective Python module provide a surprisingly easy way to create professional looking TUI (Text User Interface) applications like the solution required in our example. This particular application takes them temporary file and submits it via a WebDAV PUT operation to the workflow engine provided by OpenGroupware Coils. The route in the workflow engine reformats the dreadful output of the COBOL application into the required format (using the format support in route Read and Write operations). The client detects the route is complete by watching the URL specified in the header of the initial PUT, and then downloads the required data. If anything goes wrong along the way the client can retry, or provide the user the option to abort. All along the way the client provides detailed feedback to the user and a familiar dialog-box and prompt interface when feedback is required.

Service BASIC HTTP Authentication with Python

While building the core parts of OpenGroupware COILS I noticed there don't appear to be any example of providing HTTP's BASIC authentication scheme anywhere on the interweb. Or any other authentication schemes for that matter. To remedy that here is a rough outline, including the example code, of how it is implemented in COILS. First spin up an HTTP server:

import BaseHTTPServer

class HTTPServer(BaseHTTPServer.HTTPServer):
pass

from coils.net.handler   import HTTPRequestHandler
from coils.net.server    import HTTPServer
....
HTTP_HOST = 'localhost'
HTTP_PORT = 8080
httpd = HTTPServer((HTTP_HOST, HTTP_PORT), HTTPRequestHandler)
httpd.serve_forever()

Your app will almost certainly provide a custom HTTPRequestHandler to delegate the requests to whatever logic your application provides. In the BaseHTTPRequestHandler a GET request is handed to do_GET, a POST request to do_POST, etc... In our case the object that handles the request will decide what to do based on the request type so we channel all requests to our generic process_request method. process_request looks up the object targeted by the request via marshall_handler() [not shown] and then calls that objects do_request method. The important part for authentication is to catch the exception raised by the object handling the request if that object thinks the request is not authenticated and requires authentication. In this case that is the COILS' AuthenticationException; your application has to provide something equivalent. In order to make the client try again with authentication you need to send a "WWW-Authenticate" header telling the client to use basic authentication and what realm to use. See RFC2617 for details on Basic authentication in general

class HTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

def process_request(self):
  """Respond to a request"""
  try:
    """ find the object mapped to the specified request
        The marshall_handler() is a COILS specific things so it isn't
        shown in this example. """
    handler = self.marshall_handler()
    handler.do_request(self)
  except AuthenticationException, err:
    """ An AuthenticationException has an error code of 401
        We need to add an authentication header so the client will know
        to respond the the failure with the appropriate credentials
        TODO: Provide a digest realm once digest authentication is supported
    """
    self.send_response(err.error_code())
    self.send_header('WWW-Authenticate', 'Basic realm="OpenGroupware COILS"')
    self.end_headers()
    self.wfile.write('Authentication failure')
  except CoilsException, err:
    # An Coils Exception has an error code of 500
    self.send_response(err.error_code(), err.error_text())
    self.end_headers()
  except Exception, err:
    # Yikes, something generic web very wrong
    self.send_response(500, err)
    self.end_headers()

def do_GET(self):
  """Respond to a GET request."""
  self.process_request()

def do_POST(self):
  """Respond to a POST request"""
  self.process_request()

In order to actually process the authentication request COILS uses an Authenticator object. Our DBAuthenticator object provides basic authentication against accounts with passwords stored directly in the database [verses accounts from LDAP or trusting an external authorization mechanism such as Kerberos]. Explanation for the steps to process a Basic authentication operation have been added to the code example:

from base64 import b64decode
from crypt  import crypt
class DBAuthenticator(Authenticator):

def _authenticate(self, context, request):
  Authenticator._authenticate(self, context, request)
  authorization = request.headers.get('authorization')
  if (authorization == None):
    raise AuthenticationException('Authentication Required')
  (kind, data) = authorization.split(' ')
  if (kind == 'Basic'):
    # Authentication method is  "Basic"
    (username, _, password) = b64decode(data).partition(':')
    """ This method provided by the parent class goes to the ORM and
        retrieves the account object for the specified username. It
        will throw an authentication exception if no such username is
        found (user entered it wrong?) or a generic Coils Exception if
        multiple objects match the username (that doesn't make sense) -
        either will stop the authentication process, but the authentication
        exception should reprompt the client to try again. """
    account = Authenticator._getLogin(self, username)
    secret = account.password
    if (secret == crypt(password, secret[:2])):
      # Password matches, user is authenticated
      self.loginId = account.objectId
      self.login   = account.login
    else:
      # Password does not match, authentication failes
      raise AuthenticationException('Incorrect username or password')
  else:
    # Authorization header indicated an authentication type other than BASIC
    CoilsException('Unsupported HTTP Authenticated Mech')

See, that easy easy.

Python XML-RPC Presentation

Presented on using XML-RPC with Python to the Grand Rapids Python Users Group (GRPUG).

Presentation File

While it is admittedly a runt of an RPC solution I've found XML-RPC to be very useful and have used it extensively in both my work on OpenGroupware and other projects.