Implementing Queue Expiration w/RabbitMQ

The latest versions of RabbitMQ support a feature where idle queues can be automatically deleted from the server.  For queues used in an RPC or workflow model this can save a lot of grief - as the consumers for these queues typically vanish leaving the queue behind. Over time these unused queues accumulate and consume resources on the server(s). If you are using pyamqplib setting the expiration on a queue is as simple as:

import amqplib.client_0_8 as amq
connection = amq.Connection(host="localhost:5672", userid=*, password=*, virtual_host="/", insist=False)
channel = connection.channel()
queue = channel.queue_declare(queue="testQueue", durable=True, exclusive=False, auto_delete=False, arguments={'x-expires': 9000})
channel.exchange_declare(exchange='testExchange', type="fanout", durable=False, auto_delete=False)
channel.queue_bind(queue="testQueue", exchange='exchange')

Now if that queue goes unused for 9 seconds it will be dropped by the server [the value is in milliseconds]. So long as the queue has consumers it will persist, but once the last consumer has disconnected and no further operations have occurred - poof, you get your resources back.

Finding Address Coordinates using Python, SOAP, & the Bing Maps API

Bing maps provides a SOAP API that can be easily accessed via the Python suds module.  Using the API it is trivial to retrieve the coordinates of a postal address.  The only requirement is to acquire a Bing API application key; this process is free, quick, and simple.

import sys, urllib2, suds

if __name__ == '__main__':  
    url = 'http://dev.virtualearth.net/webservices/v1/geocodeservice/geocodeservice.svc?wsdl'
    
    client = suds.client.Client(url)
    client.set_options(port='BasicHttpBinding_IGeocodeService')
    request = client.factory.create('GeocodeRequest')

    credentials = client.factory.create('ns0:Credentials')
    credentials.ApplicationId = 'YOUR-APPLICATION-KEY'
    request.Credentials = credentials

    #Address
    address = client.factory.create('ns0:Address')
    address.AddressLine = "535 Shirley St. NE"
    address.AdminDistrict = "Michigan"
    address.Locality = "Grand Rapids"      
    address.CountryRegion = "United States"
    request.Address = address

    try:        
        response = client.service.Geocode(request)    
    except suds.client.WebFault, e:        
        print "ERROR!"        
        print(e)
        sys.exit(1)

    locations = response['Results']['GeocodeResult'][0]['Locations']['GeocodeLocation']
    for location in locations:        
        print(location)

If you need to make the request via an HTTP proxy server expand the line client = suds.client.Client(url) to:

    proxy = urllib2.ProxyHandler({'http': 'http://YOUR-PROXY-SERVER:3128'})\
    transport = suds.transport.http.HttpTransport()
    transport.urlopener = urllib2.build_opener(proxy)
    client = suds.client.Client(url, transport=transport)

The results will be Bing API GeocodeLocation objects that have an Longitude and Latitude properties.  Note that you may receive multiple coordinates for an address as there are multiple mechanism for locating an address; the method corresponding to the coordinates is a string in the CalculationMethod property of the GeocodeLocation objects.

Suppressing SNMP Connection Messages

You have, of course, done the responsible sys-admin thing and setup an NMS (be it ZenOSS, OpenNMS, Nagios, whatever...).  Then there is the concommitant action of configuring SNMP services on all the relevant hosts.  All is good.  But running SNMP on several distributions churns out the log messages;  when you go to the logs to research a problem you have to filter out and sort through thousands upon thousands of pointless messages like:

Aug  1 16:08:38 flask-yellow snmpd[1976]: Connection from UDP: [192.168.1.38]:52021
Aug  1 16:08:38 flask-yellow last message repeated 24 times

Argh.  Detail logging is good, but pointless noise is not.  The solution isn't very well documented but you can bring this to a stop. 

Step 1.) Make sure you have net-snmp 5.3.2.2 or later.  This should not be a problem as even RHEL5/CentOS5 provide this version via update.

    $ rpm -q net-snmp
    net-snmp-5.3.2.2-9.el5_5.1

Step 2.) Edit /etc/sysconfig/snmpd.options or your system's equivalent making sure you do not pass the "-a" option to the SNMP daemon.  The "-a" option enables the logging of the source IP addresses of all incoming requests.  If you want to know about these kind of events iptables and ulog are more reliable and efficient methods for capturing that information.

    # OPTIONS="-Lsd -Lf /dev/null -p /var/run/snmpd.pid -a"
    OPTIONS="-Lsd -Lf /dev/null -p /var/run/snmpd.pid"

Step 3.) Edit the /etc/snmpd/snmpd.conf verifying you have the dontLogTCPWrappersConnects directive set to 1 (true).  Add this directive to the configuration file if it is not specified.
Step 4.) Restart the SNMP service.

Now when you go to look into the log files it is again possible to hear the breeze, the singing of the birds, and the distant growling of that guy from Kazakstan who is trying to crack your SSH daemon.

Automated Backup of IOS Router Configuration

Who hasn't had the experience of remoting to a router and making a configuration change... and not saving that change. Inevitably that is the weekend that facility will experience a power outage long enough to deplete their UPS. And then you get that dreaded text message from NetOps that a facility is down. Argh! Fortunately Cisco IOS 12.x and later supports a cron like service known as "kron". One of the handiest uses for kron is to configure automatic backup of the router's configuration to a TFTP server.

kron occurrence backup at 0:00 Thu recurring
 policy-list backup
!
kron policy-list backup
 cli write
 cli show running-config | redirect tftp://192.168.1.38/brtgate.config
!

This creates a batch of commands named "backup" [where in typical ISO fashion everything is referred to as a "policy"] that will be executed every Thursday morning. This batch commits the running configuration to flash memory ["cli write"] and copies the running configuration to the specified TFTP server ["cli show running-config | redirect tftp://192.168.1.38/brtgate.config"]. The rather odd looking use of "redirect" is because the IOS "copy" command is interactive and interactive commands cannot be run via "kron".

Remember that the file on the TFTP server has to exist, even if zero sized, and be world writable; otherwise the redirect will fail with a permission denied error.

block_dump logging

There are lots of tools for studying the systems use of CPU and memory, but I/O is generally harder to track down.  A useful trick is available via the block dump.  Setting the value to "1" turns on block access logging to the kernel ring-buffer [aka dmesg] and a value of "0" turns it back on.  This means it can be turned on by a simple:

echo "1" > /proc/sys/vm/block_dump

This logs the accesses to the block storage as:

[ 2032.934178] postmaster(11528): READ block 5058592 on dm-3 (16 sectors)
[ 2032.934200] postmaster(11528): READ block 5058624 on dm-3 (32 sectors)
[ 2032.934240] postmaster(11528): READ block 3172800 on dm-3 (16 sectors)
[ 2032.945328] banshee-1(11267): dirtied inode 1051864 (banshee.db-journal) on dm-0
[ 2032.945336] banshee-1(11267): dirtied inode 1051864 (banshee.db-journal) on dm-0
[ 2033.042671] python(11518): READ block 9017928 on dm-2 (32 sectors)
[ 2033.055771] python(11518): dirtied inode 267260 (expatbuilder.pyc) on dm-2
[ 2033.055808] python(11518): READ block 9017960 on dm-2 (40 sectors)
[ 2033.412972] nautilus(11078): dirtied inode 410492 (dav:host=127.0.0.1,port=8080,ssl=false) on dm-0
[ 2033.413001] nautilus(11078): READ block 50855560 on dm-0 (40 sectors)
[ 2033.431011] nautilus(11078): dirtied inode 410596 (dav:host=127.0.0.1,port=8080,ssl=false-ab9de673.log) on dm-0
[ 2033.431044] nautilus(11078): READ block 50855736 on dm-0 (64 sectors)
[ 2034.221831] jbd2/dm-2-8(386): WRITE block 21261800 on dm-2 (8 sectors)
[ 2034.221887] jbd2/dm-2-8(386): WRITE block 21261808 on dm-2 (8 sectors)

 Handy.

Manually Adding an ACL To An Object

In OpenGroupware the ACLs applied to an object are stored in the "object_acl" table.  If, for example, I want to add the list, view, read, write, and administer privileges for the team 11,530 for object 1,6829,810 the correct SQL to execute is:

INSERT INTO object_acl
  (object_acl_id, sort_key, action, object_id, auth_id, permissions)
VALUES (nextval('key_generator'), 0, 'allowed', 16829810, 11530, 'lvrwa')

The important points are:

1. Use the "key_generator" sequence to assign the "object_acl_id" value.  This is the object id of the ACL itself;  all object ids are assigned from the key_generator sequence.
2. The value of "sort_key" is always 0.  This value isn't actually used for anything.
3. The value of "action" must be either "allowed" or "denied".  In most cases "allowed" is what you want in order to grant access.
4. "object_id" is the object id of the object to which the ACL is applied in contrast to "auth_id" is the context to which the privileges, specified in "permissions", are either granted [if "action" is "allowed"] or revoked [if "action" is "denied"].  The value of "auth_id" should be the object id of an account or a team.
5. The permissions string is always lower case.  Permission flags are documented in WMOGAG.

The ACLs in "object_acl" are the primary access control mechanism for all entities excepting Projects and Appointments.

Bootstrapping "opengoupware.us"

Thanks to the addition of the "pages" feature in Blogspot I've finally created a site at opengroupware.us, The OpenGroupware [Legacy] project website (I won't even bother to link to it) has been worthless for some time, and the information about the constellation of projects beyond legacy is very scattered. opengroupware.us is an attempt to at least create an index of that information as well as resurrect some of the good content from the abyss that is the docs plone.  If you want to submit content to opengroupware.us or want to help edit / maintain content just let me know any I'll add your Blogspot account to the site permissions.

Coils: Features merged into "default"

This morning the following features were merged into OpenGroupware Coils "default" branch, and will be available in the next release.

•  The scheduler service no longer depends on the ticktock heartbeat.  It checks the run queue in its work method.
• Format(s) now support logging of rejected records to a buffer.  This feature is available via a new parameter on the readAction: "rejectionsLabel".  This is documented in the readAction's wiki page.
•  The above motivated me to abstract how ActionCommand goes about creating [output] messages.  It is now simple for an action to create additional messages [beyond it's 'default' output message] by just calling the store_in_message(label, wfile, mimetype='application/octet-stream') method.
• New CLI tools: "coils-list-schedule" and "coils-unschedule-process" that let the administrator view and modify the workflow scheduler from the command line.  Yes, we still need a "coils-schedule-process". 
• getopt isn't the greatest command line option parser,  would be great if someone would look into swapping that out with something better.
• The above motivated me to add an initialize_tool(name, argv, arguments=['',[]]) function to coils.core.utility.  This provides a simple way for a tool [vs. a service] to bootstrap itself onto the OpenGroupware Coils service bus so that it can receive callbacks, etc... Return value is an AdministrativeContext (with a registered Broker for IPC) and dictionary of unconsumed command line parameters. initialize_tool automatically consumes the --store=, --add-bundle=, and --ban-bundle parameters in order to initialize the Coils environment.

In the last week or so there has been significant work on tidying things up, and the deployment instructions on the wiki have been updated as well.  Getting a working OpenGroupware Coils instance should now be very straight-forward.   Big work ahead is the move to WSGI which is underway in the NetComm branch (which needs to be remerged with default, btw).  And with the impending release of openSUSE 11.3, and its inclusion of Evolution 1.30.x,  the GroupDAV address book support can be completely tested and polished.  Evolution 1.30.x contains several enhancements to make it's WebDAV address book support fully compatible with GroupDAV.

Keeping an OIE Route After Completion

Once an OpenGroupware Integration Engine route has successfully completed it is automatically garbage collected.  This prevents the system from filling up with messages and version information.  But if an external application, like ogo-curses-putter, wants to retrieve the output of the route... it's gone!  To facilitate this use-case the garbage collection can be suppressed per-route by setting the {http://www.opengroupware.us/oie}preserveAfterCompletion property of the route entity with a value of "YES".  If this property value exists the processes derived from that route will not be garbage collected.  The simplest way to set the property is to retrieve the route via "route::get" and use your context's property manager:

from coils.core import *
ctx = AdministrativeContext()
r = ctx.run_command('route::get', name='MTAStockOrder_TEST')
ctx.property_manager.set_property(r, 'http://www.opengroupware.us/oie', 'preserveAfterCompletion', 'YES')
ctx.commit() 

The above will set the value of {http://www.opengroupware.us/oie}preserveAfterCompletion to "YES" for the route named "MTAStockOrder_TEST". The property manager will update the value of the property if such a property already exists, or it will create a new property with the specified value.

Performing PROPFIND from Python

Surprising, when I went looking for examples of performing PROPFIND requests from Python I found very little information. For testing OpenGroupware Coil's WebDAV presentation I needed a simple way to perform the PROPFIND required to list the entities in a collection and then request each entity. With a bit of scratching through the thin documentation I was able to come up with the following:

1. import httplib, urllib2, base64, sys
2. from lxml import etree
4. PROPFIND = u'''<?xml version="1.0" encoding="utf-8"?>
5. <propfind xmlns="DAV:">
6. <prop>
7. <getetag/>
8. </prop>
9. </propfind>'''
11. SERVER_HOST = '127.0.0.1'
12. SERVER_PORT = 8080
13. SERVER_PATH = '/dav/Contacts'
14. CLIENT_AGENT = 'Whitemice rules/so very true'
16. auth_string = 'Basic {0}'.format(base64.encodestring('adam:fred123')[:-1])
18. urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler()))
20. connection = httplib.HTTPConnection(SERVER_HOST, SERVER_PORT)
21. connection.putrequest('PROPFIND', SERVER_PATH)
22. connection.putheader('Authorization', auth_string)
23. connection.putheader('User-Agent', CLIENT_AGENT)
24. connection.putheader('Depth', 1)
25. connection.putheader('Content-Length', str(len(PROPFIND)))
26. connection.endheaders()
27. connection.send(PROPFIND)
28. response = connection.getresponse()
29. if response.status == 207:
30.   data = response.read()
31.   connection.close()
32.   response = None
33.   namespace_prefix_map = { 'D' : 'DAV:' }
34.   document = etree.fromstring(data)
35.   for path in document.xpath('/D:multistatus/D:response/D:href/text()',
36.       namespaces=namespace_prefix_map):
37.     if (path != SERVER_PATH):
38.       connection = httplib.HTTPConnection('127.0.0.1', 8080)
39.       connection.putrequest('GET', path)
40.       connection.putheader('Authorization', auth_string)
41.       connection.putheader('User-Agent', CLIENT_AGENT)
42.       connection.endheaders()
43.       response = connection.getresponse()
44.       if response.status != 200:
45.         print 'Error retrieving {0}'.format(path)
46.         sys.exit(1)

This will make the PROPFIND for all the items in the collection, requesting the getetag property, and then from the response perform an HTTP GET for every item. Even better would be if it requested isCollection and knew better than to perform GETs on collections.  I hope this simple example will be of use to someone.

Disabling GNOME Automount

When you connect a mass storage device to a computer running the GNOME desktop environment it automatically mounts the device in /media and places a short-cut to the device on your desktop.  This is an excellent default behavior;  but if you are working with various devices sometimes it can get in the way.  To correctly disable this feature simply execute:

gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount false

The devices will still appear in computer:/// (in nautilus) where you can right click on them to select the mount action if and when you want them to be mounted.  If and when you want to re-enable to auto-mounting feature execute:

gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount true

Configuring OpenLDAP's dynlist in cn=config

Step#1: Make sure the dynlist module is loaded on the server.  What modules are loaded are typically controlled by the "cn=modules{0}, cn=config" object.  If your server doesn't have a modules object it probably isn't loading any dynamic modules.  Create an LDIF file and import it.  Our "cn=modules{0}, cn=config" looks like:

dn: cn=modules{0}, cn=config
olcModuleLoad: {0}accesslog.la
olcModuleLoad: {1}auditlog.la
olcModuleLoad: {2}constraint.la
olcModuleLoad: {3}dynlist.la
olcModuleLoad: {4}memberof.la
olcModuleLoad: {5}ppolicy.la
olcModuleLoad: {6}refint.la
olcModuleLoad: {7}seqmod.la
olcModuleLoad: {8}syncprov.la
olcModuleLoad: {9}sssvlv.la
olcModuleLoad: {10}translucent.la
olcModuleLoad: {11}unique.la
olcModuleLoad: {12}back_monitor.la
olcModulePath: /usr/lib/openldap2.4
objectClass: olcModuleList
cn: modules{0}

NOTE: Double check that the "olcModulePath" is the absolute path to the directory containing the OpenLDAP modules.

Step#2: Check that your schema has the groupOfURLs objectclass defined.  If you attempt to configure the dynlist module without that schema available you still crash slapd.  To define the dynamicGroup schema (if it is missing) you can import the following LDIF into your cn=config:

dn: cn=dynamicGroup, cn=schema, cn=config
olcObjectClasses: {0}( 2.16.840.1.113730.2.33         NAME 'groupOfURLs'   
     SUP top STRUCTURAL         MUST cn         MAY ( memberURL $ businessCat
 egory $ description $ o $ ou $                 owner $ seeAlso ) )
olcAttributeTypes: {0}( 2.16.840.1.113730.1.198         NAME 'memberURL'   
     DESC 'Identifies an URL associated with each member of a group. Any type
  of labeled URL can be used.'         SUP labeledURI )
objectClass: olcSchemaConfig
cn: dynamicGroup

Step#3: Create an olcOverlayConfig object in the scope of your Dit database.  For example, our Dit is the 1st HDB database on the server so the appropriate object is:

dn: olcOverlay=dynlist,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynamicList
olcOverlay: dynlist

Step#4: Now you are ready to actually use the dynlist module.  A common use-case is to create dynamic mail alias objects;  with dynlist you don't need to maintain mail aliases, they will automatically contain everyone who matches the relevent criteria.  Provided you use the traditional nisMailAlias objectclass in order to define mail aliases adding the attribute -

olcDlAttrSet: {0}nisMailAlias labeledURI rfc822mailmember:mail

 - to "olcOverlay=dynlist,olcDatabase={1}hdb,cn=config" will enable dynamic mail aliases.  Specifically any nisMailAlias containing a labeledURI attribute will be expanded by the query specified in that attribute.  The results of that query will be rewritten by the optional rfc822mailmember:mail clause which will rename the mail attributes resulting from the query into the rfc822mailmember attribute required by consumers of the nisMailAlias objects.  So rather than populating the mail alias object with rfc822mailmember attributes manually, you extend the object with the auxilliary labeledURIObject objectclass and define the query in the labeledURI attribute.

dn: cn=gr_parts, ou=ListAliases, ou=Aliases, ou=Mail, ou=SubSystems, o=Morrison Industries,c=US
mail: gr_parts@morrison-ind.com
labeledURI: ldap:///ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US?mail?one?(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))
objectClass: nisMailAlias
objectClass: top
objectClass: labeledURIObject
cn: gr_parts

The object will immediately populate with rfc822mailmember attributes derived from the mail attribute of those objects matching the specified filter: "(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))".  The third parameter, "one", of the URI is the scope of the query so only objects immediately subordinate to "ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US" are candidates for the filter.

Python Curses In Action, even on AIX.

So, what if you have an old COBOL application that you want to integrate with some web services or a website? You'd need a "green-screen" application that can take that COBOL applications temporary output file, send it up to the web service, wait for a response, replace the temporary file, and then return control to the COBOL application (or whatever "green screen" application comes next). It sounds simpler than it is in practice: what if the web service takes awhile to complete [or has human involvement on the other end?!] or the web service isn't available right at the moment you try to send your request? For a real production environment your "green screen" application would need to deal with all of those things.
And you need that "green screen" application to work on both LINUX and AIX!

Fortunately pware provides Python 2.6 for AIX, including curses! The curses library and respective Python module provide a surprisingly easy way to create professional looking TUI (Text User Interface) applications like the solution required in our example. This particular application takes them temporary file and submits it via a WebDAV PUT operation to the workflow engine provided by OpenGroupware Coils. The route in the workflow engine reformats the dreadful output of the COBOL application into the required format (using the format support in route Read and Write operations). The client detects the route is complete by watching the URL specified in the header of the initial PUT, and then downloads the required data. If anything goes wrong along the way the client can retry, or provide the user the option to abort. All along the way the client provides detailed feedback to the user and a familiar dialog-box and prompt interface when feedback is required.

New OGo Help Desk Feature

Enhancement Bug#2027 "Allow help desk users to create tasks on behalf of users" has been resolved as of r2274. This update requires a database schema change -

ALTER TABLE job ADD COLUMN owner_id INT;

UPDATE job SET owner_id = creator_id;

The database scripts pg-build-schema.psql and pg-update-1.x-to-5.5.psql have been updated. NOTE: Be careful not to run this part of pg-update-1.x-to-5.5.psql more than once or you risk modifying actual data in your database if you use the OGoHelpDeskRoleName default.

This feature allows a member of the team whose named is defined in the OGoHelpDeskRoleName default to create tasks with an owner other than themselves. Delegated and archived task lists now display tasks based on owner rather than creator. By default the owner is the creator so this has no effect on normal task behaviour. Currently the help desk feature, setting of the owner to another user, is only available via the zOGI API. The modifications to the zOGI API are documented on the Task entity.

If your help desk team name contains spaces be sure to use proper quoting - Defaults write NSGlobalDomain OGoHelpDeskRoleName '"all intranet"' - or you may not get the results you expect. Obviously if you set OGoHelpDeskRoleName to "all intranet" all users will be able to create tasks on behalf of other users.

Service BASIC HTTP Authentication with Python

While building the core parts of OpenGroupware COILS I noticed there don't appear to be any example of providing HTTP's BASIC authentication scheme anywhere on the interweb. Or any other authentication schemes for that matter. To remedy that here is a rough outline, including the example code, of how it is implemented in COILS. First spin up an HTTP server:

import BaseHTTPServer

class HTTPServer(BaseHTTPServer.HTTPServer):
pass

from coils.net.handler   import HTTPRequestHandler
from coils.net.server    import HTTPServer
....
HTTP_HOST = 'localhost'
HTTP_PORT = 8080
httpd = HTTPServer((HTTP_HOST, HTTP_PORT), HTTPRequestHandler)
httpd.serve_forever()

Your app will almost certainly provide a custom HTTPRequestHandler to delegate the requests to whatever logic your application provides. In the BaseHTTPRequestHandler a GET request is handed to do_GET, a POST request to do_POST, etc... In our case the object that handles the request will decide what to do based on the request type so we channel all requests to our generic process_request method. process_request looks up the object targeted by the request via marshall_handler() [not shown] and then calls that objects do_request method. The important part for authentication is to catch the exception raised by the object handling the request if that object thinks the request is not authenticated and requires authentication. In this case that is the COILS' AuthenticationException; your application has to provide something equivalent. In order to make the client try again with authentication you need to send a "WWW-Authenticate" header telling the client to use basic authentication and what realm to use. See RFC2617 for details on Basic authentication in general

class HTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

def process_request(self):
  """Respond to a request"""
  try:
    """ find the object mapped to the specified request
        The marshall_handler() is a COILS specific things so it isn't
        shown in this example. """
    handler = self.marshall_handler()
    handler.do_request(self)
  except AuthenticationException, err:
    """ An AuthenticationException has an error code of 401
        We need to add an authentication header so the client will know
        to respond the the failure with the appropriate credentials
        TODO: Provide a digest realm once digest authentication is supported
    """
    self.send_response(err.error_code())
    self.send_header('WWW-Authenticate', 'Basic realm="OpenGroupware COILS"')
    self.end_headers()
    self.wfile.write('Authentication failure')
  except CoilsException, err:
    # An Coils Exception has an error code of 500
    self.send_response(err.error_code(), err.error_text())
    self.end_headers()
  except Exception, err:
    # Yikes, something generic web very wrong
    self.send_response(500, err)
    self.end_headers()

def do_GET(self):
  """Respond to a GET request."""
  self.process_request()

def do_POST(self):
  """Respond to a POST request"""
  self.process_request()

In order to actually process the authentication request COILS uses an Authenticator object. Our DBAuthenticator object provides basic authentication against accounts with passwords stored directly in the database [verses accounts from LDAP or trusting an external authorization mechanism such as Kerberos]. Explanation for the steps to process a Basic authentication operation have been added to the code example:

from base64 import b64decode
from crypt  import crypt
class DBAuthenticator(Authenticator):

def _authenticate(self, context, request):
  Authenticator._authenticate(self, context, request)
  authorization = request.headers.get('authorization')
  if (authorization == None):
    raise AuthenticationException('Authentication Required')
  (kind, data) = authorization.split(' ')
  if (kind == 'Basic'):
    # Authentication method is  "Basic"
    (username, _, password) = b64decode(data).partition(':')
    """ This method provided by the parent class goes to the ORM and
        retrieves the account object for the specified username. It
        will throw an authentication exception if no such username is
        found (user entered it wrong?) or a generic Coils Exception if
        multiple objects match the username (that doesn't make sense) -
        either will stop the authentication process, but the authentication
        exception should reprompt the client to try again. """
    account = Authenticator._getLogin(self, username)
    secret = account.password
    if (secret == crypt(password, secret[:2])):
      # Password matches, user is authenticated
      self.loginId = account.objectId
      self.login   = account.login
    else:
      # Password does not match, authentication failes
      raise AuthenticationException('Incorrect username or password')
  else:
    # Authorization header indicated an authentication type other than BASIC
    CoilsException('Unsupported HTTP Authenticated Mech')

See, that easy easy.

Database Changes to OpenGroupware v5.5

For those building from trunk, or currently pulling packages from OBS, there is a new table required as of r2256.

CREATE TABLE ctags (
entity VARCHAR NOT NULL,
ctag INTEGER NOT NULL DEFAULT 0
);
INSERT INTO ctags (entity) VALUES ('Person');
INSERT INTO ctags (entity) VALUES ('Enterprise');
INSERT INTO ctags (entity) VALUES ('Date');
INSERT INTO ctags (entity) VALUES ('Job');
INSERT INTO ctags (entity) VALUES ('Team');

This is part of adding ctag support to OGo (specifically ZideStore). ctags allow a client to *very* quickly detect if the contents of a collection have changed; such as the /public/Contacts folder. Thus avoiding doing a PROPFIND on the entire folder to determine if a re-sync is needing. ctag support is not complete or working yet, but without this table a server post-r2256 will fails some operations with a database error.

The subversion repository contains an update script for migrating a v5.4 database to v5.5.

Python XML-RPC Presentation

Presented on using XML-RPC with Python to the Grand Rapids Python Users Group (GRPUG).

Presentation File

While it is admittedly a runt of an RPC solution I've found XML-RPC to be very useful and have used it extensively in both my work on OpenGroupware and other projects.

Logging *USEFUL* Web Access To PostgreSQL

It is pretty easy to configure syslog to log system messages to a PostgreSQL database; and that sure beats flat files. With some GRANT/REVOKE goodness this setup can be made quite secure - only allowing the message import process to insert messages and denying the ability to everyone else to modify the contents of the database. But all that data is still pretty ugly to deal with as the messages are just long strings; so some method is needed to break down that "data" into "information". I had the specific need to do this with SQUID weblogs. I already had syslog logging to a PostgreSQL database so the first step was just to have SQUID use syslog to log access. That is easily accomplished like:

cache_access_log syslog:LOG_LOCAL4

After a restart SQUID messages show up in the LOCAL4 facility (table: facility_local4) with the priority of "info". Then I created a table to hold the parsed messages:

CREATE TABLE proxy_audit (
timestamp TIMESTAMP,
elapsed INT,
source VARCHAR(40),
status VARCHAR(15),
size INT,
method VARCHAR(15),
url VARCHAR(128),
domain VARCHAR(60),
identity VARCHAR(25),
mimetype VARCHAR(45));

Last a trigger on the LOCAL4 log parses incoming messages into this table:

CREATE OR REPLACE FUNCTION p_proxy_audit() RETURNS trigger AS '
BEGIN
IF (new.hostname = ''hod-a'' AND new.priority = ''info'') THEN
INSERT INTO proxy_audit (timestamp, elapsed, source, status, size,
method, url, domain, identity, mimetype)
VALUES(''epoch''::TIMESTAMP + (split_part(split_part(new.message, '' '', 1), ''.'', 1)::INT) * ''1 second''::INTERVAL,
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 1)::INT,
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 2),
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 3)::VARCHAR(15),
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 4)::INT,
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 5)::VARCHAR(15),
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 6)::VARCHAR(128),
split_part(split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 6), ''/'', 3)::VARCHAR(60),
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 7)::VARCHAR(25),
split_part(TRIM(substring(new.message, strpos(new.message, '' ''))), '' '', 9));
END IF;
RETURN new;
END
' LANGUAGE plpgsql;
CREATE TRIGGER t_proxy_audit AFTER INSERT
ON facility_local4 FOR EACH ROW
EXECUTE PROCEDURE p_proxy_audit()

Once you get allot of data in your table, which happens quickly, you'll probably discover you need an index.

CREATE INDEX proxy_audit_i1 ON proxy_audit(identity, timestamp);

Now looking at a user's web usage is straight-forward. Finally.

OBS Packages Updated

OpenGroupware packages on the OBS have been updated to r2207. This revision provides improved GroupDAV version 1 compatibility and adds GroupDAV version 2 support. The upside of those improvements is that the new ZideOne Outlook plugin should work well with this revision (or later) of OpenGroupware. Numerous enhancements and bug fixes in Logic and zOGI are also included.

NOTE: The GroupDAV version 2 specification has not been published yet.

Bug#394

OpenGroupware revision 2181 closes Bug#394. It is now possible to set a default to determine the default storage backend for new projects. (Wow, with GNUstep based stuff one ends up using the term "default" so often in different ways....)

To select database as your default project storage backend:
Defaults write NSGlobalDomain OGoDefaultProjectStorageBackend Database

To select the filesystem as your default project storage backend:
Defaults write NSGlobalDomain OGoDefaultProjectStorageBackend FileSystem

Most people I assume will want "Database". Remember that default values are CASE SENSITIVE!

If you (at least on r2181) attempt to create a project without selecting FileSystem/Database you'll get a popup warning: "Please specify your project storage!" If the default is set then the preferred storage backend is preselected (and can be changed).

Aside: If you use FileSystem projects make sure you have the SkyFSPath default set and that the referred to directory has the correct permissions.