Manually Adding an ACL To An Object

In OpenGroupware the ACLs applied to an object are stored in the "object_acl" table.  If, for example, I want to add the list, view, read, write, and administer privileges for the team 11,530 for object 1,6829,810 the correct SQL to execute is:
INSERT INTO object_acl
  (object_acl_id, sort_key, action, object_id, auth_id, permissions)
VALUES (nextval('key_generator'), 0, 'allowed', 16829810, 11530, 'lvrwa')
The important points are:
  1. Use the "key_generator" sequence to assign the "object_acl_id" value.  This is the object id of the ACL itself;  all object ids are assigned from the key_generator sequence.
  2. The value of "sort_key" is always 0.  This value isn't actually used for anything.
  3. The value of "action" must be either "allowed" or "denied".  In most cases "allowed" is what you want in order to grant access.
  4. "object_id" is the object id of the object to which the ACL is applied in contrast to "auth_id" is the context to which the privileges, specified in "permissions", are either granted [if "action" is "allowed"] or revoked [if "action" is "denied"].  The value of "auth_id" should be the object id of an account or a team.
  5. The permissions string is always lower case.  Permission flags are documented in WMOGAG.
The ACLs in "object_acl" are the primary access control mechanism for all entities excepting Projects and Appointments.


Python Exceptions and "auto" Instantiation

Interesting thread on the python-list as to what happens regarding raising exceptions. The low-down: If you "raise ExceptionClass" and instance of that ExceptionClass is always created.

For example:
try: raise KeyError
except KeyError: pass

From the documentation:
raise evaluates the first expression as the exception object. It must be either a subclass or an instance of BaseException. If it is a class, the exception instance will be obtained when needed by instantiating the class with no arguments.

Some exception classes require arguments so the naked raise will fail with a TypeError.
 raise UnicodeDecodeError
Traceback (most recent call last):
  File "", line 1, in
TypeError: function takes exactly 5 arguments (0 given)

Seems to me it is clearer to just do the raise with instance syntax:
  raise KeyError()
except KeyError as e:


SQLAlchemy & Upcoming Birthdays

OpenGroupware Coils uses SQLAlchemy  as it's ORM. One of the desired features was a Logic command that efficiently returns contacts with upcoming birthdays.  In raw SQL this query would be very simple to write - but how to do it in SQLAlchemy? The answer: "sql.expression.extract" which will create an expression column equivalent to EXTRACT. With EXTRACT it is possible to compare to the year-of-day represented by a date.  The Python code looks like:
db = self._ctx.db_session()
# Get the current day-of-year
doy = datetime.today().timetuple().tm_yday
# Deal with year wrap-around
floor = doy - 2
if (floor < 1): floor +=365
ceiling = doy + 14
if (ceiling > 365): ceiling -= 365
# Create a field that is the SQL expression DOY(Contact.birth_date)
orm_doy = sql.expression.extract('doy', Contact.birth_date)
# Create the query
query = db.query(Contact).filter(and_(sql.expression.between(orm_doy, floor, ceiling),
                                      Contact.birth_date != None,
                                      Contact.is_account == self.accounts,
                                      Contact.status != 'archived'))
In this example "self.accounts" is an attribute with a value of 0 or 1.


Bootstrapping "opengoupware.us"

Thanks to the addition of the "pages" feature in Blogspot I've finally created a site at opengroupware.us, The OpenGroupware [Legacy] project website (I won't even bother to link to it) has been worthless for some time, and the information about the constellation of projects beyond legacy is very scattered. opengroupware.us is an attempt to at least create an index of that information as well as resurrect some of the good content from the abyss that is the docs plone.  If you want to submit content to opengroupware.us or want to help edit / maintain content just let me know any I'll add your Blogspot account to the site permissions.


Coils: Features merged into "default"

This morning the following features were merged into OpenGroupware Coils "default" branch, and will be available in the next release.
  •  The scheduler service no longer depends on the ticktock heartbeat.  It checks the run queue in its work method.
  • Format(s) now support logging of rejected records to a buffer.  This feature is available via a new parameter on the readAction: "rejectionsLabel".  This is documented in the readAction's wiki page.
  •  The above motivated me to abstract how ActionCommand goes about creating [output] messages.  It is now simple for an action to create additional messages [beyond it's 'default' output message] by just calling the store_in_message(label, wfile, mimetype='application/octet-stream') method.
  • New CLI tools: "coils-list-schedule" and "coils-unschedule-process" that let the administrator view and modify the workflow scheduler from the command line.  Yes, we still need a "coils-schedule-process". 
    • getopt isn't the greatest command line option parser,  would be great if someone would look into swapping that out with something better.
  • The above motivated me to add an initialize_tool(name, argv, arguments=['',[]]) function to coils.core.utility.  This provides a simple way for a tool [vs. a service] to bootstrap itself onto the OpenGroupware Coils service bus so that it can receive callbacks, etc... Return value is an AdministrativeContext (with a registered Broker for IPC) and dictionary of unconsumed command line parameters. initialize_tool automatically consumes the --store=, --add-bundle=, and --ban-bundle parameters in order to initialize the Coils environment.
In the last week or so there has been significant work on tidying things up, and the deployment instructions on the wiki have been updated as well.  Getting a working OpenGroupware Coils instance should now be very straight-forward.   Big work ahead is the move to WSGI which is underway in the NetComm branch (which needs to be remerged with default, btw).  And with the impending release of openSUSE 11.3, and its inclusion of Evolution 1.30.x,  the GroupDAV address book support can be completely tested and polished.  Evolution 1.30.x contains several enhancements to make it's WebDAV address book support fully compatible with GroupDAV.


Keeping an OIE Route After Completion

Once an OpenGroupware Integration Engine route has successfully completed it is automatically garbage collected.  This prevents the system from filling up with messages and version information.  But if an external application, like ogo-curses-putter, wants to retrieve the output of the route... it's gone!  To facilitate this use-case the garbage collection can be suppressed per-route by setting the {http://www.opengroupware.us/oie}preserveAfterCompletion property of the route entity with a value of "YES".  If this property value exists the processes derived from that route will not be garbage collected.  The simplest way to set the property is to retrieve the route via "route::get" and use your context's property manager:

from coils.core import *
ctx = AdministrativeContext()
r = ctx.run_command('route::get', name='MTAStockOrder_TEST')
ctx.property_manager.set_property(r, 'http://www.opengroupware.us/oie', 'preserveAfterCompletion', 'YES')
The above will set the value of {http://www.opengroupware.us/oie}preserveAfterCompletion to "YES" for the route named "MTAStockOrder_TEST". The property manager will update the value of the property if such a property already exists, or it will create a new property with the specified value.


Performing PROPFIND from Python

Surprising, when I went looking for examples of performing PROPFIND requests from Python I found very little information. For testing OpenGroupware Coil's WebDAV presentation I needed a simple way to perform the PROPFIND required to list the entities in a collection and then request each entity. With a bit of scratching through the thin documentation I was able to come up with the following:
  1. import httplib, urllib2, base64, sys
  2. from lxml import etree
  3. PROPFIND = u'''<?xml version="1.0" encoding="utf-8"?>
  4. <propfind xmlns="DAV:">
  5. <prop>
  6. <getetag/>
  7. </prop>
  8. </propfind>'''
  9. SERVER_HOST = ''
  10. SERVER_PORT = 8080
  11. SERVER_PATH = '/dav/Contacts'
  12. CLIENT_AGENT = 'Whitemice rules/so very true'
  13. auth_string = 'Basic {0}'.format(base64.encodestring('adam:fred123')[:-1])
  14. urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler()))
  15. connection = httplib.HTTPConnection(SERVER_HOST, SERVER_PORT)
  16. connection.putrequest('PROPFIND', SERVER_PATH)
  17. connection.putheader('Authorization', auth_string)
  18. connection.putheader('User-Agent', CLIENT_AGENT)
  19. connection.putheader('Depth', 1)
  20. connection.putheader('Content-Length', str(len(PROPFIND)))
  21. connection.endheaders()
  22. connection.send(PROPFIND)
  23. response = connection.getresponse()
  24. if response.status == 207:
  25.   data = response.read()
  26.   connection.close()
  27.   response = None
  28.   namespace_prefix_map = { 'D' : 'DAV:' }
  29.   document = etree.fromstring(data)
  30.   for path in document.xpath('/D:multistatus/D:response/D:href/text()',
  31.       namespaces=namespace_prefix_map):
  32.     if (path != SERVER_PATH):
  33.       connection = httplib.HTTPConnection('', 8080)
  34.       connection.putrequest('GET', path)
  35.       connection.putheader('Authorization', auth_string)
  36.       connection.putheader('User-Agent', CLIENT_AGENT)
  37.       connection.endheaders()
  38.       response = connection.getresponse()
  39.       if response.status != 200:
  40.         print 'Error retrieving {0}'.format(path)
  41.         sys.exit(1)

This will make the PROPFIND for all the items in the collection, requesting the getetag property, and then from the response perform an HTTP GET for every item. Even better would be if it requested isCollection and knew better than to perform GETs on collections.  I hope this simple example will be of use to someone.


OIE Progress, and lots of it.

The workflow engine [OIE] in the OpenGroupware Coils project has reached a point a general usefulness.  OIE is accessible via WebDAV for creating, browsing, and starting flows.  The engine compiles BPML 1.0 markup to an internal format (adding additional markup-notations, such as XPDL,  shouldn't be much harder than writing a SaX parser to produce the internal format).  The foreach flow control structure is implemented;  switch, until, and while are simply stubs at this point.  switch is actually compiled but can't be executed yet.  Message scope is handled correctly.  Even with just linear execution and foreach a great many business processes can be modelled, especially ETL type flows.

The actions implemented so far are:
  • read - Translate a message using a format; fixed record length and XLS formats have been implemented.
  • select - SELECT from a defined RDBMS.
  • write - Write a StandardXML message out using a specified format.
  • assign - XPath select a value or assign a static value to a message.
  • getEntity - Retrieve an entity from the groupware database.
  • xpath - Create a new message from an XPath query.
  • ldapSearch - Search an LDAP DSA, results are in DSMLv1.0.
  • regularExpressionFind - Perform regexp matches.
  • readJSON - Translate JSON data into XML.
  • sendMail - Send a message via an SMTP server, including support for attachments.
All commands support label substitution in appropriate parameters.

As of yesterday there is also an insert action for inserting data into a RDMS table;  this action hasn't yet been as heavily tested as the others,  but it completes the tool set required for simple ETL flows.

Disabling GNOME Automount

When you connect a mass storage device to a computer running the GNOME desktop environment it automatically mounts the device in /media and places a short-cut to the device on your desktop.  This is an excellent default behavior;  but if you are working with various devices sometimes it can get in the way.  To correctly disable this feature simply execute:
gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount false
The devices will still appear in computer:/// (in nautilus) where you can right click on them to select the mount action if and when you want them to be mounted.  If and when you want to re-enable to auto-mounting feature execute:
gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount true


Passing a column set to an SQLalchemy query

In implementing the List method of the EntityAccessManager provided by the Contacts bundle in OpenGroupware Coils it seemed like it would be very efficient to allow the consumer to request what set of attributes it needed;  for instance, if a WebDAV client's PROPFIND request didn't ask for a given property, why request the corresponding attribute in the query?  Especially since the result set for PROPFIND queries are frequently very large [on the order of 20,000 records or so].  But how to pass a set of attributes as a parameter?

def List(ctx, attributes):
  db = query(attributes).filter(....)
  return db.all

does not work;  where attributes is a list, or a set of more than one attribute. Some fiddling around and I discover that the following works:

def List(ctx, *attributes):
  db = query(*attributes).filter(...)
  return db.all()

ctx = AssumedContext(10100)
a = BundleManager.get_access_manager('Contact', ctx)
a.List(ctx, (Contact.object_id, Contact.version))

That single asterisk is the trick.

Configuring OpenLDAP's dynlist in cn=config

Step#1: Make sure the dynlist module is loaded on the server.  What modules are loaded are typically controlled by the "cn=modules{0}, cn=config" object.  If your server doesn't have a modules object it probably isn't loading any dynamic modules.  Create an LDIF file and import it.  Our "cn=modules{0}, cn=config" looks like:

dn: cn=modules{0}, cn=config
olcModuleLoad: {0}accesslog.la
olcModuleLoad: {1}auditlog.la
olcModuleLoad: {2}constraint.la
olcModuleLoad: {3}dynlist.la
olcModuleLoad: {4}memberof.la
olcModuleLoad: {5}ppolicy.la
olcModuleLoad: {6}refint.la
olcModuleLoad: {7}seqmod.la
olcModuleLoad: {8}syncprov.la
olcModuleLoad: {9}sssvlv.la
olcModuleLoad: {10}translucent.la
olcModuleLoad: {11}unique.la
olcModuleLoad: {12}back_monitor.la
olcModulePath: /usr/lib/openldap2.4
objectClass: olcModuleList
cn: modules{0}

NOTE: Double check that the "olcModulePath" is the absolute path to the directory containing the OpenLDAP modules.

Step#2: Check that your schema has the groupOfURLs objectclass defined.  If you attempt to configure the dynlist module without that schema available you still crash slapd.  To define the dynamicGroup schema (if it is missing) you can import the following LDIF into your cn=config:

dn: cn=dynamicGroup, cn=schema, cn=config
olcObjectClasses: {0}( 2.16.840.1.113730.2.33         NAME 'groupOfURLs'   
     SUP top STRUCTURAL         MUST cn         MAY ( memberURL $ businessCat
 egory $ description $ o $ ou $                 owner $ seeAlso ) )
olcAttributeTypes: {0}( 2.16.840.1.113730.1.198         NAME 'memberURL'   
     DESC 'Identifies an URL associated with each member of a group. Any type
  of labeled URL can be used.'         SUP labeledURI )
objectClass: olcSchemaConfig
cn: dynamicGroup

Step#3: Create an olcOverlayConfig object in the scope of your Dit database.  For example, our Dit is the 1st HDB database on the server so the appropriate object is:

dn: olcOverlay=dynlist,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynamicList
olcOverlay: dynlist

Step#4: Now you are ready to actually use the dynlist module.  A common use-case is to create dynamic mail alias objects;  with dynlist you don't need to maintain mail aliases, they will automatically contain everyone who matches the relevent criteria.  Provided you use the traditional nisMailAlias objectclass in order to define mail aliases adding the attribute -
olcDlAttrSet: {0}nisMailAlias labeledURI rfc822mailmember:mail
 - to "olcOverlay=dynlist,olcDatabase={1}hdb,cn=config" will enable dynamic mail aliases.  Specifically any nisMailAlias containing a labeledURI attribute will be expanded by the query specified in that attribute.  The results of that query will be rewritten by the optional rfc822mailmember:mail clause which will rename the mail attributes resulting from the query into the rfc822mailmember attribute required by consumers of the nisMailAlias objects.  So rather than populating the mail alias object with rfc822mailmember attributes manually, you extend the object with the auxilliary labeledURIObject objectclass and define the query in the labeledURI attribute.

dn: cn=gr_parts, ou=ListAliases, ou=Aliases, ou=Mail, ou=SubSystems, o=Morrison Industries,c=US
mail: gr_parts@morrison-ind.com
labeledURI: ldap:///ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US?mail?one?(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))
objectClass: nisMailAlias
objectClass: top
objectClass: labeledURIObject
cn: gr_parts

The object will immediately populate with rfc822mailmember attributes derived from the mail attribute of those objects matching the specified filter: "(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))".  The third parameter, "one", of the URI is the scope of the query so only objects immediately subordinate to "ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US" are candidates for the filter.


Python Curses In Action, even on AIX.

So, what if you have an old COBOL application that you want to integrate with some web services or a website? You'd need a "green-screen" application that can take that COBOL applications temporary output file, send it up to the web service, wait for a response, replace the temporary file, and then return control to the COBOL application (or whatever "green screen" application comes next). It sounds simpler than it is in practice: what if the web service takes awhile to complete [or has human involvement on the other end?!] or the web service isn't available right at the moment you try to send your request? For a real production environment your "green screen" application would need to deal with all of those things.
And you need that "green screen" application to work on both LINUX and AIX!

Fortunately pware provides Python 2.6 for AIX, including curses! The curses library and respective Python module provide a surprisingly easy way to create professional looking TUI (Text User Interface) applications like the solution required in our example. This particular application takes them temporary file and submits it via a WebDAV PUT operation to the workflow engine provided by OpenGroupware Coils. The route in the workflow engine reformats the dreadful output of the COBOL application into the required format (using the format support in route Read and Write operations). The client detects the route is complete by watching the URL specified in the header of the initial PUT, and then downloads the required data. If anything goes wrong along the way the client can retry, or provide the user the option to abort. All along the way the client provides detailed feedback to the user and a familiar dialog-box and prompt interface when feedback is required.