Configuration of NTP for Samba4

In an Active Directory domain, which is a Kerberos domain, time-synchronization is critical.  If a workstation or servers clock drifts to much from the clock of the KDC neither users or services will be able to participate in the domain.  But Active Directory clients [such as Windows 7] require an MS-SNTP server that cryptographically signs the response.  This means that the NTP server shipped with almost all current distributions cannot provide time services to these devices - support for this interoperability was added in NTP 4.2.6 (see NTP Bug#1028).  Appropriately versioned NTP packages are available for openSUSE 11.3 from namtrac's repository and will be included in openSUSE 11.4 (see openSUSE Bug#657194)..
With a good version of NTP the required NTP configuration is:
ntpsigndsocket /opt/ad/samba4/var/run/ntp_signd/
restrict default mssntp
Samba4 will create a socket which NTP can use to have its responses signed - the location of this socket is what is indicated by the NTP directive "ntpsigndsocket".  Note that this is a directory path to where NTP will expect to find a domain socket named "socket" (not intuitive!).  It is Samba which creates and listens to this socket; NTP uses the socket as a consumer and only opens the socket when it needs signing services.  The default location where Samba will initially create this socket is controlled via the "ntp signd socket directory" directive in Samba's "smb.conf".  In most cases this won't be set explictely.  You can verify the default value using the Samba "testparams" command with the "--verbose" option which will list both defined directives and the default values of undefined directives [including "ntp signd socket directory"].
Running the NTP daemon as "strace ntpd -n" will keep ntpd in the foreground and allow you to see the files and sockets that it opens and closes.  Once properly configured issue the "w32tm /resync /rediscover" option on a Windows 7 AD member and you should receive the response:
Sending resync command to local computer
The command completed successfully.
On the server you will see (in the strace output) and exchange between NTP and Samba4 via the signing socket.  It will look something like:
recvmsg(21, {msg_name(-1734344)={...},
msg_iov(17832102453714092032)=0x14ffe5894c, msg_controllen=120259084288,
msg_control=0x848, msg_flags=MSG_DONTROUTE|MSG_PROXY}, 0) = 68
recvmsg(21, 0xffe5891c, 0)              = -1 EAGAIN (Resource
temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {1291827449, 637483079}) = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 4
connect(4, {sa_family=AF_FILE,
path="/opt/ad/samba4/var/run/ntp_signd//socket"}, 110) = 0
write(4, "\0\0\0@", 4)                  = 4
write(4, "\0\0\0\0\0\0\0\0\1\0\0\0P\4\0\0\34\v\21\360\0\0\0\0\0\0002\337
\177\177\1\0"..., 64) = 64
read(4, "\0\0\0P", 4)                   = 4
read(4, "\0\0\0\0\0\0\0\3\0\0\1\0\34\v\21\360\0\0\0\0\0\0002\337\177\177
\1\0\320\2527c"..., 80) = 80
sendto(21, "\34\v\21\360\0\0\0\0\0\0002\337\177\177\1\0\320\2527c>\360
\220\303\320\2527\201\3478 \330"..., 68, 0, {sa_family=AF_INET,
sin_port=htons(123), sin_addr=inet_addr("")}, 16) = 68
If that doesn't happen verify that both Samba4 and NTP have access to the socket; this may include permissions issues as well as SELinux / AppArmour policies.  Once it works just run NTP normally - you have working secure time synchronization.

Manually Adding an ACL To An Object

In OpenGroupware the ACLs applied to an object are stored in the "object_acl" table.  If, for example, I want to add the list, view, read, write, and administer privileges for the team 11,530 for object 1,6829,810 the correct SQL to execute is:
INSERT INTO object_acl
  (object_acl_id, sort_key, action, object_id, auth_id, permissions)
VALUES (nextval('key_generator'), 0, 'allowed', 16829810, 11530, 'lvrwa')
The important points are:
  1. Use the "key_generator" sequence to assign the "object_acl_id" value.  This is the object id of the ACL itself;  all object ids are assigned from the key_generator sequence.
  2. The value of "sort_key" is always 0.  This value isn't actually used for anything.
  3. The value of "action" must be either "allowed" or "denied".  In most cases "allowed" is what you want in order to grant access.
  4. "object_id" is the object id of the object to which the ACL is applied in contrast to "auth_id" is the context to which the privileges, specified in "permissions", are either granted [if "action" is "allowed"] or revoked [if "action" is "denied"].  The value of "auth_id" should be the object id of an account or a team.
  5. The permissions string is always lower case.  Permission flags are documented in WMOGAG.
The ACLs in "object_acl" are the primary access control mechanism for all entities excepting Projects and Appointments.


Samba4 & PHPLDAPAdmin

Samba4 includes a PHP LDAP admin configuration sample. Unfortunately it doesn't match up to the current version of PHP LDAP Admin. The configuration you want to put into config/config.php is:
$servers->setValue('server','name','Samba4 AD Server');

Then you should get an PHP LDAP Admin login screen for your shiny Samba4 AD DSA. For the AD uninitiated the DN of the Administrator's account is "CN=Administrator,CN=Users,DC=ad,DC=mormail,DC=com" (for example, if you AD domain is "ad.mormail.com").  Using that DN and the domain administrator's password you should be able to login.
Another trick is to put that LDAPI LDAP URI into the /etc/openldap/ldap.conf  file so you can use the OpenLDAP LDAP CLI utilities [ldapsearch, ldapadd, ldapmodify, etc...] provided by your distribution.

Python Exceptions and "auto" Instantiation

Interesting thread on the python-list as to what happens regarding raising exceptions. The low-down: If you "raise ExceptionClass" and instance of that ExceptionClass is always created.

For example:
try: raise KeyError
except KeyError: pass

From the documentation:
raise evaluates the first expression as the exception object. It must be either a subclass or an instance of BaseException. If it is a class, the exception instance will be obtained when needed by instantiating the class with no arguments.

Some exception classes require arguments so the naked raise will fail with a TypeError.
 raise UnicodeDecodeError
Traceback (most recent call last):
  File "", line 1, in
TypeError: function takes exactly 5 arguments (0 given)

Seems to me it is clearer to just do the raise with instance syntax:
  raise KeyError()
except KeyError as e:


SQLAlchemy & Upcoming Birthdays

OpenGroupware Coils uses SQLAlchemy  as it's ORM. One of the desired features was a Logic command that efficiently returns contacts with upcoming birthdays.  In raw SQL this query would be very simple to write - but how to do it in SQLAlchemy? The answer: "sql.expression.extract" which will create an expression column equivalent to EXTRACT. With EXTRACT it is possible to compare to the year-of-day represented by a date.  The Python code looks like:
db = self._ctx.db_session()
# Get the current day-of-year
doy = datetime.today().timetuple().tm_yday
# Deal with year wrap-around
floor = doy - 2
if (floor < 1): floor +=365
ceiling = doy + 14
if (ceiling > 365): ceiling -= 365
# Create a field that is the SQL expression DOY(Contact.birth_date)
orm_doy = sql.expression.extract('doy', Contact.birth_date)
# Create the query
query = db.query(Contact).filter(and_(sql.expression.between(orm_doy, floor, ceiling),
                                      Contact.birth_date != None,
                                      Contact.is_account == self.accounts,
                                      Contact.status != 'archived'))
In this example "self.accounts" is an attribute with a value of 0 or 1.


Ohio LinuxFest 2010

Last weekend (Sept. 10th - 11th) was Ohio LinuxFest 2010 - and it was the biggest one yet.  The growth each year of this conference is impressive;  both in attendees and vendors.  While the website is persistently void of detailed information[1] this is a great conference for users of Open Source software and I recommend everyone attend if at all possible.  While once a Saturday only conference it seems to have expanded, without much fanfare, to include Friday.  This year there was a free all-day training session by Zenoss on Friday that I would have desperately loved to attend but was announced too late to make that possible.  Next year I hope to be there for the Friday sessions.  We (I and a sys-admin from Kalamazoo) did make if for the pre-party however - which is also something not-to-miss;  it is a great opportunity to hobnob with other users, including some well-known luminaries.  The nearly unlimited drink-tickets makes that even more entertaining.

Saturday's conference track began with Stormy Peters' excellent keynote on "Who is stealing your desktop?" about the dangers of cloud computing and the, almost stealth, proprietary services that so many users have come to rely on.  It raises the question if Open Source "won" only to then give up and accept yet another hegemony of corporations.  This same sentiment was echoed in Phil Robb's [of HP] talk "Irrelevance of the desktop".

Then it was on to regular part of the conference.  This is where I have one serious gripe about OLF: the topic selection.  It seems so "LINUX Journal".  They aren't "bad" topics, but it seems a lot of sessions are spent on promotional, evangelistic, or quite esoteric fare.  I'm generally opposed to almost all forms of "evangelism" so I'll admit I'm biased - but talking about promoting Open Source or how Open Source is better or the history of Open Source at an Open Source conference... doesn't this seem like preaching-to-the-choir and self-congratulation?  The very first year I went to OLF I learned about what was new with PostgreSQL [from a PostgreSQL developer!], about Apache mod_rewrite, about SMTP grey listing, etc... really solid detailed stuff.  I'd sit down and the presenter would dive right in.  I still have the notes from that conference and I still use all that information in my daily job.  That is lacking in recent OLF years where I come away with only a couple of actually useful bits of information.  I still go because it is simply fun and the feel you get for the direction of certain projects and vendors is valuable in its own way.

Probably the best presentation was David Nalley's (ke4qqq) talk on SELinux.  SE has always seemed like a black box - a black box to promptly be disabled!  Only a few sentences into his talk and David had knocked the sides of that box.  Straight-to-the-point and beautiful.  Over time I will now proceed to re-enable SELinux at least on my DMZ hosts.  There was some overlap between this an Deral Heiland's "And Now They Own You" - which could also have been named: "PEOPLE!  FIREWALL EGRESS TOO!".

Oh, and Asterisk 1.8 will support IPv6!  Another side-effect of OLF is to install a deep sense of Asterisk envy in those saddled with lousy / proprietary phone systems.  CalDAV integration with the PBX... really... why didn't someone thing of that feature before?  How simply spectacular.  Oh, wait,  I have Nortel phone systems... @*(&$&*(@*(& &*(@^$&*@^* @&^@$@!

[1] Even now the schedule page lists the "Pre-Party" details "TBD" (to-be determined).  Fortunately a quick Twitter search discovered where it was.


Bootstrapping "opengoupware.us"

Thanks to the addition of the "pages" feature in Blogspot I've finally created a site at opengroupware.us, The OpenGroupware [Legacy] project website (I won't even bother to link to it) has been worthless for some time, and the information about the constellation of projects beyond legacy is very scattered. opengroupware.us is an attempt to at least create an index of that information as well as resurrect some of the good content from the abyss that is the docs plone.  If you want to submit content to opengroupware.us or want to help edit / maintain content just let me know any I'll add your Blogspot account to the site permissions.


Coils: Features merged into "default"

This morning the following features were merged into OpenGroupware Coils "default" branch, and will be available in the next release.
  •  The scheduler service no longer depends on the ticktock heartbeat.  It checks the run queue in its work method.
  • Format(s) now support logging of rejected records to a buffer.  This feature is available via a new parameter on the readAction: "rejectionsLabel".  This is documented in the readAction's wiki page.
  •  The above motivated me to abstract how ActionCommand goes about creating [output] messages.  It is now simple for an action to create additional messages [beyond it's 'default' output message] by just calling the store_in_message(label, wfile, mimetype='application/octet-stream') method.
  • New CLI tools: "coils-list-schedule" and "coils-unschedule-process" that let the administrator view and modify the workflow scheduler from the command line.  Yes, we still need a "coils-schedule-process". 
    • getopt isn't the greatest command line option parser,  would be great if someone would look into swapping that out with something better.
  • The above motivated me to add an initialize_tool(name, argv, arguments=['',[]]) function to coils.core.utility.  This provides a simple way for a tool [vs. a service] to bootstrap itself onto the OpenGroupware Coils service bus so that it can receive callbacks, etc... Return value is an AdministrativeContext (with a registered Broker for IPC) and dictionary of unconsumed command line parameters. initialize_tool automatically consumes the --store=, --add-bundle=, and --ban-bundle parameters in order to initialize the Coils environment.
In the last week or so there has been significant work on tidying things up, and the deployment instructions on the wiki have been updated as well.  Getting a working OpenGroupware Coils instance should now be very straight-forward.   Big work ahead is the move to WSGI which is underway in the NetComm branch (which needs to be remerged with default, btw).  And with the impending release of openSUSE 11.3, and its inclusion of Evolution 1.30.x,  the GroupDAV address book support can be completely tested and polished.  Evolution 1.30.x contains several enhancements to make it's WebDAV address book support fully compatible with GroupDAV.


Keeping an OIE Route After Completion

Once an OpenGroupware Integration Engine route has successfully completed it is automatically garbage collected.  This prevents the system from filling up with messages and version information.  But if an external application, like ogo-curses-putter, wants to retrieve the output of the route... it's gone!  To facilitate this use-case the garbage collection can be suppressed per-route by setting the {http://www.opengroupware.us/oie}preserveAfterCompletion property of the route entity with a value of "YES".  If this property value exists the processes derived from that route will not be garbage collected.  The simplest way to set the property is to retrieve the route via "route::get" and use your context's property manager:

from coils.core import *
ctx = AdministrativeContext()
r = ctx.run_command('route::get', name='MTAStockOrder_TEST')
ctx.property_manager.set_property(r, 'http://www.opengroupware.us/oie', 'preserveAfterCompletion', 'YES')
The above will set the value of {http://www.opengroupware.us/oie}preserveAfterCompletion to "YES" for the route named "MTAStockOrder_TEST". The property manager will update the value of the property if such a property already exists, or it will create a new property with the specified value.


Performing PROPFIND from Python

Surprising, when I went looking for examples of performing PROPFIND requests from Python I found very little information. For testing OpenGroupware Coil's WebDAV presentation I needed a simple way to perform the PROPFIND required to list the entities in a collection and then request each entity. With a bit of scratching through the thin documentation I was able to come up with the following:
  1. import httplib, urllib2, base64, sys
  2. from lxml import etree
  3. PROPFIND = u'''<?xml version="1.0" encoding="utf-8"?>
  4. <propfind xmlns="DAV:">
  5. <prop>
  6. <getetag/>
  7. </prop>
  8. </propfind>'''
  9. SERVER_HOST = ''
  10. SERVER_PORT = 8080
  11. SERVER_PATH = '/dav/Contacts'
  12. CLIENT_AGENT = 'Whitemice rules/so very true'
  13. auth_string = 'Basic {0}'.format(base64.encodestring('adam:fred123')[:-1])
  14. urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler()))
  15. connection = httplib.HTTPConnection(SERVER_HOST, SERVER_PORT)
  16. connection.putrequest('PROPFIND', SERVER_PATH)
  17. connection.putheader('Authorization', auth_string)
  18. connection.putheader('User-Agent', CLIENT_AGENT)
  19. connection.putheader('Depth', 1)
  20. connection.putheader('Content-Length', str(len(PROPFIND)))
  21. connection.endheaders()
  22. connection.send(PROPFIND)
  23. response = connection.getresponse()
  24. if response.status == 207:
  25.   data = response.read()
  26.   connection.close()
  27.   response = None
  28.   namespace_prefix_map = { 'D' : 'DAV:' }
  29.   document = etree.fromstring(data)
  30.   for path in document.xpath('/D:multistatus/D:response/D:href/text()',
  31.       namespaces=namespace_prefix_map):
  32.     if (path != SERVER_PATH):
  33.       connection = httplib.HTTPConnection('', 8080)
  34.       connection.putrequest('GET', path)
  35.       connection.putheader('Authorization', auth_string)
  36.       connection.putheader('User-Agent', CLIENT_AGENT)
  37.       connection.endheaders()
  38.       response = connection.getresponse()
  39.       if response.status != 200:
  40.         print 'Error retrieving {0}'.format(path)
  41.         sys.exit(1)

This will make the PROPFIND for all the items in the collection, requesting the getetag property, and then from the response perform an HTTP GET for every item. Even better would be if it requested isCollection and knew better than to perform GETs on collections.  I hope this simple example will be of use to someone.


OIE Progress, and lots of it.

The workflow engine [OIE] in the OpenGroupware Coils project has reached a point a general usefulness.  OIE is accessible via WebDAV for creating, browsing, and starting flows.  The engine compiles BPML 1.0 markup to an internal format (adding additional markup-notations, such as XPDL,  shouldn't be much harder than writing a SaX parser to produce the internal format).  The foreach flow control structure is implemented;  switch, until, and while are simply stubs at this point.  switch is actually compiled but can't be executed yet.  Message scope is handled correctly.  Even with just linear execution and foreach a great many business processes can be modelled, especially ETL type flows.

The actions implemented so far are:
  • read - Translate a message using a format; fixed record length and XLS formats have been implemented.
  • select - SELECT from a defined RDBMS.
  • write - Write a StandardXML message out using a specified format.
  • assign - XPath select a value or assign a static value to a message.
  • getEntity - Retrieve an entity from the groupware database.
  • xpath - Create a new message from an XPath query.
  • ldapSearch - Search an LDAP DSA, results are in DSMLv1.0.
  • regularExpressionFind - Perform regexp matches.
  • readJSON - Translate JSON data into XML.
  • sendMail - Send a message via an SMTP server, including support for attachments.
All commands support label substitution in appropriate parameters.

As of yesterday there is also an insert action for inserting data into a RDMS table;  this action hasn't yet been as heavily tested as the others,  but it completes the tool set required for simple ETL flows.

Disabling GNOME Automount

When you connect a mass storage device to a computer running the GNOME desktop environment it automatically mounts the device in /media and places a short-cut to the device on your desktop.  This is an excellent default behavior;  but if you are working with various devices sometimes it can get in the way.  To correctly disable this feature simply execute:
gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount false
The devices will still appear in computer:/// (in nautilus) where you can right click on them to select the mount action if and when you want them to be mounted.  If and when you want to re-enable to auto-mounting feature execute:
gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount true


Passing a column set to an SQLalchemy query

In implementing the List method of the EntityAccessManager provided by the Contacts bundle in OpenGroupware Coils it seemed like it would be very efficient to allow the consumer to request what set of attributes it needed;  for instance, if a WebDAV client's PROPFIND request didn't ask for a given property, why request the corresponding attribute in the query?  Especially since the result set for PROPFIND queries are frequently very large [on the order of 20,000 records or so].  But how to pass a set of attributes as a parameter?

def List(ctx, attributes):
  db = query(attributes).filter(....)
  return db.all

does not work;  where attributes is a list, or a set of more than one attribute. Some fiddling around and I discover that the following works:

def List(ctx, *attributes):
  db = query(*attributes).filter(...)
  return db.all()

ctx = AssumedContext(10100)
a = BundleManager.get_access_manager('Contact', ctx)
a.List(ctx, (Contact.object_id, Contact.version))

That single asterisk is the trick.

Configuring OpenLDAP's dynlist in cn=config

Step#1: Make sure the dynlist module is loaded on the server.  What modules are loaded are typically controlled by the "cn=modules{0}, cn=config" object.  If your server doesn't have a modules object it probably isn't loading any dynamic modules.  Create an LDIF file and import it.  Our "cn=modules{0}, cn=config" looks like:

dn: cn=modules{0}, cn=config
olcModuleLoad: {0}accesslog.la
olcModuleLoad: {1}auditlog.la
olcModuleLoad: {2}constraint.la
olcModuleLoad: {3}dynlist.la
olcModuleLoad: {4}memberof.la
olcModuleLoad: {5}ppolicy.la
olcModuleLoad: {6}refint.la
olcModuleLoad: {7}seqmod.la
olcModuleLoad: {8}syncprov.la
olcModuleLoad: {9}sssvlv.la
olcModuleLoad: {10}translucent.la
olcModuleLoad: {11}unique.la
olcModuleLoad: {12}back_monitor.la
olcModulePath: /usr/lib/openldap2.4
objectClass: olcModuleList
cn: modules{0}

NOTE: Double check that the "olcModulePath" is the absolute path to the directory containing the OpenLDAP modules.

Step#2: Check that your schema has the groupOfURLs objectclass defined.  If you attempt to configure the dynlist module without that schema available you still crash slapd.  To define the dynamicGroup schema (if it is missing) you can import the following LDIF into your cn=config:

dn: cn=dynamicGroup, cn=schema, cn=config
olcObjectClasses: {0}( 2.16.840.1.113730.2.33         NAME 'groupOfURLs'   
     SUP top STRUCTURAL         MUST cn         MAY ( memberURL $ businessCat
 egory $ description $ o $ ou $                 owner $ seeAlso ) )
olcAttributeTypes: {0}( 2.16.840.1.113730.1.198         NAME 'memberURL'   
     DESC 'Identifies an URL associated with each member of a group. Any type
  of labeled URL can be used.'         SUP labeledURI )
objectClass: olcSchemaConfig
cn: dynamicGroup

Step#3: Create an olcOverlayConfig object in the scope of your Dit database.  For example, our Dit is the 1st HDB database on the server so the appropriate object is:

dn: olcOverlay=dynlist,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynamicList
olcOverlay: dynlist

Step#4: Now you are ready to actually use the dynlist module.  A common use-case is to create dynamic mail alias objects;  with dynlist you don't need to maintain mail aliases, they will automatically contain everyone who matches the relevent criteria.  Provided you use the traditional nisMailAlias objectclass in order to define mail aliases adding the attribute -
olcDlAttrSet: {0}nisMailAlias labeledURI rfc822mailmember:mail
 - to "olcOverlay=dynlist,olcDatabase={1}hdb,cn=config" will enable dynamic mail aliases.  Specifically any nisMailAlias containing a labeledURI attribute will be expanded by the query specified in that attribute.  The results of that query will be rewritten by the optional rfc822mailmember:mail clause which will rename the mail attributes resulting from the query into the rfc822mailmember attribute required by consumers of the nisMailAlias objects.  So rather than populating the mail alias object with rfc822mailmember attributes manually, you extend the object with the auxilliary labeledURIObject objectclass and define the query in the labeledURI attribute.

dn: cn=gr_parts, ou=ListAliases, ou=Aliases, ou=Mail, ou=SubSystems, o=Morrison Industries,c=US
mail: gr_parts@morrison-ind.com
labeledURI: ldap:///ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US?mail?one?(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))
objectClass: nisMailAlias
objectClass: top
objectClass: labeledURIObject
cn: gr_parts

The object will immediately populate with rfc822mailmember attributes derived from the mail attribute of those objects matching the specified filter: "(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))".  The third parameter, "one", of the URI is the scope of the query so only objects immediately subordinate to "ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US" are candidates for the filter.


Python Curses In Action, even on AIX.

So, what if you have an old COBOL application that you want to integrate with some web services or a website? You'd need a "green-screen" application that can take that COBOL applications temporary output file, send it up to the web service, wait for a response, replace the temporary file, and then return control to the COBOL application (or whatever "green screen" application comes next). It sounds simpler than it is in practice: what if the web service takes awhile to complete [or has human involvement on the other end?!] or the web service isn't available right at the moment you try to send your request? For a real production environment your "green screen" application would need to deal with all of those things.
And you need that "green screen" application to work on both LINUX and AIX!

Fortunately pware provides Python 2.6 for AIX, including curses! The curses library and respective Python module provide a surprisingly easy way to create professional looking TUI (Text User Interface) applications like the solution required in our example. This particular application takes them temporary file and submits it via a WebDAV PUT operation to the workflow engine provided by OpenGroupware Coils. The route in the workflow engine reformats the dreadful output of the COBOL application into the required format (using the format support in route Read and Write operations). The client detects the route is complete by watching the URL specified in the header of the initial PUT, and then downloads the required data. If anything goes wrong along the way the client can retry, or provide the user the option to abort. All along the way the client provides detailed feedback to the user and a familiar dialog-box and prompt interface when feedback is required.