Accounts has been slightly on hold since the wages needed to be run and I didn’t see that accounts or operations would be happy with fugures potentially changing.

The main project has been setting up a notification service to set up the service layer correctly. I’ve finally got the server working so I’m just building a framework. I thought of porting parts of djabberd projects into PHP but I’m just  looking at parts of it but XMPP is certainly a useful tool in getting machines to speak to each other and to develop event driven services.

I’ve also managed to complete a SOAP service in PHP to listen for status updates and just doing the final tests to it now. Once its up it can be repurposed for other companies. One of the things that I think  will come up is how to store XML files most efficiently as MySQL 5 appears to be tied to uploading files rather than just taking POST strings. I’m thinking of using something like Oracle’s BDB XML database (though the license appears to preclude our uses) or eXist but that is something to come back to much later.

I’ve been thinking about the Open Correspondence site and the best way to allow it to be extended by other people. I think that the best way forward to create an internal XML format which the load command can use and anybody can use to create their own files and databases. Its along the lines of the stuff I partially did some work on in the Open Shakespeare project.

Given the boss is away, time for finishing more things off next week. I’ve also created a Trac instance for internal purposes but I think it’ll help on that bane if developing live – documentation.

However if you think about the LLEN command, you can pop as many items as you want, drop them into an array and iterate across them. I need to try this but you could feasibly call items from the middle of the array by changing the start and end points in LLEN. Normally I’d do something like  <list name> LLEN 0, -1 for all items or <list name> LLEN 0, 2 for the first two but if you change 0 to something else where you know there are 30 items but only want 5 from position 20 then you could pop in LLEN 20, 5 to achieve the result. It is not really germaine to the queueing that I’ve been looking at (for system updates where I need everything or just the first item) but could be a useful adaptation for somebody else.

The main challenge this week has been reading Excel attachments from email. PHP’s imap library  allows you to read the structure of an email but is curiously reticent in retrieving data if you have mime parts. I spent ethe best part of a day and a half getting a script to iterate over an incoming email, filter the parts so that it just explored the attachments mime type and then retrive any attachments either from a flat structure or iterating over each part before calling imap_fetchbody(). So far the fix appears to work and has allowed me to create a prototype mail service for receiving email data. It seems odd that in the era of web services that financial data is still sent by insecure methods but we must accomodate.

I’ve also been looking at PHP’s SOAP library to create a status update service which will probably utilise Service Orientated Architecture to create a stable, scalable service. Initially I created a WSDL file using the Eclipse IDE but that threw all sorts of issues and ended up using Zend’s WSDL generator tool running across the existing server. Must look into this but there might be a conflict in versions of WSDL as well as first time learning curve. I’m hoping to get the first version of the service up this week.

I suspect that this week is going to complete the commission and service status services as well as possibly doing some documentation as it is beginning to pile up.

I started with reading an article  on the Zend site that offers an overview of setting up a table to run with a Full-Text index. As the article mentions, you have to ensure that the column which is being searched is either VARCHAR or TEXT as MySQL. If it is not in either form, then just alter the column using

ALTER TABLE <tablename> MODIFY <column> TEXT

(or VARCHAR but TEXT is probably preferable). What the Zend article does not mention is that the table type needs to by MyISAM rather than use InnoDB (which means that transactions won’t work on the table). Having made the alteration, I ran the query:

SELECT * , MATCH (<table>) AGAINST (‘<search term>’) AS score FROM <table> WHERE MATCH (<table>) AGAINST (‘<search term>’)

The table returns all the columns with a score against the term.

The SQL code just needs calling as you would any other form of database code. I’m still playing with this but I’ve been ordering the table by the score descending (ORDER BY score DESC) so that the most relevant results are posted for the user.

I do think that I need to do some pre-processing on my own results set to highlight relevance and to extract further semantic meanings for results. For example the publisher ‘Chapman and Hall’ that I could run on the Dickens letters (http://austgate.co.uk/dickens/search.php?term=Chapman&submit=Submit+Query) could equally pull up other businesses or people. I still need to write a parser that can make some sort of judgement even if it is a guess.

I’m sure as I carry on developing the engine and bringing everything together for the project, I’ll have further thoughts on the creation of an engine and creating a more advanced version. This does at least give me a start using current tools (though it is perhaps not as good as Xapian but sometimes you have to at least learn some of the basics).