Coding Respite.

At the beginning of May my main hard drive broke  and while waiting for a new drive I caught up on my reading backlog. This list has grown extensive and includes

  • The latest C11++ revision of The C++ Programming Language by Bjarne Stroustrup
  • Effective C++: 55 Specific Ways to Improve Your Programs and Designs  by Scott Meyers
  • The STL Tutorial and Reference Guide by David R Musser
  • Effective STL By Scott Meyers

I have read most of these before  but not up to date revisions and given this was pre-C11++ it seemed like a good idea to read them; I know there are newer standards but I tend to use C11++ as a base line for writing code.

Google was also used to track down several tutorials on the C++ STL that include these sites (Search site for STL articals or any other targeted topic)

My main machine used to  run Windows 10 but after I received a new hard drive I decided to bite the bullet and finally install  and use Linux. After some thought I opted for one the most popular distros Linux Mint to replace it.

Bar all the standard default packages I have installed the following

  • Netbeans
  • Elipse
  • Mono
  • QTCreator

I already used Netbeans for C++ development under windows 10 and cross compiling for Raspbian.  The  new packages are for future use as I wish to get into Java and QT development; initially for GUI related work. Mono is for later when I may learn about .NET and C#.

As I  use it mainly for development and I do not play  games these days then I after two months of use I really don’t miss windows at all.

E-mail Archival Program (Pendulum).

This began as one of the example programs for the Antikythera class library ArchiveMailBox; but it was decided to refactor its code and create a more fully featured application. To this end the code was separated out into 4 namespaces

  1. Pendulum – Main program control logic.
  2. Pendulum_ProcCmdLine – Command line option processing.
  3. Pendulum_File –  All file related functionality.
  4. Pendulum_MailBox – IMAP Server and mailbox code.

Pendulum: control flow logic that connects to a server and selects all specified mailboxes sequentially and searches for either all messages or all new message from a given UID. From this a vector of message UID(s) are returned and the message bodies for each UID are then downloaded and written to an .EML file (archived).

Pendulum_ProcCmdLine: command line processing code that uses the BOOST program options library to translate any options passed on the command line into a structure that is then used to drive the program.

Pendulum_File:  EML file creation and archive directory scan code. Each archive EML file created has as part of its filename the e-mails UID. Before new e-mails for a mailbox are searched for the highest UID for an archived e-mail is found an this used as the basis of a search (i.e. new e-mail searched for).

Pendulum_MailBox:  IMAP server connection and mailbox handling. It is possible whilst communicating with the server for the connection to fail; instead of just failing a reconnect is tried and any current command is repeated.

The current version of Pendulum can be found here and it is very much a work in progress.

Possible changes in future.

  1. Addition of mailbox ignore list for use with –all so that for instance Deleted/Junk/Spam mailboxes are not archived.
  2. The ability to work with multiple servers
  3. Use of threads.
  4. QT based UI (Know nothing of Qt but it looks like the best choice for a portable/flexible way to provide a cross platform user interface).

Refactoring Antikythera classes.

Spent time splitting Antikythera class library into name spaces to aid future work and usage; it was a good time to do this exercise as the number of classes is still small and manageable.

  • Antik::IMAP (CIMAP, CIMAPParse, CIMAPBodyStruct)
  • Antik::SMTP (CSMTP)
  • Antik::ZIP (CZIP, ZIPIO)
  • Antik::File (CTask, CApprise, CMIME)
  • Antik::Util (CRedirect, CLogger)

Time has also been spent on a number of example programs that use the classes facilities.

  1. IMAPCommandTerminal – A Simple IMAP command console/terminal that logs on to a given IMAP server and executes commands typed in. The raw command responses are echoed back as default but parsed responses are displayed if specified in program options.
  2. DownloadAllAttachments – Log on to a given IMAP server and download attachments found in any e-mail in a specific mailbox to a given local folder. The final destination folder is a base name with the mailbox name attached.
  3. ArchiveMailBox – Log on to a given IMAP server and download all e-mails for a given mailbox and create an .eml file for them in a specified destination folder. The .eml files are created within a sub-folder with the mailbox name and with filenames consisting of the mail UID prefix and the subject line. If parameter –updates is set then the date of the newest .eml in the destination folder is used as the basis of the IMAP search (ie. only download new e-mails). Note: MIME encoded words in the email subject line are decoded to the best ASCII fit available.
  4. WaitForMailBoxEvent Log on to a IMAP server and wait for a status change in a specified mailbox. By default it will use IDLE but polling every time period using NOOP is also supported. This is not directly useful but may be applied to other situations where the functionality is needed.
  5. SMTPSendMail A command line program to log on to an SMTP server and send an email to given recipients. The mails details such as contents, subject and any attachments are configured via command line arguments.
  6. ArchiveFolder A command line program that writes the contents of a source folder to a ZIP archive; traversing it recursively and adding any sub-folder contents. It compresses each file with deflate unless its size does not decrease in which case it simply stores the file.
  7. ExtractToFolder A command line program that extracts the contents of a ZIP archive to a specified destination folder. Note: Any destination folders are created by the program before a file is extracted as the class will not do this.
  8. ZIPArchiveInfo This is a command line program to scan a ZIP archive and output information about it.

The aim is to add more over time and add unit tests as and when new areas are added to the library.

ZIP archives.

I wanted to add an new task action function to the FPE that will take any new files added to a watch folder and append them to a specified ZIP archive (if the ZIP archive does not exist then it will be created). For this purpose two new classes CZIP and CZIPIO  where written  and added to the Antikythera classes. The first contains the high level logic to create an archive and add/extract files to/from an archive; the latter contains the low level record file I/O for reading and writing from archives and is the base class of CZIP ( though it may be used standalone to access archives.

Searching the web provided several useful sites that outlined the ZIP file format but the most detailed description was from Appnotes at the PKWARE web site.  Version 2.0 was chosen as the baseline for the format with extensions to support > 4 GB files added with ZIP64 support.  Files are stored within the archive in two formats

  1. store – exact copy of the file
  2. deflate – compressed format implemented using the ZLIB library.

Note: ZLIB is also used to calculate a files 32 CRC checksum.

In the future support may be added for the removal of files but at present it is not needed.  The production of theses classes and the modification of the FPE to use them  has taken most of my development time up throughout March and I hope to move onto something different in April.

Refactoring IMAP classes.

Spent a week looking at the whole of the IMAP class code and refactored it so that the parser only returns a pointer to one structure rather than one of a number of command specific structure. Also the class now handles instances where the server decides to disconnect a connection without any BYE command warning; this does happen frequently enough to warrant attention. If this does happen it is signaled though an exception and together with the boolean in the reply that signals a BYE response was sent it it possible for client code to detect all disconnects and actuate a reconnect.

Whats next on the list of things to do ? A class to create and manipulate ZIP archives,

CMailIMAP and the rest of the band.

This class started off as just a way to add the ability to append an email containing a file attachment generated by the FPE to a particular IMAP mail box. The initial version just used libcurl to connect to the server and issue the command too attach the email. Out of this grew a fully fledged class that provides the ability to connect to an IMAP server and issue commands and receive the raw text results. The transport and connection are again provided by libcurl as with CMailSMTP and all IMAP4 commands are supported along with the IDLE and NOOP which enable a mailbox to be interrogated to see if its state has changed (e.g. new mail has arrived).

This was all that was really needed for the FPE functionality but the decision was made to extend the work with two more support classes that would enable a higher level program interface than that provided by CMailIMAP.CMailIMAPParse originally began as part of CMailIMAP contains all static methods and data and is used to parse the text response provided by CMailIMAP. It takes the raw text response returned by a command, parses it into a command specific structure; a pointer to which is returned. The structure contains a group of common fields per command that include returned status and also any error message that may have been sent. The decoded information provided as part of these structures allows a high level language to gather information and process further IMAP commands. For example a program may search a selected mailbox given certain criteria, receive, parse the response and then issue further commands based on the structure containing the UIDs of the found email.

All the above enabled the creation of a simple IMAP command terminal example program that enables a connection to an IMAP server and commands to be issued. This provides the ability to manipulate email and mailboxes but does not give a way to download/upload email content such as attachments. The in depth map of an emails structure is provided by its BODYSTRUCTURE which can be requested through a FETCH command; other structures exist that describe email details such as its ENVELOPE but the BODYSTRUCTURE is the most detailed. An IMAP list structure that represents this can be returned by CMailIMAPParse and in order to parse and read this CIMAPBodyStruct was born.

This parses the BODYSTRUCTURE list into a tree structure and enables this tree to be walked and a user provided function (with data) called on the tree nodes (individual body parts). The provided function can then interrogate the node for data to use in any way. The class contains an example  built-in that extracts information on all attachments and places this information inits local data (i.e. attachment type, encoding, file name etc). This built in provided the functionality needed to create another example program that searches for all attachments in emails within a given mailbox and downloads them to a local directory.

I have spent al ot of time of this IMAP functionality during January and February and its still a work in progress as learn more about IMAP .

Six monthly update.

I the past six months I have worked on  number of  personal projects:

– Simple CRUD contacts database application that uses an HTML user interface, client side JavaScript (Query/AJAX) to communicate with a server side Node/JavaScript component that talks to an MySQL or SQLite database back-end.
– Generic data importer written in Node/JavaScript to enable the importing of comma separated value files into either an MySQL or SQLite database by simply copying them to a watch folder.
– File processing engine (FPE) application written in Node/JavaScript that takes files copied/moved into a configured watch folder and processes it using a set script.
– C++ variant of the FPE written in C11++ on Linux. This not only involved the creation of the core program logic but also the of numerous support classes for such things as file event watching, STMP mail creation and an IMAP command processor.

The above all helped increase my knowledge/experience of the following HTML, CSS, JavaScript, Node, C11++, SQL, SQLite, BOOST C++ Libraries, JQuery, AJAX, RESTful APIs, and Linux.

So what is ahead for the next six months ? Well for the near future I intend to refactor some of the classes that I have created in particular separate out the IMAP decode response code from CMailIMAP into  separate class, write a high level functional interface that uses CMailIMAP to do things like fetch and decode mail attachments.  I also want to come back to Node/JavaScript and work on that more;probably through new projects.

Class CMailSMTP

The JavaScript variant of the  FPE supports a simplistic printer spooler using a node SMTP package to attach a file to an email and send it to HPs ePrint service for printing  on a local HP printer. As a exercise it was decided to add this functionality to the C++ variant of the FPE and in order to provide this some way of creating an email, attach a file and sending it was needed.

Several high level C/C++ libraries exist to perform this functionality but it was decided to write as much of the SMTP protocol handling code as possible and so it was decided to use libcurl which is suitably low level that a significant part of the protocol handling had to be written. Another reason for using is that it is written in highly portable C and available on most platforms/operating systems and it supports most internet based protocols so useful in a wide variety of future projects.

From this was born the class CMailSMTP which enables the caller to set email server URL/ account details,  any recipients for the email, attach mutiple files encoded in either 7bit or base64 and then send the email. The class uses libcurl to connect to the server including login, setting up an SSL connection)and perform the actual email transfer but constructing the contents of the email including headers and encoded data are performed by the class itself.

The class itself only dependencies at present are C11+,  libcurl and some Linux specific code that reads file /etc/mime.types to setup a table for file extension to MIME type mapping ( changing this to be more portable is penciled in to be looked at in future).


CFileTask, CFileApprise and exceptions.

Both these classes can generate exceptions that if created are passed up the call chain without any internal catch clauses; with the exception of an error 22 which can be generated by inotify_rm_watch() while removing a watch but is valid behavior and processing is allowed to continue.

But both classes processing loops CFileTask::monitor() and CFileApprise:watch() are designed to run as a separate thread so  any exceptions caught inside these are lost to the main code although they could be reported via a suitable error message.

To make sure that any exceptions generated  are passed to the main code each class has a variable  std::exception_ptr thrownException that is used to store the current exception when one is generated and the looping function exiting the thread. To check if any where generated the caller uses class method getThrownException() which is non null if the reason for exit was an exception. The caller can then do whatever they require with it which in the case of the FPE is re-throw so it is caught locally.

The CFileTask uses a CFileApprise object run on a separate thread so before it exits its main loop it checks CFileApprise::getThrownException() and if set then copies that value to its own thrownException (basically passing the thrown exception back up the call chain). Note that this mechanism does rely on the object instance staying in scope and not being destroyed. CFileApprise also will generate an event for an exception which may be picked by the callers event loop and dealt with as appropriate.

Refactoring The Task Class.

I have finally gotten to factoring all of the file event handing code from the task class with the creation of the CFileApprise class. This is similar to the old Task class except it can generate a number of events beyond just file added. It now supports the following

  1. Event_add –  File added to watched folder hierarchy
  2. Event_change –  File changed
  3. Event_unlink –  File deleted from watched folder hierarchy
  4. Event_addir – Directory added to watched folder hierarchy
  5. Event_unlinkdir – Directory deleted from watched folder hierarchy
  6. Event_error – Exception error

with each event having the following entries in its structure

  • event id
  • full path and name of file that created event

The main class constructor still has a central watch folder from which any addition  / removal of a directory results automatically in a new watch being created or removed. There is now a secondary constructor which only takes an options parameter which allows watches to be manually added and removed for files /directories. At present any directories added still have the recursive behavior for the handling of watches but this constructor is intended for full manual  and it is intended to remove this.

The Task class (renamed CFileTask)  has thinned dramatically and when created it uses a CFileApprise object on a separate thread and loops waiting for Event_add events and then calling the task action function for it. It should be noted that the class still has the same interfaces and functionality as before but the file event handling as mentioned above has moved to CFileApprise.

Now that all of the inotify code is contained within CFileApprise this make porting the FPE and CFileTask to other platforms a lot easier and is the intention in the near future. Not having access to a Macintosh these days then a Windows version will probably be looked at first.

The current state of the above classes and their source can be found at my GitHub here