Coding Respite.

At the beginning of May my main hard drive broke  and while waiting for a new drive I caught up on my reading backlog. This list has grown extensive and includes

  • The latest C11++ revision of The C++ Programming Language by Bjarne Stroustrup
  • Effective C++: 55 Specific Ways to Improve Your Programs and Designs  by Scott Meyers
  • The STL Tutorial and Reference Guide by David R Musser
  • Effective STL By Scott Meyers

I have read most of these before  but not up to date revisions and given this was pre-C11++ it seemed like a good idea to read them; I know there are newer standards but I tend to use C11++ as a base line for writing code.

Google was also used to track down several tutorials on the C++ STL that include these sites (Search site for STL articals or any other targeted topic)

My main machine used to  run Windows 10 but after I received a new hard drive I decided to bite the bullet and finally install  and use Linux. After some thought I opted for one the most popular distros Linux Mint to replace it.

Bar all the standard default packages I have installed the following

  • Netbeans
  • Elipse
  • Mono
  • QTCreator

I already used Netbeans for C++ development under windows 10 and cross compiling for Raspbian.  The  new packages are for future use as I wish to get into Java and QT development; initially for GUI related work. Mono is for later when I may learn about .NET and C#.

As I  use it mainly for development and I do not play  games these days then I after two months of use I really don’t miss windows at all.


E-mail Archival Program (Pendulum).

This began as one of the example programs for the Antikythera class library ArchiveMailBox; but it was decided to refactor its code and create a more fully featured application. To this end the code was separated out into 4 namespaces

  1. Pendulum – Main program control logic.
  2. Pendulum_ProcCmdLine – Command line option processing.
  3. Pendulum_File –  All file related functionality.
  4. Pendulum_MailBox – IMAP Server and mailbox code.

Pendulum: control flow logic that connects to a server and selects all specified mailboxes sequentially and searches for either all messages or all new message from a given UID. From this a vector of message UID(s) are returned and the message bodies for each UID are then downloaded and written to an .EML file (archived).

Pendulum_ProcCmdLine: command line processing code that uses the BOOST program options library to translate any options passed on the command line into a structure that is then used to drive the program.

Pendulum_File:  EML file creation and archive directory scan code. Each archive EML file created has as part of its filename the e-mails UID. Before new e-mails for a mailbox are searched for the highest UID for an archived e-mail is found an this used as the basis of a search (i.e. new e-mail searched for).

Pendulum_MailBox:  IMAP server connection and mailbox handling. It is possible whilst communicating with the server for the connection to fail; instead of just failing a reconnect is tried and any current command is repeated.

The current version of Pendulum can be found here and it is very much a work in progress.

Possible changes in future.

  1. Addition of mailbox ignore list for use with –all so that for instance Deleted/Junk/Spam mailboxes are not archived.
  2. The ability to work with multiple servers
  3. Use of threads.
  4. QT based UI (Know nothing of Qt but it looks like the best choice for a portable/flexible way to provide a cross platform user interface).

Refactoring Antikythera classes.

Spent time splitting Antikythera class library into name spaces to aid future work and usage; it was a good time to do this exercise as the number of classes is still small and manageable.

  • Antik::IMAP (CIMAP, CIMAPParse, CIMAPBodyStruct)
  • Antik::SMTP (CSMTP)
  • Antik::ZIP (CZIP, ZIPIO)
  • Antik::File (CTask, CApprise, CMIME)
  • Antik::Util (CRedirect, CLogger)

Time has also been spent on a number of example programs that use the classes facilities.

  1. IMAPCommandTerminal – A Simple IMAP command console/terminal that logs on to a given IMAP server and executes commands typed in. The raw command responses are echoed back as default but parsed responses are displayed if specified in program options.
  2. DownloadAllAttachments – Log on to a given IMAP server and download attachments found in any e-mail in a specific mailbox to a given local folder. The final destination folder is a base name with the mailbox name attached.
  3. ArchiveMailBox – Log on to a given IMAP server and download all e-mails for a given mailbox and create an .eml file for them in a specified destination folder. The .eml files are created within a sub-folder with the mailbox name and with filenames consisting of the mail UID prefix and the subject line. If parameter –updates is set then the date of the newest .eml in the destination folder is used as the basis of the IMAP search (ie. only download new e-mails). Note: MIME encoded words in the email subject line are decoded to the best ASCII fit available.
  4. WaitForMailBoxEvent Log on to a IMAP server and wait for a status change in a specified mailbox. By default it will use IDLE but polling every time period using NOOP is also supported. This is not directly useful but may be applied to other situations where the functionality is needed.
  5. SMTPSendMail A command line program to log on to an SMTP server and send an email to given recipients. The mails details such as contents, subject and any attachments are configured via command line arguments.
  6. ArchiveFolder A command line program that writes the contents of a source folder to a ZIP archive; traversing it recursively and adding any sub-folder contents. It compresses each file with deflate unless its size does not decrease in which case it simply stores the file.
  7. ExtractToFolder A command line program that extracts the contents of a ZIP archive to a specified destination folder. Note: Any destination folders are created by the program before a file is extracted as the class will not do this.
  8. ZIPArchiveInfo This is a command line program to scan a ZIP archive and output information about it.

The aim is to add more over time and add unit tests as and when new areas are added to the library.

ZIP archives.

I wanted to add an new task action function to the FPE that will take any new files added to a watch folder and append them to a specified ZIP archive (if the ZIP archive does not exist then it will be created). For this purpose two new classes CZIP and CZIPIO  where written  and added to the Antikythera classes. The first contains the high level logic to create an archive and add/extract files to/from an archive; the latter contains the low level record file I/O for reading and writing from archives and is the base class of CZIP ( though it may be used standalone to access archives.

Searching the web provided several useful sites that outlined the ZIP file format but the most detailed description was from Appnotes at the PKWARE web site.  Version 2.0 was chosen as the baseline for the format with extensions to support > 4 GB files added with ZIP64 support.  Files are stored within the archive in two formats

  1. store – exact copy of the file
  2. deflate – compressed format implemented using the ZLIB library.

Note: ZLIB is also used to calculate a files 32 CRC checksum.

In the future support may be added for the removal of files but at present it is not needed.  The production of theses classes and the modification of the FPE to use them  has taken most of my development time up throughout March and I hope to move onto something different in April.

Refactoring IMAP classes.

Spent a week looking at the whole of the IMAP class code and refactored it so that the parser only returns a pointer to one structure rather than one of a number of command specific structure. Also the class now handles instances where the server decides to disconnect a connection without any BYE command warning; this does happen frequently enough to warrant attention. If this does happen it is signaled though an exception and together with the boolean in the reply that signals a BYE response was sent it it possible for client code to detect all disconnects and actuate a reconnect.

Whats next on the list of things to do ? A class to create and manipulate ZIP archives,

CMailIMAP and the rest of the band.

This class started off as just a way to add the ability to append an email containing a file attachment generated by the FPE to a particular IMAP mail box. The initial version just used libcurl to connect to the server and issue the command too attach the email. Out of this grew a fully fledged class that provides the ability to connect to an IMAP server and issue commands and receive the raw text results. The transport and connection are again provided by libcurl as with CMailSMTP and all IMAP4 commands are supported along with the IDLE and NOOP which enable a mailbox to be interrogated to see if its state has changed (e.g. new mail has arrived).

This was all that was really needed for the FPE functionality but the decision was made to extend the work with two more support classes that would enable a higher level program interface than that provided by CMailIMAP.CMailIMAPParse originally began as part of CMailIMAP contains all static methods and data and is used to parse the text response provided by CMailIMAP. It takes the raw text response returned by a command, parses it into a command specific structure; a pointer to which is returned. The structure contains a group of common fields per command that include returned status and also any error message that may have been sent. The decoded information provided as part of these structures allows a high level language to gather information and process further IMAP commands. For example a program may search a selected mailbox given certain criteria, receive, parse the response and then issue further commands based on the structure containing the UIDs of the found email.

All the above enabled the creation of a simple IMAP command terminal example program that enables a connection to an IMAP server and commands to be issued. This provides the ability to manipulate email and mailboxes but does not give a way to download/upload email content such as attachments. The in depth map of an emails structure is provided by its BODYSTRUCTURE which can be requested through a FETCH command; other structures exist that describe email details such as its ENVELOPE but the BODYSTRUCTURE is the most detailed. An IMAP list structure that represents this can be returned by CMailIMAPParse and in order to parse and read this CIMAPBodyStruct was born.

This parses the BODYSTRUCTURE list into a tree structure and enables this tree to be walked and a user provided function (with data) called on the tree nodes (individual body parts). The provided function can then interrogate the node for data to use in any way. The class contains an example  built-in that extracts information on all attachments and places this information inits local data (i.e. attachment type, encoding, file name etc). This built in provided the functionality needed to create another example program that searches for all attachments in emails within a given mailbox and downloads them to a local directory.

I have spent al ot of time of this IMAP functionality during January and February and its still a work in progress as learn more about IMAP .

Six monthly update.

I the past six months I have worked on  number of  personal projects:

– Simple CRUD contacts database application that uses an HTML user interface, client side JavaScript (Query/AJAX) to communicate with a server side Node/JavaScript component that talks to an MySQL or SQLite database back-end.
– Generic data importer written in Node/JavaScript to enable the importing of comma separated value files into either an MySQL or SQLite database by simply copying them to a watch folder.
– File processing engine (FPE) application written in Node/JavaScript that takes files copied/moved into a configured watch folder and processes it using a set script.
– C++ variant of the FPE written in C11++ on Linux. This not only involved the creation of the core program logic but also the of numerous support classes for such things as file event watching, STMP mail creation and an IMAP command processor.

The above all helped increase my knowledge/experience of the following HTML, CSS, JavaScript, Node, C11++, SQL, SQLite, BOOST C++ Libraries, JQuery, AJAX, RESTful APIs, and Linux.

So what is ahead for the next six months ? Well for the near future I intend to refactor some of the classes that I have created in particular separate out the IMAP decode response code from CMailIMAP into  separate class, write a high level functional interface that uses CMailIMAP to do things like fetch and decode mail attachments.  I also want to come back to Node/JavaScript and work on that more;probably through new projects.