blue wave mail packet file structures
the blue wave offline mail system mail packet file structures revision level 2 copyright 1990-1994 by cutting edge computing all rights reserved created by george hatchew documentation by martin pollard and george hatchew revision 2.01 - january 18, 1994 =================== table of contents =================== introduction filename conventions files in blue wave packets byte ordering in file structures using the file structures unused and reserved structure fields the *.inf file (inf_header & inf_area_info) the *.mix file (mix_rec) the *.fti file (fti_rec) the *.dat file the *.xti file (xti_rec) the *.upl file (upl_header & upl_rec) the *.upi (upi_rec) and *.net (net_rec) files the *.req file (req_rec) the *.pdq file (pdq_header & pdq_rec) appendix a - how to create a blue wave mail packet appendix b - how to create a blue wave reply packet appendix c - the blue wave structures and turbo pascal appendix d - serial numbers in mail and reply packets ============================ copyright and restrictions ============================ the blue wave packet structures were created by george hatchew, and are the copyrighted property of cutting edge computing. permission is granted for third parties to use these structures in their own pro- grams, without any royalties or licenses required. cutting edge computing reserves the right to make any changes to these structures, at any time. as such, third parties are requested not to make any unauthorized changes to these structures, as cutting edge computing is not bound to follow these changes. any proposed changes should be brought to the attention of cutting edge computing, where they may be included in future revisions of the structures. authors that use these structures are allowed to claim that their programs are "blue wave compatible". however, to avoid confusion and complications, authors are not allowed to use "blue wave" as any part of the name of their programs (as "blue wave" is a product line from, as well as a trademark of, cutting edge computing). =================== trademark notices =================== the following are products, trademarks, or registered trademarks of the following individuals and/or companies: arc - system enhancement associates blue wave - george hatchew and cutting edge computing fidonet - tom jennings and fido software megareader - kip compton ms-dos - microsoft corp. pkzip - pkware inc. qwk - mark "sparky" herring silver xpress - hector santos and santronics software turbo pascal, borland pascal - borland international xrs - michael y. ratledge any omissions from this list are purely unintentional. blue wave mail packet file structures - revision level 2 introduction ============ the world of offline mail has virtually exploded since the late 1980s, due mostly to the ever-increasing interest in electronic mail networks (such as fidonet and the internet). as the flow of mail increased, more and more users became aware of the benefits of downloading mail, reading it offline, and uploading replies at a later date, thus maxi- mizing efficiency and minimizing the time spent online. several competing formats for storage of offline mail have come into existence during this period, with the minimalist qwk format emerging as the dominant one due to its open specifications. (the megareader, silver xpress, and xrs, formats also exist, but never really achieved "critical mass" due to the proprietary nature of their file formats.) qwk enjoys widespread popularity, but its technical limitations make it less than suitable for handling the wide variety of electronic mail that currently exists (and which will appear in the coming years). the blue wave format was designed as a superior method of providing offline mail capabilities, particularly for networks based on the fidonet standard (which means full support for fidonet-style private mail, or netmail). its design is simple enough that virtually any programmer can create a blue wave-compatible product in a short amount of time, yet flexible enough to provide plenty of room for future needs (such as fax capabilities). it also has basic support for non- fidonet style mail, such as that required by internet mail, usenet newsgroups, and qwk-based network mail. note that this is a reference document, not a programming tutorial. a tutorial on programming is beyond the scope of this document. thus, we do not recommend the use of these structures by the novice. filename conventions ==================== the blue wave format was originally designed for the blue wave series of offline mail readers and doors running on an intel-compatible pc using ms-dos (or a dos-compatible operating system). this means that filenames are limited to the dos standard "8.3" format (up to eight characters, optionally followed by a period and a one to three charac- ter extension, with no distinction made between upper and lower case letters). for maximum compatibility across different platforms, programs utilizing the blue wave format should limit filenames to the dos format as well. additionally, dos allows for some non-alphanumeric characters to be used in filenames. these characters, while suitable for dos, may cause problems on non-dos platforms. therefore, it would be wise to restrict the allowable characters in filenames to uppercase letters ("a" to "z") and digits ("0" to "9"). files in blue wave packets ========================== there are two main components to the blue wave system: mail packets, which consist of messages obtained from the host system (such as a bbs), and reply packets, which consist of messages written by the user via an offline mail reader (such as the reader that bears its name, the blue wave offline mail reader). each type of packet contains its own set of unique files. blue wave mail and reply packet filenames are based around what is called a "packet id". the packet id is a one to eight character string that uniquely identifies a particular host system, and is used as the basis for all packet files. "packets", as defined here, are groups of files contained in an archive file, which uses the packet id as the base filename and is created using a file archive utility (such as arc or pkzip, or the equivalent for non-dos platforms). the three- character extension for a mail packet is comprised of the first two letters of the day of the week, followed by a digit from 0 to 9. fidonet sysops will recognize this as the same scheme used for the extensions on echomail packets. (an alternate scheme is to use a pure numerical extension, i.e. "001" through "999".) the extension for a reply packet is "new". (note that door implementations should include code to keep track of the last mail packet extension used, so that multiple mail packets created on the same day won't have the same filename.) examples of packet archive names and internal filenames, based around the packet id, are given below (after the list of files). a mail packet consists of, at minimum, the following files: *.inf information about the host system and its message areas, as well as information about the user who obtained the mail packet. *.fti the headers for each message in the mail packet. the headers consist of such things as the from:, to:, and subject: fields, and the date/time the message was written. *.mix an index file that points to the messages for each message area, designed for quick access to messag- es. *.dat the text for all messages in the mail packet. optional text/ansi files may also be included in the archive. in addition to the "reader" files specified in the *.inf header, there are two other types of files, not defined in the *.inf header, that may be included. the first, a list of new files available for down- load, can be included as "newfiles.*" (any extension is valid). the second, system bulletins, can be included as "blt*.*" (any filename is valid, as long as it begins with "blt"). the methods used to display these bulletins is implementation dependent. a reply packet consists of, at minimum, the following files: *.upl contains the information (name, network address, message attributes, filename of message text, etc.) for each reply message. replaces the *.upi and *.net files (see below) used in older blue wave implementations. *.upi contains the information (name, network address, message attributes, filename of message text, etc.) for each non-netmail reply message. this file has been obsoleted by the *.upl file, but is documented here for compatibility purposes (as some older systems are not yet compatible with the *.upl file). *.net contains the information (name, network address, message attributes, filename of message text, etc.) for each netmail reply message. this file has been obsoleted by the *.upl file, but is documented here for compatibility purposes (as some older systems are not yet compatible with the *.upl file). *.req an optional file that specifies the information on file requests made through the offline mail read- er. *.pdq an optional file that specifies the information on remote configuration (such as adding and dropping message areas) made through the offline mail reader. to clarify, let's say a bbs is using the packet id of "wildblue". a mail packet from that bbs would contain (at minimum) the files wild- blue.inf, wildblue.fti, wildblue.mix, and wildblue.dat, and when archived, would be called "wildblue.su1" (the extension may differ, depending on the criteria described above). consequently, a reply packet destined for that bbs would contain (at minimum) the files wildblue.upl, wildblue.net and wildblue.upi (for compatibility), plus the individual files that comprise reply messages, and when archived, would be called "wildblue.new". the text of each reply message is stored in individual files in the reply packet. each *.upl record contains the name of the text file corresponding to the particular reply message. the naming convention used to assign names to each text file is up to the programmer. (the blue wave reader uses "xxx.yyy", which stands for "message 'xxx' in area 'yyy', but you are not limited to this format as long as the filename is properly stored in the *.upl, *.upi, or *.net record.) byte ordering in file structures ================================ since the blue wave packet structures were initially written for ibm pcs and compatible systems, the format for multi-byte fields in the data structures is expected to be in intel format (i.e. the least significant byte first, followed by the most significant byte[s]). some cpus, particularly the motorola 68000 series, store multi-byte fields as most significant byte first. if you are writing a blue wave compatible program for a system that does not store data in the intel format, you will have to write a routine that will convert data bet- ween the two formats. using the file structures ========================= the file structures, as presented here, are provided as a header file for use with the c programming language. simply use the #include statement in your program source code to incorporate the header file: #include "bluewave.h" each file structure is defined as a data structure ("struct") using the "typedef" feature, making it easy to define variables. for exam- ple, to define a variable used to store the *.inf file header, simply use: inf_header infhdr; in your program. to make the structures as compatible across platforms as possible, all data types used in the structures are user-defined via "typedef". for example, to use a 16-bit unsigned integer, the data type "tword" is used instead of "unsigned int". this way, data fields are guaranteed to be the same size across platforms. (for more information, refer to the information contained in the bluewave.h file.) also, if your program is being written for a cpu that does not store data in intel format (as described earlier), you should insert: #define big_endian before you include the bluewave.h file. this will define the data types as arrays of bytes, making it easier to create routines that will convert data fields between intel format and the format native to your cpu. (refer to "byte ordering in file structures" for more information.) several of the file structures -- the *.inf and *.upl headers -- include fields that define the lengths of the other file structures used in mail and reply packet files. these fields are used to ensure that programs can use future releases of the file structures without breaking... as long as programs are coded to use them, that is. door authors should take the few extra lines of code to fill in the structure length fields. reader authors need to take the time to code for possible extensions to this file format. if the data fields are longer than expected, simply do a seek to move to the next record, and ignore the extra information. (if the data fields are shorter than expected, a simple "please upgrade your reader" should suffice. however, you should never encounter a record size smaller than the ones defined here.) any extra information that is sent in the packets probably would not be crucial, and you may be able to continue with reading the packet anyway. (it should be noted that all blue wave doors earlier than the version 3.0 series set these fields to 0, as this extensibility was not added until recently. if the structure sizes are 0, readers should assume that all records are of the sizes defined in the header file as the "original_xxxx_len" macros, and should use these macros when field lengths of 0 are encountered. there is no definition for the original length of the *.upl structures, as the older doors did not recognize the *.upl file.) to see an example of how to use these structure length fields, refer to the comments in the bluewave.h file. there, you will see a c code snippet that not only demonstrates the length fields, but the original_xxxx_len macros as well. important note: all blue wave file structures must be stored in "packed" format (i.e. the compiler must not insert padding between fields in order to force fields onto word boundaries). most compilers default to "packed" mode, but if yours does not, you must use the appropriate settings or preprocessor directives to set "packed" mode. failure to do so will all but guarantee that your program will gener- ate incompatible blue wave packets! unused and reserved structure fields ==================================== some fields and flags in the blue wave structures are either not defined, or are marked as being reserved for future use. these areas are not to be used by programmers, unless otherwise indicated. they are reserved for future expansion and enhancement of the blue wave packet structures, and if you use them for your own purposes, you run the risk of making your program incompatible with future updates of the file structures. furthermore, future structure updates will assume that these unused areas are "garbage-free" (i.e. they are filled with 0 values). in order to cover all bases, then, all unused areas should be set to 0. this can be easily done with the standard c function memset(). for example, using: memset(&infhdr, 0, sizeof(inf_header)); before you begin adding information to the *.inf header structure will ensure that all unused fields are set to 0. this should be done before adding information to *any* blue wave file structure. the *.inf file (inf_header & inf_area_info) =========================================== the *.inf file consists of two "parts": a header, which contains information about the host system and the user to whom the packet is intended, and a series of records that contain the information on all message bases available on the host system. (the latter is used in a reader both for posting replies and offline configuration.) the header structure is known as inf_header, and the record structure is known as inf_area_info. most of the fields in inf_header are self-explanatory. the following fields, however, deserve extra attention: ver packet version level. this is a crucial field, as it allows doors and readers to determine the revision level of the file structures used to create the packet. pro- grams should check this field to insure that they can properly handle the packet; doors should store in this field the current packet revision level, shown at the top of this document. readerfiles files that can be displayed by the reader. these are usually bulletins, sign-on banners, etc. up to five files can be specified. keywords specifies the keywords used while bundling messages. door authors that wish to provide keyword support should refer to the blue wave user documentation for more information. filters same as above, but specifies the filters used while bundling messages. macros same as above, but specifies the macros used in the door to specify message bundling operations. can_forward indicates that the user can forward messages to other message areas from within a reader. inf_header_len length of inf_header used in mail packet. door authors are required to fill this field; reader authors should use this field to properly parse the *.inf file. the current crop of blue wave doors properly fill this field, but older versions may not. refer to the header file for more information on using this field. inf_areainfo_len same as above, but specifies the length of the inf_area_info structure. mix_structlen same as above, but specifies the length of the *.mix file structure (mix_rec). fti_structlen same as above, but specifies the length of the *.fti file structure (fti_rec). uses_upl_file a non-zero value in this field indicates that the door which create the mail packet can process reply packets stored in the *.upl format. if zero, the door can only handle reply packets stored in the older *.upi and *.net formats. (these formats are discussed later.) packet_id the packet id used by the host system, which can be used by readers to properly access mail packet control files even if the mail packet itself has been renamed. (if this field is not filled, readers should assume that the root name of the mail packet is the packet id.) the inf_area_info structure fields are defined as follows: areanum the area number on the host system for which the record defines, specified as an ascii string. this field will correspond to the similar field in the *.mix structure (dis- cussed later). this does not necessarily have to be a number, so a case-insensitive match should be performed when doing any comparisons using this field. echotag the area tag name for which the record de- fines. this field is used in the reply packet to link reply messages to the destina- tion areas on the host system, and must be unique (i.e. there cannot be two or more identical area tags). title the message area description. area_flags specifies the unique characteristics of the message area for which the record defines. most of the flags are self-explanatory, as specified in the header file; the network area flags, however, need to be clarified. if the message area is part of a network (i.e. fidonet), the inf_echo flag should be set; this will allow a reader to properly handle network mail. if the area is for private network mail, the inf_netmail flag should also be set. network_type specifies the network type for which the defined message area belongs. note that the network type is specified as a full byte value, not as a bit flag. (if the inf_echo flag is not set, the network type can be ignored, obviously.) the *.mix file (mix_rec) ======================== each record in the *.mix file points to the beginning of the area in the *.fti file that contains the header information for messages that were obtained from each message area. note that *only* the areas for which messages were extracted will have a corresponding *.mix record. each structure field is defined as follows: areanum corresponds to the identical field in the *.inf record for the particular message area. it is used by the reader to coordinate the information between the *.inf and *.mix records. again, case-insensitive comparisons should be performed when using this field, as this field does not have to contain a numeri- cal value. totmsgs total number of messages in this area. up to 65,535 messages per area are allowed (the limit of an unsigned 16-bit integer). numpers total number of messages in this area that are directed specifically to the user who bundled the mail packet. this value is usually obtained during message bundling by comparing the name in the "to:" field to the user name for every message, and incrementing a counter if they match. (reader authors can feel free to disregard this value and perform their own search for personal messages, once the packet is loaded into the program.) msghptr pointer to the first record in the *.fti file that corresponds to this message area. note that this specifies the byte offset into the *.fti file, *not* the record number; thus, programmers can use the seek() or lseek() functions to quickly point to the proper section of the *.fti file. the *.fti file (fti_rec) ======================== the *.fti record specifies the header information for messages in the mail packet, along with additional information such as the length of the message and the pointer to the message stored in the *.dat file. these records are stored sequentially in the *.fti file, with all the headers for the first message area stored first, followed by the headers for the second message area, and so forth. most of the fields are self-explanatory, with the following fields described in more detail: msgnum the message number as stored on the host system. this provides an "absolute" message number for use with message threading. replyto indicates the absolute message number for which this message is a reply (the "previous" message). if zero, there is no previous message. replyat indicates the absolute message number of a reply to this message (the "next" message). if zero, there is no next message. msgptr pointer to the start of the message text in the *.dat file. this is a byte offset, thus a programmer can use the seek() or lseek() functions to quickly point to the proper position in the *.dat file. msglength length of the message text (in bytes) plus 1. the addition of 1 is done to compensate for the required space character in front of the message text (described below). flags bit-mapped message status flags. note that for fidonet network messages, these flags are *not* stored in the exact same order as specified in the fidonet technical standards, even though they are identical in function. the *.dat file |