Mailfromd NEWS -- history of user-visible changes. 2020-07-26 See the end of file for copying conditions. Please send Mailfromd bug reports to Version 8.8, 2020-07-26 * DKIM signing The new function 'dkim_sign' is available in the 'eom' handler for signing the current message using DKIM. The typical use is prog eom do dkim_sign("example.org", "s2048", "/etc/pem/my-private.pem") done * DKIM verification The 'dkim_verify' function verifies the DKIM signature of the message. Example usage: require status prog eom do set result dkim_verify(current_message()) if result == DKIM_VERIFY_OK # success elif result == DKIM_VERIFY_PERMFAIL # signature verification failed elif result == DKIM_VERIFY_TEMPFAIL # message not signed, the key is not available, or the like. fi done * Functions header_delete_nth, header_replace_nth were removed There's no reliable way to address Nth header in the message using the Milter API. * MFL changes ** Enumeration constants Enumeration constants are defined using the following syntax: [qualifier] const do name1 [expr0] ... nameN [exprN] done expr0-exprN are optional expressions evaluating to constant numeric values. In the absence of exprN, the nameN gets defined to the value of the previous enumeration item plus one. expr0 default to 0. For a detailed discussion, see the manual, section 4.8 "Constants". Version 8.7, 2019-01-03 * The --callout-socket option New option --callout-socket=URL instructs mailfromd to use URL to pass callout requests to the callout server listening on URL. It is equivalent to the callout-url configuration statement, which it overrides. This option is used by mtasim to avoid clobbering the existing callout sockets when starting new mailfromd instance. * NS lookup MFL functions This release implements the following new MFL functions: number primitive_hasns (string DOM) Returns 1 if the domain DOM has at least one NS record and 0 otherwise. Throws an error if DNS lookup fails. require 'dns' number hasns (string DOM) Same as above, but returns 0 on DNS lookup failures. string getns (string DOM ; number RESOLVE, number SORT) Returns a whitespace-separated list of all the NS records for the domain DOM. If optional parameter RESOLVE is 1, the returned list contains IP addresses. Optional SORT controls whether the entries are sorted. * Bugfixes ** Callout functions return true on checking the null return address (<>) ** Arguments in transaction between mailfromd and calloutd are quoted ** Avoid false failures in testsuite due to libadns warnings ** configure --with-dbm=T accepts any T supported by mailutils ** The 'dbdel' built-in silently ignores non-existing keys Version 8.6, 2018-07-24 * New configure option --with-dbm This option allows you to select any DBM flavor supported by mailutils as the default DBM implementation for mailfromd. * Fix byte compilation of mfl-mode.el * Minor fixes in DNS resolver * Case-insensitive comparison of SPF record marker Version 8.5, 2018-04-13 * Ensure proper integer promotion (was broken on certain 64-bit architectures). * Ensure case-insensitive comparison of SPF record markers. * Fix primitive_resolve() and resolve() * Fix assembling of fragmented TXT records. * Fix resolving of queries containing invalid characters. Version 8.4, 2017-11-03 * Requires Mailutils 3.4 Version 8.3, 2017-11-02 * GNU adns required This version requires the adns resolver library (https://www.gnu.org/software/adns). * Removed arbitrary limits on the sizes of DNS RRs The following configuration statements are removed: ** runtime.max-dns-reply-a ** runtime.max-dns-reply-ptr ** runtime.max-dns-reply-mx ** max-match-mx ** max-callout-mx * Removed caching of SPF results. The following MFL global variable have been removed: ** spf_ttl ** spf_cached ** spf_database ** spf_negative_ttl * New MFL functions string ptr_validate (string IP) Tests whether the DNS reverse-mapping for IP exists and correctly points to a domain name within a particular domain. Version 8.2, 2017-10-18 The purpose of this release is to simplify packaging with alpha version of Mailutils * Requires Mailutils 3.1.92 or newer Version 8.1, 2016-11-09 * Requires Mailutils 3.1 Version 8.0, 2016-11-09 This version is a major rewrite. Main changes: * Requires Mailutils 3.0 * New daemon calloutd The calloutd utility is a stand-alone callout daemon. It allows you to run sender address verification from a separate server. See the section 'Milter and Callout servers' below, for a detailed explanation. * New utility mfdbtool In previous versions database management tasks were performed by mailfromd itself when it was called with appropriate options (--list, --delete, --expire). Now these options are gone, and all database management tasks are carried out by a stand alone utility mfdbtool. * Changes to mailfromd configuration ** enable-vrfy The `enable-vrfy' statement enables the use of SMTP VRFY statement prior to normal callout sequence. If VRFY is supported by the remote server, mailfromd will rely on its reply and will not perform normal callout. This feature is provided for the completeness sake. Its use is not recommended, because many existing VRFY implementations always return affirmative result, no matter is the requested email handled by the server or not. ** The `listen' statement withdrawn Use `server milter' statement instead, e.g.: server milter { listen "inet://127.0.0.1:7788"; } ** The `--remove' option withdrawn ** The `backlog' statement The new `backlog' statement is provided for configuring the size of the queue of pending connections. The statement is available in `server' block, e.g.: server milter { id main; listen unix:///var/lib/mailfromd/mailfrom; backlog 16; } * Changes to MFL ** Initial #!/ comment If a source begins with `#!/' or `#! /' characters, mailfromd treats this as a start of a multi-line comment, which is closed by the `!#' on a line by themselves. This feature allows for compensating for the deficiences of the traditional `#!' script magic. For example, if a mailfromd script must be invoked with some additional option passed to mailfromd (say --no-user-conf), it can now be written as: #!/bin/sh exec /usr/sbin/mailfromd --no-config --run $0 $@ !# func main(...) returns number do ... done ** The use of % before variable names is no longer supported. The % is now used as modulo operator (see below). ** New operators << bitwise shift left >> bitwise shift right % modulo ** New pragma prereq The "#pragma prereq" statement ensures that the version of mailfromd used to compile the source file is correct. It takes version number as its arguments and produces a compilation error if the actual mailfromd version number is earlier than that. E.g. #pragma prereq 7.0.94 ** New exception e_exists This exception is emitted by the dbinsert built-in if the requested key is already present in the database. ** New built-in functions void _expand_dataseg (number N) Expands the run-time data segment by at least @var{n} words. number _reg (number r) Returns the value of the register @var{r} at the moment of the call. Symbolic names for run-time registers are provided in the module "_register". void _wd ([number N]) Enters a time-consuming loop which waits for N seconds (by default -- indefinitely). Before entering the loop, a diagnostic message is printed on the syslog facility "crit", listing the PID of the process and suggesting the command to be used to attach to it. number access (string FILE, number MODE) Interface to the access(2) function. Checks whether the calling process can access the FILE using MODE. Returns true on success. Symbolic values for MODE" are provided in module "status". number callout_transcript ([number N]) Returns the current state of the callout SMTP transcript. With argument N, sets the callout transcript to that value (0 - disabled, 1 - enabled). string default_callout_server_url (void) Returns the URL of the default callout server. string fd_delimiter (number FD) Returns line delimiter string for file @var{FD}. void fd_set_delimiter (number FD, string DELIM) Sets line delimiter for file FD. string getenv (string VAR) Searches the environment list for the variable VAR and returns its value. If the variable is not defined, raises the e_not_found exception. string ltrim (string STR [, string CSET) Returns a copy of the input string STR with any leading characters present in CSET removed. If the latter is not given, white space is removed (spaces, tabs, newlines, carriage returns, and line feeds). string message_nth_header_name (number MD, number N) Returns name of the Nth header in message MD. If there is no such header, the e_range exception is raised. string message_nth_header_name (number MD, number N) Returns value of the Nth header in message MD. If there is no such header, the e_range exception is raised. void message_to_stream (number FD, number MD[, string FLTCHAIN]) Copies message MD to stream descriptor FD. The descriptor must have been obtained by a previous call to "open". Optional FLTCHAIN supplies the name of Mailutils filter chain, through which the data will be passed before writing to FD. number message_from_stream (number FD; string FLTCHAIN) Converts contents of the stream identified by FD to a mail message. Returns identifier of the created message. string message_body_to_stream (number FD, number MD[, string FLTCHAIN]) Copies body of the message MD to stream descriptor FD. The descriptor must have been obtained by a previous call to "open". Optional FLTCHAIN supplies the name of Mailutils filter chain, through which the data will be passed before writing to FD. boolean message_body_is_empty (number MD) Returns true if the body of message MD has zero size or contains only whitespace characters. If the "Content-Transfer-Encoding" header is present, its value is used to decode body before processing. void replbody_fd (number FD) Replaces body of the current message with the content of the stream FD. Use this function if the body is very big, or if it is returned by an external program. Notice that this function starts reading from the current position in FD. Use rewind() to ensure that entire stream is copied. void rewind (number FD) Rewinds the stream identified by FD to its beginning. string rtrim (string STR [, string CSET) Returns a copy of the input string STR with any trailing characters present in CSET removed. If the latter is not given, white space is removed (spaces, tabs, newlines, carriage returns, and line feeds). void shutdown (number FD, number HOW) Interface to the shutdown(2) call. Causes all or part of a full-duplex connection FD to be closed. FD must be either a socket descriptor or a two-way pipe socket descriptor (returned by "open(|&...)"), otherwise the call to shutdown() is completely equivalent to close(). The HOW argument identifies which part of the connection to shut down. void set_from (string EMAIL[, string ARGS]) Sets envelope sender address to EMAIL, which must be a valid email address. Optional ARGS supply arguments to ESMTP MAIL FROM command. void send_message (number MD [, string TO, string FROM, string MAILER]) Send the message identified by descriptor MD. Optional arguments supply recipient email addresses (TO), sender email address (FROM), and the URL of the mailer to use. number spawn (string CMD [, number IN, number OUT, number ERR]) Runs the command CMD. The syntax of CMD argument is the same as for the NAME argument to open(), which begins with "|", excepting that the "|" sign is optional. void syslog (number PRIO, string TEXT) Sends TEXT to syslog using facility/priority PRIO. number tempfile ([string DIR]) Creates a nameless temporary file and returns its descriptor. Optional DIR specifies the directory where to create the file (default - "/tmp"). void unlink (string NAME) Unlinks (deletes) the file NAME. On error, throws the e_failure exception. number vercmp (string A, string B) Compares two strings as version numbers. The result is negative if B precedes A, positive if B is later than A, and zero, if they refer to the same version. ** message_burst New function `message_burst' converts an RFC-934 digest message into a MIME message: number message_burst(number nmsg; number flags) The descriptor of the input message is given as its argument (nmsg). On success, the function returns a descriptor of the newly created message. If the input message is not a digest, the e_format exception is raised. The global variable `burst_eb_min_length' sets the minimal length of the encapsulation boundary for digests. ** spf_mechanism Upon successful return from spf_check_host (or spf_test_record), the built-in variable spf_mechanism contains a whitespace-separated list of mechanisms that were matched when evaluating the query. This includes any include: or redirect= statements traversed during evaluation. The mechanisms are listed in reverse order (latest first). ** clamav and sieve These two functions take a descriptor of the message as their first argument. The new prototypes are: number sieve(number msg, string script ; number flags, string file, number line) number clamav(number msg, string url) This change is incompatible with previous versions. To use these functions in the eom handler, pass current_message as their first argument, e.g.: prog eom do if clamav(current_message(), "tcp://127.0.0.1:3344") ... ** sa and spamc New function `spamc' is an improved version of the old `sa' function: number spamc(number nmsg, string url, number prec, number command) Arguments are: nmsg - descriptor of the message to be processed, url - URL of the spamd server, prec - precision, command - command to send to spamd. Allowed values for command argument are: SA_SYMBOLS Process the message and return 1 or 0 depending on whether it is diagnosed as spam or not. Store SpamAssassin keywords in the global variable sa_keywords. SA_REPORT Process the message and return 1 or 0 depending on whether it is diagnosed as spam or not. Store entire SpamAssassin report in the global variable sa_keywords. SA_LEARN_SPAM Learn the supplied message as spam. SA_LEARN_HAM Learn the supplied message as ham. SA_FORGET Forget any prior classification of the message. The function `sa' is rewritten as a wrapper over `spamc' ** open The open call now supports the "|<" prefix to its first argument: set fd open("|< progname arg") It starts the program with its stdin closed and stdout open for reading. The usual stderr redirection is allowed between the '<' and the command line. ** New M4 macros New macro `string_list_iterate' is provided to compensate for the lack of array data type in MFL. The macro splits its argument into segments separated by a supplied delimiter and, for each segment, executes MFL code given as its last argument. For example: string_list_iterate(path, ":", seg, ` if access(seg, F_OK) echo "%seg exists" fi') This code treats the `path' argument as a UNIX path string. Each directory component is tested for existence and is output if the test succeeds. ** New library functions require 'callout' number callout(string EMAIL) require 'callout' number callout_open(string URL) require 'callout' void callout_close(number FD) require 'callout' number callout_do(number FD, string EMAIL[, string REST]) require 'header_rename' void header_prefix_pattern (string PATTERN[, string PREFIX]) Rename all headers matching PATTERN (see glob(7)) by prefixing them with PREFIX. If prefix is not given, remove such headers. require 'header_rename' void header_prefix_all (strin NAME[, string PREFIX]) Rename all headers with given NAME by prefixing them with PREFIX. If prefix is not given, remove such headers. ** debug_spec changed signature ** listens and portprobe The `listens' function was moved to the `portprobe' module. It is actually an alias to the `portprobe' function. If your filter uses `listens', require the `portprobe' module. ** _pollhost, _pollmx, stdpoll, strictpoll These functions have been moved to the `poll' module, which must be required prior to using any of them. ** message_header_count This function takes an optional string argument: number message_header_count (number NMSG [, string NAME]) If NAME is supplied, only headers with this name are counted. * mtasim Mtasim now sends SMFIC_CONNECT (i.e. invokes the "connect" handler). Unless connection parameters are supplied, the handler is called with family 0 (FAMILY_STDIO) and "localhost" as the host name. Connection parameters can be supplied from the command line (using the --sender-sockaddr option), or from the interactive shell, using the \S command. See documentation, chapter 12 "`mtasim' -- a testing tool", for a detailed discussion. * Bugfixes ** Next in do-while loops The `next' keyword bypassed conditional when used in a do-while loop, making it effectively endless. This is fixed. Version 7.0, 2010-08-07 * Incompatible changes The following features, that had been marked as deprecated in 6.0, are now removed: - old-style functional notation - the use of functional operators - implicit concatenations - #pragma option - #pragma database * Milter and Callout servers This release introduces a notion of a `server', i.e. a special `mailfromd' entity responsible for handling a particular task. Two kinds of servers are supported: Milter servers, which handle the milter requests, and Callout servers, which run callout (or sender verification) SMTP sessions and update the cache database accordingly. When a callout server is enabled, sender verification functions work the following way. First, the usual sender verification is performed with a set of so-called `soft' timeout values. If this verification yields a definite answer, that answer is stored in the cache database and returned to the calling procedure as usual. In that regard, this release works exactly as its predecessors did. If, however, the verification is aborted due to a timeout, the caller procedure is returned the e_temp_failure exception, and the session is scheduled for processing by a callout server. The latter processes the request using a set of `hard' timeouts, which are normally much longer than `soft' ones (their default values are those required by RFC 2822; see below for a detailed description). This callout session runs independently of the milter session that initiated it and which, having initiated the callout, returns a temporary error to the sender, thereby urging it to retry the connection later. In the meantime, the callout server has a chance to finish the requested sender verification and store its result in the cache database. When the sender retries the delivery, the milter server will obtain the already cached result from the database. If the callout server has not finished the request by the time the sender retries the connection, the latter is again returned a temporary error, and the process continues until the callout is finished. Milter servers are declared using the following configuration statement: server { # Server ID. id ; # Listen on this URL. listen ; # Maximum number of instances allowed for this server. max-instances ; # Single-process mode. single-process ; # Reuse existing socket (default). reuseaddr ; } If the type is `callout', the `server' block statement may also contain the following sub-statement: default ; When arg is `yes', this server is marked as the default callout server for all milter servers declared in the configuration. Alternatively, you may use a remote callout server run by a separate daemon 'calloutd'. In that case, the URL of a callout server is declared using the `callout-url' statement: callout-url ; * Timeout control for callout SMTP sessions Callout SMTP sessions initiated by polling functions are controlled by two sets of timeouts: `soft' and `hard'. Soft timeouts are used by the mailfromd milter servers. Hard timeouts are used by callout servers. When a soft timeout is exceeded, the calling procedure is delivered the e_temp_failure exception and the session is scheduled for processing by a callout server. The latter re-runs the session using hard timeouts. If a hard timeout is exceeded, the address is marked as `not_found' and is stored in the cache database with that status. Normally, soft timeouts are set to shorter values, suitable for use in MFL scripts without disturbing the calling SMTP session. Hard timeouts are set to large values, as requested by RFC2822, which guarantee obtaining a definitive answer (see below for the default values). Individual timeouts may be set in the configuration file using the following statement: smtp-timeout [soft | hard] { # Initial SMTP connection timeout. connection ; # Timeout for initial SMTP response. initial-response ; # Timeout for HELO resonse. helo ; # Timeout for MAIL response. mail ; # Timeout for RCPT response. rcpt ; # Timeout for RSET response. rset ; # Timeout for QUIT response. quit ; }; The default timeout settings are: Timeout Soft Hard --------------------------------- connection 10s 5m initial-response 30s 5m helo I/O 5m mail I/O 10m rcpt I/O 5m rset I/O 5m quit I/O 2m The entries marked with I/O, unless set explicitly by a corresponding `smtp-timeout soft' entry, are set to the value of I/O timeout (see the `io-timeout' configuration statement and the `--timeout' command line option), which defaults to 3s. * Automatic export of the "i" macro. The "i" Sendmail macro is automatically requested for the lowest used stage handler. This ensures the log messages are marked with the corresponding queue ID. Notice, however, that the MTA is free to honor or ignore the request. Generally speaking, this works well with Sendmail starting from 8.14.0 and MeTA1 starting from 1.0.PreAlpha29.0. The "i" macro is almost useless with Postfix 2.6 or higher, because it defines it only at the EOM stage. * Language changes ** Use of % in front of identifiers It is no longer necessary to use % in front of an identifier. To reference a variable, simply use its name, e.g.: set x var + z However, old syntax is still supported, so the following statement will also work: set x %var + %z It will, however, generate a warning message. Of course, the use of % to reference variables within strings remains mandatory. ** User-defined exceptions In addition to the built-in exception codes, you may define your own exceptions. This is done using the `dclex' statement. For example: dclex myerror This statement declares a new exception identifier `myerror'. This identifier may then be used in `catch' and `throw' statements, e.g.: throw myerror "My error encountered" For a detailed discussion, see the manual, subsection 4.19.2 "User-defined Exceptions". ** The try-catch construct This version introduces a `try-catch' construct similar to that used in another programming languages. The construct is: try do STMTLIST-1 done catch EX-LIST do STMTLIST-2 done where SMTLIST-1 and STMTLIST-2 are lists of MFL statements and EX-LIST is a list of exceptions. The control flow is as follows. First, the statements from STMTLIST-1 are executed. If the execution finishes successfully, control is passed to the first statement after the `catch' block. Otherwise, if an exception is signalled and this exception is listed in EX-LIST, the execution is passed to the STMTLIST-2. The `try-catch' construct allows a better and more flexible error recovery. For a detailed discussion, see the manual, subsection 4.19.3 "Exception Handling". ** Handling of stderr in open("|...") If the open function is used to start an external program (i.e. the argument to open() begins with a `|' or `|&'), the standard error of the program is closed prior to starting it. The following special constructs are provided for redirecting it to an output file or syslog: - "|2>null: COMMAND" Standard error is redirected to /dev/null. - "|2>file:NAME COMMAND Standard error is redirected to the file NAME. If the file exists, it will be truncated. - "|2>>file:NAME COMMAND Standard error is appended to the file NAME. If file does not exist, it will be created. - "|2>syslog:FACILITY COMMAND" - "|2>syslog:FACILITY.PRIORITY COMMAND" Standard error is redirected to the given syslog facility and, if specified, priority. If the latter is omitted, LOG_ERR is assumed. ** Message Modifications and Accept. Calling `accept' undoes any modifications to the message applied by, e.g., header_add and the like. This is due to requirements of the Milter protocol. This behavior caused several false bug reports in the past. Starting with this version, calling `accept' after any modifications to the message results in the following warning message: RUNTIME WARNING near /etc/mailfromd.mf:36: `accept' causes previous message modification commands to be ignored; call mmq_purge() prior to `accept', to suppress this warning It is only a warning and the `accept' action itself is, of course, honored. If you see this diagnostics in your log, do the following: - if the behavior was intended, call mmq_purge, as suggested (see below for a description of this function); - if it was not, read the manual, section 5.8, "Message Modification Queue", for information on how to handle it. ** #pragma miltermacros The new `#pragma miltermacros' declares macros that are referenced at a given stage. It is helpful when automatic macro negotiation is in use and mailfromd is unable to trace all macros referenced from each handler, i.e. when at least one of the handlers: - calls functions that reference MTA macros; - refers to macros via macro_defined() and getmacro() functions. See the manual, subsection 4.2.5, "Pragma miltermacros", for a detailed description. ** MFL functions *** progress The `progress' function notifies the MTA that the filter is still processing the message and urges it to restart its timeouts. It is available only in the `eom' handler. *** sleep The `sleep' function takes an optional second argument. If given, it specifies the number of microseconds to wait. For example, to sleep for 1.5 seconds, use: sleep(1,500000) *** dequote string dequote(string email) The function `dequote' removes angle brackets surrounding its argument and returns the resulting string. If there are no angle brackets, or if they are unbalanced, it returns unchanged argument. *** debug_spec string debug_spec([string modnames[, number minlevel]]) Returns the current debugging level specification. Optional parameters supply conditions for abridging the amount of information returned. The `minlevel' parameters instructs the function to return only those specifications that have the level part greater than or equal to the given value. The `modnames' parameter (a comma-separated list of module name parts) selects only those specification that match the supplied module names. *** mmq_purge The function `mmq_purge' purges internal message modification queue. This undoes the effect of the following functions, if they had been called previously: rcpt_add, rcpt_delete, header_add, header_insert, header_delete, header_replace, replbody, quarantine. ** New built-in constant `__git__'. The `__git__' built-in constant is defined for alpha versions only. Its value is the GIT tag of the recent commit corresponding to that version of the package. * smap The smap utility has been removed. A new project was established that provides and largely expands its functionality. See http://smap.software.gnu.org.ua, for additional information, including links to file downloads. * Version output for alpha versions. Starting from this release, all alpha versions output additional information when invoked with the `--version' option. The new output looks like this: mailfromd (mailfromd) 6.0.91 [release-6.0-17-ga3fa5da] where the string between brackets is the recent GIT tag (see also the description of `__git__' constant above). If this release contains some uncommitted changes, the suffix `-dirty' is appended to it. * mtasim New option --milter-timeout sets timeouts for Milter I/O operations. Version 6.0, 2009-12-12 * Overview This release is aimed to fix the logical inconsistencies that affected the previous versions, in preparation to the new major release that will follow. Some features that have been previously declared as deprecated are now removed, whereas some obsolete features are marked deprecated now and will be removed in the future version. These changes are described in the chapter `Incompatible changes'. A special mechanism is provided to facilitate migration to the new syntax. It is described in chapter `Upgrade procedure'. Please read it carefully before upgrading. * Incompatible changes Historically, the filter script file contained both the actual filter source code, and run-time configuration directives for mailfromd (in the form of `#pragma option' and `#pragma database' statements). Such a mixture of concerns is counter-productive in the long run. Starting from version 6.0 the two concerns are separated. The filter source code is kept in the file `$sysconfdir/mailfromd.mf' (see below), and the run-time configuration is provided by the configuration file `$sysconfdir/mailfromd.conf'. (see the docs, chapter "Mailfromd Configuration"). Thus, changing run-time configuration does not imply changing the filter program itself and vice-versa. This allows to implement a full-fledged module system. Consequently, the `#pragma option' and `#pragma database' statements are deprecated. In previous versions, the filter script file had the `.rc' suffix. This was wrong, because this suffix usually marks a configuration file, which `mailfromd.rc' was not. Starting with this release the script file is renamed to `mailfromd.mf'. For compatibility reasons, in the absence of this file, the legacy file `mailfromd.rc' is recognized and parsed. ** Pies withdrawn The `pies' utility has been removed from the package. It is moved into a stand-alone package called `Pies'. See http://pies.software.gnu.org.ua, for more information about the package, including pointers to file downloads. ** Unquoted literals The use of unquoted literals is no longer allowed. ** Unnamed parameters and short type names in function declarations The following style of function declarations is deprecated: func foo(string,number) The use of first letter instead of the full type name is deprecated as well. ** Operational notation Historically, MFL allowed to call functions of single argument using operational notation, i.e. to write: foo 2 instead of foo(2) Such a usage is deprecated and its support will be discontinued in future versions. ** Implicit concatenation Implicit concatenation of variables (e.g. `%bar %baz') is still supported but issues a deprecation warning. It will be removed in future versions. Use `.' operator instead. See `Explicit concatenation', below. Notice, however, that this change does not affect adjacent literals, which are implicitly concatenated as before. In other words, the following statement is OK: set foo "string" "ent test" but the following is not: set foo %bar %baz It should be written as set foo %bar . %baz ** #require The `#require' keyword is deprecated. Use `require' instead. See `Module system', for details. ** message_read_line, message_read_body_line and EOF These functions raise EOF exception if there are no more lines to read. In previous version, e_io was signalled in this case. * Upgrade procedure To remove the deprecated features from your scripts and upgrade them for the new configuration system, follow the steps below: 1. Run `mailfromd --lint'. It will show a list of warnings, similar to this (though, perhaps, much longer): mailfromd: Warning: using legacy script file /usr/local/etc/mailfromd.rc mailfromd: Warning: rename it to /usr/local/etc/mailfromd.mf or use script-file statement in /usr/local/etc/mailfromd.conf to disable this warning mailfromd: /usr/local/etc/mailfromd.rc:19: warning: this pragma is deprecated: use relayed-domain-file configuration statement instead mailfromd: /usr/local/etc/mailfromd.rc:23: warning: this pragma is deprecated: use io-timeout configuration statement instead mailfromd: Info: run script /tmp/mailfromd-newconf.sh to fix the above warnings 2. At the end of the run mailfromd creates a shell script named `/tmp/mailfromd-newconf.sh'. To fix the above warnings, run this script: $ sh /tmp/mailfromd-newconf.sh It will edit and patch all MFL sources that need upgrading. A backup copy of each source file will be created. The name of each backup file is constructed by appending `.bak' to the original file name. 3. Now retry `mailfromd --lint'. It should show no warnings now. Remember, that your script file is now named `mailfromd.mf'. * New features ** Explicit concatenation The concatenation of two string operands is indicated by the `.' (dot) operator between them: set boo %bar . %baz Implicit concatenation (e.g. `%bar %baz') is still supported but issues a deprecation warning. It will be removed in future versions. ** Explicit type casts The syntax for an explicit type cast is: TYPE(EXPR), where TYPE is the corresponding type name, e.g.: set val string(10 + %a) ** Module system A module is a logically isolated part of code that implements a separate concern or feature. Each module occupies a separate compilation unit (i.e. file). The functionality provided by the module is incorporated into the main program by "requiring" this module or by "importing" the desired components from it. To require a module, use the following syntax: require MODNAME where MODNAME is the name of the module, which corresponds to the name of its compilation unit, without the `.mf' suffix. Unless MODNAME is a valid identifier in MFL, it should be enclosed in quotes (single or double). Note, that the old syntax `#require FILENAME' is deprecated but it still supported (see `Upgrade procedure', below). To import a subset of symbols from a module, use the following syntax: from MODNAME import NAMES. where MODNAME is as described above, and NAMES is a comma-separated list of the symbol names to import. Note that the final dot is mandatory. Each NAME may also be a regular expression, in which case only those symbols that match the expression are imported, or a transform expression, which allows to rename imported symbols on the fly. See Mailfromd Manual, subsection 4.20.1, "Declaring Modules", for a detailed description of the import syntax in general and transform expressions in particular. A couple of examples to illustrate the concept: 1) from A import foo,bar. From module A (file A.mf) import symbols foo and bar. The importing module can then refer to these symbols by their name. 2) from A import '/foo.*[0-9]/'. From module A (file A.mf) import all symbols that match the regular expression between //. 3) from A import '/foo.*[0-9]/s/.*/my_&/'. From module A import all symbols that match the given regular expression and rename them, by prefixing each of them with the string 'my_'. Thus, e.g. function `foo_1' becomes `my_foo_1', etc. ** Module declaration A module is declared using the following syntax: module MODNAME [INTERFACE-TYPE]. The final dot is mandatory. The optional INTERFACE-TYPE defines the type of the module interface. If it is `public', then all the symbols declared in this module are made public (importable) by default, unless explicitly declared otherwise (See `Symbol scope', below). If it is `static' all symbols, not explicitly marked as public, become static by default. Default is `public'. The module definition is terminated by the logical end of its compilation unit, i.e. either by the end of file, or by the keyword `bye' (see below), if it is used. ** The `bye' keyword. Special keyword `bye' may be used to prematurely end the current compilation unit before the physical end of the containing file. Any material between `bye' and end of file is ignored by the compiler. ** New built-in constant `__module__' The `__module__' constant keeps the name of the current module. For the main script file, its value is "top". ** Symbol scope The qualifiers `static' or `public' may be used in front of the declaration to set the scope of visibility of the symbol. Symbols defined as `static' are visible only within the current module, whereas the ones declared as `public' are visible outside as well. The default scope is declared in the module declaration (see below). Examples: static const ONE 1 public func foo() See also `Precious variables', below. ** Global variables and Milter abort When Milter abort request is received, all global variables, excepting precious ones (see below), are reset to their initial values. In particular, this occurs when the MTA receives the RSET command. ** Precious variables A new keyword "precious" may be used in front of a variable declaration to inform Mailfromd that this variable must retain its value across Milter abort requests. For example: precious number rcpt_counter prog envrcpt do set rcpt_counter %rcpt_counter + 1 done In this code, the value of rcpt_counter variable will reflect the total number of SMTP "RCPT TO" commands received during this session, including those that have been cancelled by RSET commands. `Precious' may be combined with a scope qualifier. In that case the order of their appearance is irrelevant. E.g. the two declarations below are equivalent: static precious string rcpt_list precious static string rcpt_list The following built-in variables are implicitly declared as precious: ehlo_domain, mailfrom_address, milter_client_family, milter_client_address, milter_server_family, milter_server_address. * New built-in variables ** number milter_client_family Address family of the client connection. ** string milter_client_address Address (IPv4 or UNIX socket) of the Milter client. ** number milter_server_family Address family of the Milter server socket. ** string milter_server_address Address (IPv4 or UNIX socket) the server socket is bound to. * Changes to MFL functions ** getmx The host names or IPs in the getmx return are sorted in order of increasing MX priority. ** is_greylisted If the `is_greylisted' function returns 1, it sets the global variable `greylist_seconds_left' to the number of seconds left to the end of greylisting period. * Bugfixes ** ismx function correctly handles MX names that have more than one A records. Version 5.2, 2009-08-27 * `next' statement The definition of `next' statement has been fixed to match `continue' in other programming languages. Namely, `next' passes control to STMT2 in the loop definition: loop for STMT1, while EXPR1, STMT2 * Process titles. The process titles visible in the output of the ps(1) command reflect the actual states of the corresponding mailfromd subprocesses. For example: $ ps axw|grep mailfromd 11251 ? S 0:00 mailfromd: n5AIs7aL019522: MAIL FROM SIZE=1417 BODY=8BITMIME 19980 ? S 0:00 mailfromd: n5AJ5kFO021484: aborting 30497 ? S 0:00 mailfromd: HELO s6.newveoron.com * GeoIP support Two new built-in functions are added to the MFL: - string geoip_country_code_by_addr(string ip[, bool tlc]) This function looks up the country code by IP address in string form. - string geoip_country_code_by_name(string name[, bool tlc]) This function looks up the country code by host name. By default both functions return 2 letter country code (ISO 3166-1 alpha-2). When a non-zero value is given as the `tlc' argument, these functions return 3 letter country code (ISO 3166-1 alpha-3). Both functions raise the `e_not_found' exception if they fail to determine the country code. The GeoIP functions are available only if the GeoIP library (see http://www.maxmind.com/app/c) is installed on the host. Applications may test whether the GeoIP support is present and enable corresponding code blocks conditionally by testing if the `WITH_GEOIP' m4 macro is defined. E.g.: m4_ifdef(`WITH_GEOIP',` add "X-Originator-Country" geoip_country_code_by_addr($client_addr) ') * The gethostname function The gethostname function takes an optional argument: string gethostname ([bool FQN]) If FQN is specified and is `true', the function attempts to determine a fully qualified host name. * The localdomain function New function is defined in the library module `localdomain': string localdomain() It returns the domain name of the box mailfromd is executed on. It is more reliable than getdomainname, because it uses DNS to determine the fully qualified domain name. * `Safedb' functions. Two new `safedb' interfaces are implemented: string safedbmap (string db, string key [, string defval, number null]) void safedbdel (string db, string key [, number null]) A new global variable is added, which controls the verbosity of all safedb functions. If the variable safedb_verbose is set to 1, these functions log the detailed diagnostics about intercepted exceptions before returning to the caller. * `Open' function The `open' function can be used to connect to TCP/IP sockets. To do so, its first argument must be a valid socket URL prefixed with `@'. E.g.: number fd open("@ inet://127.0.0.1:25") The I/O operations on `fd' will write to and read from the opened socket. * System user database functions These functions provide interfaces to corresponding POSIX calls: - string getpwnam (string NAME) - string getpwuid (number UID) The return value is the string in the usual /etc/passwd format. If argument is not found in the system password database, these functions raise the e_not_found exception. In addition, the following two functions are provided for checking whether the given key is present in the system password database: - boolean mappwnam (string NAME) - boolean mappwuid (number UID) These functions never raise exceptions. * New I/O functions - string read (number RD, number N) Reads N bytes from the resource descriptor RD. - string getdelim (number RD, string DELIM) Reads the string terminated by DELIM from the resource descriptor RD. DELIM must contain exactly one character. * Sockmap functions The sockmap.mf module provides functions for interfacing with MeTA1 "sockmaps": - string sockmap_lookup(number FD, string MAP, string ARG) - string sockmap_single_lookup(string URL, string MAP, string ARG) * pies Signals can be specified in tag list of `return-code' statement. A signal is given either by its name, or as SIG+n, where n is its number. Valid signal names are: SIGHUP, SIGINT, SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGIOT, SIGBUS, SIGFPE, SIGKILL, SIGUSR1, SIGSEGV, SIGUSR2, SIGPIPE, SIGALRM, SIGTERM, SIGSTKFLT, SIGCHLD, SIGCONT, SIGSTOP, SIGTSTP, SIGTTIN, SIGTTOU, SIGURG, SIGXCPU, SIGXFSZ, SIGVTALRM, SIGPROF, SIGWINCH, SIGPOLL, SIGIO, SIGPWR, SIGSYS. Examples of usage: return-code (SIGABRT, EX_USAGE) { ... } return-code SIG+6, EX_USAGE { ... } New action is allowed in `return-code' block: exec COMMAND; It executes the given COMMAND before other actions. * Debugging New function mailutils_set_debug_level allows to set global Mailutils debug level from your MFL scripts. This is useful for debugging scripts that use mailbox or message accessing functions. Version 5.1, 2009-05-13 * Milter v6. The version 6 of Milter protocol is implemented, which is compatible with Sendmail 8.14.0 and newer. While being backward compatible with the earlier versions, it allows you to use the new `prog data' handler. It also supports macro negotiation, a feature that enables Mailfromd to ask the MTA to export the macros it needs for each particular handler. This means that if you are using Sendmail 8.14.0 or higher (or Postfix 2.5 or higher), you no longer need to worry about exporting macro names in sendmail.cf file. The same feature is also implemented on the server side, in mtasim and pmult. Consequently, using `define-macros' in pmult configuration file is not strictly necessary. However, keep in mind that due to the specifics of MeTA1, the number of symbols that may be exported for each stage is limited (Mailfromd manual, section 11.1.2). * Reject and tempfail actions: Functional notation The reply actions `reject' and `tempfail' allow functional notation, i.e. their arguments can be supplied as to a function: reject(550, 5.7.7, "IP address does not resolve") An important feature of this notation is that all three arguments are MFL expressions, which means that you can now compute the reply codes at run time: reject(550 + %n, "5.7." %x, "Transaction rejected") An argument can be omitted, in which case the default value is used, e.g.: reject(550 + %n, , "Transaction rejected") * New functions A set of new functions is added that allow to access the headers from the current message in a uniform fashion. These functions are available in the following handlers: eoh, body, eom. - number current_header_count([string name]) Return number of headers in the current message. With an argument - return number of headers that have this name. - string current_header_nth_name(number n) Return the name of the nth header. N is 1-based. - string current_header_nth_value(number n) Return the value of the nth header. N is 1-based. - string current_header(string name[, number index]) Return the value of the named header, e.g.: set s current_header("Subject") Optional second argument specifies the header instance, if there are more than 1 header of the same name, e.g.: set s current_header("Received", 2) Index is 1-based. All current_header functions raise the e_not_found exception if the requested header is not found. New system information functions are added: - string gethostname () Return the host name of this machine. - string getdomainname () Return the domain name of this machine. - string uname (string format) Return system information formatted according to the format specification. * New pragma `dbprop' This pragma defines user database properties. It takes two or three arguments: #pragma dbprop where is the name of the database or a shell globbing pattern, is the word "null" if the terminating null byte is included in the key length, and is the database file mode, either in octal or in usual `ls' notation (e.g. rw-r-----). Either of or may be omitted. If both are given, they may appear in any order. * Token Bucket Filter The new function is provided: bool tbf_rate(string key, number cost, number interval, number burst_size) It implements a classical token bucket filter algorithm. Tokens are added to the bucket identified by the `key' at constant rate of 1 token per `interval' microseconds, to a maximum of `burst_size' tokens. If no bucket is found for the specified key, a new bucket is created and initialized to contain `burst_size' tokens. For example: if not tbf_rate($f "-" ${client_addr}, 1, 10000000, 20) tempfail 450 4.7.0 "Mail sending rate exceeded. Try again later" fi This adds a token every 10 seconds with a burst size of 20 and a cost of 1. In other words, it allows to sent up to 20 emails within the first 10 seconds after sending the very first email from the given email/host address pair. After that, that pair is allowed to send at most 1 message per 10 seconds. One of possible implementations for this function is to limit the total size of messages tranferred per given amount of time. To do so, the tbf_rate must be used in `prog eom'. The `cost' value must contain the number of bytes in an email (or email bytes * number of recipients), the `interval' must be set to the number of bytes per microsecond a given user is allowed to send, and the `burst_size' must be large enough to accommodate a couple of large emails. E.g.: prog eom do if not tbf_rate($f "-" ${client_addr}, message_size(current_message()), 10240, # At most 10 Kb/ms 2000000) tempfail 450 4.7.0 "Data sending rate exceeded. Try again later" fi done The `tbf_rate' implementation is contributed by John McEleney and Ben McKeegan. * Greylisting A new implementation of the `greylist' function is provided. In the contrast to the traditional implementation, which keeps in the database the time when the greylisting was activated for the given key, the new one stores the time when the greylisting period is set to expire. This implementation allowed to implement the `is_greylisted' function: bool is_greylisted(string key) which returns True if the `key' is currently greylisted, and False otherwise. This implementation is based on the patch by Con Tassios. By default, the traditional implementation is used, which ensures backward compatibility with the previous versions. To switch to the new implementation, use the following pragmatic comment at the beginning of your script: #pragma greylist con-tassios or #pragma greylist ct * The rate builtin The rate builtin function now takes an optional `threshold' argument: number rate(string key, number interval, [number mincnt, number threshold]) If the observed rate (per interval seconds) is higher than the threshold, the rate function does not increment the hit counters for that key. That way messages that were not accepted do not affect the calculated rate. Normally, the threshold argument should be equal to the value used in the right side of comparison operator, e.g.: if rate($f "-" ${client_addr}, %rate_interval, 4, %maxrate) > %maxrate tempfail 450 4.7.0 "Mail sending rate exceeded. Try again later" fi The threshold argument is made optional in order to provide backward compatibility with the prior releases of mailfromd. Nevertheless, its use is strongly encouraged. To simplify the task, the new function `rateok' is provided (see below). * The rateok function A new library function is provided: bool rateok(string key, number sample_span, number threshold; number mincnt) This is a higher-level interface to the rate function. This function returns True if the mail sending rate for `key', computed for the interval of `sample_span' seconds is less than the `threshold'. Optional `mincnt' parameter supplies the minimal number of mails needed to obtain the statistics. It defaults to 4. An example of rateok usage follows: #require rateok prog envfrom do if not rateok($f "-" ${client_addr}, interval("1 minute"), 40) tempfail 450 4.7.0 "Mail sending rate exceeded. Try again later" fi done This example limits the rate to 40 mails per minute. * Rate expiration In addition to the usual expiration algorithm, the rate records are also expired if no mails were received during a time span greater than the value of the 2nd argument to the rate (or rateok) function. * The __statedir__ built-in constant. The __statedir__ built-in constant is now expanded to the current value of the program state directory. In prior releases it used to expand to the default program state directory. A new built-in constant __defstatedir__ is introduced, which expands to the value of the default program state directory. * The __preproc__ built-in constant. Similarly, the __preproc__ built-in constant, which used to signify the default preprocessor command line, now expands to its current value. The new constant __defpreproc__ expands to the default preprocessor command line. * Bugfixes ** Second argument to envfrom and envrcpt ** write without third argument ** sa_format_report_header: fix formatting ** Limit use of file descriptors by message capturing eom functions ** fix implementation of `restex' instruction. ** fix inconsistencies in message capturing code. Version 5.0, 2008-12-26 * Requires Mailutils 2.0 or newer. * Incompatible changes ** body handler The type of first parameter ($1) is now generic pointer, not a string. It can be converted to a usual string using the `body_string' function (see below). * Changes to MFL ** Function aliases Functions can have several names. Alternative function names, or aliases, are introduced by `alias' statement, placed between the function declaration and return type declaration, e.g.: func foo() alias bar alias baz do ... done Any number of aliases is allowed. ** Functions with variable number of arguments Ellipsis as the last argument in a list of formal arguments to a function indicates that this function takes a variable number of arguments. For example: func foo (string a ; string b, ...) Actual arguments passed in a list of variable arguments have string data type. A special construct is provided to access these arguments: $(expr) where expr is any valid MFL expression, evaluating to a number. This construct returns exprth argument from the variable argument list. ** getopt and vaptr New function `getopt' is provided. New operator `vaptr' is available for converting function argument list to the second argument of `getopt'. * New MFL functions ** rcpt_add(string rcpt) Adds a recipient to the message envelope. ** rcpt_delete(string rcpt) Removes the given recipient from the envelope. ** header_add(string hdr, string value [, number index]) Adds a header "hdr: value" to the message. This function differs from the `add' action in two regards: 1. It allows to construct the header name, whereas `add' requires it to be a literal string; 2. Optional `index' argument specifies the location in the header list where this header should be inserted. ** header_insert(string hdr, string value, number index) Equivalent to header_add(hdr, value, index). ** header_delete(string hdr [, number index]) Delete header `hdr' from the envelope. This function differs from the `delete' action in two regards: 1. It allows to construct the header name, whereas `delete' requires it to be a literal string; 2. Optional `index' argument allows to select a particular header instance to delete. ** header_replace(string hdr, string value [, number index]) Replace the value of the header `hdr' with `value'. Optional argument `index' specifies the number of `hdr' instance to replace (1-based). This function differs from the `replace' action in two regards: 1. It allows to construct the header name, whereas `replace' requires it to be a literal string; 2. Optional `index' argument allows to select a particular header instance to replace. ** quarantine(string reason) Quarantines the message using the given reason. ** replbody(string text) Replaces the body of the message with the given `text' ** body_string string body_string(pointer text, number length) This function converts a generic pointer to an MFL string. It can be used only in `body' handler, e.g. the following fragment collects the entire message body in a variable and then uses it in the `eom' handler: string text prog body do set text %text body_string($1,$2) done prog eom do echo %text done ** Macro Access Two functions are provided for accessing Sendmail macros. - string getmacro(string name) Return the value of Sendmail macro `name'. It is entirely equivalent to `${name}' construct, except that allows to construct the name programmatically, while the `${name}' construct requires it to be a literal string. - number macro_defined(string name) Return true if Sendmail macro `name' is defined. ** String functions - string replstr(string s, number n) - string sa_format_score(number score, number prec) - string sa_format_report_header(string report) ** Character type functions - number isalnum(string s) - number isalpha(string s) - number isascii(string s) - number isblank(string s) - number iscntrl(string s) - number isdigit(string s) - number isgraph(string s) - number islower(string s) - number isprint(string s) - number ispunct(string s) - number isspace(string s) - number isupper(string s) - number isxdigit(string s) ** Mailbox and message manipulation - number current_message() - void mailbox_append_message(number mbx, number msg) - void mailbox_close(number mbx) - number mailbox_get_message(number mbx, number nmsg) - number mailbox_messages_count(number nmbx) - number mailbox_open(string url[, string mode, string perms]) - number message_body_lines(number nmsg) - void message_body_rewind(number nmsg) - number message_body_size(number nmsg) - void message_close(number nmsg) - number message_count_parts(number nmsg) - string message_find_header(number nmsg, string header[, number idx]) - number message_get_part(number nmsg, number idx) - bool message_has_header(number msg, string header[, number idx]) - number message_header_count(number nmsg) - number message_header_lines(number nmsg) - number message_header_size(number nmsg) - bool message_is_multipart(number nmsg) - number message_lines(number nmsg) - string message_read_body_line(number nmsg) - string message_read_line(number nmsg) - void message_rewind(number nmsg) - number message_size(number nmsg) ** System functions: umask ** string verp_extract_user(string email, string domain) If `email' is a valid VERP-style email address for `domain', this function returns the user name, corresponding to that email. Otherwise, it returns an empty string. ** string sa_format_report_header(string text) Format a SpamAssassin report `text' in order to include it in a RFC 822 header. This function selects the score listing from `text', and prefixes each line with "* ". ** string sa_format_score(number score, number prec) Format `score' as a floating-point number with `prec' decimal digits. This function is convenient for formatting SpamAssassin scores for use in message headers and textual reports. It is defined in module sa.mf. @smallexample sa_format_score(5000, 3) @result{} "5.000" @end smallexample ** Sequential access to DBM - number dbfirst (string name) - number dbnext (number dn) - string dbkey (number dn) - string dbvalue (number dn) * Changes to MFL functions ** sa This function takes an optional third argument, which controls what kind of data is returned in the sa_keywords variable. If the third argument is not supplied or is zero, the sa_keywords variable contains a string of comma-separated SpamAssassin keywords identifying this message. This is compatible with previous versions of Mailfromd. Otherwise, if the third argument is not 0, the value of sa_keywords is a @dfn{spam report} message. It is a multi-line textual message, containing detailed description of spam scores in a tabular form. The function `sa_format_report_header' can be used to format it for use in a message header. ** substring Third argument (`end') can be a negative number, meaning offset from the end of the string. Thus: substring("mailfrom",4,-1) => "from" substring("mailfrom",4,-2) => "fro" ** index and rindex Both functions take an optional third argument indicating where to start searching, e.g.: index("string of rings", "ring") => 2 index("string of rings", "ring", 3) => 10 * New global variables. ** last_poll_greeting Keeps the initial SMTP reply from the last poll. ** last_poll_helo Keeps the reply to HELO (EHLO) command from the last poll. * New operation mode. When given `--run' command line option, mailfromd looks for a function named `main' and invokes it, passing the rest of command line as its arguments. The function `main' must be declared as: func main(...) returns number The return value from this function is used as the exit code. Command line arguments may be processed using `getopt' builtin function. * Two stack growth policies. The stack can be grown either by fixed size blocks, or exponentially. In first case, the size of the block is divisible by expansion chunk, which is 4096 words by default. The expansion chunk size can be specified as a second argument to pragma stacksize, e.g.: #pragma stacksize 10240 8192 Exponential stack growth means that each time the evaluator needs to have more stack space, it expands the stack to twice the size it had before. This growth policy is selected if the word `twice' appears as the second argument to pragma stacksize: #pragma stacksize 10240 twice All numbers may be suffixed with usual size suffixes (k, m, g, t, meaning kilowords, megawords, etc). Additionally, maximum stack size may be given as third argument: #pragma stacksize 10m twice 200m * Milter `ports' can be specified using URL notation, e.g.: inet://127.0.0.1:1234 instead of inet:1234@127.0.0.1 unix:///var/run/socket instead of unix:/var/run/socket * ACLs Access to Milter can be controlled using access control lists. * Removed depecated features: 1. Command line options --ehlo, --postmaster-email, and --mailfrom * mtasim New command line options `--user' and `--group' allow to specify user name and a list of additional groups when the program is run with `root' privileges. * New programs: ** smap. Smap is a general-purpose remote map for MeTA1. ** pmult Pmult is a pmilter to milter multiplexer. Pmilter is a new filter protocol implemented in MeTA1, an MTA which should in future replace Sendmail. Pmult listens for Pmilter commands from the server, translates them into equivalent Milter commands and passes the translated requests to a preconfigured set of Milter filters. When the filters reply, the reverse operation is performed: Milter responses are translated into their Pmilter equivalents and are sent back to the server. Pmult allows to use existing milters with MeTA1 without any modifications. ** pies Pies is a process invocation and execution supervisor. * GNU Emacs MFL Mode is improved * Bugfixes ** Fix program evaluator bug that manifested itself on machines where sizeof(unsigned long) > sizeof(usnigned). ** Fix stack reallocation. ** Header modification functions affected subsequent messages in the same SMTP session. This is fixed. Version 4.4, 2008-03-10 * The --domain option is withdrawn. This option was declared as deprecated in version 4.0. Now it is withdrawn and its short version (-D) is reused for another purpose (see -D option, below). * New command line options -D and -U Both options work like their m4 counterparts: the `-D' command line option defines a preprocessor symbol, the `-U' option undefines it. * Constant vs. variable shadowing In previous versions of mailfromd name clashes between constants and variables went unnoticed, which sometimes lead to hard-to-diagnose errors. This bug is fixed in this version. If a constant is defined which has the same name as a previously defined variable (the constant "shadows" the variable), the compiler prints the following diagnostic message: :: Warning: Constant name `name' clashes with a variable name :: Warning: This is the location of the previous definition A similar diagnostics is issued if a variable is defined whose name coincides with a previously defined constant (the variable "shadows" the constant). In any case, the %NAME notation refers to the last defined symbol, be it variable or constant. If a variable shadows a constant, the scope of the shadowing depends on the storage class of the variable. For automatic variables and function parameters, it ends with the final `done' closing the function. For global variables, it lasts up to the end of input. * Exception names. To minimize chances of name clashes, all symbolic exception codes has been renamed by prefixing them with the `e_', thus, e.g. `divzero' became `e_divzero', etc. The `ioerr' exception code is renamed to `e_io'. For consistency, the following most often used codes are available without the `e_' previx: success, not_found, failure, temp_failure. The use of old exception codes is still possible by defining a preprocessor symbol OLD_EXCEPTION_CODES, for example: mailfromd -DOLD_EXCEPTION_CODES * match_dnsbl and match_rhsbl Both functions malfunctioned in versions from 4.0 up to 4.3.1 due to a name clash between the exception code `range' and their third argument. This is fixed. Additionally, previous versions of match_dnsbl silently ignored invalid first argument. Now, the e_invip exception is signalled in this case. Version 4.3.1, 2008-03-01 * Fix program evaluator bug that manifested itself on machines where sizeof(unsigned long) > sizeof(usnigned). Version 4.3, 2008-02-10 * write built-in The `write' built-in function takes an optional third argument, specifying the number of bytes to write. This form is particualry useful in `body' handler for writing $1, because it is not null- terminated, e.g.: prog body do write(fd, $1, $2) ... done * sieve New built-in function `sieve' provides an interface to Sieve interpreter. * Restore previous meaning of --enable-syslog-async. Unless this option is given to ./configure, asynchronous syslog implementation will not be compiled. * Bugfixes: ** Fix compilation on Sun. ** Fix header deletion (delete action). ** Variable shadowing was broken if a rehash happened between vardcl and forget_autos. Version 4.2, 2007-10-23 * Licensed under the GPLv3 * New command line options --syslog-async and --no-syslog-async These options allow to select the implementation of syslog to use at run time. * RFC 2821 compatibility The sender verification engine used to send whitespace character between `MAIL FROM:' and `RCPT TO:' commands and their argument. This violated RFC 2821, which requires the argument to follow the command without any intermediate whitespace. It is fixed in this release. * Sample threshold for `rate' function. The `rate' function takes an optional third argument allowing to specify the minimum number of received mails needed to obtain the sending rate value. The default is 2, which is probably too conservative. The following example raises it to 10: if rate($f "-" ${client_addr}, interval("1 hour"), 10) > %maxrate ... fi * mtasim is improved In particular, it accepts MAIL FROM: and RCPT TO: without extra space after the colon, as required by RFC. The milter interface is also improved. * The `resolve' function ignores TXT records. In other words, resolve and primitive_resolve are guaranteed to return either an A or a PTR record. Version 4.1, 2007-06-11 * National Language Support. The program includes National Language Support. Polish and Ukrainian translations are available. * NLS Functions NLS functions allow to localize your filter scripts for a particular language. The following functions are implemented: bindtextdomain, dgettext, dngettext, textdomain, gettext, ngettext. In addition, macros _() and N_() are also provided. * GNU Emacs MFL Mode This release comes with the file `mfl-mode.el', providing MFL mode for GNU Emacs. This mode facilitates editing MFL source files. By default, the new mode is installed whenever configure determines the presense of GNU Emacs on your machine. See the documentation, node `Using MFL Mode' for the detailed discussion of this mode including customization information. * Input files are preprocessed before compilation. The default preprocessor is M4, but this can be changed (or disabled) at configuration time (see `DEFAULT_PREPROCESSOR' variable and `--with-preprocessor' command line option). * New atom $# Returns the number of the arguments passed to the function. * New atom @parm Returns the position of parameter `parm' in the function argument list. It can be used, for example, to check whether an optional argument value is passed to the function, e.g.: func foo(string x; number n) do if $# > @n /* `n' is passed */ ... The default preprocessor setup script provides a macro `define' designed to be used for this purpose: func foo(string x; number n) do if defined(n) /* `n' is passed */ ... * sprintf The built-in function `sprintf' is available with the same semantics as its C counterpart. * Discontinued support for deprecated features: ** `&code' form to specify an exception code is discontinued. ** pragma options retry, io-retry, and connect-retry * Bugfixes: ** Built-in listen ignored optional second argument. ** Debug specification incorrectly gave preference to the global level over the source level. This is fixed, so that `--debug=40,dns=10' means level 10 for calls from `dns.c', and level 40 for all the rest. Version 4.0, 2007-05-12 Note for users of 3.1.x: see also the notes for previous alpha (3.1.9x) versions. * SIGHUP handling SIGHUP instructs `mailfromd' to restart itself. * rc.mailfromd reload The `reload' option given to `rc.mailfromd' instructs it to send SIGHUP to the running instance of the program. * mtasim The `mtasim' utility is an MTA simulator for testing and debugging mailfromd filter scripts. It supports stdio (-bs) and daemon (-bd) modes, has GNU readline support and `expect' facility, which makes it useful in automated test cases. See the documentation, chapter `mtasim'. * `begin'/`end' handlers The `begin' and `end' special handlers may be used to supply startup and cleanup code for the filter program. The `begin' special handler is executed once for each SMTP session, after the connection has been established but before the first milter handler has been called. Similarly, an `end' handler is executed exactly once, after the connection has been closed. Neither of handlers takes any arguments. See the documentation, section `begin/end'. * Cache control Use function `db_set_active' to enable or disable given cache database. E.g. # Disable DNS cache: db_set_active("dns", 0) # Enable it back again: db_set_active("dns", 1) Similarly, the function `db_get_active' returns a number indicating whether the given cache database is used or not. ** Bugfixes * Fix a long-standing bug in parsing the --predict option argument. Is there anybody using this option at all? Version 3.1.91, 2007-04-23 * Non-blocking syslog This version is shipped with non-blocking syslog implementation by Simon Kelley. You may wish to enable it if you noticed that the number of mailfromd processes grows uncontrollably and the processes are hung for prolonged amounts of time. Usually this indicates that the daemon blocks in syslog() calls. Read the description of `--enable-syslog-async' option in chapter `Building' for the detailed discussion of this (try `info -f doc/mailfromd.info --index-search syslog-async'). * SPF support The function check_host() tests the SPF record for the given identity/host name. The syntax is: number check_host(string ip, string domain, string sender, string helo) See the documentation, node `SPF functions' for the detailed description of this function and the related functions and variables. * next and pass Use `pass' instead of `next'. The `next' keyword has changed its semantics: it is now used to resume the next iteration of the enclosing loop statement (see below). For compatibility with the previous versions, its use outside of a loop statement is still allowed, but a warning is issued. You are encouraged to replace all occurrences of `next' in your configuration scripts with `pass'. * Loop The loop statement is implemented. Its syntax is: loop [name] [for ,] [while ,] [] do ... done [while ] See the documentation, section `Loop Statements'. * break and next The `break' statement exits from the enclosing loop. The `next' statement resumes the next iteration of the enclosing loop statement. Both statements take an optional argument specifying the identifier (name) of the loop to break from (or continue), this allows to build complex iterations consisting of nested loops. For example, in this code (line numbers added for clarity): 1 loop outer for set i 1, while %i < %N 2 do 3 ... 4 loop for set j 1, while %j < %i 5 do 6 if foo(%j) 7 break outer 8 fi 9 done 10 done 11 accept if the call to `foo' in line 6 returns true, the control is immediately passed to `accept' in line 11. * Resizable stack The runtime stack of the MFL grows automatically as the need arises. Thus, `#pragma stacksize' sets the initial size of the stack, and the `Out of stack space' error, which was common in the previous versions, now can occur only if there is no more virtual memory left. Whenever the stack gets expanded, mailfromd issues a warning message to the logs, notifying of the new stack size, e.g.: warning: stack segment expanded, new size=8192 You can use these messages to adjust your stack size configuration settings. * Runtime stack traces New command line option --stack-trace enables dumping stack traces on runtime errors. This might help localize the source of the error. The trace looks like: mailfromd: RUNTIME ERROR near ../mflib/match_cidr.mf:30: invalid CIDR (boo%) mailfromd: Stack trace: mailfromd: 0077: test.mf:7: match_cidr mailfromd: 0096: test.mf:13: bar mailfromd: 0110: test.mf:18: foo mailfromd: Stack trace finishes mailfromd: Execution of the configuration program was not finished Each trace line describes one stack frame, the lines appear in the order of most recently called to least recently called. Each frame consists of: 1. Value of program counter at the time of its execution 2. Source code location, if available 3. Name of the function called The same output can be obtained by calling function stack_trace() in your filter program. See the documentation, section `Runtime Errors', for the detailed description and examples. * connect handler Connect handler is implemented. * envfrom and envrcpt Both handlers take an additional second argument, containing the rest of the SMTP command line. * New functions - string db_name(string fmt) Return full file name of the database file corresponding to format `fmt' - number db_expire_interval(string fmt) Return the expiration period for db format `fmt' - string getmx(string domain [, number resolve]) Returns a whitespace-separated list of MX names (if `resolve' is not given or is 0) or MX IP addresses (if `resolve'==1) for `domain'. If `domain' has no MX records, empty string is returned. If the DNS query fails, `getmx' raises an appropriate exception. This interface differs from that of version 3.1.4 in that the calls to getmx(domain) and getmx(domain,1) can return different number of entries (see the docs for an example). * #pragma regex stack The `#pragma regex' statement can keep a stack of regex flags. The stack is maintained using `push' and `pop' commands. The statement #pragma regex push [options] saves current regex flags on stack and then optionally modifies them as requested by options. The statement #pragma regex pop [options] does the opposite: restores the current regex flags from the top of stack and applies [options] to it. This statement is useful in include files to avoid disturbing user regex settings. E.g.: #pragma regex push +extended +icase . . . #pragma regex pop * Optional arguments in user-defined functions User-defined functions can take optional arguments. In a declaration, optional arguments are separated from the mandatory ones by a semicolon. For example: func foo(number a, number b; string c) This function is declared with two mandatory arguments (a and b), and an optional one (c). Subsequently it can be invoked either as foo(x, y, z) or foo(x, y) When invoking such functions, any missing arguments are replaced with default values: - 0 for numeric arguments - "" for string arguments Thus, continuing our previous example, the invocation `foo(x, y)' is equivalent to `foo(x, y, "")'. * New statement #include_once This statement works exactly like `#include' except that it keeps track of the included files. If the requested file has already been included, `#include_once' returns silently, while `#include' issues an error message. * New statement #require Requires use of the named module, e.g.: #require dns See the documentation, section `Modules', for the description of MFL module system. * Internet address manipulation functions - number ntohl (number N) - number htonl (number N) - number ntohs (number N) - number htons (number N) - number inet_aton (string S) - string inet_ntoa (number N) - number len_to_netmask (number N) - number netmask_to_len (number N) * DNS functions DNS functions are reimplemented in two layers: 1. Primitive calls: - string primitive_hostname (string IP) - string primitive_resolve (string S [, string DOMAIN]) - number primitive_hasmx (string DOMAIN) - number primitive_ismx (string DOMAIN, string IP) These functions throw an exception if the requested lookup operation fails. 2. Traditional calls: - string hostname (string IP) - string resolve (string S [, string DOMAIN]) - number hasmx (string DOMAIN) - number ismx (string DOMAIN, string IP) These are implemented in MFL and work exactly as their predecessors in 3.1.x branch. To use the traditional calls, add the following statement at the beginning of your script file: #require dns (see the documentatio, section `Modules' for the description of #require statement) * Function `match_cidr' This function has been reimplemented in MFL. To use it, add `#require match_cidr' at the top of your script source (see the documentatio, section `Modules' for the description of #require statement) * Catch arguments Catch takes two positional arguments: $1 gives the exception code, $2 is the diagnostic string, explaining what happened in detail. * New statement `throw' The `throw' statement throws the given exception. For example: throw invcidr "invalid CIDR (%cidr)" * New command line option -v (--variable) allows to alter initial value of a global variable, e.g: mailfromd -v ehlo_domain=mydomain.org The old way of assigning values to the globals from the command line (by prefixing an assignment with a percent sign) is discontinued. * Pragmatic options `mailfrom' and `ehlo' Both options create ambiguities in the language and are therefore deprecated. They are still understood but a deprecation warning is issued when the parser sees them. To update your scripts: Change #pragma option mailfrom @var{value} to set mailfrom_address @var{value} Change #pragma option ehlo @var{value} to set ehlo_domain @var{value} * Code generation redone to decrease memory requirements and make compiled code self-sufficient. * New statement `const' defines a named constant, e.g.: const n 10 const s "String" if $f != s ... fi In program text, constants are referred to by their name. In strings, they are referred to using variable syntax (e.g. "%s"). * It is allowed to initialize variables in declarations. For example, instead of number x set x 1 you can write number x 1 * Built-in macro __statedir__ New built-in macro `__statedir__' expands to the name of the default program state directory. * Changing state directory at run-time It is possible using `--state-directory' command line option or `#pragma option state-directory' statement. * Bugfixes ** Fix incorrect packet length calculation in connect Milter handler. The bug manifested itself with the following log messages: - In mailfromd log: MailfromFilter: shan_connect: wrong length - In Sendmail log: milter_read(mailfrom): cmd read returned 0, expecting 5 Milter(mailfrom): to error state ** Fix coredumps on printing void returns with --dump-tree ** Fix coredumps on optimizing conditionals like if 0 do_something fi ** Fix function context checks. Previous versions bailed out on using context-limited built-ins (like `sa' and `clamav') in functions. It is fixed. The context limit of the built-in propagates to the function it is used in, that is defining func sa_wrapper(string url) do if sa(%url, 3) discard fi done makes function `sa_wrapper' limited for use in `prog eom' only. ** Fix passing of string handler arguments between handlers, as in prog helo do set x $1 done prog envfrom do if %x = "dom.ain" ... (from 3.1.3) ** If, during sender verification, the remote server replies with 4xx to MAIL FROM command, do not try next sender address, but tempfail immediately. Version 3.1.90, 2006-12-13 * `==' can be used as well as `=' to test for equality This is a bit of syntactic sugar for seasoned C programmers, who seem to type == instinctively (like the author does). * resolve takes an optional second argument The argument specifies the domain. For example, the following calls are equivalent: resolve("puszcza", "gnu.org.ua") = resolve("puszcza.gnu.org.ua") resolve("22.0.120.213", "in-addr.arpa") = hostname("213.130.0.22") * listens takes an optional second argument The second argument specifies the port to use instead of the default 25. * New functions - message_header_decode(string TEXT, [string CS]) The TEXT must be encoded as per RFC 2047. The function decodes it and returns the resulting string. Optional argument CS specifies the character set for the output string. Default is UTF-8. - message_header_encode(string TEXT, [string ENC, string CS]) Encodes TEXT according to RFC 2047. Optional arguments: ARG Meaning Default value ------+------------------+------------------- ENC Encoding quoted-printable CS Character set UTF-8 Valid values for ENC are quoted-printable, Q, base64, B - unfold (string TEXT) Unfold TEXT as defined in RFC 2822, section 2.2.3. Return unfolded string. * Default logging facility is LOG_MAIL * While checking sender validity, issue RSET if the previous MAIL FROM returned 4xx. Bugfixes: * `Make install' creates state directory if it does not exist * `clamav' function could cause coredumps when using socket ports. * `resolve' function altered its argument if it was a CNAME. * A typo in gram.y prevented some correct `matches' conditions from being compiled. In particular, strip_domain_part.mf triggered this error. Version 3.1, 2006-12-07 * Incompatible changes. For detailed instructions on how to upgrade from version 3.0, please see http://mailfromd.software.gnu.org.ua/upgrade.html 1. The package refuses to compile without DBM 2. The command line option --config-file (-c) is no longer supported. To use an alternative filter script, give its name as an argument in the command line, e.g. mailfromd my-script.rc For backward compatibility, the invocation `mailfromd --config-file my-script.rc' still works but produces a warning message. The semantics of `-c' will change in the next release. 3. The function `dbmap' takes an optional third argument. If it is 1, then the length of the lookup key will include the terminating null character. In previous versions dbmap always counted the terminating null character in the key length. So, you should add the non-zero third argument to the calls to dbmap to preserve their functionality. 4. Variables, implicitly declared within a function, are automatic. Previous versions created them as global. * Language changes ** Hex and octal constants Usual C notation (0xNNN for hex and 0NNN for octal) is accepted. ** Bitwise operators: &, |, ^ (logical and, or, xor) and ~ (twos-complement) ** Search path for include files The `#include' statement handles its argument the same way C preprocessor does: the argument is searched in the include file path, if enclosed in angle brackets (<>), and in the current working directory first and then in the include file path, if it is enclosed in double quotes. The default include file path is /usr/share/mailfromd/include:/usr/local/share/mailfromd/include plus $prefix/share/mailfromd/include if $prefix is not `/usr' or `/usr/local'. The command line option -I (--include) adds the named directory in front of the default include path. ** Code optimization Parse tree is optimized before code generation. This can be controlled using -Olevel option, where `level' is the optimization level. Currently implemented levels are 0 (no optimization) and 1 (full optimization), which is the default. ** All variables are now strongly typed. The declaration of the variable has the form: `TYPE NAME', where TYPE is one of `string' or `number', and NAME is the variable name. For compatibility with the previous versions, the declaration is optional. If it is absent, the first assignment to the variable defines its type. Subsequent assignments will implicitly cast the value being assigned to the type of the variable. ** New style of function declarations. Named parameters. Functions should be defined as: func NAME (PARAM-LIST) returns TYPE where TYPE is as described in the previous paragraph and PARAM-LIST is a comma-separated list of parameter declarations in the form TYPE NAME. Consequently, instead of the positional notation, parameters can be referenced by their names: func sum(number a, number b) returns number do return %a + %b done Within the function body, the named parameters can be handled the same way as other variables, in particular they can be assigned new values using `set' instruction. For compatibility with the previous version, old type of function declarations is supported as well. ** Automatic variables Automatic variables are defined within a function or handler. Their scope of visibility ends with the terminating `done' statement. Automatic variables are declared and referenced the same way as global ones. To declare an automatic variable, use `TYPE NAME' notation. Variable declarations can be intermixed with executable statements. The following example defines two automatic variables for the function `foo': func foo() do number a string s ... If a variable is declared implicitly within a function or handler, it is declared automatic. See the documentation for the detailed description and examples. ** New functions: *** I/O functions: open, close, write, getline See http://mailfromd.software.gnu.org.ua/manual/bi/io.html *** Time functions: time, strftime See http://mailfromd.software.gnu.org.ua/manual/bi/system.html *** System functions: system See http://mailfromd.software.gnu.org.ua/manual/bi/system.html *** DBM functions: dbput, dbdel See http://mailfromd.software.gnu.org.ua/manual/bi/db.html *** String functions: substr, index, rindex See http://mailfromd.software.gnu.org.ua/manual/bi/string.html *** Debugging functions: debug, cancel_debug, program_trace, cancel_program_trace See http://mailfromd.software.gnu.org.ua/manual/bi/debug.html *** Mail sending functions: send_mail, send_text, send_dsn See http://mailfromd.software.gnu.org.ua/manual/bi/mail.html ** The legacy function numrcpt() has been withdrawn Use %rcpt_count instead. ** Built-in macros Built-in macros have names beginning and ending with double underscore. As their name implies, the macros are expanded to constant values. The following built-in macros are defined: 1. __file__ expands to the name of the current source file 2. __line__ expands to the number of line in the current source file 3. __function__ expands to the name of the current lexical context, i.e. the function or handler name. 4. __package__ expands to the string containing package name ("mailfromd") 5. __version__ expands to the textual representation of the program version (e.g. "3.0.90") 6. __major__ expands to the major version number 7. __minor__ expands to the minor version number 8. __patch__ expands to the version patch level, or 0 if it is not defined. Built-in macros can be used in variable context. For example, to use them within a string or here-document, prepend them with % as if they were regular variables, e.g.: echo "%__file__:%__line__: Checkpoint" * The envfrom and envrcpt handlers print entire argument array in the debugging output. * New DNS caching scheme. All DNS lookups are cached on global basis, as opposed to the per-session basis in previous versions. The cache is stored in the DBM database `dns'. It can be listed and otherwise operated upon using usual mailfromd commands. If a lookup gives a positive result, the TTL from the DNS record is used as the record expiration interval. For negative lookups, the default interval of 3600 seconds is used. It can be altered by the following pragmatic comment: #pragma database dns negative-expire-interval N * New command line option --xref Produces a cross-reference listing of global variables. * Fuller SMTP timeout control In order to more fully control SMTP transactions, new timeout value is introduced: initial-response-timeout. This is the maximum time to wait for the remote to issue the initial SMTP response. This value is especially useful for dealing with the servers that impose a delay before the initial reply (most notably "CommuniGate Pro" ones"). The default value is 30 seconds which should be enough for most normal servers. See the documentation, node "SMTP Timeouts" for the detailed discussion. * No more `retry' options. The `retry' options and pragmas have been removed. The new timeout control scheme warrants that the polling will take at most the given interval of time. In particular, that affects: ** Command line option `--retry' ** Pragma options io-retry and connect-retry * Bugfixes ** Switch statements without the default branch produced incorrect code (the very first branch was used as the default one). This is fixed. ** Fix handling of escape sequences at the beginning of a string and before the beginning of an interpreted sequence within the string. ** Fix the declarations of the built-in functions `toupper' and `tolower'. ** Fix storing the macro values obtained from Sendmail ** Collect zombie subprocesses as soon as possible ** Fix arithmetical expression syntax in rc.mailfromd ** Fix multiple from address handling ** Fix race condition when using GDBM Version 3.0, 2006-11-05 * The mailfromd binary is now installed in ${prefix}/sbin. Please, update your scripts. You are encouraged to update the startup script (run `cp etc/rc.mailfromd /wherever-your-startup-lives'), since the new version contains lots of enhancements (see below). * The package no longer uses libmilter. * Several `from' addresses can be specified both with polling functions and in `#pragma option mailfrom' statement. In this case the probing will try each address until either the remote party accepts it or the list of addresses is exhausted, whichever happens first. This can help if a remote host is picky about sender addresses. * After discussions with Jan, the final part of the standard poll method has been redone. Now the last-resort poll (i.e. querying the domain part of the sender email, treated as an MX) is done only if the domain has no MX records. * New option --dump-macros shows all Sendmail macros used in the configuration file, by milter states. It is useful to create Sendmail `Milter.macros.*' (confMILTER_MACROS_*) statements. * rc.mailfromd stript two new options: - rc.mailfromd configtest [FILE] Checks configuration file syntax. If FILE is not given, the default one is assumed. - rc.mailfromd macros [-c] [FILE] Generate milter export statements for Sendmail configuration files. Optional FILE specifies alternative mailfromd configuration file. By default, `.mc' statements are generated. Specifying `-c' option instructs the script to create `.cf' statements instead. * New pragmatic options `connect-retry' and `connect-timeout' set retry count and timeout values for initial connections. The corresponding values for I/O operations are set using `io-retry' and `io-timeout' options. The pragmatic options `retry' and `timeout' are retained for backward compatibility. They are synonymous to their `io-' counterparts. The default values are: #pragma option connect-retry 1 #pragma option connect-timeout 30 #pragma option io-retry 3 #pragma option io-timeout 3 * New function `sa' checks the message for spam via SpamAssasin spamd interface. It supplies additional data via the global variables sa_score, sa_threshold, and sa_keywords. * New function `clamav' checks the message for viruses via ClamAV daemon interface. Additional data (the virus name, if found) is stored in the global variable clamav_virus_name. * New function `ismx' returns true if the IP address or hostname given by its second argument is one of the MX records of the domain name given by the first argument. * New variables `last_poll_host', `last_poll_send', `last_poll_recv' contain the host name of the lastly polled host, the command sent and the first line of the reply received. The variables are set by polling functions. * New variable `cache_used' is set to true if cache data were used instead of polling, and to false otherwise. The variable is set by stdpoll and strictpoll built-in functions (and by `on poll' statement, accordingly). * New string functions: `length' and `substring' * Here document syntax expanded. ** Removing leading whitespace Inserting a single space between the dash and the terminator word in the beginning of the here-document construct, as in: <<- WORD ... WORD instructs parser to remove leading white space characters from each line of the document body, including terminating WORD. ** Expansion of macros and variables Variables and sendmail macros are expanded when used within a double-quoted string or a here-document body. The variable expansion and backslash interpretation is suppressed by quoting the WORD, e.g.: <<\WORD ... WORD or <<'WORD' ... WORD ** Numeric escape sequences Two new kinds of escape sequences are supported: \0ooo, where o is any octal digit \xhh, where h is any hex digit ** Back-references. References to the parenthesized subexpressions of the previous regular expression are expanded both in the code and in double-quoted string literals. For example: if domainpart $f matches '\(.*\).com' set d \1 fi * Bugfixes ** Fix berkeley 4.x support ** Fix expiration of the greylist and rate databases. ** Fix returning multiline replies. The last line of the reply was not taken into account unless it ended with a newline. ** Fix type casting of arguments to user-defined functions ** Fix argument passing in function calls generated for `on poll' statements. Version 2.0 * Program requires Mailutils version 1.0 or newer * Support for old DBM and NDBM has been withdrawn. * Added support for Berkeley DB versions 3.x and 4.x * INCOMPATIBLE CHANGES ** To use version 1.x configuration files, the following changes should be applied to them: 1. The entire code section should be enclosed in the following statement: prog envfrom do ... done See the section `Handler declarations' below for the detailed description. 2. Convert any `rate' statements to function calls, e.g.: if rate $f 180 / minute should be rewritten as if rate($f, interval(minute)) > 180 See the section `rate(key, interval)' below for the detailed description. See also section "Special test functions" in the mailfromd documentation. ** Format of cache and rates database has changed: the key field now includes the trailing nul character, which is also reflected in its length. This allows for empty (zero-length) keys. To convert existing databases to the new format, run mailfromd --compact --all after compiling the package. ** The database management options (--list,--delete,--expire) do not take any argument. To specify that the option refers to the rate database, use --format=rate option. * Language features ** Compiled code Configuration file handling completely rewritten. The file is parsed into a pseudo-code program, which is executed considerably faster than the parse tree in 1.x branch. ** Sendmail macros Multiletter Sendmail macros can be used with and without surrounding curly braces, i.e. ${rcpt_count} and $rcpt_count are both valid. ** File inclusion Configuration file can include other files. The syntax is: #include "FILENAME" Inclusion depth is not limited. ** Adjacent expressions concatenate: $f => "gray@gnu.org.ua" ${client_addr} => "127.0.0.1" $f "-" $client_addr => "gray@gnu.org.ua-127.0.0.1" ** Arithmetical expressions The four usual arithmetical expressions are now supported. ** Comparisons In addition to equal (=) and not equal (!=) the following comparison operators are supported: <, <=, >, >=. Their precedence and associativity is the same as in C. ** Type casting The rules for implicit type casts are: 1. Both arguments to an arithmetical operation are cast to numeric types. 2. All arguments of the concatenation operation are cast to string type. 3. Both arguments to `match' or `fnmatch' function are cast to string type. 4. The argument of the unary negation (arithmetical or boolean) is cast to numeric type. 5. Otherwise, if the arguments to a binary operation have different types, the right-hand side argument is cast to the type of the left-hand side argument. There is no special syntactic sugar for explicit type casting. To cast an expression to string, concatenate an empty string to it: %var "" To cast an expression to numeric, add it to zero: %var + 0 ** Single-quoted strings In addition to double-quoted strings, single-quoted ones are also supported. The single-quoted strings are not subject to backslash expansion, thus they are particularly useful for writing regular expressions: if $f matches '.*\.com' ** Splitting strings between several lines. Double-quoted strings can be split over several lines, by placing a backslash immediately before the newline. For example: "A very\ long string" produces "A very long string". ** Here document syntax and multiline sendmail replies Mailfromd supports "here document" syntax: <<[-]WORD ... WORD Optional '-' instructs parser to remove leading tabulation characters from each line of the document body, including terminating WORD. This allows for here documents to follow normal program indentation. The backslash expansion is performed on the contents of the here document, unless WORD is quoted (i.e. either 'WORD' or \WORD). This is particularly useful in providing multiline Sendmail replies: reject 550 5.0.0 <<-EOT This service is not available now. Please, refer to http://some.site for more information on the subject. EOT ** Handler declarations The program consists of a set of handler declarations. Each handler is defined as prog STATE do ... done where `...' represents the actual code, and STATE is the milter state this handler is defined for. Recognized milter states are: helo, envfrom, envrcpt, header, eoh, body, eom. ** Positional arguments ($N notation) Handlers can take several positional arguments, which can be dereferenced in the program using $N notation, where N is a decimal number. The semantics of the positional arguments depends on the state for which the handler is designed. For example: prog helo do /* For helo, $1 means helo domain */ if $1 matches "gnu.org.ua" ... fi done ** Internal variables (% notation) Mailfromd variables can be defined using `set' statement: set VAR expr where VAR is the variable name. The variables are dereferenced using %VAR notation. Their values are retained between handlers, for example: prog helo do set helo_domain $1 done prog envfrom do if $f = "gray" and %helo_domain matches "gnu.org.ua" ... fi done The `set' statement can be used both inside and outside of program handlers or functions. When used outside them, it declares a global variable and assigns it an initial value. The variable will be initialized to that value at the beginning of each message. ** Built-in function syntax Built-in functions of more than 1 argument are invoked using the usual C-like syntax, e.g.: name(arg1, arg2, arg3) Built-in functions of one argument can be invoked with or without parentheses: name arg name(arg) ** User defined functions User defined functions are declared using the following syntax: func NAME ( TYPELIST ) returns TYPE do ... done where NAME is the name of the function, TYPELIST is a comma-separated list of parameter types, and TYPE is the return type of the function. Types are designated by a single letter: 'n' denotes the numeric type, 's' denotes the string type. Within the function body, the arguments are denoted using positional argument notation (see `Positional arguments' above). The `return' statement is used to return a value from the function. For example: func sum(n,n) returns n do return $1 + $2; done ** Switch statement The mailfromd language now has switch statement. Its syntax is similar to that of C: switch EXPR do case VAL [or VAL...] : STMT . . . default: STMT done where EXPR is any valid mailfromd expression, VAL is a literal value (numeric or string), STMT is any mailfromd statement or a list of statements. For example: switch %x do case 1 or 2: echo "Accepting mail" accept default: reject done ** Catch statement The new `catch' statement can be used to handle exceptional conditions occurring within a function. An exceptional condition is signalled when the filter script program encounters a condition it is not able to handle. See the documentation, section "Exceptions" for the details. The syntax of the `catch' statement is: catch EXCEPTION-LIST do HANDLER-BODY done where EXCEPTION-LIST is the list of exception types, separated by the word `or'. Special form `catch *' catches all exceptions. The HANDLER-BODY is the list of statements comprising the handler body. ** On statement The support for `on' statement has been entirely rewritten. The statement is implemented as a wrapper around `catch'. Instead of `poll' you can specify any function call as its selector value. For example, this statement: prog envfrom do on poll $f do when success: accept when not_found or failure: reject 550 5.1.0 "Sender validity not confirmed" when temp_failure: tempfail 450 4.1.0 "Try again later" done done is actually a shortcut for: function poll_wrapper(s) returns n do catch &success or ¬_found or &failure or &temp_failure do return $1 done return stdpoll($1, %ehlo_domain, %mailfrom_address) done prog envfrom do switch poll_wrapper($f) do case &success: accept case ¬_found or &failure: reject 550 5.1.0 "Sender validity not confirmed" case &temp_failure: tempfail 450 4.1.0 "Try again later" done done (See also the description of %ehlo_domain and %mailfrom_address variables) ** MX matching The `matches' and `fnmatches' operations has been extended to allow operations on MX lists. The test if ${client_addr} mx matches '.*\.gnu\.org' yields true if the list of MXs for the IP address ${client_addr} contains a host name that matches the regular expression '.*\.gnu\.org'. The left side argument can be either a hostname or IP address or an RFC 2822 email address. In the latter case, its domain part is used to query for the MX records. The similar 'mx fnmatches' construction is also available. Both `mx' tests can throw one of the following exceptions: not_found, failure, temp_failure. ** #pragma database New pragma `database' controls various aspects of databases used by the program. The pragma has four forms: 1. Store the database DBNAME in the given FILENAME #pragma database DBNAME file FILENAME 2. Set the expiration interval for DBNAME: #pragma database DBNAME expire-interval INTERVAL INTERVAL can be any valid interval specification (see the next section). If DBNAME is "cache", two additional forms are understood: 3. Set the expiration period for positive cache records: #pragma database cache positive-expire-interval INTERVAL 4. Set the expiration period for negative cache records: #pragma database cache negative-expire-interval INTERVAL The forms 3. and 4. are equivalent to #pragma option positive-expire-interval INTERVAL and #pragma option negative-expire-interval INTERVAL correspondingly. ** Time interval specification Time intervals can be specified as an English text, e.g. "1 hour 35 minutes". The following pragmas take a time interval specification as their argument: #pragma option timeout #pragma option milter-timeout #pragma option expire-interval #pragma option negative-expire-interval #pragma option positive-expire-interval #pragma option rates-expire-interval #pragma option lock-retry-timeout * New built-in functions: ** dbmap(dbname, key) Returns true if `key' is found in the database file `dbname', E.g.: dbmap("/etc/mail/aliases.db", $f) ** domainpart(str) Returns the domain part of `str' if it is a valid email address, otherwise returns `str' itself. ** greylist(key, interval) Returns true if the `key' is found in the greylist database (controlled by `#pragma database greylist' pragma). The argument `interval' gives the greylisting interval in seconds. The function sets internal variable %greylist_seconds_left to the number of seconds left to the end of greylisting period. Sample usage: set gltime 500 if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, %gltime) if %greylist_seconds_left = %gltime tempfail 470 "You are greylisted for " %gltime " seconds" else tempfail 470 "Still greylisted for " %greylist_seconds_left "seconds" fi fi ** hasmx(host) Returns true if `host' has any MX records. The function can throw one of the following exceptions: not_found, failure, temp_failure. ** interval(string) Converts its argument, which should be a valid time interval specification, to seconds ** localpart(str) Returns the local part of `str' if it is a valid email address, otherwise returns unchanged `str'. ** match_cidr(ip, cidr) Returns true if the IP address `ip' matches the network block `cidr'. For example: match_cidr(${client_addr}, "213.130.0.0/19") Possible exceptions: invip, if the first parameter is not a valid IP, and invcidr if the second parameter is not a valid CIDR. ** stdpoll(email, domain, mailfrom) Performs standard poll for `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. Can raise one of the following exceptions: failure, temp_failure. In `on' statement context, it is synonymous to `poll' without explicit `host'. ** strictpoll(email, domain, mailfrom, host) Performs strict poll for `email' on host `host'. See the description of stdpoll for the detailed information. In `on' statement context, it is synonymous to `poll host'. ** _pollhost(ip, email, domain, mailfrom) Poll SMTP host `ip' for email address `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. In contrast to strictpoll function, this function does not use cache database and does not fall back to MX poll if the poll tempfails. The function can throw one of the following exceptions: failure, temp_failure. ** _pollmx(domain, email, domain, mailfrom) Poll MX-s of the `domain' for email address `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. In contrast to stdpoll function, _pollmx function does not use cache database and does not fall back to host poll if the poll fails. The function can throw one of the following exceptions: failure, temp_failure. ** tolower(string) Returns a copy of the string str, with all the upper-case characters in string translated to their corresponding lower-case counterparts. Non-alphabetic characters are left unchanged. ** toupper(string) Returns a copy of the string str, with all the lower-case characters in string translated to their corresponding upper-case counterparts. Non-alphabetic characters are left unchanged. ** rate(key, interval) Returns the mail sending rate for `key' per `interval'. This function replaces `rate' statement from 1.x branch. To convert old rate statements use the following algorithm: Old statement: if rate KEY LIMIT '/' EXPR New statement: if rate(KEY, interval("EXPR")) > LIMIT For example: Old statement: rate $f 180 / 1 hour 25 minutes New statement: if rate($f, interval("1 hour 25 minutes")) > 180 ** resolve(host) Returns the IP address corresponding to `host' or "0" if it cannot be resolved. ** validuser(user) Returns true if `user' is a valid local account. It uses mailutils authentication mechanisms. * Global variables ** %ehlo_domain Name of the domain used by polling functions in EHLO or HELO command. It is set by `#pragma option ehlo' directive, or via --ehlo command line option. ** %mailfrom_address Email address used by polling functions in 'MAIL FROM' command. Set by `#pragma option mailfrom' directive or via --mailfrom command line option. ** %rcpt_count The variable %rcpt_count keeps the number of recipients given so far. The variable is defined in the envrcpt state. * Database expiration The operation of `mailfromd --expire' has been completely redesigned to avoid skipping some keys when using GDBM. * Database compaction New option --compact starts "database compaction" process, which removes all expired entries and empty blocks from the database. It also converts any obsolete (not nul-terminated) keys to nul-terminated ones. During this process the original database file is locked for writing, so the running mailfromd instance is able to read entries from it, but cannot write or update it. The existed database will be replaced with the compacted version only if there were no errors during the process. If you wish to ignore any failed reads (keys that were not retrieved), use the --ignore-failed-reads option. * Database locking Before accessing, any database file is locked using kernel locking. By default, if the first attempt to lock the file fails, two more attempts are undertaken in 1 second intervals. If the lock cannot be acquired after the last attempt, the database file is opened in read- only mode. The number of locking attempt and the timeout value are controlled by command line options --lock-retry-count and --lock-retry-timeout, or the corresponding pragmas: #pragma option lock-retry-count #pragma option lock-retry-timeout * Selecting the database format and file ** New option -H (--format) specifies which format is the database being operated upon by any of the database management options. Recognized formats are: cache Poll cache database rate Sending rate database greylist Greylisting database (see below) ** New option --all can be used with one of the options --expire or --compact to apply the operation to all configured databases. This is useful to invoke mailfromd as a crontab job. ** New option --file allows to explicitely specify the database file name. Notice, that in contrast to the previous version, the name should include the suffix. * Debugging ** New option --lint (-l, --syntax-check): check the configuration file syntax. ** The argument to --debug option should be the numeric debug level. The use of characters 'c', 'd', 'l', 'y' is discouraged (although still supported for a while). See below for the alternatives. ** New option --dump-code dumps the listing of the assembled code on screen (similar to the earlier --debug=c) ** New option --dump-tree dumps the parse tree in human-readable form (earlier --debug=d option) ** New option --dump-grammar-trace prints grammar parser traces while parsing the configuration file (earlier --debug=g option). ** New option --dump-lex-trace dumps lexical analizer traces (earlier --debug=l option). ** New option -L (--log-tag) sets the identifier used in syslog messages. ** New option --source-info includes source line information into the debugging messages. Previously this information was included by default. * New option --group (-g) allows to retain the given supplementary group when switching to user privileges. By default mailfromd does not retain any supplementary groups. The use of this option may be necessary if your mailfromd script needs to access some databases that have restrictive access privileges. For example, if mailfromd runs with the privileges of user 'mail' (the default) and needs to access /etc/mail/aliases.db, which is usually owned by root.smmsp and has access rights 0640, you should run mailfromd --group smmsp * New option --source (-S) sets the source address mailfromd will use for any TCP connections. The configuration file equivalent is #pragma option source * To set up the local account validation (see the description of `validuser' function above) mailfromd uses authentication options from mailutils. See the mailutils documentation, chapter `Authorization and Authentication Principles' for the detailed description of these. * Code generation optimized to avoid unnecessary instructions and to reduce code size. * MX lookups no longer recurse to parent domains. Previously, if the domain "some.domain.com" had no MX records, mailfromd would lookup for MXs of "domain.com" and use these instead. This is no longer the case. * Added testsuite Version 1.4 * Configuration Added possibility to link against the forked version of libmilter (--with-forks). The patch for sendmail-8.13.1 is included (etc/sendmail-8.13.1.diff). * Configuration file ** New unary expression `listens' checks if the host listens on port 25. ** Several `#pragma option relay' statements accumulate * Bugfixes ** Fixed coredump on incorrect libmilter socket specification. ** Fixed `poll for EMAIL as EMAIL'. Version 1.3 * Rewritten DNS resolver functions in order to take into account CNAMEs. * Updated Makefiles to allow for compilation with the CVS Mailutils * Improved documentation. Version 1.2 * Implemented sending rate control. This feature allows to impose a limit on the number of messages a user can send within a given interval. If this number is exceeded, the connection is refused until enough time passes to keep the rate within the given limit. Version 1.1 Mostly bugfixes. Version 1.0 Lots of major improvements. Implemented two methods of sender address verification, controlled by a sophisticated configuration file. Sender domains and emails can also be distinguished basing on POSIX regex or shell-style globbing patterns. Version 0.2 First release. ========================================================================= Copyright information: Copyright (C) 2005-2020 Sergey Poznyakoff Permission is granted to anyone to make or distribute verbatim copies of this document as received, in any medium, provided that the copyright notice and this permission notice are preserved, thus giving the recipient permission to redistribute in turn. Permission is granted to distribute modified versions of this document, or of portions of it, under the above conditions, provided also that they carry prominent notices stating who last changed them. Local variables: mode: outline paragraph-separate: "[ ]*$" eval: (add-hook 'write-file-hooks 'time-stamp) time-stamp-start: "changes. " time-stamp-format: "%:y-%02m-%02d" time-stamp-end: "\n" end: