File (im_file) :: NXLog Documentation (2023)

This module can be used to read log messages from files.The file position can be persistently saved across restarts in order to avoid reading from the beginning again when NXLog is restarted.External rotation tools are also supported.When the module is not able to read any more data from the file, it checks whether the opened file descriptor belongs to the same filename it opened originally.If the inodes differ, the module assumes the file was moved and reopens its input.

im_file uses a one second interval to monitor files for new messages.This method was implemented because polling a regular file is not supported on all platforms.If there is no more data to read, the module will sleep for 1 second.

By using wildcards, the module can read multiple files simultaneously and will open new files as they appear.It will also enter newly created directories if recursion is enabled.

The module needs to scan the directory content for wildcarded file monitoring.This can present a significant load if there are many files (hundreds or thousands) in the monitored directory.For this reason it is highly recommended to rotate files out of the monitored directory either using the built-in log rotation capabilities of NXLog or with external tools.

Configuration

The im_file module accepts the following directives in addition to the common module directives.The File directive is required.

File

This mandatory directive specifies the name of the input file to open.It may be given more than once in a single im_file module instance.The value must be a string type expression.For relative filenames you should be aware that NXLog changes its working directory to "/" unless the global SpoolDir is set to something else.On Windows systems the directory separator is the backslash (\).For compatibility reasons the forward slash (/) character can be also used as the directory separator, but this only works for filenames not containing wildcards.If the filename is specified using wildcards, the backslash (\) should be used for the directory separator.Filenames on Windows systems are treated case-insensitively, but case-sensitively on Unix/Linux.

Wildcards are supported in filenames and directories.Wildcards are not regular expressions, but are patterns commonly used by Unix shells to expand filenames (also known as "globbing").

?

Matches a single character only.

*

Matches zero or more characters.

\*

Matches the asterisk (*) character.

\?

Matches the question mark (?) character.

[…​]

Used to specify a single character. The class descriptionis a list containing single characters and ranges of charactersseparated by the hyphen (-). If the first character of the classdescription is ^ or !, the sense of the description isreversed (any character not in the list is accepted). Anycharacter can have a backslash (\) preceding it, which isignored, allowing the characters ] and - to be used in thecharacter class, as well as ^ and ! at the beginning.

By default, the backslash character (\) is used as an escape sequence.This character is also the directory separator on Windows.Because of this, escaping of wildcard charactersis not supported on Windows, see the EscapeGlobPatterns directive.However, string literals are evaluated differently depending on the quotation type.Single quoted strings are interpreted as-is without escaping,e.g. 'C:\t???*.log' stays C:\t???\*.log.Escape sequences in double quoted strings are processed, for example "C:\\t???\*.log" becomes C:\t???\*.log after evaluation.In both cases, the evaluated string is the same and gets separated into parts with different glob patterns at different levels.In the previous example the parts are c:, t???, and *.log. NXLog matches these at the proper directory levels to find all matching files.

ActiveFiles

This directive specifies the maximum number of filesNXLog will actively monitor. If there are modifications tomore files in parallel than the value of this directive, thenmodifications to files above this limit will only get noticed afterthe DirCheckInterval (all datashould be collected eventually). Typically there are only a few logsources actively appending data to log files, and the rest of thefiles are dormant after being rotated, so the default value of 10files should be sufficient in most cases. This directive is also onlyrelevant in case of a wildcarded File path.

CloseWhenIdle

If set to TRUE, this boolean directive specifies thatopen input files should be closed as soon as possible after there isno more data to read. Some applications request an exclusive lock onthe log file when written or rotated, and this directive canpossibly help if the application tries again to acquire the lock. Thedefault is FALSE.

DirCheckInterval

This directive specifies how frequently, inseconds, the module will check the monitored directory formodifications to files and new files in case of a wildcardedFile path. The default is twice the value ofthe PollInterval directive (ifPollInterval is not set, the defaultis 2 seconds). Fractional seconds may be specified. It isrecommended to increase the default if there are many files whichcannot be rotated out and the NXLog process is causing highCPU load.

ReadOrder

This optional directive specifies the reading orderof the elements in a directory. The accepted values are none, CtimeOldestFirst,CtimeNewestFirst (Ctime is file creating time),MtimeOldestFirst, MtimeNewestFirst (Mtime is file modification time),NameAsc and NameDesc (sort is done according to ASCII codes of name characters).If directive is not specified then none is used as a defaultwhich means that the order of entries read from the directory is not specified.

Exclude

This directive can specify a file or a set of files (using wildcards) to be excluded.More than one occurrence of the Exclude directive can be specified.

InputType

See the InputType directive in the list of common module directives.If this directive is not specified the default is LineBased (the module will use CRLF as the record terminator on Windows, or LF on Unix).

This directive also supports data converters, see the description in theInputType section.

NoEscape

This boolean directive specifies whether the backslash (\) infile paths should be disabled as an escape sequence. This is especiallyuseful for file paths on Windows. By default, NoEscape is FALSE (backslashescaping is enabled and the path separator on Windows must be escaped).

OnEOF

This optional block directive can be used to specify a group ofstatements to execute when a file has been fully read (on end-of-file). Onlyone OnEOF block can be specified per im_file module instance. Thefollowing directives are used inside this block.

Exec

This mandatory directive specifies the actions to execute after EOFhas been detected and the grace period has passed. Like the normalExec directive, the OnEOF Exec can be specifiedas a normal directive or a block directive.

GraceTimeout

This optional directive specifies the time in seconds to waitbefore executing the actions configured in theExec block or directive. The default is 1second.

PollInterval

This directive specifies how frequently the module willcheck for new files and new log entries, in seconds. If thisdirective is not specified, it defaults to 1 second. Fractionalseconds may be specified (PollInterval 0.5 will check twice everysecond).

ReadFromLast

This optional boolean directive instructs the module toonly read logs which arrive after NXLog is started. This directivecomes into effect if a saved position is not found, for example on firststart, or when the SavePos directive is FALSE.When the SavePos directive is TRUE and apreviously saved position is found, the module will always resume reading fromthe saved position. If ReadFromLast is FALSE, the module will read alllogs from the beginning of the file. This can result in a lot of messagesand is usually not the expected behavior. If this directive is not specified,it defaults to TRUE.

The following matrix shows the outcome of this directive in conjunctionwith the SavePos directive:

ReadFromLastSavePosSaved PositionOutcome

TRUE

TRUE

No

Reads events that are logged after NXLog is started.

TRUE

TRUE

Yes

Reads events from saved position.

TRUE

FALSE

No

Reads events that are logged after NXLog is started.

TRUE

FALSE

Yes

Reads events that are logged after NXLog is started.

FALSE

TRUE

No

Reads all events.

FALSE

TRUE

Yes

Reads events from saved position.

FALSE

FALSE

No

Reads all events.

FALSE

FALSE

Yes

Reads all events.

Recursive

If set to TRUE, this boolean directive specifies that input filesset with the File directive should be searchedrecursively under sub-directories. For example, /var/log/error.log willmatch /var/log/apache2/error.log. Wildcards can be used in combinationwith Recursive: /var/log/*.log will match/var/log/apache2/access.log. This directive only causes scanning under thegiven path and does not affect the processing of wildcarded directories:/var/*/qemu/debian.log will not match/var/log/libvirt/qemu/debian.log. The default is FALSE.

RenameCheck

If set to TRUE, this boolean directive specifies that input files should be monitored for file rotation to avoid re-reading the same content.The module considers a file rotated when it detects a new file with the same inode and size as another deleted input file.Note that the file system can reuse the inode number of a deleted file.Therefore, if a new log file has the same inode and size as a deleted file, it will yield a false positive, and the module will falsely detect it as a rotated/renamed file.The default value of RenameCheck is FALSE – the module considers renamed files to be new and will re-read the file content.

When using file rotation, it is better to use a naming scheme that does not match the wildcard specified in the File directive so rotated files are no longer monitored than relying on the RenameCheck directive.
SavePos

If this boolean directive is set to TRUE, the file positionwill be saved when NXLog exits. The file position will beread from the cache file upon startup. The default is TRUE, the fileposition will be saved if this directive is not specified. This directiveaffects the outcome of the ReadFromLastdirective. The SavePos directive can be overridden by the globalNoCache directive.

Functions

The following functions are exported by im_file.

string file_name()

Return the name of the currently open file which the log was read from.

integer record_number()

Returns the number of processed records (including the current record) of the currently open file since it was opened or truncated.

Creating and populating fields

im_file populates the $raw_event core field with the log message read from file.Further processing of this field can be done to parse the message into structured data or convert it to a different output format, such as JSON or XML.See Parsing and converting log records below for an example and Parsing various log formats in the NXLog User Guide for more information on parsing log records.

Examples

Example 1. Forwarding logs from a file to a remote host

This configuration will read from a file and forward messages via TCP.No additional processing is done.

nxlog.conf

<Input messages> Module im_file File "/var/log/messages"</Input><Output tcp> Module om_tcp Host 192.168.1.1:514</Output><Route messages_to_tcp> Path messages => tcp</Route>

Example 2. Parsing and converting log records

This configuration reads logs from file and parses the $raw_event field with a regular expression.If the regular expression matches, fields are created according to the captured groups, otherwise the log record is dropped.Finally, the record is converted to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json> Module xm_json</Extension><Input messages> Module im_file File '/path/to/log/file' <Exec> if $raw_event =~ /(?x)^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\+\d\d:\d\d), (.+),(.+)$/ { $EventTime = parsedate($1); $Severity = $2; $Message = $3; } else { drop(); } to_json(); </Exec></Input>

Input sample

2021-11-05T14:03:40+01:00,INFO,The service started successfully

Output sample in JSON format

{ "EventReceivedTime": "2021-11-05T14:04:24.244343+01:00", "SourceModuleName": "messages", "SourceModuleType": "im_file", "EventTime": "2021-11-05T14:03:40.000000+01:00", "Severity": "INFO", "Message": "The service started successfully"}
Top Articles
Latest Posts
Article information

Author: Madonna Wisozk

Last Updated: 03/29/2023

Views: 6229

Rating: 4.8 / 5 (48 voted)

Reviews: 95% of readers found this page helpful

Author information

Name: Madonna Wisozk

Birthday: 2001-02-23

Address: 656 Gerhold Summit, Sidneyberg, FL 78179-2512

Phone: +6742282696652

Job: Customer Banking Liaison

Hobby: Flower arranging, Yo-yoing, Tai chi, Rowing, Macrame, Urban exploration, Knife making

Introduction: My name is Madonna Wisozk, I am a attractive, healthy, thoughtful, faithful, open, vivacious, zany person who loves writing and wants to share my knowledge and understanding with you.