Tailnmail analyzed old logs after server down for some days

[gp4git]

Hello,
for maintenance reasons, our PG linux server was down from Friday October 27 to Monday October 30.

In our postgresql.conf we use : log_rotation_age = 1d
In tail-nmail config we have : FILE1: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces

So after one year activity, we have 31 PG log files in our LOG directory (from <CLUSTER_NAME>-01.log to <CLUSTER_NAME>-31.log) on which PG rotates each night.

When the server went up, this morning (October 30th), the first run of Tailnmail analyzed files <CLUSTER_NAME>-27 to <CLUSTER_NAME>-30 considering they were all from October but due to the weekend shutdown files 28 and 29 were in fact files from last September. Only files -27 and -30 files should have been analyzed.

Here is the debug output of this tnm run :
Opened config file "/u02/pgsql/admin/9.4/<CLUSTER_NAME>/scripts/tnm_pgsql.cfg"
$opt1 = {
'configfile' => {
'log_line_prefix' => 1,
'include' => 1,
'file.HASH(0x1270388)' => 1,
'file' => 1,
'email' => 1,
'[email protected]' => 1,
'exclude.checkpoint complete:' => 1,
'include.FATAL:' => 1,
'exclude.automatic analyze' => 1,
'include.LOG:' => 1,
'exclude.checkpoint starting:' => 1,
'offset' => 1,
'include.ERROR:' => 1,
'lastfile' => 1,
'customsubject' => 1,
'include.PANIC:' => 1,
'exclude.(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg_|seq)|SET |$))' => 1,
'mailsubject' => 1,
'exclude.automatic vacuum' => 1,
'exclude' => 1
},
'lastfile' => {
'1' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log'
},
'log_line_prefix' => ''user=%u,db=%d,'',
'include' => [
'PANIC:',
'FATAL:',
'LOG:',
'ERROR:'
],
'file' => [
{
'lastfile' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log',
'original' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-%d.log',
'suffix' => '1',
'name' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log',
'offset' => '5870628'
}
],
'customsubject' => 1,
'email' => [
'[email protected]'
],
'mailsubject' => 'HOST (Serveur de PRODUCTION <CLUSTER_NAME>) Postgres errors UNIQUE : NUMBER',
'offset' => {
'1' => '5870628'
},
'exclude' => [
'checkpoint starting:',
'checkpoint complete:',
'automatic analyze',
'automatic vacuum',
'(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|seq)|SET |$))'
]
};
$arg1 = {
'verbose' => 1,
'log_line_prefix' => ''user=%u,db=%d,'',
'mailserver' => 'example.com',
'find_line_number' => 1,
'file' => [],
'sortby' => 'count',
'maxsize' => 80000000,
'mailuser' => 'example',
'tsepnosub' => 0,
'debug' => 1,
'duration_limit' => 0,
'sqlstate' => 0,
'dryrun' => 0,
'timewarp' => 0,
'pgmode' => 1,
'nomail' => 0,
'duration' => -1,
'statement_size' => 3000,
'mailport' => 465,
'tsep' => undef,
'flatten' => 1,
'type' => 'normal',
'skip_non_parsed' => 1,
'smtp' => 0,
'tempfile_limit' => 0,
'tempfile' => -1,
'hideflatten' => 1,
'mailcom' => '/usr/sbin/sendmail',
'maxemailsize' => 10000000,
'reset' => 0,
'pretty_query' => 1,
'help' => 0,
'mailmode' => 'sendmail',
'mailpass' => 'example',
'offset' => -1,
'quiet' => 0,
'showonly' => 0,
'version' => 0,
'mailsig' => [],
'mailzero' => 0,
'rewind' => 0,
'pglog' => 'pg'
};
Log line prefix regex: (?-xism:^(()()user=[[\w]],db=[[\w]],)(.*))
Log line prefix regex2: (?-xism:^user=,db=,)
Log line prefix regex3: (?-xism:^user=)
Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log Offset: 5,870,628 Size: 5,871,159 Maxsize: 80,000,000
Adding exclusion: checkpoint starting:
Adding exclusion: checkpoint complete:
Adding exclusion: automatic analyze
Adding exclusion: automatic vacuum
Adding exclusion: (execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|seq)|SET |$))
Exclusion: (?-xism:checkpoint starting:)|(?-xism:checkpoint complete:)|(?-xism:automatic analyze)|(?-xism:automatic vacuum)|(?-xism:(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|_seq)|SET |$)))
Adding inclusion: PANIC:
Adding inclusion: FATAL:
Adding inclusion: LOG:
Adding inclusion: ERROR:
Inclusion: (?-xism:PANIC:)|(?-xism:FATAL:)|(?-xism:LOG:)|(?-xism:ERROR:)
MATCH at line 44725 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
MATCH at line 44726 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
MATCH at line 44727 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
MATCH at line 44728 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
MATCH at line 44729 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log
Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log: 5
Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log)
Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log)
Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log)
Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log Offset: 0 Size: 11,310,774 Maxsize: 80,000,000
MATCH at line 6768 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 6769 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 6770 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 6771 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 7363 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 8170 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 8344 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
MATCH at line 21717 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log
Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log: 67
Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log
File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log Offset: 0 Size: 6,128,680 Maxsize: 80,000,000
No new lines found in file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log
Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log Offset: 0 Size: 554,364 Maxsize: 80,000,000
MATCH at line 955 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 956 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 957 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 958 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 960 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 961 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 995 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 1039 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 1123 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 1175 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 1941 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2043 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2098 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2166 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2300 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2443 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2701 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
MATCH at line 2936 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log
Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log: 18
Sending mail to: [email protected]
Performing final cleanup
Saving new config file

Regards