Skip to content

Commit 30c3ca5

Browse files
anurag4DSBanurag4DSB
committed
ft: ZENKO-1456-sysBackupMongoDB
1 parent 5a204b8 commit 30c3ca5

File tree

3 files changed

+206
-0
lines changed

3 files changed

+206
-0
lines changed

‎SysBackup/README.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
SysBackup
2+
================
3+
4+
This is a system for backup and restore for two components of Zenko for
5+
disaster recovery purposes:
6+
* Metadata in MongoDB
7+
* Zookeeper queues
8+
9+
Currently the backup process is automated. The process/script for backup will
10+
be run as a cronjob inside the container: zenko-backup.
11+
12+
MongoDB metadata backup
13+
-----------------------
14+
15+
The tool used to dump mongoDB documents and collections is `mongodump`. The
16+
backup will be a combination of a fullbackup and multiple incremental backups.
17+
Everything will be automated inside the container zenko-backup using a cronjob.
18+
The cronjob will take a full backup everyday using the below command:
19+
20+
```shell
21+
mongodump --host $MONGODB_HOST --port $MONGODB_PORT --oplog --gzip -o \
22+
$BACKUP_DATA_PATH/mongodb-backup/`date \+\%Y\%m\%d_\%s`
23+
```
24+
25+
This command will assume that MONGODB_HOST, MONGODB_PORT and BACKUP_DATA_PATH
26+
environment variables are defined in the help chart.
27+
28+
For incremental backup the script `incrementalMongodbBackup.sh` will be used.
29+
This is a simple bash script to analyze the full backups or previous
30+
incremental backups and create backups post the previous oplog entry. It will
31+
avoid overlapping entries. This script will be run every 5 minutes.
32+
33+
Currently local storage attached to the pod will be used for backups. As the
34+
backups are compressed using the `--gzip` option, the backups will roughly take
35+
8-10% of the actual mongoDB metadata size.
36+
37+
<!-- TODO: Add tool like rclone to upload backups to cloud. -->
38+
39+
MongoDB metadata restore
40+
-----------------------
41+
<!-- TODO: write the process for restoring to a new zenko cluster. -->
42+
43+
The tool used to restore mongoDB documents and collection is `mongorestore`.
44+
The restore process is a combination of a full restore of the full backup made
45+
and a script to restore all the incremental backups. Currently the restore
46+
process is manual. To restore the full backup the following command should be
47+
used:
48+
49+
```shell
50+
mongorestore --gzip --host <mongoDB host --port 27017 <path-to-fullbackup> --oplogReplay --verbose
51+
```
52+
53+
The helper script `incrementalMongodbRestore.sh` restores all incremental
54+
backups. The script assumes that the full backup has been restored using
55+
`mongorestore` command as stated previously. The script requires 3 variables
56+
with the execution:
57+
* FULL_DUMP_DIRECTORY: directory path to the full backup
58+
* OPLOGS_DIRECTORY: Directory path to incremental backup
59+
* OPLOG_LIMIT: Defines the time till which you want the backup for
60+
61+
The script needs the names of the backup files to be the same as when
62+
backed up.
63+
64+
65+
<!-- TODO: write the process of zookeeper backups -->

‎SysBackup/incrementalMongodbBackup.sh

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/bin/bash
2+
3+
function initStaticParams
4+
{
5+
MONGODB_SERVER=$MONGODB_HOST
6+
MONOGDB_PORT=$MONGODB_PORT
7+
MONGODB_USER=NA
8+
MONGODB_PWD=NA
9+
OUTPUT_DIRECTORY=$BACKUP_DATA_PATH/mongodb-backup
10+
LOG_PATH=$OUTPUT_DIRECTORY/logs
11+
LOG_FILE="$LOG_PATH/backup.log"
12+
13+
LOG_MESSAGE_ERROR=1
14+
LOG_MESSAGE_WARN=2
15+
LOG_MESSAGE_INFO=3
16+
LOG_MESSAGE_DEBUG=4
17+
LOG_LEVEL=$LOG_MESSAGE_DEBUG
18+
SCRIPT=`readlink -f ${BASH_SOURCE[0]}`
19+
ABSOLUTE_SCRIPT_PATH=$(cd `dirname "$SCRIPT"` && pwd)
20+
}
21+
22+
23+
24+
function log
25+
{
26+
MESSAGE_LEVEL=$1
27+
shift
28+
MESSAGE="$@"
29+
30+
if [ $MESSAGE_LEVEL -le $LOG_LEVEL ]; then
31+
echo "`date +'%Y-%m-%dT%H:%M:%S.%3N'` $MESSAGE" >> $LOG_FILE
32+
fi
33+
}
34+
35+
initStaticParams
36+
37+
mkdir -p $OUTPUT_DIRECTORY
38+
mkdir -p $LOG_PATH
39+
40+
log $LOG_MESSAGE_INFO "[INFO] starting incremental backup of oplog"
41+
42+
43+
LAST_OPLOG_DUMP=`ls -t ${OUTPUT_DIRECTORY}/*.bson 2> /dev/null | head -1`
44+
45+
if [ "$LAST_OPLOG_DUMP" != "" ]; then
46+
log $LOG_MESSAGE_DEBUG "[DEBUG] last incremental oplog backup is $LAST_OPLOG_DUMP"
47+
LAST_OPLOG_ENTRY=`bsondump ${LAST_OPLOG_DUMP} 2>> $LOG_FILE | grep ts | tail -1`
48+
if [ "$LAST_OPLOG_ENTRY" == "" ]; then
49+
log $LOG_MESSAGE_ERROR "[ERROR] evaluating last backuped oplog entry with bsondump failed"
50+
exit 1
51+
else
52+
TIMESTAMP_LAST_OPLOG_ENTRY=`echo $LAST_OPLOG_ENTRY | jq '.ts[].t'`
53+
INC_NUMBER_LAST_OPLOG_ENTRY=`echo $LAST_OPLOG_ENTRY | jq '.ts[].i'`
54+
START_TIMESTAMP="Timestamp( ${TIMESTAMP_LAST_OPLOG_ENTRY}, ${INC_NUMBER_LAST_OPLOG_ENTRY} )"
55+
log $LOG_MESSAGE_DEBUG "[DEBUG] dumping everything newer than $START_TIMESTAMP"
56+
fi
57+
log $LOG_MESSAGE_DEBUG "[DEBUG] last backuped oplog entry: $LAST_OPLOG_ENTRY"
58+
else
59+
log $LOG_MESSAGE_WARN "[WARN] no backuped oplog available. creating initial backup"
60+
fi
61+
62+
if [ "$LAST_OPLOG_ENTRY" != "" ]; then
63+
mongodump -h $MONGODB_SERVER --authenticationDatabase=admin -d local -c oplog.rs --query "{ \"ts\" : { \"\$gt\" : $START_TIMESTAMP } }" -o - > ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson 2>> $LOG_FILE
64+
RET_CODE=$?
65+
else
66+
TIMESTAMP_LAST_OPLOG_ENTRY=0000000000
67+
INC_NUMBER_LAST_OPLOG_ENTRY=0
68+
mongodump -h $MONGODB_SERVER --authenticationDatabase=admin -d local -c oplog.rs -o - > ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson 2>> $LOG_FILE
69+
RET_CODE=$?
70+
fi
71+
72+
if [ $RET_CODE -gt 0 ]; then
73+
log $LOG_MESSAGE_ERROR "[ERROR] incremental backup of oplog with mongodump failed with return code $RET_CODE"
74+
fi
75+
76+
FILESIZE=`stat --printf="%s" ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson`
77+
78+
if [ $FILESIZE -eq 0 ]; then
79+
log $LOG_MESSAGE_WARN "[WARN] no documents have been dumped with incremental backup (no changes in mongodb since last backup?). Deleting ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson"
80+
rm -f ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson
81+
else
82+
log $LOG_MESSAGE_INFO "[INFO] finished incremental backup of oplog to ${OUTPUT_DIRECTORY}/${TIMESTAMP_LAST_OPLOG_ENTRY}_${INC_NUMBER_LAST_OPLOG_ENTRY}_oplog.bson"
83+
fi

‎SysBackup/incrementalMongodbRestore.sh

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/bin/bash
2+
3+
# OPLOG_LIMIT: only include oplog entries before the provided Timestamp
4+
# The timestamp is a unix timestamp
5+
# If your desaster happened for example on 2019-03-15 11:40 and you want to restore until 11:39
6+
# your timestamp is 'date -d "2019-03-15 11:39:00" +%s'
7+
8+
FULL_DUMP_DIRECTORY=$1
9+
OPLOGS_DIRECTORY=$2
10+
OPLOG_LIMIT=$3
11+
12+
if [ "$FULL_DUMP_DIRECTORY" == "" ]; then
13+
echo "usage: restoreOpLogs.sh [NAME_OF_DIRECTORY_OF_FULL_DUMP] [DIRECTORY_TOP_OPLOGS]"
14+
exit 1
15+
fi
16+
17+
if [ "$OPLOGS_DIRECTORY" == "" ]; then
18+
echo "usage: restoreOpLogs.sh [NAME_OF_DIRECTORY_OF_FULL_DUMP] [DIRECTORY_TOP_OPLOGS]"
19+
exit 1
20+
fi
21+
22+
if [ "$OPLOG_LIMIT" == "" ]; then
23+
echo "usage: restoreOpLogs.sh [NAME_OF_DIRECTORY_OF_FULL_DUMP] [DIRECTORY_TOP_OPLOGS] [OPLOG_LIMIT]"
24+
exit 1
25+
fi
26+
27+
28+
29+
FULL_DUMP_TIMESTAMP=`echo $FULL_DUMP_DIRECTORY | cut -d "_" -f 2 | cut -d "/" -f 1`
30+
LAST_OPLOG=""
31+
ALREADY_APPLIED_OPLOG=0
32+
33+
mkdir -p /tmp/emptyDirForOpRestore
34+
35+
for OPLOG in `ls $OPLOGS_DIRECTORY/*.bson`; do
36+
OPLOG_TIMESTAMP=`echo $OPLOG | rev | cut -d "/" -f 1 | rev | cut -d "_" -f 1`
37+
if [ $OPLOG_TIMESTAMP -gt $FULL_DUMP_TIMESTAMP ]; then
38+
if [ $ALREADY_APPLIED_OPLOG -eq 0 ]; then
39+
ALREADY_APPLIED_OPLOG=1
40+
echo "applying oplog $LAST_OPLOG"
41+
mongorestore --authenticationDatabase=admin --oplogFile $LAST_OPLOG --oplogReplay --dir /tmp/emptyDirForOpRestore --oplogLimit=$OPLOG_LIMIT
42+
echo "applying oplog $OPLOG"
43+
mongorestore --authenticationDatabase=admin --oplogFile $OPLOG --oplogReplay --dir /tmp/emptyDirForOpRestore --oplogLimit=$OPLOG_LIMIT
44+
else
45+
echo "applying oplog $OPLOG"
46+
mongorestore --authenticationDatabase=admin --oplogFile $OPLOG --oplogReplay --dir /tmp/emptyDirForOpRestore --oplogLimit=$OPLOG_LIMIT
47+
fi
48+
else
49+
LAST_OPLOG=$OPLOG
50+
fi
51+
done
52+
53+
if [ $ALREADY_APPLIED_OPLOG -eq 0 ]; then
54+
if [ "$LAST_OPLOG" != "" ]; then
55+
echo "applying oplog $LAST_OPLOG"
56+
mongorestore --authenticationDatabase=admin --oplogFile $LAST_OPLOG --oplogReplay --dir $OPLOGS_DIRECTORY --oplogLimit=$OPLOG_LIMIT
57+
fi
58+
fi

0 commit comments

Comments
 (0)