Skip to content

Commit 191771d

Browse files
drgergdrgerg
committed
Rebuild Instructions
1 parent 92506e6 commit 191771d

File tree

3 files changed

+371
-0
lines changed

3 files changed

+371
-0
lines changed

‎Docs/allBuild.md

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
## Building Brilliant (home to 'allApp.py' and others)
2+
3+
(alternate notes in OneNote)
4+
5+
**Assumptions:** Things that were already in place.
6+
7+
- In all cases, the OS is Ubuntu.
8+
- Brilliant is the name of the machine I'm working on. I'm replacing the storage drive and updating the OS.
9+
- Valiant is the name of the machine I am working at. Windows Subsystem for Linux running Ubuntu.
10+
- Brilliant is in the 'hosts' file on Valiant, so I don't need to type the IP address.
11+
- ssh keys are in place on Valiant for use in public key authentication (passwordless login).
12+
- mySQL databases must be moved from the old drive to the new one. They must be 'exported' before the old drive is deactivated.
13+
- The username for the primary user must be the same on the new drive/OS as on the old one.
14+
- The contents of the /home/{user}/ folder must be copied intact to the new drive. That's where my Python program files live.
15+
16+
17+
Curly brackets should not be typed in commands. They indicate a private value that should be known only to you.
18+
For example: {user} is your username.
19+
20+
## Getting Started ##
21+
22+
### First: Before removing the old drive, prepare to move mySQL data.
23+
24+
**Either ssh into Brilliant, or do this locally on Brilliant**
25+
26+
mysql -u {user} -p
27+
28+
Now you're logged into mySQL, so from now on we're at the mySQL prompt.
29+
30+
select @@datadir;
31+
show databases;
32+
quit;
33+
This gives you info on the location and names of your databases.
34+
35+
**Now export your databases** at the Linux commandline.
36+
37+
mysqldump outdata > outdata-20210220.sql -u {user} -p
38+
39+
The result is 'outdata-20210220.sql' in your current directory.
40+
Repeat for each database you want to export / backup.
41+
42+
### Second: Install the new drive.
43+
44+
**Locally on Brilliant:**
45+
Prepare the new drive.
46+
a. Disconnect the old drive from the computer.
47+
b. Connect the new drive.
48+
c. Install the OS (Remember the user name and the machine name need to be the same as they were before.)
49+
50+
Get ssh-server up and running on the new drive.
51+
52+
sudo install ssh-server
53+
sudo nano /etc/sshd_config
54+
55+
Uncomment #port 22 and replace 22 with your {port} number.
56+
57+
sudo service sshd restart
58+
59+
**Make the old drive available to the system in some fashion.**
60+
61+
I used a USB interface to connect the old drive to Brilliant.
62+
Then copy these folders over to the new drive:
63+
64+
The /home/{user}/all folder.
65+
The /home/{user}/.ssh folder.
66+
67+
68+
**Get mySQL up and running on the new drive.**
69+
70+
sudo apt install mysql-server
71+
sudo mysql (logs you in as root)
72+
create user {user}@localhost identified by '{password}';
73+
grant all privileges on *.* to {user}@localhost;
74+
flush privileges;
75+
quit;
76+
77+
That should have done it. But NO, I had been using 5.7.33 and the new mySQL was 8.0.23.
78+
Every time I brought my databases in, mysql went squirrely.
79+
80+
I finally had to figure out how to install 5.7.3 again. It went like this:
81+
82+
wget https://dev.mysql.com/get/mysql-apt-config_0.8.12-1_all.deb
83+
sudo dpkg -i mysql-apt-config_0.8.12-1_all.deb
84+
Select Bionic
85+
Select mySQL 5.7 server
86+
sudo apt-get update
87+
sudo apt install -f mysql-client=5.7.33-1ubuntu18.04
88+
sudo apt install -f mysql-community-server=5.7.33-1ubuntu18.04
89+
sudo apt install -f mysql-server=5.7.33-1ubuntu18.04
90+
91+
I did assign a password to the 'root' account. I tried to use the 'empty' method first time through these steps, but it didn't work, so I had to start over.
92+
93+
sudo mysql_secure_installation
94+
95+
sudo ufw status verbose
96+
sudo ufw allow {port}/tcp
97+
sudo ufw allow from 192.168.1.1/24 to any port 3306
98+
sudo ufw allow 80,443/tcp
99+
100+
sudo ufw enable
101+
102+
**Create a mySQL user for each node of PiNet for allSysReport.service (zAllSysChk.py)**
103+
104+
**ssh-copy-id -i id_rsa -p {port} {user}@{machine}**
105+
106+
This needs to be done from Brilliant (/home/{user}/.ssh) to every other PiNet machine <b> including Brilliant </b>. I know, that doesn't exactly make sense, but it is true.
107+
108+
109+
**Locally on Valiant**
110+
111+
Copy the ssh keys to Brilliant and reset known_hosts to accept the new Brilliant.
112+
113+
ssh-copy-id -i {publickeyfile} -p{port} {user}@brilliant
114+
ssh-keygen -f "/home/{user}/.ssh/known_hosts" -R "[brilliant]:{port}"
115+
116+
**Log into Brilliant.**
117+
118+
ssh -p {port}
119+
120+
Start installing some of the dependencies.
121+
122+
sudo apt install python3-pip
123+
sudo apt-get install nginx
124+
sudo pip3 install gunicorn
125+
pip3 install mysql-connector
126+
sudo apt-get install python3-numpy
127+
pip3 install noaa-sdk
128+
sudo pip3 install -U Flask
129+
sudo pip3 install flask_wtf
130+
pip install pyyaml ua-parser user-agents
131+
sudo apt install mailutils
132+
sudo apt-get install msmtp msmtp-mta
133+
134+
**Configure nginx**
135+
136+
sudo nano /etc/nginx/nginx.conf
137+
138+
Once in there, look for these various entries and make them look like this, character for character, no # in front: (All this comes from e-tinkers linked above.)
139+
140+
multi_accept on;
141+
keepalive_timeout 30;
142+
server_tokens off;
143+
gzip_vary on;
144+
gzip_proxied any;
145+
gzip_comp_level 5;
146+
gzip_http_version 1.1;
147+
148+
Add a line just before the <b>gzip_types</b> line that says this:
149+
150+
gzip_min_length 256;
151+
152+
Honestly, I don't know if this is necessary, but it was in the instructions, and I can say it didn't hurt anything, so, plug this in just below the # gzip_types ... line that is there. (Yes, you can just use this to replace the existing line.)
153+
154+
gzip_types
155+
application/atom+xml
156+
application/javascript
157+
application/json
158+
application/rss+xml
159+
application/vnd.ms-fontobject
160+
application/x-font-ttf
161+
application/x-web-app-manifest+json
162+
application/xhtml+xml
163+
application/xml
164+
font/opentype
165+
image/svg+xml
166+
image/x-icon
167+
text/css
168+
text/plain
169+
text/x-component
170+
text/javascript
171+
text/xml;
172+
173+
**Set up msmtp to handle notifications**
174+
175+
The instructions can be found in the doorPi (ohd) docs here on Github. https://github.com/casspop/ohd/blob/master/SetupRaspianForOhd.md
176+
I don't see any reason to copy and paste that over here.
177+
178+
179+

‎Docs/mysql.conf.d

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#
2+
# The MySQL database server configuration file.
3+
#
4+
# You can copy this to one of:
5+
# - "/etc/mysql/my.cnf" to set global options,
6+
# - "~/.my.cnf" to set user-specific options.
7+
#
8+
# One can use all long options that the program supports.
9+
# Run program with --help to get a list of available options and with
10+
# --print-defaults to see which it would actually understand and use.
11+
#
12+
# For explanations see
13+
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
14+
15+
# This will be passed to all mysql clients
16+
# It has been reported that passwords should be enclosed with ticks/quotes
17+
# escpecially if they contain "#" chars...
18+
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
19+
20+
# Here is entries for some specific programs
21+
# The following values assume you have at least 32M ram
22+
23+
[mysqld_safe]
24+
socket = /var/run/mysqld/mysqld.sock
25+
nice = 0
26+
27+
[mysqld]
28+
#
29+
# * Basic Settings
30+
#
31+
user = mysql
32+
pid-file = /var/run/mysqld/mysqld.pid
33+
socket = /var/run/mysqld/mysqld.sock
34+
port = 3306
35+
basedir = /usr
36+
datadir = /var/lib/mysql
37+
tmpdir = /tmp
38+
lc-messages-dir = /usr/share/mysql
39+
skip-external-locking
40+
#
41+
# Instead of skip-networking the default is now to listen only on
42+
# localhost which is more compatible and is not less secure.
43+
#bind-address = 127.0.0.1
44+
bind-address = 192.168.1.150
45+
#
46+
# * Fine Tuning
47+
#
48+
key_buffer_size = 16M
49+
max_allowed_packet = 16M
50+
thread_stack = 192K
51+
thread_cache_size = 8
52+
# This replaces the startup script and checks MyISAM tables if needed
53+
# the first time they are touched
54+
myisam-recover-options = BACKUP
55+
#max_connections = 100
56+
#table_open_cache = 64
57+
#thread_concurrency = 10
58+
#
59+
# * Query Cache Configuration
60+
#
61+
query_cache_limit = 1M
62+
query_cache_size = 16M
63+
#
64+
# * Logging and Replication
65+
#
66+
# Both location gets rotated by the cronjob.
67+
# Be aware that this log type is a performance killer.
68+
# As of 5.1 you can enable the log at runtime!
69+
#general_log_file = /var/log/mysql/mysql.log
70+
#general_log = 1
71+
#
72+
# Error log - should be very few entries.
73+
#
74+
log_error = /var/log/mysql/error.log
75+
#
76+
# Here you can see queries with especially long duration
77+
#slow_query_log = 1
78+
#slow_query_log_file = /var/log/mysql/mysql-slow.log
79+
#long_query_time = 2
80+
#log-queries-not-using-indexes
81+
#
82+
# The following can be used as easy to replay backup logs or for replication.
83+
# note: if you are setting up a replication slave, see README.Debian about
84+
# other settings you may need to change.
85+
#server-id = 1
86+
#log_bin = /var/log/mysql/mysql-bin.log
87+
expire_logs_days = 10
88+
max_binlog_size = 100M
89+
#binlog_do_db = include_database_name
90+
#binlog_ignore_db = include_database_name
91+
#
92+
# * InnoDB
93+
#
94+
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
95+
# Read the manual for more InnoDB related options. There are many!
96+
#
97+
# * Security Features
98+
#
99+
# Read the manual, too, if you want chroot!
100+
# chroot = /var/lib/mysql/
101+
#
102+
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
103+
#
104+
# ssl-ca=/etc/mysql/cacert.pem
105+
# ssl-cert=/etc/mysql/server-cert.pem
106+
# ssl-key=/etc/mysql/server-key.pem

‎allLogReader.py

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env python3
2+
#
3+
## allLogReader.py - Supports reading logs across the vast PiNet
4+
#
5+
# Copyright (c) 2020, 2021 - Gregory Allen Sanders.
6+
7+
# This program is free software: you can redistribute it and/or modify
8+
# it under the terms of the GNU General Public License as published by
9+
# the Free Software Foundation, either version 3 of the License, or
10+
# (at your option) any later version.
11+
12+
# This program is distributed in the hope that it will be useful,
13+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15+
# GNU General Public License for more details.
16+
17+
# You should have received a copy of the GNU General Public License
18+
# along with this program. If not, see <http://www.gnu.org/licenses/>.
19+
#
20+
import time,os,logging,argparse,traceback,signal,sys
21+
22+
parserALR = argparse.ArgumentParser()
23+
parserALR.add_argument('-d', '--debug', help="Turn on debugging output to et.log file.", action="store_true")
24+
parserALR.add_argument('-uet', help="Unix Epoch Time value.", action="store")
25+
ALRHome = os.getcwd()
26+
logger = logging.getLogger(__name__)
27+
argsALR = parserALR.parse_args()
28+
29+
logger.debug('Checked for commandline argument: argsALR.uet.')
30+
if argsALR.uet:
31+
ALRRaw = float(argsALR.uet)
32+
print(' Input UALR: ' + str(ALRRaw))
33+
34+
35+
if argsALR.debug:
36+
logging.basicConfig(filename=ALRHome + '/ALR.log', format='[%(name)s]:%(levelname)s: %(message)s. - %(asctime)s', datefmt='%D %H:%M:%S', level=logging.DEBUG)
37+
logging.info("Debugging output enabled")
38+
else:
39+
logging.basicConfig(filename=ALRHome + '/ALR.log', format='%(asctime)s - %(message)s', datefmt='%a, %d %b %Y %H:%M:%S', level=logging.INFO)
40+
#
41+
logger.info(" - - - - allLogReader.py DATA LOGGING STARTED - - - - ")
42+
logger.info(" allLogReader.py INITIAL CONFIGURATION COMPLALRE ")
43+
logger.info("'HOME' path is: " + ALRHome)
44+
45+
#
46+
## - - - - - TEST CODE BELOW HERE - - - -
47+
#
48+
49+
def main():
50+
logger.info('And so it begins.')
51+
# 1) Get a list of .log files
52+
# 2)
53+
54+
55+
## - - - - - - END TEST CODE - - - - - - -
56+
#
57+
58+
def SignalHandler(signal, frame):
59+
if signal == 2:
60+
sigStr = 'CTRL-C'
61+
logger.info('* * * ' + sigStr + ' caught. * * * ')
62+
print("\nSignalHandler invoked")
63+
logger.info("Shutting down gracefully")
64+
logger.debug("Wrote to log in SignalHandler")
65+
logger.info("That's all folks. Goodbye")
66+
logger.info(" - - - - allLogReader.py DATA LOGGING STOPPED INTENTIONALLY - - - - ")
67+
sys.exit(0)
68+
69+
if __name__ == "__main__":
70+
signal.signal(signal.SIGINT, SignalHandler) ## This one catches CTRL-C from the local keyboard
71+
signal.signal(signal.SIGTERM, SignalHandler) ## This one catches the Terminate signal from the system
72+
try:
73+
if argsALR.uet:
74+
main(argsALR.uet)
75+
else:
76+
uet = time.time()
77+
print("\nYou didn't supply a Unix Epoch Time value. Here's one: " + str(uet) + '\n Learn more by typing "./et.py --help".\n')
78+
logger.debug('No time value was provided, so we provided this: ' + str(uet))
79+
pass
80+
logger.info(" - - - - allLogReader.py COMPLETED ITS MISSION - - - - ")
81+
except Exception:
82+
logger.info("Exception caught at bottom of try.", exc_info=True)
83+
error = traceback.print_exc()
84+
logger.info(error)
85+
logger.info("That's all folks. Goodbye")
86+
logger.info(" - - - - allLogReader.py DATA LOGGING STOPPED BY EXCEPTION - - - - ")

0 commit comments

Comments
 (0)