I recently had an issue with ec2 instance volumes being filled by files in /var/lib/mysql/#rocksdb.
RockDb was filling the volume with log files, meaning the instance wouldn't boot. An easy enough fix, if I could get access to my ec2 instance with a full disk!
This particular instance was running Amazon Linux 2023.
AWS console actions to mount volume on other ec2 instance
- stop machine with full disk
- Detach volume
- Attach to another instance
- (Fix stuff (below) via SSH)
- Detach and reattach to original ec2 instance
SSH session on foster parent instance
- SSH onto second instance
- Find the drive in OS... mine was nvme1n1 (see below)
- Mount drive
- Find & remove large unneeded files
Finding device
[root@ip-(redacted) ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zram0 253:0 0 407M 0 disk [SWAP]
nvme0n1 259:0 0 8G 0 disk
├─nvme0n1p1 259:1 0 8G 0 part /
├─nvme0n1p127 259:2 0 1M 0 part
└─nvme0n1p128 259:3 0 10M 0 part /boot/efi
nvme1n1 259:4 0 10G 0 disk
├─nvme1n1p1 259:5 0 10G 0 part
└─nvme1n1p128 259:6 0 1M 0 part
[root@ip-(redacted) ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 204M 0 204M 0% /dev/shm
tmpfs 82M 472K 82M 1% /run
/dev/nvme0n1p1 8.0G 4.5G 3.6G 56% /
tmpfs 204M 0 204M 0% /tmp
/dev/nvme0n1p128 10M 1.3M 8.7M 13% /boot/efi
tmpfs 41M 0 41M 0% /run/user/1000
We can see from lsblk above, the 2nd volume is nvme1n1 and it has 2 partitions.
df -h shows us it isn't mounted yet.
Mount volume & find large files
[root@ip-(redacted) dev]# mount /dev/nvme1n1p1 /mnt/targetdir
mount: /mnt/targetdir: mount point does not exist.
[root@ip-(redacted) dev]# mkdir /mnt/targetdir
[root@ip-(redacted) dev]# mount /dev/nvme1n1p1 /mnt/targetdir
[root@ip-(redacted) dev]# cd /mnt/targetdir/
[root@ip-(redacted) targetdir]# du -a . | sort -n -r | head -
Having removed a bunch of large files I reattached the volume to the original ec2 instance and booted it up.
Prevent RocksDb creating so many large log files
I created a simple conf file:
/etc/my.cnf.d/my-custom.cnf
In that file was a simple config value:
[mariadb]
rocksdb_keep_log_file_num=3
After restarting MariaDb service I can see disk usage (df -h) drops dramatically as the database server cleansup excess log files.
Will I need those files again in the future? No idea! I've not needed them in the years this server has been running.... fingers crossed!