Elasticsearch disk space management and migration

Previous incidents

https://github.com/ISISScientificComputing/autoreduce-documents/blob/master/on-call/postmortem/2021-06-15.md

Mounting & formatting the volume

1. Create new volume on https://openstack.stfc.ac.uk/project/instances/

2. Mount it to elasticsearch instance

   • At this point, this is just an empty unformatted data blob
   • Note the location, shown in the volume details, at which it is mounted - the steps below assume it was mounted at /dev/vdb, which should be the default for first additional volume

3. mkfs /dev/vdb - Make it into a filesystem

   • This will just use all defaults, which means ext2 filesystem on CentOS7

4. mkdir /mnt/external - Create the mounting point, in this case /mnt/external is used, but any folder works

5. Add entry into /etc/fstab to mount it

   • /dev/vdb /mnt/external ext2 defaults 0 0 - simple mount

6. Refresh the mounts with sudo mount -a

Changing the Elasticsearch path.data

1. service elasticsearch stop - to stop the service.

If starting clean just go to N, else do the following to copy the data first:

2. mkdir /mnt/external/elasticsearch && rsync --info=progress2 -ah /var/lib/elasticsearch/ /mnt/external/elasticsearch
   • This will take