The purpose of this section is to provide examples on how you can use different tools to create a backup of your data on the Catalyst Cloud.
Before you continue with the examples below, there are a few assumptions that are made which you will need to consider before jumping in further:
You are familiar with the Linux command line and the Openstack CLI tools.
You have installed the OpenStack command line tools and sourced an openrc file, as explained in this section of the documentation.
While both of the methods we describe in this tutorial will create a backup of your data, there are some differences between them in how your backup is created, stored and maintained.
When using Openstack to create your backup, a point in time snapshot of your data is created. From this snapshot, Openstack creates a volume; this is your backup volume. A copy of this volume is then placed in object storage in the back end for redundancy.
Because this backup is created using a point in time snapshot, it is a ‘crash consistent’ solution that is able to restore your data to the specific point in time the original backup was taken. It does not ensure that the data is in a particular state after the restore.
Duplicity on the other hand, creates a backup that will make multiple attempts to capture a given file if it has been changed. This is due to Duplicity being a file oriented backup rather than a point in time backup. As mentioned earlier, being a file oriented backup, Duplicity allows you to perform file-level restoration from your backup should you need to.
Depending on the type of backup you want to create and the solution that best suits your situation, you may decide to use one method or another. Generally the basic Openstack backup solution is easier for maintaining a crash consistent copy of your data somewhere ready to restore while Duplicity is able to perform a more rigorous capture of the state of your files.
To create a backup using the openstack command line tools, we first need to find the original volume we are trying to back up. To show a list of the currently available volumes, you can use the following command:
$ openstack volume list
+--------------------------------------+---------------------+-----------+------+-----------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+---------------------+-----------+------+-----------------+
| 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee | backup-vol-original | available | 5 | |
+--------------------------------------+---------------------+-----------+------+-----------------+
# we can then export our volume ID for later use:
$ export volumeID='81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee'
Once we have the volume that we want to back up we can construct our backup command. The syntax for creating a backup using the openstack CLI is:
$ openstack volume backup create [--incremental] [--force] <VOLUME>
Where <VOLUME> is the name or ID of the original volume you wish to back up.
Because we exported our volume ID earlier we can use the following to create our
initial backup:
$ openstack volume backup create $volumeID
Now that we have our backup created, we can view it with the following commands.
Note
If your volumes is currently attached to a running instance the default backup
command will fail. However, you can still create a volume backup using the
--force parameter. This will allow you to create a backup even if your
original volume is in the in-use state.
$ openstack volume backup list
+--------------------------------------+------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+-----------+------+
| 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c | None | None | available | 5 |
+--------------------------------------+------+-------------+-----------+------+
# Once we have our backup ID we can view more information about it like so:
$ openstack volume backup show 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| availability_zone | nz-por-1a |
| container | volumes_backup_nz-por-1 |
| created_at | 2021-09-20T04:13:08.000000 |
| data_timestamp | 2021-09-20T04:13:08.000000 |
| description | None |
| fail_reason | None |
| has_dependent_backups | False |
| id | 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c |
| is_incremental | False |
| name | None |
| object_count | 104 |
| size | 5 |
| snapshot_id | None |
| status | available |
| updated_at | 2021-09-20T04:14:59.000000 |
| volume_id | 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee |
+-----------------------+--------------------------------------+
For future backups of the original volume we can
make use of the incremental optional argument. Instead of creating an
entirely new backup, the incremental argument will create a snapshot with the
differences between our current backup volume and the updated state of the
original volume.
# When we create our incremental backup we still use the ID of the original volume that we are backing up.
$ openstack volume backup create --incremental $volumeID
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | cbbefa42-XXXX-XXXX-XXXX-XXXXXXX36f00 |
| name | None |
+-------+--------------------------------------+
# Now if we take a look at our first backup we will see that the `has_dependant_volume` property is set to True:
$ openstack volume backup show 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| availability_zone | nz-por-1a |
| container | volumes_backup_nz-por-1 |
| created_at | 2021-09-20T04:13:08.000000 |
| data_timestamp | 2021-09-20T04:13:08.000000 |
| description | None |
| fail_reason | None |
| has_dependent_backups | True |
| id | 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c |
| is_incremental | False |
| name | None |
| object_count | 104 |
| size | 5 |
| snapshot_id | None |
| status | available |
| updated_at | 2021-10-04T00:33:04.000000 |
| volume_id | 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee |
+-----------------------+--------------------------------------+
$ export first_backup="376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c"
# If we then take a look at our new backup, we will see that it has the 'is_incremental' property set to True:
$ openstack volume backup show cbbefa42-XXXX-XXXX-XXXX-XXXXXXX36f00
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| availability_zone | nz-por-1a |
| container | volumes_backup_nz-por-1 |
| created_at | 2021-10-04T00:32:22.000000 |
| data_timestamp | 2021-10-04T00:32:22.000000 |
| description | None |
| fail_reason | None |
| has_dependent_backups | False |
| id | cbbefa42-XXXX-XXXX-XXXX-XXXXXXX36f00 |
| is_incremental | True |
| name | None |
| object_count | 1 |
| size | 5 |
| snapshot_id | None |
| status | available |
| updated_at | 2021-10-04T00:33:04.000000 |
| volume_id | 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee |
+-----------------------+--------------------------------------+
$ export second_backup="cbbefa42-XXXX-XXXX-XXXX-XXXXXXX36f00"
The last thing that we need to cover is how to restore your volume using one of these backups. The syntax for the restore command is as follows:
$ openstack volume backup restore <BACKUP_ID> <VOLUME_ID>
Depending on which backup we want to use, Openstack will perform different actions when restoring our volume.
If we choose our original backup to restore from, then Openstack will perform a full restore of our backup. This will restore the volume to the point in time our original backup was created.:
$ openstack volume backup restore $first_backup $volume_ID
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| backup_id | 376a741c-XXXX-XXXX-XXXX-XXXXXXX7881c |
| volume_id | 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee |
| volume_name | backup-vol-original |
+-------------+--------------------------------------+
If we choose to restore from our incremental backup, then Openstack will first organize a list of the backups we have made starting with the initial backup. It will then perform a full restore starting from that backup and then layer on the additional backups afterward:
$ openstack volume backup restore $second_backup $volume_ID
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| backup_id | cbbefa42-XXXX-XXXX-XXXX-XXXXXXX36f00 |
| volume_id | 81599985-XXXX-XXXX-XXXX-XXXXXXXXXXee |
| volume_name | backup-vol-original |
+-------------+--------------------------------------+
Duplicity is a band-width efficient backup utility capable of providing encrypted, digitally signed, versioned, remote backups in a space efficient manner.
Duplicity creates an initial archive that is a full backup. All subsequent backups are incremental and only save the difference between the latest (full or incremental) backup. A full backup and corresponding series of incremental backups can be recovered to any point in time covered by the incremental backups. If an incremental backup is missing from the backup chain then any subsequent incremental backup file cannot be recovered.
Duplicity is released under the terms of the GNU General Public License (GPL), and as such is free software.
If you’re using a major Linux distribution, you should be able to find a pre-compiled package in the repositories. If not, then a tar file is available at Duplicity.
sudo apt-get update
sudo apt-get install duplicity
Because we are going to authenticate against keystone, it is also necessary to
install python-keystoneclient.
sudo apt-get install python-keystoneclient
or
pip install python-keystoneclient
If you intend to create encrypted backups you will also require a GPG key. The
gpg --gen-key command line tool can create a local one for you, see
(GnuPG) for more information on this.
Duplicity requires certain environment variables to be set. One option would be to source a simple bash script like this. The data for these variables can be obtained from your OpenStack RC file.
#!/bin/bash
# Swift credentials for Duplicity
export SWIFT_USERNAME="somebody@example.com"
export SWIFT_TENANTNAME="mycloudtenant"
export SWIFT_AUTHURL="https://api.cloud.catalyst.net.nz:5000"
export SWIFT_AUTHVERSION="3"
export SWIFT_USER_DOMAIN_NAME="default"
export SWIFT_PROJECT_DOMAIN_NAME="default"
# With Keystone you pass the keystone password.
echo "Please enter your OpenStack Password: "
read -sr PASSWORD_INPUT
export SWIFT_PASSWORD=$PASSWORD_INPUT
In order to source this file, run the following from the command line
source <filename.sh>
This will need be done before each Duplicity run if the variables are not already set.
Firstly, lets check our connectivity to the object store. If we run the following for an existing empty container, in this case ‘first-container’, we should see something like this
$ duplicity collection-status swift://first-container
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /home/ubuntu/.cache/duplicity/cd3fc2f113a80b76b6xxxxxx7b16aee5
Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
Now we can run our first backup. For this example we will use a single local file called foo.sh.
Note
if you do not have a valid gpg key you will need to append --no-encryption
to the end of your duplicity commands.
$ duplicity foo.sh swift://first-container
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1484012914.11 (Tue Jan 10 01:48:34 2017)
EndTime 1484012914.11 (Tue Jan 10 01:48:34 2017)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 1
SourceFileSize 44 (44 bytes)
NewFiles 1
NewFileSize 44 (44 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 44 (44 bytes)
TotalDestinationSizeChange 231 (231 bytes)
Errors 0
-------------------------------------------------
We can verify the state of our backups with:
$ duplicity collection-status swift://first-container
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Jan 10 01:48:25 2017
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /home/ubuntu/.cache/duplicity/cd3fc2f113a80b76b6xxxxxx7b16aee5
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Tue Jan 10 01:48:25 2017
Chain end time: Tue Jan 10 01:48:25 2017
Number of contained backup sets: 1
Total number of contained volumes: 1
Type of backup set: Time: Num volumes:
Full Tue Jan 10 01:48:25 2017 1
-------------------------
No orphaned or incomplete backup sets found.
and check to see if there are local files that have not yet been backed up by running
duplicity verify swift://first-container .
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Jan 10 01:48:25 2017
GnuPG passphrase for decryption:
Verify complete: 595 files compared, 0 differences found.
Warning
If you wish to back up the root ‘/’ directory, it is advisable to add
--exclude /proc as this may cause Duplicity to crash on the weird stuff
in there.