There are several different methods available to you for interacting with the object storage service. The following sections cover the most common tools that people use. Each of these examples shows some of the basic commands that you can use to create and edit your object storage containers.
When using the object storage service, your data must be stored in a container (also referred to as a bucket.) So our first step is to create at least one container prior to uploading any data. To create a new container, navigate to the “Containers” section on the dashboard and click “Create Container”.
You will need to provide a name for the container as well as select the appropriate access level and replication policy type before clicking “Submit”.
Warning
Please do not use an underscore “_” when naming your container. The
reasoning for this is explained in the programmatic examples below.
Note
Setting “Public” level access on a container means that anyone with the container’s URL can access the contents of that container.
You should now see the newly created container. As this is a new container, it currently does not contain any data. Click on the upload button next to “Folder” to add some content.
Click on the “Browse” button to select the file you wish to upload and once selected click “Upload File”.
In the containers view the Object Count has gone up to one and the size of the container is now 5 Bytes.
For several of the methods detailed below, you will have to prepare your command line environment before continuing with the examples. The key things that need to be prepared are:
You must Source an OpenRC file.
You must ensure that you have the correct role for using object storage on your cloud project. See here for more details.
Once you have met these requirements, you can continue with any of the method you below:
Warning
Please do not use an underscore “_” when naming your containers. This will
cause an error with the S3 api and you will receive errors through your CLI
when using this method. As a general practice we recommend avoiding
the use of an underscore regardless of the method you choose below.
The following is a list of the most commonly used commands that will help you interact with the object storage service via the openstack command line.
To view the current containers on your project, you can use the openstack
container list command:
$ openstack container list
mycontainer-1
mycontainer-2
To view the objects stored within a container:
openstack object list <container_name>
$ openstack object list mycontainer-1
+-------------+
| Name |
+-------------+
| file-1.txt |
| image-1.png |
+-------------+
To create a new container: openstack container create <container_name>
$ openstack container create mynewcontainer
+---------+----------------+----------------------------------------------------+
| account | container | x-trans-id |
+---------+----------------+----------------------------------------------------+
| v1 | mynewcontainer | tx000000000000000146531-0057bb8fc9-2836950-default |
+---------+----------------+----------------------------------------------------+
To add a new object to a container:
openstack object create <container_name> <file_name>
$ openstack object create mynewcontainer hello.txt
+-----------+----------------+----------------------------------+
| object | container | etag |
+-----------+----------------+----------------------------------+
| hello.txt | mynewcontainer | d41d8cd98f00b204xxxxxx98ecf8427e |
+-----------+----------------+----------------------------------+
To delete an object from a container: openstack object delete <container> <object>
$ openstack object delete mynewcontainer hello.txt
To delete a container: openstack container delete <container>
Note
this will only work if the container does not contain any objects.
$ openstack container delete mycontainer-1
To delete a container and all of the objects within the container:
openstack container delete --recursive <container>
$ openstack container delete --recursive mycontainer-1
For this section of the documentation, we will cover the basic features of using the Swift object storage API. For a more in depth understanding of the features that are offered via this API we recommend reading through the official OpenStack documentation
Region |
Version |
Endpoint |
|---|---|---|
nz-por-1 |
1 |
https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_%tenantid% |
3 |
||
nz-hlz-1 |
1 |
https://object-storage.nz-hlz-1.catalystcloud.io:443/v1/AUTH_%tenantid% |
3 |
Note
While our geo-replicated storage does backup to all three of our cloud regions (Porirua, Hamilton and Wellington) It is preferable that you interact with the service with either the Porirua or Hamilton region endpoints. Your data will still be replicated/stored in the Wellington region if that is the replication policy you have opted for even if you access it from either of the other regions.
Like the the other methods in this tutorial section, you will need to source an openRC file to interact with the object storage APIs. However, there is an additional environment variable that we must manually set to be able to interact correctly with the swift API. We need to create an OS_STORAGE_URL variable so that swift is able to correctly authenticate using your openstack credentials. We do this by taking one of the API endpoints from above, and adding our project_ID to it.
To get your project_ID, you can run the following command:
$ openstack project show <name of the project you sourced your OpenRC with>
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | 7xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54 |
| tags | [] |
+-------------+----------------------------------+
We then take the ID that we find from this output and we combine it with the Auth API from the region we want to operate in; in this case we are using the Porirua region. We then export this environment variable as “OS_STORAGE_URL” like so:
$ export OS_STORAGE_URL="https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_7xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54"
After you have your environment variables sourced, you will also need to install the standard client library for swift, which in this case is the python-swiftclient. You can add this library to your current Python environment using the code snippet below:
# Make sure you have pip and virtualenv installed
sudo apt-get install python-pip python-virtualenv
# Create a new virtual environment for Python and activate it
virtualenv venv
source venv/bin/activate
# Install Python Swiftclient library on your virtual environment
pip install python-swiftclient
The code below demonstrates how you can use the python-swiftclient to interact with your object storage containers while making use of the environment variables that we have already created. The following script will:
create a container on your project
add a file to the container
list all of your containers and their contents.
To use this file, save it as a ‘.py’ and run it from your command line.
#!/usr/bin/env python
import swiftclient
import os
token = os.environ['OS_TOKEN']
stourl = os.environ['OS_STORAGE_URL']
conn = swiftclient.Connection(
preauthtoken = token,
preauthurl = stourl,
insecure = False,
auth_version = 1,
)
# Create a new container
container_name = 'mycontainer'
conn.put_container(container_name)
# Put an object in it
conn.put_object(container_name, 'hello.txt',
contents='Hello World!',
content_type='text/plain')
# List all containers and objects
for container in conn.get_account()[1]:
cname = container['name']
print ("container\t{0}".format(cname))
for data in conn.get_container(cname)[1]:
print ('\t{0}\t{1}\t{2}'.format(data['name'], data['bytes'], data['last_modified']))
If you are using an username and password to authenticate with the Swift API rather than a token, you will need to make some changes to the start of the script above. Once these changes have been made you should be able to authenticate and perform the same actions using username and password authentication instead of token authentication.
Note
You may need to change or set some additional environment variables for the following code snippet to work. However, if you have
authenticate using the --no-token flag on your openRC file, these should already be set.
Replace the starting section of the previous file with the following:
#!/usr/bin/env python
import swiftclient
import os
# Read configuration from environment variables (openstack.rc)
auth_username = os.environ['OS_USERNAME']
auth_password = os.environ['OS_PASSWORD']
auth_url = os.environ['OS_AUTH_URL']
options = {
'tenant_name': os.environ['OS_PROJECT_NAME'],
'region_name': os.environ['OS_REGION_NAME'],
'user_domain_name': os.environ['OS_USER_DOMAIN_NAME'],
'project_domain_id': os.environ['OS_PROJECT_DOMAIN_ID']
}
# Establish the connection with the object storage API
conn = swiftclient.Connection(
authurl = auth_url,
user = auth_username,
key = auth_password,
insecure = False,
os_options = options,
auth_version = '3'
)
# ...You will then need to remove the previous piece of code that created a "conn=swiftclient.Connection" using the os_token variable.
The Swift object storage service has an Amazon S3 emulation layer that supports common S3 calls and operations.
See also
Swift3 middleware emulates the S3 REST API on top of OpenStack. Swift is documented fully here.
Region |
Endpoint |
|---|---|
nz-por-1 |
|
nz_wlg_2 |
|
nz-hlz-1 |
You need valid EC2 credentials in order to interact with the S3 compatible API. You can obtain your EC2 credentials from the dashboard (under Access & Security, API Access), or by using the command line tools:
$ openstack ec2 credentials create
If you are using planning to use boto to interact with the API, you will need boto installed on your current Python environment. The example below illustrates how to install boto on a virtual environment:
# Make sure you have pip and virtualenv installed
sudo apt-get install python-pip python-virtualenv
# Create a new virtual environment for Python and activate it
virtualenv venv
source venv/bin/activate
# Install Amazon's boto library on your virtual environment
pip install boto
The code below is an example of a file using boto to interact with the S3 compatible API.
#!/usr/bin/env python
import boto
import boto.s3.connection
access_key = 'fffff8888fffff888ffff'
secret = 'bbbb5555bbbb5555bbbb555'
api_endpoint = 'object-storage.nz-por-1.catalystcloud.io'
port = 443
mybucket = 'mytestbucket'
conn = boto.connect_s3(aws_access_key_id=access_key,
aws_secret_access_key=secret,
host=api_endpoint, port=port,
calling_format=boto.s3.connection.OrdinaryCallingFormat())
# Create new bucket if not already existing
bucket = conn.lookup(mybucket)
if bucket is None:
bucket = conn.create_bucket(mybucket)
# Store hello world file in it
key = bucket.new_key('hello.txt')
key.set_contents_from_string('Hello World!')
# List all files in test bucket
for key in bucket.list():
print (key.name)
# List all buckets
for bucket in conn.get_all_buckets():
print ("{name}\t{created}".format(
name = bucket.name,
created = bucket.creation_date,
))
To access object storage using cURL it is necessary to provide credentials to authenticate any requests you make.
This can be done by sourcing your OpenRC file and retrieving your account specific details via the Swift command line tools; then exporting the required variables as shown below.
$ source openstack-openrc.sh
# we then need to create a OS_STORAGE_URL environment variable.
# To create this variable we will need to find our project ID:
$ openstack project show <name of the project you sourced your OpenRC with>
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | 1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54 |
| tags | [] |
+-------------+----------------------------------+
We then export this ID with the storage API for the region we are working in. For this example we will use the Porirua region:
$ export OS_STORAGE_URL="https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54
# Then we grab our auth token for later use:
$ swift stat -v
StorageURL: https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54
Auth Token: 5f5a043e1bd24a8fa8xxxxxxcca8e0fc
Containers: 48
Account: AUTH_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Objects: 156
Bytes: 11293750551
Containers in policy "policy-0": 48
Objects in policy "policy-0": 156
Bytes in policy "policy-0": 11293750551
X-Account-Project-Domain-Id: default
Server: nginx/1.8.1
X-Timestamp: 1466047859.45584
X-Trans-Id: tx4bdb5d859f8c47f18b44d-00578c0e63
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes
$ export token="5f5a043e1bd24a8fa8xxxxxxcca8e0fc"
To create a new container, use the following cURL request:
curl -i -X PUT -H "X-Auth-Token: $token" $storageURL/mycontainer
Then run the following command to get a list of all available containers for your project:
curl -i -X GET -H "X-Auth-Token: $token" $storageURL
You can optionally specify alternative output formats. For example: to have XML or JSON returned use the following syntax:
curl -i -X GET -H "X-Auth-Token: $token" $storageURL?format=xml
curl -i -X GET -H "X-Auth-Token: $token" $storageURL?format=json
To view the objects within a container, simply append the container name to the cURL request:
curl -i -X GET -H "X-Auth-Token: $token" $storageURL/mycontainer
To upload a file to your container, use the following cURL format:
curl -i -T <my_object> -X PUT -H "X-Auth-Token: $token" $storageURL/mycontainer
To delete a file from your container, use this code:
curl -X DELETE -H "X-Auth-Token: <token>" <storage URL>/mycontainer/myobject
Finally, to delete a container you can use the following syntax.
Note
A container must be empty before you try and delete it. Otherwise the operation will fail.
curl -X DELETE -H "X-Auth-Token: <token>" <storage URL>/mycontainer
Using Heat you are able to manage a large number of resources programmatically, by utilizing a stack that will construct and monitor your resources for you. You can create a stack by using a pre-designed template. The following example assumes that you have some knowledge of Heat and how to use these template files.
The following code snippet contains the minimum required components to construct an object storage container using Heat:
heat_template_version: 2015-04-30
description: >
Creating a swift container using HEAT
resources:
swift_container:
type: OS::Swift::Container
properties:
PurgeOnDelete: FALSE
name: heat-container
For more information on object storage containers and what customization options you can select for them, please see the Openstack Heat documentation.
Once you have your template constructed, you should make sure to validate it before creating any resources. You can do this by running the following code:
$ openstack orchestration template validate -t <your-template-name>
If your template is constructed correctly then the output of this command should return a copy of your template. If there is an error inside your template, you will instead be notified of the error in the output.
Once you have ensured your template is valid, you can construct your stack:
$ openstack stack create -t <template> <stack-name>
The stack_status indicates that creation is in progress. Use the
event list command to check on the stack’s orchestration progress:
$ openstack stack event list <stack-name>
2020-11-09 22:53:56Z [container-stack]: CREATE_IN_PROGRESS Stack CREATE started
2020-11-09 22:53:57Z [container-stack.swift_container]: CREATE_IN_PROGRESS state changed
2020-11-09 22:54:01Z [container-stack.swift_container]: CREATE_COMPLETE state changed
2020-11-09 22:54:01Z [container-stack]: CREATE_COMPLETE Stack CREATE completed successfully
Once your status has reached CREATE_COMPLETE you should be able to see the resources on your project.
This tutorial assumes that you have some familiarity with Terraform. The minimum knowledge this tutorial assumes is an understanding of how Terraform scripts are written and how they function. We also assume that you have installed all of the prerequisite tools to run Terraform scripts.
Below is an example template that contains the basic information required for Terraform to create an object storage container on the cloud. You can view the full list of customization options for this resource on the Terraform documentation
provider "openstack" {
}
resource "openstack_objectstorage_container_v1" "container_1" {
name = "tf-test-container-1"
metadata = {
test = "true"
}
content_type = "application/json"
}
Once you have saved this script you will need to switch to the correct directory and run the following commands to create your new object storage container. The first command will outline what resources are going to be made and managed by Terraform and what their outputs will be:
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# openstack_objectstorage_container_v1.container_1 will be created
+ resource "openstack_objectstorage_container_v1" "container_1" {
+ content_type = "application/json"
+ force_destroy = false
+ id = (known after apply)
+ metadata = {
+ "test" = "true"
}
+ name = "tf-test-container-1"
+ region = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
After you review the Terraform plan and ensure that it has all the resources you want to be created, you can use the following code to create your new objects:
$ terraform apply
... #truncated for brevity
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
openstack_objectstorage_container_v1.container_1: Creating...
openstack_objectstorage_container_v1.container_1: Creation complete after 5s [id=tf-test-container-1]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Once you have reached this step, you should have an object storage container created and managed by Terraform. If you want to delete this container in the future, as well as any other resources created in your plan, you can use the following code to delete them:
$ terraform destroy
... # truncated for brevity
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
openstack_objectstorage_container_v1.container_1: Destroying... [id=tf-test-container-1]
openstack_objectstorage_container_v1.container_1: Destruction complete after 1s
Destroy complete! Resources: 1 destroyed.