Create an s3 storage gateway

 

1. Install Globus Connect Server

The Globus Connect Server needs to be installed on your Jetstream instance. When you’ve logged into your instance you can execute the commands below. These commands will install Globus Connect Server version 5.4 on your CentOS instance.

1 2 3 4 5 sudo yum install http://downloads.globus.org/toolkit/gt6/stable/installers/repo/rpm/globus-toolkit-repo-latest.noarch.rpm sudo yum-config-manager --enable Globus-Connect-Server-5-Stable sudo yum-config-manager --enable Globus-Toolkit-6-Stable sudo yum install yum-plugin-priorities sudo yum install globus-connect-server54

2. Create the endpoint

The next step is to establish the endpoint on your system. The “endpoint” is Globus’s name for the installation of Globus.

2.1. Create service credentials

The first step in establishing your endpoint is to register it with Globus and obtain credentials for the server itself. These credentials allow the endpoint to securely identify itself to—and interact with Globus services.

2.1.1 Open the Globus Developers Console in a web browser.

2.1.2. Click “Register a new Globus Connect Server v5.”

2.1.3. You’ll need to login. Choose “XSEDE” for the organization to authenticate with.

 

 

2.1.4. If this is your first time using the Developers Console create a new project.
Otherwise, click “Add another project.”

 

2.1.5. Fill out the form to create your project. Use a name you’ll remember, like “Jetstream S3 Access.”

2.1.6. From the “Add…” menu for the project click “Add a new Globus Connect Server.”

2.1.7. Fill out the form. The display name will be used to identify this endpoint to users when they access
it for the first time.

 

2.1.8. Click “Generate a New Client Secret” and fill out the form.

2.1.9. Save the Client ID and Client Secret values. You will need them in the next step.

 

 

2.2. Configure the endpoint

To finish creating the endpoint, run the globus-connect-server endpoint setup command. This command creates an endpoint in the Globus Transfer service.

The globus-connect-server endpoint setup command takes a number of command-line options. The following are required to create an endpoint: Display name - Name for the endpoint; organization - organization operating the endpoint; client-id - the client ID generated in the previous step.

1 2 3 4 sudo globus-connect-server endpoint setup "Ssudarsh's Jetstream S3 Access" \ --organization "Indiana University" \ --client-id "7d3dcf7c-0059-4517-b5be-ca924bb8c14e" \ --owner "ssudarsh@iu.edu"

The command will ask for the Client Secret you generated in the previous step. This command will take a few minutes to complete.

2.3. Start the Server

Run the globus-connect-server node setup command to configure and start the Globus services. This command must be run as the root user. You will need to provide your Client Secret again.

1 2 3 sudo globus-connect-server node setup \ --client-id 7d3dcf7c-0059-4517-b5be-ca924bb8c14e \ --ip-address your_ip_address

2.4. Log into the endpoint

This command will print out a URL to follow to authenticate with Globus, and return an access code that needs to pasted into the command-line tool.

1 globus-connect-server login localhost

2.5. View endpoint configuration

1 globus-connect-server endpoint show

2.6. Set the endpoint as managed

In order to use the S3 feature, your software needs to be added to Jetstream’s Globus subscription. This may only be done by a member of the Jetstream staff, so send a support request.

To make this request, email help@xsede.org with the subject “Requesting Jetstream Globus s3 access for the <XYZ PROJECT>”. The project should be your Jetstream project ID. In your email, be sure to include the UUID of your endpoint. After you send this email, you’ll have to wait for a response from the staff.

2.7. Create s3 storage gateway

Note: If your endpoint is not managed by the Jetstream Globus subscription see 2.6 above.

This step requires one or more existing buckets. You can create buckets on Openstack Horizon or Openstack CLI. (Openstack Horizon calls buckets “containers”)

1 2 3 4 5 6 7 8 9 10 globus-connect-server storage-gateway create s3 \ "Ssudarsh's Jetstream S3 Access" \ --domain xsede.org \ --s3-user-credential \ --s3-endpoint https://iu.jetstream-cloud.org:8080 \ --bucket research-data-bucket1 --bucket research-data-bucket2 Storage Gateway ID: ea79b37e-681f-4d42-969d-38b9a916f5f3