Troubleshooting and FAQ

My XSEDE account and password formerly worked to login to https://use.jetstream-cloud.org, and it still works at https://portal.xsede.org but now I get a banner saying that my account does not have access.

1. Is your XSEDE account still a member of a project that includes a Jetstream allocation?
   
   

   1. Check https://portal.xsede.org/group/xup/allocations/usage to see if you're still a member

   2. If you are no longer a member, check with the Project PI about regaining access.

2. Did your XSEDE project recently expire but you need to restore access?

   1. If you are the PI, see Requesting additional SUs or storage for Jetstream -- Supplemental Allocations

3. Did your project recently renew or get a supplement?

   1. Occasionally, accounts fail to be transferred from the previous version of the Project's allocation to the new version of the allocation. In this case, email help@jetstream-cloud.org to create a support ticket. Be sure and include the Project code (example: TG-AAA000000) and your XSEDE account name to facilitate manually re-activating your account.

4. Have you in the past used compute resources at the Texas Advanced Computing Center (TACC) but haven't used them in more than 6 months?
   
   

   1. If you have a Deactivated , PendingMigration, or otherwise disabled account with the Texas Advanced Computing Center (TACC), you will need to reactivate that account by changing your password here:

      https://portal.tacc.utexas.edu/password-reset

5. Do you use extended character sets such as those commonly used for asian characters, rather than unicode or western sets?

   1. Try reseting your passwords while using unicode or western character sets.
      Asian characters sets have occasionally been found to confuse some of the authentication systems Jetstream uses.

My XSEDE password doesn't work for logging into my Atmosphere-launched VM

This is not a problem. That is by design.

Please read this page to solve the issue: My XSEDE password does not work for logging into my Atmosphere launched VM

If you get a connection refused after multiple failed login attempts, you may need to use the unban utility mentioned in the section below titled You get an ssh "connection refused" message using a terminal-based shell client or Putty

VM gets stuck trying to provision the network or in deploying.

or

VM has IP address and shows active/green but is unreachable

or

Previously behaving and unmodified VM develops problems.

Try these in this order:

1. Refresh: Refresh your browser and confirm that the Atmosphere interface has synchronized with the latest Jetstream information.

2. Wait: Please be aware that while provisioning is generally rapid, it can sometimes take several minutes during times of heavy activity. (wait 5-10 minutes)

3. Redeploy/Reboot: Click Projects(on the top menu) and select the Project with the instance that is stuck.
   Then click on the instance name that's stuck.
   From the Actions menu on the right try
   • Redeploy first (waiting about 5-10 minutes before proceeding),
   • Reboot if that fails (waiting about 5-10 minutes before going to the next step), and
   • Hard Reboot if the first two actions fail (waiting about 5-10 minutes).

4. If everything otherwise seems in order deploying but the instance seems to be delayed in getting an IP address or in showing the WebShell buttons, the instance may be updating system packages. You will have to wait until this finalizes, then redeploy. Jetstream staff regularly update FEATURED IMAGES to address this. For images not maintained by Jetstream you will need to contact the owner.

If none of these Actions work, DO NOT DELETE THE INSTANCE. Make note of the Alias (UID) and IP Address of the Instance, and email that information to help@jetstream-cloud.org or click the button.

If you have enabled a host-based firewall with ufw, iptables, or firewalld, you need to permit these IPs to ssh to your host on Atmosphere or deployment will fail.

Required IPs

Do I need to apply security updates to my VM?

It's always a good practice to apply periodic updates. A good recommendation is to check for updates once a week at least.

From the Jetstream Policies and Best Practices page:

The Ubuntu 18.04, 16.04 and Centos 7 featured images are piloting unattended security updates. Nodes will not reboot, but they will apply any update marked as a security update. It's still a good idea to update your VM, just in case.

• To apply the updates:
  CentOS: sudo yum update
  Ubuntu: sudo apt-get update then sudo apt-get upgrade

If the kernel or glibc/libc packages are being updated, rebooting is necessary to implement those updates

Always run updates before requesting a new custom image - An actively updating instance may be slow to deploy and may require a redeploy or even a reboot after updating in order to fully successfully deploy.

Hostname mismatch

If you’re having hostname mismatch issues with some software, try the following:

1. type:   su

2. type:   crontab -e

3. add this line:
   
   

   1 @reboot /bin/hostname | awk -F'.' '{print $1}' > /etc/hostname

   
   
   

4. save
   
   

Volume fails to detach

Volumes won't detach if they are actively being used. The unix umount will fail to protect you to prevent file corruption and/or to prevent you from accidentally stopping a process/job you didn't mean to kill. This, in turn, will prevent the Atmosphere detach from proceeding.

If your volume won't detach, try these steps:

1. make sure the instance to which the volume is attached is actively running

   

2. make sure no jobs or process are running on the instance are using the volume
   The command:

   1 fuser -vm /vol_b

   will show all processes using the volume named /vol_b

3. Kill those jobs (after verifying they don't need to be running):

   1 fuser -km /vol_b

   
   
   

Volume fails to attach

1. make sure the instance to which the volume should be attached is actively running

   

My instance has been locked by an administrator

1. Are you running an insecure web-service like Apache or MongoDB with no security features enabled ?

   1. MongoDB: https://docs.mongodb.com/manual/administration/security-checklist/

   2. Apache: https://httpd.apache.org/docs/2.4/misc/security_tips.html

2. Have you recently requested help with the instance? If so the administrator is likely working on solving your problem.

Web Desktop does not launch

Web Desktop should work for any Jetstream Atmosphere Featured Image. If it does not, you might try doing a redeploy on the instance.

If it's not a featured image, you might wish to try using the instructions for an external VNC viewer. 

API command line access to TACC fails with "requires authentication" error

Add the following two lines to the openrc.sh script if they are not there:

missing TACC openrc.sh env variables

btrfs in conjunction with OpenStack's ceph has known kernel bug

please use a different file system. e.g XFS, EXT4

Mouse does not work in web desktop

Laptops with touchscreens are unable to use the mouse in the web desktop using Chrome and Firefox on Windows 10.

There is a bug in the web browser code in both Chrome and Firefox for using a touchscreen PC with the web desktop of Jetstream. This is not a bug we can easily work around since it's in the browser code itself.

While we do not recommend using Internet Explorer, it does not have the issue. Using a VNC Viewer also is a viable workaround.

web-shell works for my Ubuntu 16.04 version or newer instance, but I cannot ssh to it directly from an external ssh client

• Have you added keys to your Atmosphere settings ?

• Are you using DSA ssh-keys? DSA keys are deprecated for Ubuntu 16.04 and newer instances. Please use RSA keys.

• Are you being asked for a password? Since password authentication is turned off by default, you may have a problem with your keys.

Users with a brace " { " or parenthesis "(" in their passwords may experience errors with OpenStack command line commands. 

• There is a known issue with braces in openstack passwords when using the OpenStack command line client. At this time, do not use a brace in your password.

• A potential work around is to enclose your password in single quotes in your .openrc

You get an ssh "connection refused" message using a terminal-based shell client or Putty

Probable cause: fail2ban has locked you out from vnc or terminal-based shell sessions

You can use the unban utility that is part of the Jetstream Atmosphere EZ-tools.

Use the web shell and do the following:

unban -l

will show you all currently banned IPs. Find your IP in the list and do:

unban -i your_ip_address

OLD Method:

• 1 sudo iptables -L -n

  • Look for your IP number in that output, if it's there, proceed

• 1 sudo fail2ban-client status

Should show you fail2ban is working and your jail name(s):

• 1 sudo fail2ban-client set ssh-iptables unbanip YOUR_IP_NUMBER

Jetstream can't be accessed from certain countries

Certain organizations and individuals are subject to trade sanctions, embargoes, and other restrictions under US law or by individual cloud provider policy. These restrictions apply to both domestic and foreign transactions.

• A list of countries may be found here along with a link to US government ITAR information.
  This list is subject to change based on US policy.

You get a "403 Forbidden"  or "403 Authentication credentials were not provided” error/screen trying to access the web shell or web desktop.

Your Globus credentials have become stale. Log out of Atmosphere by clicking on your username in the upper right hand corner of the Atmosphere interface and select "Sign Out" from the menu. Then log in again and try using the web shell or desktop.

I have an Ubuntu18-based instance that has been stuck deploying for quite some time.

Click on your instance and check the version:

• Based on Ubuntu 18.04 Devel and Docker v1.12

If the version is < 1.8, you will have a known bug that introduces a configuration conflict that prevents rebooting/redeployment.

All Ubuntu versions ≥1.8 have been patched to fix this networking bug.

If possible, please rebuild your instance with a more recent version, otherwise, please contact help@jetstream-cloud.org for assistance in applying the fix.

When I try to login after having selected XSEDE as my authorization provider, I get a page with the error message:

Unknown user or wrong password

• Try logging into https://portal.xsede.org 
  If you can login to the XSEDE portal, then it's likely that your username and password combination are correct.

• Are you treating your username and password as if they were case-sensitive?
  Best results will occur if you treat your credentials as if they were case-sensitive when using Jetstream, Globus, or XSEDE.

My CentOS based instance shows active but isn't reachable via ping or SSH or my CentOS based instance is hung up at "Active Networking" in the booting phase.

Following an upgrade rebuilding virtual networks is now taking multiple hours. In the best case scenario, the failover happens successfully and no VMs are affected. If a networking node is rebooted before all of the rebuild has completed, there may be a gap where dhcp requests are not being answered. Red Hat based OSes like CentOS stop attempting to get a dhcp address after a certain period.

Rebooting the instance should bring the instance back online.

I want to use Microsoft Windows on Jetstream

While technically possible, Jetstream presently does not support Microsoft Windows at all. It will likely not be integrated into Atmosphere during the life of the current Jetstream project. If you wish to explore it with the understanding that we cannot provide support beyond normal VM operations, you can refer to Microsoft Windows on Jetstream

I have created a custom image in the Atmosphere interface but it isn't deploying properly in the API interface. (or vice versa)

We cannot recommend strongly enough not using Atmosphere images on the API side of Jetstream or vice-versa.

The Atmosphere user interface is designed to be extremely easy to use, particularly for beginning users but the trade-off is that Atmosphere enforces certain practices behind the scenes.

The API user interface is close to raw Openstack, makes almost no assumptions, requires advanced knowledge of Openstack and Linux, but is therefore much more flexible.

API and Atmosphere base images provided by Jetstream staff require SUBSTANTIAL work to make them CLEANLY work in the other environment.

My Debian/Ubuntu instance gives an error when I update packages.

/etc/profile.d/cyverse10-utils.sh: Syntax error: "(" unexpected (expecting "}")

A package you are installing is forcing the dpkg package manager to use sh instead of bash. This potentially conflicts with various Jetstream configuration scripts. You will need to use :

       sudo dpkg-reconfigure dash

       and answer no

This will avoid the conflict.

I have a period "." in my instance name

Currently, we advise users to avoid placing periods "." anywhere in the names of their instances as this appears to be triggering Dynamic DNS features that then prevent a successful deployment.

Can I transfer ownership of a VM to another user?

The short answer is no. However, there are some methods for handling this. Please review Can I transfer/change ownership on a running VM to another user?

I'm unable to unshelve my m1.xxlarge or s1.xxlarge instance – it begins the process and reverts to shelved_offloaded.

I’m unable to launch an m1.xxlarge instance - it goes to an error status very shortly after launch.

§ Please note that new s1.* instances are no longer supported. Please use m1.* instances with attached volumes instead.

While there are a number of reasons this may happen, the primary reason is usually that there are no hypervisors available for this size instance. Xxlarge instances require an almost entirely empty node to run on. If there is more than a pair of m1.tiny instance or m1.small instances, there won't be enough room to spawn an xxlarge instance on that node. 

Because Jetstream has become a popular resource, it's become likely, especially on the IU cloud, that there is contention for getting entire nodes available for XXLarge instance usage. 

The only options when this happens is to launch a smaller instance size, launch an xxlarge on the other cloud, wait, or keep trying. You can contact support via help@xsede.org if you want to verify that this is the issue. 

You may want to benchmark your workflows and see if your code/workflow scales to need 44 cores and how efficiently it uses them. If you could use 24 cores, you'll generally have fewer issues shelving and unshelving. If you truly need 44 cores, you may encounter this from time to time - it's an unfortunate side effect of Jetstream's popularity.

My instance reverts to a shelved state without me shelving it.

1. Is the instance attached to a valid ACTIVE XSEDE allocation with service units still available?

   1. If the allocation is expired or out of services units, the instance will auto-shelve.

2. Was the allocation recently renewed, converted from STARTUP to RESEARCH, or otherwise updated at XSEDE?

   1. Instances can remain attached to the older, invalid incarnation of the allocation.
      In these cases you often see a “BLANK” allocation source drop-down menu.
      Try selecting a valid allocation from the drop-down menu, even if it means re-selecting the one you had before.

Both of these solutions are related to https://iujetstream.atlassian.net/wiki/spaces/JWT/pages/537296963

Obsolete Issues

Old Web-Shell (Gate-One) does not connect (via ssh) 

First, try the new Guacamole-based Web-Shell. 

If your Old Web-Shell / Gate-One stalls at a prompt like:

1. then close the session by clicking on the "X" in the top-right menu bar:

2. then close the browser window/tab.

3. Go back to your Jetstream Atmosphere window and re-click on the  link.