Advanced Computing Center for Research and Education - ACCRE at Vanderbilt

Enjoy the nice weather today!

Sai Medury joins ACCRE as Associate System Administrator: https://buff.ly/3qnqkI7

Access to /scratch has been restored.

/scratch restored following outage this morning Update, 11/1/2021 3pm: The /scratch storage sub-system has been remounted across the cluster and the public gateways. There are a few custom gateways that will need to be rebooted and that will be coordinated with the respective groups. One of the components for the /scratch storage sub-system enter...

One of the components for the /scratch storage sub-system entered a bad state over the weekend. Our sysadmins were able to force it into a good state only to have it reoccur. The result is that some /scratch users have experienced intermittent issues when accessing files.
We plan on rebooting the component in order to clear the issue. To make sure this doesn't impact other parts of the system, we will be taking /scratch offline for 1 hour at 11am this morning.

The cluster will be be available for normal use at 10:30am this morning. The system remained stable and error free over the weekend. We were also able to catch up on tape backup operations.

https://buff.ly/3lk6kn1

During the last 70 mins we've detected a series of system incidents that
indicate the unavailability of /data and /home in the cluster
environment.

Storage issues update:
- access to /data and /home restored
- Slurm to restart at 2pm CT
- some files to be restored from tape backup
- a list of temporarily unavailable files is on the cluster
Details:

Storage issues: access to /data and /home restored; Slurm to restart at 2pm; some files will be restored from tape backup Update, 10/8/2021: After consultation with the hardware vendor, it was determined that recovery of the third and final disk group would take longer than restoring the affected files from tape backup and may not be guaranteed to succeed. Therefore, we will proceed with data recovery from this disk gr...

We continue to work on resolving the storage outage affecting /data and /home. Latest updates here:

Bitly | Forbidden | 403 This is a 403 error, and it's not as ominous as it sounds. Bitly can only show this page to people who have permission to see it. Maybe what you are looking for can be found at Bitly.com.

https://buff.ly/3odvG82

We have finished all the work within scope of this scheduled downtime and successfully completed all the system tests. All ACCRE systems are now available. You can monitor system availability here and please report any odd
behaviors via our helpdesk.

A final reminder about our upcoming downtime starting tomorrow: https://buff.ly/38hLY6i

Reminder about our upcoming downtime this Friday and Saturday: https://buff.ly/38hLY6i