Transfer Files to and from the HPC Environment

Questions

• How do I upload files to the HPC environment?
• How do I download files from the HPC cluster to my computer?
• How do I access HPC files from my Mac or Windows computer?
• How do I connect to the HPC research space from my computer?
• How do I use SFTP to transfer files to the HPC environment?
• What is the Gluster scratch space on the HPC cluster?
• Where are my HPC files stored?
• How do I transfer files larger than 5 GB to the HPC environment?

Environment

This article applies to Bowdoin faculty, students, and researchers who need to transfer files between their local computer and the HPC environment. File access methods are available from macOS, Windows 11, and Linux. Accessing HPC file storage from off campus requires a VPN connection.

Resolution

Upload and Download Files via the HPC Web Portal

The simplest way to transfer files is through the HPC Web Portal file browser at hpcweb.bowdoin.edu. The file browser supports uploads up to 5 GB per file. See Use the HPC Web Portal (Open OnDemand) in the Related Articles section for detailed file browser instructions.

For files larger than 5 GB, use one of the methods described below.

Mount the HPC Research Space from macOS

You can mount the HPC research directory directly on your Mac to drag and drop files using the Finder. Your Mac must be on the campus network or connected via VPN.

1. In the Finder, choose Go > Connect to Server.
2. In the Server Address field, type smb://microwave/hpc-research/username where username is the Bowdoin account name of the faculty member who owns the research space (for example, smb://microwave/hpc-research/jsmith).
3. Click Connect.
4. If prompted to authenticate, enter your Bowdoin email username and password and click OK.
5. An HPC-Research icon will appear on your desktop, and a Finder window will open showing the contents of the research space.

Mount the HPC Research Space from Windows 11

1. Open the Start menu and choose Run (or press Windows + R).
2. In the Open field, type \\microwave\hpc-research\username where username is the Bowdoin account name of the faculty member who owns the research space (for example, \\microwave\hpc-research\jsmith).
3. Click OK.
4. A new window will open showing the contents of the research space.

Transfer Files via SFTP (Command Line)

On macOS or Linux, you can use the sftp command to transfer files directly to any HPC machine:

1. Open a terminal window.
2. Type sftp username@dover.bowdoin.edu where username is your Bowdoin account name.
3. Enter your Bowdoin password when prompted.
4. Use put filename to upload a file, or get filename to download a file.

Gluster Temporary Scratch Space

The HPC cluster includes a dedicated high-speed Gluster filesystem at /mnt/hpc/tmp for use as temporary scratch storage while jobs are running. Gluster performs faster than Microwave and can significantly improve job performance for I/O-intensive tasks.

The HPC cluster automatically creates a personal directory in Gluster at /mnt/hpc/tmp/username. To use Gluster scratch space for a job:

1. Copy your input files from your home directory or research space to your Gluster directory:
   cp myinputfile /mnt/hpc/tmp/username/
2. In your job script, reference files in /mnt/hpc/tmp/username/ instead of your home or research directory.
3. After your job completes, copy the results back:
   cp /mnt/hpc/tmp/username/myoutputfile ~/
4. Verify the copy was successful using ls -la before relying on the destination copy.

Additional Help

If you need further assistance, you have several options:

• Bowdoin Bot: Chat with Bowdoin Bot directly from any KB page for instant answers.
• Phone: Call the Bowdoin College Service Desk at (207) 725-3030.
• In person: Visit the Tech Hub in Smith Union during business hours.
• Submit a ticket: Request assistance through the Service Catalog.