MGCF FAQ

1) I accidentally deleted a file. Can I get it back?

2) How do I capture an image from the screen?

3) How do I change my password?

4a) How do I connect from my computer running Windows ?

4b) How do I connect from my computer running Mac OSX?

4c) How do I connect from my off campus computer?

5) How do I run jobs on dino? Part I - What is the queue?

6) How do I run jobs on dino? Part II - Common SGE commands?

7) How do I run jobs on dino? Part III - Simple SGE examples.

8) How do I compile code on dino?

9) How do I run OpenMPI code on dino?

10) How do I run QChem on dino?

11) How do I run Gaussian calculations on dino?

12) How do I run DMol on dino?

13) How do I run interactive jobs on dino?

14) How do I display Gaussian output in Molekel or gOpenMol or VMD?

15) How do I make ESP/Density plots from Jaguar or Gaussian?

16) How do I use Windows software within the MGCF?


1) I accidentally deleted a file. Can I get it back?

To restore most files, you have several choices: a daily mirror, weekly mirrors, and offline mirrors.

Daily Mirror: Your files are mirrored (additions, changes and deletions!) daily at 4am. This is an almost complete copy of your home directory as it was at the time of the backup. If you just deleted a file, go to the daily mirror copy which is in a directory called /storage/export on the machine called barney.

ssh barney
cd /storage/export/home/$user
Use ls and cd to find your filename and type
cp filename ~
This will copy your filename to your home directory (that's what the ~ means).

Weekly Mirrors: There are also mirrors made once per week: Friday at 10pm on slate and Wednesday 10pm on bambam. If you failed to recover your file in time for the daily mirror, then perhaps it is still on one the weekly mirrors. Just ssh slate or bambam and check in /storage/export/home/$user and the sub folders therein.

Offline Mirrors: There are other backups that are not directly accessible for users. However if you need files from a given date, please email Kathy and she will try to restore them. Specify file and folder names along with dates.

We do not archive files. Old files are stored for a few months and then we overwrite.

Note: Our backups disregard files that are >15GB in size and *.rwf files since those are usually temporary scratch files that clog the backup process. WAVECAR files of any size are excluded from the nightly backups but those that are < 15GB in size are on the biweekly backups.


2) How do I capture an image from the screen?

***Make sure the target window or area is not obscured by any other window or image.***

The import command is part of the ImageMagick suite of tools. You use it by typing import filename.jpg followed by the Enter key.
Then either click once on the window of interest or click-drag to specify a rectangle. You will hear a beep and the file called filename.jpg is created.

Type display filename.jpg to open the editing tool of ImageMagick. Clicking anywhere in the ImageMagick window will then bring up a control panel which can be used to edit and resave the image.

Import can save the image in different formats (gif, tiff, jpg, png) if you specify the desired format at the import step. For example: import filename.tiff

The ImageMagick suite of tools is extensive, powerful and scriptable. Check it out for more sophisticated image editing needs.

3) How do I change my password?

ssh to fred and type passwd
Changes will be propagated to other machines in a few hours.

4a) How do I connect from a computer running Windows?

We recommend Mobaxterm.

Choose the Free, Personal version. Then install as usual (the Installer edition is better for most people). Make sure you allow Mabaxterm to get through the Windows firewall. Start a new Session. Choose SSH. Check the box for "Specify username" and type in your MGCF username. For "Remote Host" you can specify any facility machinename.cchem.berkeley.edu
replacing "machinename" with wilma, stone, slate, barney, betty, lava, gravel, rubble, bambam, bronto, fred, energy or nano. More details about the workstations are here.

Give your MGCF password at the prompt, and you will access your home directory just like you would in the facilty.

4b) How do I connect from a computer running Mac OSX?

You must first install XQuartz.

Once XQuartz is installed, open the Terminal program (Applications -> Utilities). In your Terminal, type:

ssh -Y username@machinename.cchem.berkeley.edu

where "username" is your MGCF username and "machinename" is one of the workstations listed here. If you are presented with a message from ssh asking whether you want to continue logging in, type yes. Give your MGCF password at the prompt, and you will get a command prompt in your MGCF home directory just like you get when you login in 175 Tan Hall. You can use most graphical programs such as gedit, Gaussview and Maestro, but they will probably run more slowly than when you are in 175 Tan Hall.

Note: Try connecting using the -X option instead of -Y if you are having trouble opening programs.

Gaussview note:On some MacOS versions, the GL libraries are incompatible. A work-around is to type setenv USE_MESAGL immediately before starting GaussView (the gv command).

File transfers: MacFusion is a nice tool. It mounts remote servers as local drives within finder allowing normal drag an drop operation. Connect to "machinename.cchem.berkeley.edu" (see section above). Use your MGCF username and the ssh protocol ("22" is the Port).

4c) How do I connect from my off campus computer?

We only accept connections from berkeley.edu and lbl.gov networks. If you are off campus, you need to start a VPN session before starting your ssh, PuTTY, fugu or filezilla connection.

5) How do I run jobs on dino? Part I - What is the queue?

Our server is named dino. This it is really just the name of a single master machine that manages a cluster of other machines. The cluster is a series of individual machines, each of which has dual 6-core AMD-64 Istanbul processors. There are 48 machines (aka compute nodes) in the cluster for a total of 576 cores (job slots).

When you submit a calculation to dino, you are actually asking the master machine (also called the head node) to assign one or more of the compute nodes to perform the calculation.

dino = head node

compute-0-0  = compute node (2 processors, 12 cores)
compute-0-1  = compute node (2 processors, 12 cores) 
compute-0-2  = compute node (2 processors, 12 cores)
compute-0-3  = compute node (2 processors, 12 cores)
 .........
compute-0-47 = compute node (2 processors, 12 cores)

Each node formally has 48GB of shared memory, or about 4GB per core. The nodes are connected via Infiniband Gigabit Ethernet.

We use a queueing system called SGE or Grid Engine to fairly allocate cpu time to users. The head node controls the queue. When you run a job in the queue, the head node locates free compute nodes, or free cores in partially-used compute nodes. No jobs should be run directly on the head node. All jobs on "dino" should actually be run via the queue which will utilize the compute nodes fairly.

There are two handy commands:
machine:/home/username 32 % qstat

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  61934 0.48760 C2H6disPt3 vlad         r     05/06/2011 19:35:41 all.q@compute-0-33.local          12        
  64195 0.45617 job10.x    nvoul        r     05/16/2011 12:58:41 all.q@compute-0-0.local            1        
  64198 0.45617 job5.x     nvoul        r     05/16/2011 12:58:41 all.q@compute-0-0.local            1        
  64199 0.45617 job10.x    nvoul        r     05/16/2011 12:58:56 all.q@compute-0-1.local            1        
  64200 0.45617 job15.x    nvoul        r     05/16/2011 12:58:56 all.q@compute-0-1.local            1        
  64202 0.45617 job5.x     nvoul        r     05/16/2011 12:58:56 all.q@compute-0-1.local            1        
  64203 0.45617 job10.x    nvoul        r     05/16/2011 12:58:56 all.q@compute-0-21.local           1        
  64206 0.45617 job5.x     nvoul        r     05/16/2011 12:58:56 all.q@compute-0-2.local            1        
  65244 0.48760 v-cell25   bean         r     05/25/2011 22:45:26 all.q@compute-0-30.local          12        
  65336 0.46474 g-PMDA_HMB fujisawa     r     05/26/2011 15:30:26 all.q@compute-0-4.local            4        
  65337 0.46474 g-PMDA_HMB fujisawa     r     05/26/2011 17:04:41 all.q@compute-0-2.local            4        
  65555 0.47046 g-xss8-631 shieh        r     05/29/2011 18:59:11 all.q@compute-0-18.local           6        
  65691 0.55617 VOxGas4    vlad         r     06/01/2011 00:01:56 all.q@compute-0-8.local           36        
  65705 0.48760 Dim_Pt3Sn  vlad         r     06/01/2011 00:05:26 all.q@compute-0-15.local          12        

Another handy command to show the current load and available processors:
machine:/home/username 33 % qstat -g c
or more easily
machine:/home/username 33 % avail
CLUSTER QUEUE                   CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE  
--------------------------------------------------------------------------------
all.q                             1.51    572      0      4    576    156      0 
The command "avail_more" also lists the partially-filled nodes.

6) How do I run jobs on dino? Part II - What are common SGE commands?

qsub

Submit a batch job to SGE. This command requires a jobscript as in the next example. Other flags can be included.

qsub jobscript

Submit jobscript to SGE. The jobscript contains all of the commands required to make the job run including resource requests like node number and type.

qsub -q queue

Submit job directly to specified queue.

qlogin

Connect to a compute node for an interactive session. This is primarily for debugging jobscripts but can be used for compiling and other activities which require user interaction. You must have a terminal session with an open connection to dino in order to use this command.

qstat

List information about queues and jobs (the basic qstat is not too helpful we set an alias for "qstat -u *").

qstat -f
(or qstat nodes)

List full information queues and jobs.

qstat -r

List full job name and time limits.

qstat -u username
(or qstat username)

List information about jobs for username.

qstat -g c
(or qstat avail
or avail or
avail_more)

List summary information about used and available cpus.

qstat -j jobnumber

List relevant data for that job, directory of input file, etc.

qdel jobnumber

Delete specified jobnumber.

7) How do I run jobs on dino? Part III - Show me simple SGE examples.

Here's a trivial script which can be run in SGE.

# test script for SGE

echo "hello world" 
echo "I'm done!" 
If we put these lines into a file called sge_test.job then we can submit to SGE with the following command:

qsub sge_test.job

The output from this job is put into a file called sge_test.job.o### (where ### is the jobid in the queue). Standard error from this job is put into a file called sge_test.job.e### (this is the same jobid).

In this example the job id was 16 so the files were sge_test.job.e16 and sge_test.job.o16. sge_test.job.e16 was empty and the contents of sge_test.job.o16 look like this:

machine:/home/username 118 % more sge_test.job.o16
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
hello world
I'm done!

The first two lines are normal shell data and the latter two lines are the output.

Just about any valid command line can be put into a script and submitted to SGE in this fashion.

Here's an example showing a more typical jobscript file:

#!/bin/tcsh -f
#$ -cwd
#$ -o test.job.o$JOB_ID
#$ -j y
#$ -N test


cd /scr
# Change to the /scr directory 

$SGE_O_WORKDIR/a.out
# Run the a.out executable (this presumes it is in $SGE_O_WORKDIR, 
# a variable provided by SGE that points to the directory from
# which the job was submitted)

If this is called test.job, type: "qsub -q short test.job" to submit. The job will be run in the queue name "short".

The test.job example introduces the concept of "#$" directives. Lines in a script which begin with "#$" are read by the SGE system as configuration settings. Handy ones are:

#$ -j y
Causes the standard error and standard output to be combined in one file.

#$ -o test.job.o$JOB_ID
defines the name of the standard error/output file to be test.job.o$JOB_ID where $JOB_ID is the job number assigned when the job starts.

#$ -N test
defines the job name that appears in the queue (qstat), in this case calling it "test".

See the other FAQs for real Chemistry examples.

8) How do I compile code on dino?


You can compile really simple code on the head node. For buggier code or compiles that require more than a few seconds of cpu time, use "qlogin" to have the queue assign an interactive session on a compute node. The compile on your assigned node.

Simple Compiling
gfortran, g++, and gcc are in the PATH by default. The Intel compilers:
C/C++, Fortran and Math Kernel Library are also installed. Before using the Intel compilers, type "intel" to include these in your PATH. You must do this each time you open a shell. You can edit your .tcshrc file to make the change permanent but this may affect some other software and is not recommended. If you need dynamic links at run time, it is best to build the "intel" setting into your job file.

Compiling OpenMPI

mpicc -o foo foo.c
mpif77 -o foo foo.f

9) How do I run OpenMPI code on dino?

Here's an SGE script:

#!/bin/tcsh -f
#$ -cwd
#$ -o water.job.o$JOB_ID
#$ -j y
#$ -N q-water
#$ -pe shared 4

#cd to location of mpihello program
cd /home/kathy/test
setenv PATH /opt/gridengine/bin/lx26-amd64:/bin:/usr/bin:/opt/rocks/bin:/usr/software/openmpi-1.4.1/gfortran/bin
setenv LD_LIBRARY_PATH /opt/gridengine/lib/lx26-amd64:/usr/software/openmpi-1.4.1/gfortran/lib
/bin/hostname > test.out
/usr/software/openmpi-1.4.1/gfortran/bin/mpirun -np 4 ./mpihello

This calls mpirun with 4 processors. Note this was compiled with gforrtran so I set the environtment variables to point to the openmpi version also compiled with gfortran. If you prefer intel compilers, add in the "intel" setting and change gfortran to intel at each point in the script.

10) How do I run QChem on dino?

QChem
Use the run_qchem command. Assuming you have an input file: glutamine.in (or .inp) you can type:
run_qchem glutamine
Appending .in or .inp is optional but the calculation expects that glutamine.in or glutamine.inp is in the current working directory. It will put output in the same directory.

General usage is :
run_qchem jobname [num_cpu save mpi bigscr]
jobname should be your input .inp or .in file name.

The default num_cpu (number of cpus) is 1. For parallel jobs in shared memory, valid choices are 2 through 12.

Use save to save scratch and/or plot files in a folder called jobname-saved at the end of the calculation. Otherwise they are deleted.

QChem has 2 parallel modes: OpenMP (shared memory - this is the default parallel mode from run_qchem) or MPI (non-shared memory). For MPI in non-shared memory (over more than one node) use the "mpi" option. Please read the manual. Shared memory is required for ADC, CCMAN, CCMAN2 and Integral Transformation. CIS and TDDFT are not parallel. You can only use up to 12 core in shared memory. If you ask for more, this script will force your job to use MPI which might not be compatible with the desired job type.

If you know that you need more than 350GB of scratch space, use the "bigscr", option. This allows scratch up to 3TB. NOTE: You must specify 12 cpus for these jobs.

The order of [num_cpu save mpi bigscr] does not matter but must follow jobname if they are used.

Examples: run_qchem jobname 4
Uses 4 core with all subjobs on the same node in shared memory.

run_qchem jobname 6 save mpi
Uses 6 core, not necessarily on the same node. Saves scratch/restart/plot data.

HF_water.in is a sample input file.

11) How do I run Gaussian calculations on dino?

To run the current version of Gaussian 09 use the run_gaussian command. You need an input file with a .com extension such as test001.com. Then type
run_gaussian test001
(appending .com is optional in this command) This runs a Gaussian calculation on test001.com in the current working directory and puts output in the same directory. The default number of cpus is 1 but you can use up to 12. run_gaussian reads the %nprocshared line in your input file and makes the proper queue request accordingly.

Type "qstat" to see jobs that are running.

Memory: For a 1-core (cpu) job you can (and should) specify up to 3800MB in the %mem= section of your input file.

Parallel: You can submit parallel jobs on up to 12 cpus on 1 node using shared memory. You must specify the number of cpus (the %nprocshared= section) and amount of memory (the %mem= section) in your input file. The amount of memory should be 3800MB x num_cpus, or the number in GB minus 1GB (for example 15GB for 4 cpus). Note that calculations do not scale linearly with increasing numbers of cpus.

Large scratch files: Certain jobs such as MP2 frequencies on large systems (or CCSD calculations) can generate very large scratch files. To cope with this we have put 3000GB of disk space on some of the nodes. To run a job on one of these nodes, put "maxdisk=3000GB" in the route section of your input file and use the command:

run_gaussian jobname bigscr

You should specify %nprocshared=12 cpus to use this option. You are not required to do so but it is advised. If you don't you could run out of scratch space or cause other jobs to do so.

Other Options: Type "run_gaussian" without arguments to see time limits and other options.

Revision D01 is the current default from run_gaussian and Gaussview. You can request other revisions:
To use revision B01, use run_gaussian.g09b01
To use revision C01, use run_gaussian.g09c01
To use revision D01, use run_gaussian.g09d01 (currently same as run_gaussian)

run_cubegen: There is also a run_cubegen script to generate MO cube files from a completed Gaussian job. This can be useful if you are generating alot of cube files and find the lag time in Gaussview to be an issue. Just type run_cubegen to see the syntax. The cube files are generated on dino and you can view them later in Gaussview at your convenience.

cubegen has more features and can be run on the command line on any workstation too. See the utilites page and the specific cubegen page.

The general idea is to generate a fchk file and then request cube files as in these examples:

formchk filename.chk filename.fchk
cubegen 0 density=scf filename.fchk density.cube
cubegen 0 potential=scf filename.fchk potential.cube
cubegen 0 MO=homo filename.fchk homo.cube 
cubegen 0 MO=lumo filename.fchk lumo.cube
Old chk files: To use checkpoint files generated by G03 on 32 bit machines, you MUST first convert them to a format that G09 (64-bit) can read. To do this, go to the directory containing the .chk file you want to convert and type:

gausschk0309 jobname (no .chk or .out etc.)

This script uses G03's 32-bit formchk utility to make a formatted checkpoint file (.fchk) then converts it back using G09's 64-bit unfchk utility.

If you want to make a formatted checkpoint file from a 32-bit G03 checkpoint file, type:

formchk03 jobname (no .chk or .out etc.)

Gaussian 03: If you need to use Gaussian 03, use the command:

run_gaussian03 jobname [num_cpu]

Jobname should be your Gaussian input com file name, with or without the 
.com extension.
The default num_cpu (number of cpus) is 1. Other valid choices are 2 through 12.
With more than 1 cpu, you must have a matching %nprocshared= entry in your 
input file.
Note that this is G03 revE and is a 64-bit version. If you have checkpoint files from a 32bit G03 version, you must convert them before using a 64bit version of Gaussian. To do this, type:

gausschk03_64bit jobname (no .chk or .out etc.)

12) How do I run DMol on dino?

You need a jobname.car and jobname.input file and then just type:
run_dmol jobname.

13) How do I run interactive jobs on dino?

If you want do some interactive testing of your job prior to sending it into the queue in the normal fashion, you need to work on a cpu that is unused by anyone else. The cluster is composed of 49 separate dual 6-core machines. The master machine bears the name of the entire cluster and is called dino. It is the front end of the 48 other 12-core machines which are the compute nodes: compute-0-0, compute-0-1,....compute-0-47. When you connect to the cluster, you are actually connecting to the master machine.

Do not run interactive jobs on this master machine. Use one of the compute nodes. Here's how (using KD's account as an example only):

dino:/home/kathy 117 % qlogin

kathy@compute-0-6 $ 
Note that the prompt changes from dino to compute-0-6.
compute-0-6 is now dedicated to the interactive session and remains dedicated until the user exits the connection:
kathy@compute-0-6 $ exit
logout

dino:/home/kathy 118 %
While the connection is active, it shows up in qstat as jobname QLOGIN.

14) How do I display Gaussian output in Molekel or gOpenMol or VMD?

One can make beautiful MO pictures in VMD. See our instructions.

Gaussian has a web page that describes how to do this with various supporting applications of which we have Gaussview, gOpenMol and Molekel.

These examples assume you have a checkpoint file. Most assume you ran your job with the pop=full keyword.

Some of these instructions assume you are using the Windows version of Gaussian. Since you are probably using the version on linux, you need to to convert your checkpoint file to cube files and then pick up the instructions from that point.

To go from checkpoints to cube files:

formchk filename.chk filename.fchk
cubegen 0 MO=homo filename.fchk homo.cube 
cubegen 0 MO=lumo filename.fchk lumo.cube
See the Gaussian utilities page for details.

You can also generate multiple MO cube files at once using our script that submits to dino.
Type:

run_cubegen
for instructions on usage.

15) How do I make ESP/Density plots from Jaguar or Gaussian?

Please see notes for Gaussian or Jaguar Documentation and the Schrodinger knowledge base on Jaguar ESP. You can control the range of ESP values to be mapped. If you want to compare multiple structures, use the same the minimum and maximum values for each. The units of the ESP are kcal/mol.

16) How do I use Windows software within the MGCF?

We run a Windows Virtual Machine inside most of the MGCF and Kavli Linux workstations. This allows us to support a few key programs that are Windows only. If you need access to any of these programs, please see our Windows WM notes.

Back to the MGCF Home Page