National Supercomputing Mission

About

 

About NSM@JNCASR


Welcome to the home page of the High Performance Computing Facility at Jawaharlal Nehru Centre for Advanced Scientific Research (JNCASR). In 2020, JNCASR was chosen to be a node under the National Supercomputing Mission (NSM) by the Government of India, and hosts the supercomputing facility called Param Yukti. Param Yukti is part of a network of supercomputing facilities in the country, and caters to the ever increasing demands of high performance computing for scientific and engineering research.

JNCASR facilitates the administration of activities under NSM and the maintenance of Param Yukti. The resources of Param Yukti are shared by the JNCASR community, users directly funded by NSM, neighboring educational institutes, and other users. Some research areas in which Param Yukti is being used are:

  • Computational Fluid Dynamics and Climate Modelling
  • Computational Physics / Materials Science
  • Computational Chemistry
  • Life Sciences and Medicine

Keep an eye on the latest announcements for calls for proposals to use the facility. The application form can be downloaded from here.

System Configuration

 

SYSTEM CONFIGURATION

PARAM Yukti provides a total of 1.8 PETAFLOPS of combined CPU and GPU compute power. PARAM Yukti comprises a total of 166 nodes   ( 156 + 10)  which are distributed as follows:

  •     Login nodes: 4
  •     Service / Master nodes: 6
  •     CPU only nodes: 75
  •     GPU nodes: 42
  •     High Memory nodes: 39

CONFIGURATION OF COMPUTE (CPU) NODES:

  •     CPU: 2 x Intel Xeon, 8268 24 Cores @ 2.9GHz
  •     RAM: 192 GB
  •     Storage: 480 GB SSD

The High Memory nodes have the same base configuration as the compute nodes but with an increased RAM capacity of 768 GB.

CONFIGURATION OF GPU ENABLED NODES:

      With 2 GPU cards: 10 Nodes

  •     CPU: 2 x Intel Xeon, 6248 20 Cores @ 2.5GHz
  •     GPU: 2 x NVIDIA V100 SXM2 cards
  •     RAM: 192 GB
  •     HDD: 480 GB SSD

     With 4 GPU cards: 32 Nodes

  •     CPU: 2 x Intel Xeon, 8268 24 Cores @ 2.9GHz
  •     GPU: 4 x NVIDIA V100 SXM2 cards
  •     RAM: 192 GB
  •     HDD: 480 GB SSD

 

INTERCONNECT

  • PRIMARY NETWORK:  100Gbps Mellanox Infiniband Interconnect network 100% non-blocking, fat tree topology.
  • SECONDARY NETWORK :  10G/1G Ethernet Networking Management network: 1G Ethernet.

STORAGE

  • A separate storage server provides a total of 1PB of storage space for user data.

The entire system operates on LINUX - CentOS 7.6, and uses SLURM job scheduler and the Lustre Parallel File System.

System Architecture

 

Param Yukti is based on a heterogeneous configuration consisting of Intel Xeon Cascade lake CPUs and NVIDIA Tesla V100 GPUs. The system is designed and built by the HPC Technologies team, Centre for Development of Advanced Computing (C-DAC).

 

Network

The compute nodes of Param Yukti are connected by InfiniBand HDR 100 Gbps high speed network and a secondary network for support and maintenance. The network topology is shown in the figure below.

 

Storage

The storage subsystem comprises of the following components:

  • 2x ES7990 Embedded Lustre Parallel File System storage appliance
  • 12x 1.92 TB SAS SSD drives for metadata pool
  • 164x 10 TB 7.2K drives for 2 x data pool (with 1x MR)
 

Software Stack

Software stack is an aggregation of software components that work in tandem to provide support for utilizing the computing resources through different means. The software stack has three layers: (i) the kernel and hardware drivers, (ii) middleware applications, management tools, etc., and (iii) application libraries, programming tools and computational software. Param Yukti supports all major programming languages and tools. For a list of available software see the resources section.


 

People

 

NSM Coordinator


Prof. Srikanth Sastry

Theoretical Sciences Unit
JNCASR
Email: nsm_coord@jncasr.ac.in
Tel: +91 80 2208 2838

Admin staff


Mrs. Nirupama V 

Email: nsmadmin@jncasr.ac.in
Tel: +91 80 2208 2945

Mr. Basavaraj T
Email: basavarajt@jncasr.ac.in
Tel: +91 80 2208 2899

Technical staff


Mr. Pandurang Bugade
Mr. Salman Khan

Email: nsmtechadmin@jncasr.ac.in
Tel: +91 80 2208 2944

Technical staff from ATOS


Mr. Punith M
Mr. Gopinath T

Email: yuktisupport@jncasr.ac.in
Tel: +91 80 2208 2944

 

Publications

Acknowledging Param Yukti in your publications

Users are required to acknowledge the use of Param Yukti resources in their research publications. Here is a sample acknowledgement:

We acknowledge the Param Yukti facility under the National Supercomputing Mission at Jawaharlal Nehru Centre For Advanced Scientific Research for the computational resources.

Publications arising from the usage of Param Yukti

2021

  1. H Bhaumik, G Foffi, S Sastry, The role of annealing in determining the yielding behavior of glasses under cyclic shear deformation. Proc. Natl. Acad. Sci. 118(16), e2100227118 (2021).
  2. Bidhan Chandra Garain, Pralok K. Samanta and Swapan K Pati. Intersystem Crossing in Boron-Based Donor–Spiro–Acceptor Organic Chromophore: A Detailed Theoretical Study. J. Phys. Chem. A 125, 6674 (2021).
  3. Nalina Vadakkayil, 'Structure and Dynamics Away from Equilibrium: Scaling results from a few condensed matter systems'. Ph.D. Thesis submitted on 22/09/2021.
  4. Nikhil V. S. Avula, Anwesa Karmakar, Rahul Kumar, and Sundaram Balasubramanian, Efficient Parametrization of Force Field for the Quantitative Prediction of the Physical Properties of Ionic Liquid Electrolytes. J. Chem. Theory Comput. 17(7), 4274–4290 (2021).
  5. S. Karmakar, S. Barman, F. A. Rahimi, T. K. Maji, Covalent grafting of molecular photosensitizer and catalyst on MOF-808: Effect of pore confinement toward visible light-driven CO2 reduction in water. Energy Environ. Sci. 14, 2429-2440 (2021).
  6. S. Karmakar, A. Ghosh, K. Prasad, F. A. Rahimi, D. Rambabu, R. Haldar, T. K. Maji, Multicolour lanthanide(III) porous 1D coordination polymers: Tunable wide spectrum emission and efficient CuII sensing. Dalton Trans. 50, 13002-13011 (2021).
  7. S. Barman, A. Singh, F. A. Rahimi, T. K. Maji, Metal-free catalysis: A Redox-active Donor-Acceptor Conjugated Microporous Polymer for Selective Visible Light-driven CO2 Reduction to CH4. J. Am. Chem. Soc. 143(39), 16284–16292 (2021).

Resources

User Account Creation

To begin using Param Yukti, you need to get an account first. Download the user account creation format from here. Depending on your category, the account creation process may be different:

Access and usage policies for users under different categories

Stage 1:

  1. JNCASR Internal users:

    Download the form from the above link. Send the duly filled account request form with the signature of the faculty member (the research advisor) to: nsmadmin@jncasr.ac.in and nsmtechadmin@jncasr.ac.in

  2. NSM funded users (PIs who have approved grants under NSM, either in the present call or earlier if the grants are still operational):

    Download the form from the above link. Send the duly filled account request form to: nsmadmin@jncasr.ac.in and nsmtechadmin@jncasr.ac.in, along with an e-copy of the approval note of sanctioned usage and computing charges. This will be forwarded after vetting by the Host Institute coordinator.

  3. Users from Neighbouring Institutions and Industry/Private Organizations:

    Download the form from the above link. Send the duly filled account request forms to the administrative email ID (nsmadmin@jncasr.ac.in or yuktisupport@jncasr.ac.in) Usage charges for using supercomputing facilities (May apply):

Usage charges for using supercomputing facilities (May apply):

CPU Rs. 0.96 per CPU core hour
GPU Rs. 42 per GPU card hour

Note: While filling the user account creation form please provide concrete information under
(i) “Brief description of the project” which will include the research goals and nature of work.
(ii) “Proposed work on PARAM Yukti” which should specify the computations planned to be undertaken.
(iii) “Requirement of resources” which should specify the estimated CPU-core hours, and GPU-hours required for the project duration specified, and any software requirements.

Please contact yuktisupport@jncasr.ac.in and nsmadmin@jncasr.ac.in if you need any other clarifications.

Stage 2:

After the submission of the duly filled account request form with the required documents and after we are getting approval from the head of the NSM Department,

  • You will receive an email to your official ID from the Yukti Support team intimating the creation of your account along with a temporary password for your account.
  • Log in to Param Yukti and change the temporary password before you start using Param Yukti. Changing the password will enable you to keep your account secure.

Your password will be valid for 90 days. On expiry, you will be prompted to change your password, on attempting to log in.

Forgot Password

Please write to yuktisupport@jncasr.ac.in to get the password reset. The password will be reset for the user and an email will be sent intimating the same. Then the user can log in with the temporary password and can set a new password of his/her choice.

Or

Please raise a ticket regarding this issue and the system administrators will resolve your problem, as described in the support page.

 


Getting StartedTo start using Param Yukti, you first need to create a user account, the details of which can be found on the account creation page. Once you have received your username and password, you can use Param Yukti as described below.

Accessing Param YuktiLogin to Param Yukti from your Linux/Unix command line:

For internal users:

$ ssh username@hostname

 

For external users:

$ ssh username@hostname -p port_number

The hostname and the port number can be found in the email you receive on creation of your account. On login, you will be redirected to one of the four login nodes. The login nodes are only used for file manipulations and visualizations. Do not run programs in the background on the login nodes. Jobs should only be submitted via the job submission script as described below.

Loading the necessary modules

To compile and run jobs, you need to first load the modules needed by your job (libraries, compilation environments, etc).

To know which modules are available to use, type the command:
$ module avail

Load the necessary modules. For instance, if your application needs the Intel compiler, do:
$ module load compiler/intel/2018.2.199

To unload the module, type:
$ module unload compiler/intel/2018.2.199

To see which module is loaded, type:
$ module list

Unload all loaded modules:
$ module purge

Submitting a job

SLURM is the JOB submission system used in Param Yukti. Create a slurm batch script as follows:

#!/bin/bash
#SBATCH --job-name=serial_job_test        # Job name
#SBATCH --mail-type=END,FAIL           # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@example.com   # Where to send mail
#SBATCH --partition=standard          # Specify partition (standard,gpu,hm)
#SBATCH --nodes=2                # Maximum number of nodes to be allocated
#SBATCH --ntasks-per-node=12         # Maximum number of tasks on each node
#SBATCH --ntasks=1               # Run on a single CPU
#SBATCH --cpus-per-task=4           # Number of CPU cores per task
#SBATCH --mem=1gb              # Job memory request
#SBATCH --time=00:05:00           # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.log     # Standard output and error log

### For GPU Jobs ###
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2 # The number of GPUs required

### Clear any previously loaded modules ###
module purge
###Load the modules necessary for the job###
module load compiler/intel/2018.2.199

###Working Directory###
cd <working_directory>

MACHINE_FILE=nodes.$SLURM_JOBID
scontrol show hostname $SLURM_JOB_NODELIST > $MACHINE_FILE

mpiexec.hydra -machinefile $MACHINE_FILE -n <NP> <executable>

Users can also download the example script from here and modify it to their needs.

Once the submit script has been created, the job can be submitted as follows:
$ sbatch script.sh

 

Managing jobs and monitoring resources

To get information about the status of which nodes are up:
$ sinfo

To show the list of submitted jobs:
$ squeue

To get the availability status of compute nodes:
$ sinfo

To delete a job
$ scancel <job-id>

To get full information about your job
$ scontrol show job <job-id>

To hold the job
$ scontrol hold <job-id>

To release the job
$ scontrol release <job-id>

 

For detailed information refer the Param Yukti user manual.

List of available applications

The following list of software is available in Param Yukti.

  • openfoam
  • abinit
  • amber
  • bowtie2
  • quantum_espresso
  • charliecloud
  • regcm
  • roms
  • vasp
  • gromacs
  • hmmer
  • lammps
  • mpiblast
  • nektar
  • nwchem
  • WRF
  • Regcm
  • openmolcas

Use the "module avail" command to know the available compilers, libraries and applications.

If users require the installation of any other software, they should contact yuktisupport@jncasr.ac.in

Support

Getting Help – Param Yukti Support

All technical issues relating to the usage of Param Yukti are handled through a ticketing system. This helps keep track of the issues and the state of their resolution. Please refer to the following steps to generate a ticket related to the issue you are experiencing. Your ticket will be assisted by the Yukti Support team. The ticket will be closed when the related issue is resolved. You can generate a new ticket for any of the issues that you are experiencing.

Steps to Create a New Ticket

  1. Go to https://paramyukti.jncasr.ac.in/support/login.php in your browser.
  2. Sign in using the username and password that you use for logging in to Param Yukti.
  3. Select a help topic from the drop-down list and then click on Create Ticket .
  4. Please fill in the details of your issue in the fields given and then click on create a ticket.

Once the ticket is generated, an acknowledgement email will be sent to your official email address. The e-mail will also contain the ticket number and reference to the ticket that you have generated. In case of any difficulty while accessing Yukti Support you can reach us via email at yuktisupport@jncasr.ac.in

Updates

Current events and announcements


Archives

OpenACC bootcamp​ under NSM:

Centre for Development of Advanced Computing (C-DAC), under the aegis of the National Supercomputing Mission (NSM) in association with NVIDIA & OpenACC, has organised a five day training program (bootcamp) on accelerating applications on Nvidia GPUs from 16-Jan-2023 and 20-Jan-2023. Apply here: click here

R&D project proposals are invited under National Super Computing Mission (NSM). Last date to submit the applications Nov 18, 2020. Apply here: http://www.serc.iisc.ac.in/projects/nsm

Call for R&D project proposals under NSM:

OpenACC Bootcamp​ under NSM:

Apply here: https://docs.google.com/forms/d/1xqw1cvzuylKSvlkn5z798OTuZZBKUteOzQbbztHw2Xo/edit?usp=sharing

JNCASR and NVIDIA with OpenACC are organizing a four-day training program (Bootcamp) on accelerating applications on Nvidia GPUs on 20th June 2022 to 24th June 2021.

Under the HR development activities of NSM, an online course on “Introduction to Deep Learning” has been planned. It will start on June 28, 2021. The details of the course and the link for registration are available here: https://iitgoa.ac.in/aishikshadl

OpenACC bootcamp​ under NSM:

Centre for Development of Advanced Computing (C-DAC) under the aegis of the National Supercomputing Mission (NSM) in association with NVIDIA & OpenACC, is organizing a two-day training program (bootcamp) on accelerating applications on Nvidia GPUs on 17-Feb-2021 and 18-Feb-2021. Apply here: https://gpuhackathons.org/event/c-dac-india-gpu-bootcamp

Contact Us

      For Technical Support
       Yukti Support
       1st floor, BMS Room
       SSL Building
       JNCASR Jakkur
       Bangalore
       Pin-560064
       Email: yuktisupport@jncasr.ac.in; nsmtechadmin@jncasr.ac.in
       Tel: +91 80 2208 2944