[gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

Tue Feb 2 13:19:37 GMT 2021

Hi Giovanni!

Thank you very much for your offer , we really would be very  grateful to be allowed to come if we run into troubles!

Well, the implementation will not happen before June or later, but may I ask only one question meanwhile?

Did you ever run into problems with IBM support or did you get a  special “OK”  from them? Or do you accept to sove any rdma specific problems without support ? (it´s only because of the FAQ “not supported” )

Have a great day and keep healthy!

Best regards walter

-----Original Message-----
From: Giovanni Bracco <giovanni.bracco at enea.it>
Sent: Montag, 1. Februar 2021 20:42
To: Walter Sklenka <Walter.Sklenka at EDV-Design.at>
Cc: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

On 30/01/21 21:01, Walter Sklenka wrote:

> Hi Giovanni!

> Thats great! Many thanks for your fast and detailed answer!!!!

> So this is the way we will go too!

>

> Have a nice weekend and keep healthy!

> Best regards

> Walter

>

I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works!

If yu have any issues I am ready to discuss!

Regards

Giovanni

> -----Original Message-----

> From: Giovanni Bracco <giovanni.bracco at enea.it<mailto:giovanni.bracco at enea.it>>

> Sent: Samstag, 30. Jänner 2021 18:08

> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>;

> Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>

> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD

> Server with only ib rdma enabled

>

> In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.

> See

> F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a

> multifabric GPFS Spectrum Scale layout," 2019 International Conference

> on High Performance Computing & Simulation (HPCS), Dublin, Ireland,

> 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813

>

> When setting up the system the main trick has been:

> just use CentOS drivers and do not install OFED We do not use IPoIB.

>

> Giovanni

>

> On 30/01/21 06:45, Walter Sklenka wrote:

>> Hi!

>>

>> Is it possible to mix OPAcards and Infininiband HCAs on the same server?

>>

>> In the faq

>> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.

>> html#rdma

>>

>>

>> They talk about RDMA :

>>

>> "RDMA is NOT  supported on a node when both Mellanox HCAs and Intel

>> Omni-Path HFIs are ENABLED for RDMA."

>>

>> So do I understand right: When we do NOT enable  the opa interface we

>> can still enable IB ?

>>

>> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers

>> (wih access to storage)  with opa interfaces which provide access to

>> remote cluster  also via OPA.

>>

>> A new cluster with HDR interfaces will be implemented soon

>>

>> They shell have access to the same filesystems

>>

>> When we add HDR interfaces to  NSD servers  and enable rdma on this

>> network  while disabling rdma on opa we would accept the worse

>> performance via opa . We hope that this provides  still better perf

>> and less technical overhead  than using routers

>>

>> Or am I totally wrong?

>>

>> Thank you very much and keep healthy!

>>

>> Best regards

>>

>> Walter

>>

>> Mit freundlichen Grüßen

>> */Walter Sklenka/*

>> */Technical Consultant/*

>>

>> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210

>> Wien

>> Tel: +43 1 29 22 165-31

>> Fax: +43 1 29 22 165-90

>> E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at> <mailto:sklenka at edv-design.at>

>> Internet: www.edv-design.at<http://www.edv-design.at> <http://www.edv-design.at/>

>>

>>

>> _______________________________________________

>> gpfsug-discuss mailing list

>> gpfsug-discuss at spectrumscale.org

>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>

>

> --

> Giovanni Bracco

> phone  +39 351 8804788

> E-mail  giovanni.bracco at enea.it<mailto:giovanni.bracco at enea.it>

> WWW http://www.afs.enea.it/bracco

>

--

Giovanni Bracco

phone  +39 351 8804788

E-mail  giovanni.bracco at enea.it<mailto:giovanni.bracco at enea.it>

WWW http://www.afs.enea.it/bracco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210202/9d3b18a2/attachment.htm>