From leonardo.sala at psi.ch Fri Mar 7 15:40:45 2025 From: leonardo.sala at psi.ch (Leonardo Sala) Date: Fri, 7 Mar 2025 16:40:45 +0100 Subject: [gpfsug-discuss] CES SMB and multi user mounts Message-ID: Dear all, we do have a CES cluster with GPFS 5.2.2.1 and Active Directory authentication, and I would like to have multiuser mounts on linux. To my understanding, in order to have this I need to have my share allowing anonymous users to get information ("restrict anonymous=0", and eventually guest access ("guest ok = True"). Despite this, I do always get the following error: Status code returned 0xc000006d STATUS_LOGON_FAILURE when mounting with:? mount.smb3 -o multiuser,sec=krb5,cifsacl,guest Has anyone succeeded in doing it? It seems that on Netapp it works, but one needs to create a share open to anyone, and then have your normal sub-shares in it. Any experience with CES? It seems to me that winbind always tries to get information from AD concerning the user mounting in multiuser mode, which in this case is root, thus it fails... Thanks! cheers leo -- Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala at psi.ch www.psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonardo.sala at psi.ch Fri Mar 7 15:42:01 2025 From: leonardo.sala at psi.ch (Sala Leonardo) Date: Fri, 7 Mar 2025 15:42:01 +0000 Subject: [gpfsug-discuss] CES SMB and multi user mounts Message-ID: Dear all, we do have a CES cluster with GPFS 5.2.2.1 and Active Directory authentication, and I would like to have multiuser mounts on linux. To my understanding, in order to have this I need to have my share allowing anonymous users to get information ("restrict anonymous=0", and eventually guest access ("guest ok = True"). Despite this, I do always get the following error: Status code returned 0xc000006d STATUS_LOGON_FAILURE when mounting with: mount.smb3 -o multiuser,sec=krb5,cifsacl,guest Has anyone succeeded in doing it? It seems that on Netapp it works, but one needs to create a share open to anyone, and then have your normal sub-shares in it. Any experience with CES? It seems to me that winbind always tries to get information from AD concerning the user mounting in multiuser mode, which in this case is root, thus it fails... Thanks! cheers leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From timm.stamer at uni-oldenburg.de Mon Mar 10 07:51:48 2025 From: timm.stamer at uni-oldenburg.de (Timm Stamer) Date: Mon, 10 Mar 2025 07:51:48 +0000 Subject: [gpfsug-discuss] CES SMB and multi user mounts In-Reply-To: References: Message-ID: Hello leo, we're using this in /etc/fstab: // We mount these shares with root rights after we have a valid krb machine ticket (klist -k). Afterwards users are able to access mounted shares with their krb user ticket. You may have to tune /etc/request-key.d/cifs.spnego.conf create cifs.spnego * * /usr/sbin/cifs.upcall %k create dns_resolver * * /usr/sbin/cifs.upcall %k Kind regards Timm Am Freitag, dem 07.03.2025 um 15:42 +0000 schrieb Sala Leonardo: > > Dear all, > > > > we do have a CES cluster with GPFS 5.2.2.1 and Active Directory > authentication, and I would like to have multiuser mounts on linux. > To my understanding, in order to have this I need to have my share > allowing anonymous users to get information ("restrict anonymous=0", > and eventually guest access ("guest ok = True"). Despite this, I do > always get the following error: > > > Status code returned 0xc000006d STATUS_LOGON_FAILURE > > > when mounting with: ?mount.smb3 -o multiuser,sec=krb5,cifsacl,guest > > > Has anyone succeeded in doing it? It seems that on Netapp it works, > but one needs to create a share open to anyone, and then have your > normal sub-shares in it. Any experience with CES? It seems to me that > winbind always tries to get information from AD concerning the user > mounting in multiuser mode, which in this case is root, thus it > fails... > > > Thanks! > > > cheers > > > leo > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6274 bytes Desc: not available URL: From xhejtman at ics.muni.cz Mon Mar 10 08:19:19 2025 From: xhejtman at ics.muni.cz (=?utf-8?B?THVrw6HFoSBIZWp0bcOhbmVr?=) Date: Mon, 10 Mar 2025 08:19:19 +0000 Subject: [gpfsug-discuss] gpfs scaleapi Message-ID: <37E195AD-FDF7-4084-911A-580F59E1FC51@ics.muni.cz> Hello, is there any documentation how to setup the scale api (I know, it is experimental), I did not find any readme and default configuration complains, that node identity is not set. (grpc socket works, but not TCP/HTTP). Thanks. -- Luk?? Hejtm?nek Linux Administrator only because Full Time Multitasking Ninja is not an official job title From abeattie at au1.ibm.com Mon Mar 10 08:30:02 2025 From: abeattie at au1.ibm.com (ANDREW BEATTIE) Date: Mon, 10 Mar 2025 08:30:02 +0000 Subject: [gpfsug-discuss] gpfs scaleapi In-Reply-To: <37E195AD-FDF7-4084-911A-580F59E1FC51@ics.muni.cz> References: <37E195AD-FDF7-4084-911A-580F59E1FC51@ics.muni.cz> Message-ID: Lukas, Reach out to your IBM Storage Technical Sales rep, They can help you engage with the Dev team Regards, Andrew Beattie Senior Technical Sales Specialist Storage for Data and AI IBM Australia and New Zealand p. +61 421 337 927 e. abeattie at au1.ibm.com From: gpfsug-discuss on behalf of Luk?? Hejtm?nek Date: Monday, 10 March 2025 at 4:21?PM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] [gpfsug-discuss] gpfs scaleapi Hello, is there any documentation how to setup the scale api (I know, it is experimental), I did not find any readme and default configuration complains, that node identity is not set. (grpc socket works, but not TCP/HTTP). Thanks. -- Luk?? Hejtm?nek Linux Administrator only because Full Time Multitasking Ninja is not an official job title _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Mar 10 10:51:40 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 10 Mar 2025 10:51:40 +0000 Subject: [gpfsug-discuss] Lenovo downloads Message-ID: <338b7b14-5a59-4de4-8802-8b075947f996@strath.ac.uk> I notice this morning that GPFS finally supports Ubuntu 24.04 How does one prod Lenovo to add the 5.2.2.2 Client side and Support nodes download to Service Connect? I have a pressing need for this to work around an issue with the kernel SMB file system driver when working with Microsoft DFS shares. TL;DR it was broken when they added IPv6 support in very early 4.x kernels and only works properly on a 6.6 or later kernel. Currently I have to maintain a CentOS 7 machine with TuxCare ELS which is far from ideal and want to switch it to something supported as soon as possible. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From scl at virginia.edu Tue Mar 11 11:49:23 2025 From: scl at virginia.edu (Losen, Stephen C (scl)) Date: Tue, 11 Mar 2025 11:49:23 +0000 Subject: [gpfsug-discuss] Setting NFS4 ACL with setxattr() Message-ID: Hi folks, I've been experimenting with python os.getxattr() and os.setxattr() for getting and setting NFS4 ACLs. I reverse engineered the format of the ACL returned by os.getxattr(), but is it documented anywhere? I discovered a permission issue when running as a non-root user. If the target file is owned by a different user but has a NFS4 ACE with WRITE_ACL enabled for me, then os.setxattr() nevertheless fails for me with permission denied. As expected, os.chmod() works for me and the chmod and mmputacl commands also work for me. If I own the file, then os.setxattr() works. Does anyone know if this is a feature or a bug? The behavior seems inconsistent. Steve Losen University of Virginia Research Computing From christof.schmitt at us.ibm.com Sun Mar 16 23:30:40 2025 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Sun, 16 Mar 2025 23:30:40 +0000 Subject: [gpfsug-discuss] Setting NFS4 ACL with setxattr() In-Reply-To: References: Message-ID: <78a4e806b7814bc82eae56e03119775371535d5e.camel@us.ibm.com> The intention with the xattr access to NFSv4 ACLs is to allow usage of the Linux nfs4-acl-tools: https://www.ibm.com/docs/en/STXKQY/gpfsclustersfaq.html#nfsacl The format should be visible in the source code of the tools, i think this would be the struct nfs4_acl: https://git.linux-nfs.org/?p=steved/nfs4-acl-tools.git;a=blob;f=include/nfs4.h;h=d15482e8a720e82d6248f311b537e1057c38adc2;hb=refs/heads/master#l129 I am not sure of the observed behavior with the WRITE_ACL permission.. Recreating that with traces would help understand which case is hit. Regards, Christof Schmitt On Tue, 2025-03-11 at 11:49 +0000, Losen, Stephen C (scl) wrote: Hi folks, I've been experimenting with python os.getxattr() and os.setxattr() for getting and setting NFS4 ACLs. I reverse engineered the format of the ACL returned by os.getxattr(), but is it documented anywhere? I discovered a permission issue when running as a non-root user. If the target file is owned by a different user but has a NFS4 ACE with WRITE_ACL enabled for me, then os.setxattr() nevertheless fails for me with permission denied. As expected, os.chmod() works for me and the chmod and mmputacl commands also work for me. If I own the file, then os.setxattr() works. Does anyone know if this is a feature or a bug? The behavior seems inconsistent. Steve Losen University of Virginia Research Computing _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Mar 18 15:04:03 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 18 Mar 2025 15:04:03 +0000 Subject: [gpfsug-discuss] Errors in dmesg Message-ID: <6b2365f7-1600-4c04-94d9-8ca1bcb67a20@strath.ac.uk> Got my Ubuntu 24.04 server up and running and it "appears" to be working. Thanks to everyone a Lenovo for helping me get access to the download. However I am see some troubling messages on the console like this [Tue Mar 18 11:21:10 2025] UBSAN: array-index-out-of-bounds in /usr/lpp/mmfs/src/gpl-linux/acl.c:314:45 [Tue Mar 18 11:21:10 2025] index 1 is out of range for type 'linux_posix_acl_entry [*]' Where index might be any of 0, 1 or 2. That to my eye doesn't look good. Is this expected or should I raise this a support issue? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From l.r.sudbery at bham.ac.uk Tue Mar 18 16:25:01 2025 From: l.r.sudbery at bham.ac.uk (Luke Sudbery) Date: Tue, 18 Mar 2025 16:25:01 +0000 Subject: [gpfsug-discuss] Replicated cluster - failure groups Message-ID: We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 data centres, with 2 DSSG per pool, and a quorum/manager node with a local tie breaker disk in a 3rd physical location. My main question is about failure groups - so far, with 2 DSS and 1 tiebreaker, we would have had 1 failure group per DSS and 1 for the tie breaker disk, giving us a total of 3. But if we did that now we would have 9 failure groups in 1 filesystem, which is more than the maximum number of replicas of the file system descriptor and not desirable, as I understand it. So we could have either: * 1 FG per physical site, and assign all 4 DSS per site to 1 FG, and a 3rd to the tiebreaker * 1 FG per pool per site, with 2 DSS in each FG. This makes sense as the pairs of DSSG will both always need to be up for all the data in the pool to be accessible. The second option would give us 5 failure groups, but what would be the advantage and disadvantages of more failure groups? Many thanks, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don't work on Monday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Mar 18 17:21:26 2025 From: scale at us.ibm.com (scale) Date: Tue, 18 Mar 2025 17:21:26 +0000 Subject: [gpfsug-discuss] Replicated cluster - failure groups In-Reply-To: References: Message-ID: There is no advantage to having more failure group than maximum number of replica supported by a file system plus 1 for tie breaker disks. In a multiple site setup, you will want 1 failure group per site in order to ensure 1 replica is placed at each site as GPFS will place replica using round-robin amount the failure groups. From: gpfsug-discuss on behalf of Luke Sudbery Date: Tuesday, March 18, 2025 at 9:28?AM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] [gpfsug-discuss] Replicated cluster - failure groups We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 data centres, with 2 DSSG per pool, and a quorum/manager node with a local tie breaker disk in a 3rd physical location. My main question is about failure groups - so far, with 2 DSS and 1 tiebreaker, we would have had 1 failure group per DSS and 1 for the tie breaker disk, giving us a total of 3. But if we did that now we would have 9 failure groups in 1 filesystem, which is more than the maximum number of replicas of the file system descriptor and not desirable, as I understand it. So we could have either: * 1 FG per physical site, and assign all 4 DSS per site to 1 FG, and a 3rd to the tiebreaker * 1 FG per pool per site, with 2 DSS in each FG. This makes sense as the pairs of DSSG will both always need to be up for all the data in the pool to be accessible. The second option would give us 5 failure groups, but what would be the advantage and disadvantages of more failure groups? Many thanks, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don?t work on Monday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From l.r.sudbery at bham.ac.uk Tue Mar 18 20:17:39 2025 From: l.r.sudbery at bham.ac.uk (Luke Sudbery) Date: Tue, 18 Mar 2025 20:17:39 +0000 Subject: [gpfsug-discuss] Replicated cluster - failure groups In-Reply-To: References: Message-ID: But any pros/cons of 5 (2 failure groups per site + tiebreaker ) vs 3? If we had 1 failure group per site, would we need to bring up all NSDs on that site (4x DSS ? 8 actual servers) to guarantee bringing up the NSD with the desc replica disk? Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don?t work on Monday. From: gpfsug-discuss On Behalf Of scale Sent: 18 March 2025 17:21 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Replicated cluster - failure groups CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. There is no advantage to having more failure group than maximum number of replica supported by a file system plus 1 for tie breaker disks. In a multiple site setup, you will want 1 failure group per site in order to ensure 1 replica is placed at each site as GPFS will place replica using round-robin amount the failure groups. From: gpfsug-discuss > on behalf of Luke Sudbery > Date: Tuesday, March 18, 2025 at 9:28?AM To: gpfsug-discuss at gpfsug.org

> Subject: [EXTERNAL] [gpfsug-discuss] Replicated cluster - failure groups We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 data centres, with 2 DSSG per pool, and a quorum/manager node with a local tie breaker disk in a 3rd physical location. My main question is about failure groups - so far, with 2 DSS and 1 tiebreaker, we would have had 1 failure group per DSS and 1 for the tie breaker disk, giving us a total of 3. But if we did that now we would have 9 failure groups in 1 filesystem, which is more than the maximum number of replicas of the file system descriptor and not desirable, as I understand it. So we could have either: * 1 FG per physical site, and assign all 4 DSS per site to 1 FG, and a 3rd to the tiebreaker * 1 FG per pool per site, with 2 DSS in each FG. This makes sense as the pairs of DSSG will both always need to be up for all the data in the pool to be accessible. The second option would give us 5 failure groups, but what would be the advantage and disadvantages of more failure groups? Many thanks, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don?t work on Monday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Paul.Sanchez at deshaw.com Tue Mar 18 21:24:40 2025 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Tue, 18 Mar 2025 21:24:40 +0000 Subject: [gpfsug-discuss] Replicated cluster - failure groups In-Reply-To: References: Message-ID: <2b133e05ccd3434bbd45d8d7dcc70b12@deshaw.com> IIRC, the filesystem descriptor disk is only in the system pool? so as long as the system pool only has 3 FGs and they correspond to your 3 sites, then the filesystem survivability characteristics are straightforward. I think that you technically could use two different FGs for the second pool and GPFS will still work as expected, but that just seems confusing to the humans. We started off with one multisite stretch cluster like you describe and 10 years later we have around 100 stretch clusters. Choosing a standard mapping between FG numbers and your sites can be a good way to reduce cognitive load on your team. -Paul From: gpfsug-discuss On Behalf Of Luke Sudbery Sent: Tuesday, March 18, 2025 16:18 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Replicated cluster - failure groups This message was sent by an external party. But any pros/cons of 5 (2 failure groups per site + tiebreaker ) vs 3? If we had 1 failure group per site, would we need to bring up all NSDs on that site (4x DSS ? 8 actual servers) to guarantee bringing up the NSD with the desc replica disk? Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don?t work on Monday. From: gpfsug-discuss > On Behalf Of scale Sent: 18 March 2025 17:21 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Replicated cluster - failure groups CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. There is no advantage to having more failure group than maximum number of replica supported by a file system plus 1 for tie breaker disks. In a multiple site setup, you will want 1 failure group per site in order to ensure 1 replica is placed at each site as GPFS will place replica using round-robin amount the failure groups. From: gpfsug-discuss > on behalf of Luke Sudbery > Date: Tuesday, March 18, 2025 at 9:28?AM To: gpfsug-discuss at gpfsug.org

> Subject: [EXTERNAL] [gpfsug-discuss] Replicated cluster - failure groups We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 data centres, with 2 DSSG per pool, and a quorum/manager node with a local tie breaker disk in a 3rd physical location. My main question is about failure groups - so far, with 2 DSS and 1 tiebreaker, we would have had 1 failure group per DSS and 1 for the tie breaker disk, giving us a total of 3. But if we did that now we would have 9 failure groups in 1 filesystem, which is more than the maximum number of replicas of the file system descriptor and not desirable, as I understand it. So we could have either: * 1 FG per physical site, and assign all 4 DSS per site to 1 FG, and a 3rd to the tiebreaker * 1 FG per pool per site, with 2 DSS in each FG. This makes sense as the pairs of DSSG will both always need to be up for all the data in the pool to be accessible. The second option would give us 5 failure groups, but what would be the advantage and disadvantages of more failure groups? Many thanks, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don?t work on Monday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christof.schmitt at us.ibm.com Tue Mar 18 22:28:48 2025 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 18 Mar 2025 22:28:48 +0000 Subject: [gpfsug-discuss] Errors in dmesg In-Reply-To: <6b2365f7-1600-4c04-94d9-8ca1bcb67a20@strath.ac.uk> References: <6b2365f7-1600-4c04-94d9-8ca1bcb67a20@strath.ac.uk> Message-ID: <6a001385d2483defe40e672aacb70c6755fef12f.camel@us.ibm.com> I see the same on a test system, and i think this is a false positive. When you check the code, there is a variable length arrary declared at the end of a struct with [0]. Apparently the UBSAN checker keys of the number, and the way to avoid the warning would be to change that to []. But this would be no functional change, just a syntax change to silence the checker. Regards, Christof Schmitt On Tue, 2025-03-18 at 15:04 +0000, Jonathan Buzzard wrote: Got my Ubuntu 24.04 server up and running and it "appears" to be working. Thanks to everyone a Lenovo for helping me get access to the download. However I am see some troubling messages on the console like this [Tue Mar 18 11:21:10 2025] UBSAN: array-index-out-of-bounds in /usr/lpp/mmfs/src/gpl-linux/acl.c:314:45 [Tue Mar 18 11:21:10 2025] index 1 is out of range for type 'linux_posix_acl_entry [*]' Where index might be any of 0, 1 or 2. That to my eye doesn't look good. Is this expected or should I raise this a support issue? JAB. -------------- next part -------------- An HTML attachment was scrubbed... URL: