From TROPPENS at de.ibm.com Thu Sep 1 17:30:56 2022 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Thu, 1 Sep 2022 16:30:56 +0000 Subject: [gpfsug-discuss] User Meeting in NYC at September 20, 2022 Message-ID: Greetings! IBM is organizing a Spectrum Scale User Meeting in New York City. We have an exciting agenda covering user stories, roadmap update, the latest insights into data fabrics, data orchestration and data management architectures, plus access to IBM experts and your peers. We look forward to welcoming you to this event. Please register here: https://www.spectrumscaleug.org/event/nyc-user-meeting-2022/ Draft Agenda: 8:30 9:00 Registration and Morning Coffee 9:00 9:10 Welcome 9:10 9:30 Strategy Update 9:30 9:50 What is new in Spectrum Scale? 9:50 10:10 What is new in ESS? 10:10 10:30 Customer Talk - TBD 10:30 11:00 Coffee and Networking 11:00 11:20 Container Update 11:20 11:40 New S3 access for AI and Analytics 11:40 12:00 Support for GPUDirect Storage 12:00 12:20 Customer Talk - NYU Langone 12:20 13:20 Lunch and Networking 13:20 13:40 Spectrum Fusion 13:40 14:00 Spectrum Scale on IBM Cloud 14:00 14:20 Customer Talk - TBD 14:20 14:40 Service Update 14:40 15:10 Coffee and Networking 15:10 15:30 Monitoring and serviceability enhancements in Spectrum Scale 15:30 16:10 Technical Deep Dive - Performance Update 16:10 16:40 Open Discussion 16:40 16:45 Wrap-up Starting 17:00 Social Event Ulf Troppens Senior Technical Staff Member Spectrum Scale Development IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen / Gesch?ftsf?hrung: David Faller Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.taylor at psi.ch Thu Sep 1 22:18:27 2022 From: joshua.taylor at psi.ch (Taylor Joshua George (PSI)) Date: Thu, 1 Sep 2022 21:18:27 +0000 Subject: [gpfsug-discuss] NF4 ACLs Message-ID: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> Hi Everyone, I'm trying implement some ACLs, however some of the documentation is a bit unclear to me. Using https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists as a reference, I'm trying to understand what to use to achieve 0660 permissions on files and 2770 on directories. So far, I've managed to achieve 0000 perms, but user with the ACL permission can chmod, or 0770 perms. Attached is a txt file with the mmgetacl output, as well as file listing on a test file, and finally, the ACL definition I used. As one can see in the attachment, the ACL requested appears differently for what it _actually_ applied. Thanks for any help! Joshua Taylor --- Paul Scherrer Institut System Engineer Science IT Infrastructure and Services department (AWI) WHGA/038 Forschungstrasse 111 5232 Villigen PSI Switzerlandd +41 56 310 52 50 -------------- next part -------------- ------. 1 e20233 p20233 70 Aug 13 14:49 mask [root at xbl-agw-1 ~]# ls -l /gpfs/perf/MX/Data10/e20233/acl-test -rwxrwx---. 1 svcusr-mx_writer p20233 0 Aug 26 10:30 /gpfs/perf/MX/Data10/e20233/acl-test [root at xbl-agw-1 ~]# mmgetacl /gpfs/perf/MX/Data10/e20233 #NFSv4 ACL #owner:e20233 #group:p20233 special:owner@:rwxc:allow:FileInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:group@:rwx-:allow:FileInherit:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:everyone@:----:allow (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED user:svcusr-mx_writer:rwx-:allow:FileInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED user:svcusr-mx_writer:rwx-:allow:DirInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED [root at xbl-agw-1 ~]# cat /gpfs/perf/scratch/josh/acl-test/svcusr-mx_writer.acl #NFSv4 ACL #owner:e20233 #group:p20233 special:owner@:rwxc:allow:FileInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:group@:rw-c:allow:FileInherit:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:everyone@:----:allow (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED user:svcusr-mx_writer:rw-c:allow:FileInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED user:svcusr-mx_writer:rw-c:allow:DirInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED From jonathan.buzzard at strath.ac.uk Fri Sep 2 09:23:48 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 2 Sep 2022 09:23:48 +0100 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> Message-ID: <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > Hi Everyone, > I'm trying implement some ACLs, however some of the documentation is a > bit unclear to me. > > Using >https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > as a reference, I'm trying to understand what to use to achieve 0660 > permissions on files and 2770 on directories. > It's not clear from this whether you are trying to achieve the equivalent of 0660 and 2770 on files and directories or have an ls show the permissions as 0660 and 2770. > So far, I've managed to achieve 0000 perms, but user with the ACL > permission can chmod, or 0770 perms. > Basically neither of the above two options is possible because there is no exact mapping between POSIX permissions and NFSv4 ACL's. For example you can't get the equivalent of the set group id permission. You can however put an inheritable ACL for a group on the directory that gives r/w plus say search directory and possibly execute permissions if you want those as well. A user with ACL permissions can change permissions that is completely expected. Note that traditional 2770 permissions are only suggestive, the file or member of the group would be able to change them to something else. In fact programs often do when you save, and Samba just completely ignores them for the most part. At least with NFSv4 ACL's you can remove the ACL permission :-) How permissions display on an ls/stat is not an exact mapping and will tend to go to something like 0000, but actual ability to access etc. the file will be based on the ACL not what you see in ls/stat. > Attached is a txt file with the mmgetacl output, as well as file > listing on a test file, and finally, the ACL definition I used. > > As one can see in the attachment, the ACL requested appears differently > for what it _actually_ applied. > What ACL schematics does the file system have? Is it NFSv4 or both? If you are wedded to POSIX style permissions perhaps change to POSIX ACL schematics on the file system? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Fri Sep 2 10:27:06 2022 From: anacreo at gmail.com (Alec) Date: Fri, 2 Sep 2022 02:27:06 -0700 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> Message-ID: I believe this ACL will achieve what you're trying to do... with POSIX permissions normally you would set 770 umask and a file create does not preserve execute by default, where a directory does. I have just now learned that with ACL's it does what you told it to do. So we have to recreate that behaviour by having a separate inheritance for both File and Directory. The last bit, how to preserve the sticky bit, I couldn't quite figure out on a whim... but when I did a "chmod g+s .", the proper "DirInherit and FileInherit:InheritOnly" were added into the ACL... and things worked as expected. $ mmgetacl . #NFSv4 ACL #owner:someuser #group:somegroup special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rw-c:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:group@:rwx-:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:group@:rw--:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:everyone@:----:allow:FileInherit:DirInherit (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (-)SYNCHRONIZE (-)READ_ACL (-)READ_ATTR (-)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED On Fri, Sep 2, 2022 at 1:25 AM Jonathan Buzzard < jonathan.buzzard at strath.ac.uk> wrote: > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > > > > Hi Everyone, > > I'm trying implement some ACLs, however some of the documentation is a > > bit unclear to me. > > > > Using > > > https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > > as a reference, I'm trying to understand what to use to achieve 0660 > > permissions on files and 2770 on directories. > > > > It's not clear from this whether you are trying to achieve the > equivalent of 0660 and 2770 on files and directories or have an ls show > the permissions as 0660 and 2770. > > > So far, I've managed to achieve 0000 perms, but user with the ACL > > permission can chmod, or 0770 perms. > > > > Basically neither of the above two options is possible because there is > no exact mapping between POSIX permissions and NFSv4 ACL's. > > For example you can't get the equivalent of the set group id permission. > You can however put an inheritable ACL for a group on the directory that > gives r/w plus say search directory and possibly execute permissions if > you want those as well. > > A user with ACL permissions can change permissions that is completely > expected. Note that traditional 2770 permissions are only suggestive, > the file or member of the group would be able to change them to > something else. In fact programs often do when you save, and Samba just > completely ignores them for the most part. At least with NFSv4 ACL's you > can remove the ACL permission :-) > > How permissions display on an ls/stat is not an exact mapping and will > tend to go to something like 0000, but actual ability to access etc. the > file will be based on the ACL not what you see in ls/stat. > > > Attached is a txt file with the mmgetacl output, as well as file > > listing on a test file, and finally, the ACL definition I used. > > > > As one can see in the attachment, the ACL requested appears differently > > for what it _actually_ applied. > > > > What ACL schematics does the file system have? Is it NFSv4 or both? > > If you are wedded to POSIX style permissions perhaps change to POSIX ACL > schematics on the file system? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacreo at gmail.com Fri Sep 2 10:36:50 2022 From: anacreo at gmail.com (Alec) Date: Fri, 2 Sep 2022 02:36:50 -0700 Subject: [gpfsug-discuss] mmchfs -k nfs4 impacts? In-Reply-To: References: <59725e39-f365-4d3e-7a53-70cdc671d21b@strath.ac.uk> <09A7D1EF-434B-49A2-9893-AADCFC39C0A7@virginia.edu> Message-ID: Stephen, (And Helge, some interesting SMB stuff at the very end), The answer is no, it won't cause a conversion or take a long time, but that is a bit of speculation based on the documentation... --- inexplicably long answer below --- The GPFS documentation quite clearly states that ACL's are managed on a per file basis, so you may have intermingled POSIX and NFS4 ACL's. Past versions going back to 3.1 seem to indicate they work the same way.. https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=lists-nfs-v4-acl-administration A simple test: $ mmlsfs ${FS} -k flag value description ------------------- ------------------------ ----------------------------------- -k all ACL semantics in effect $ touch nfsv4acl posixacl $ mmgetacl -k nfs4 nfsv4acl | mmputacl nfsv4acl $ mmgetacl -k posix posixacl | mmputacl posixacl $ mmgetacl nfsv4acl #NFSv4 ACL #owner:XXXXXXXXXXXX #group:XXX special:owner@:rw-c:allow (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (-)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED ... $ mmgetacl posixacl #owner:XXXXXXXXXXX #group:XXX user::rw-c ... $ mmgetacl -k nfs4 posixacl #NFSv4 ACL #owner:XXXXXXXXXXXX #group:XXX special:owner@:rw-c:allow (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED ... The man page for -k indicates that the user can't *ASSIGN *these ACL's it doesn't state that it won't serve or store them, so going to a more restricted mode doesn't seem to indicate that it would remove the existing ACL's. It also doesn't indicate that any conversion is happening in any circumstance (from a mmchfs). Finally the mmgetacl command has a man page that explains EXACTLY what is going on: 1. By default, mmgetacl returns the ACL in a format consistent with the file system setting, specified using the -k flag on the mmcrfs or mmchfs commands. If the setting is posix, the ACL is shown as a traditional ACL. If the setting is nfs4, the ACL is shown as an NFS V4 ACL. If the setting is all, the ACL is returned in its true form. 2. The command mmgetacl -k nfs4 always produces an NFS V4 ACL. 3. The command mmgetacl -k posix always produces a traditional ACL. 4. The command mmgetacl -k native always shows the ACL in its true form regardless of the file system setting. So the short answer is after changing -k to nfsv4 it seems mmgetacl would intuit that you want to see ACL's in nfsv4 format since POSIX isn't allowed. But you can run mmgetacl -k native to see what ACL was (and continues to be) stored for the file object. I can't change any of my FS's to not be -k all, so I couldn't test out my mmgetacl assertions. Maybe I got some details wrong here but this one seems pretty straight forward based on the documentation. Also I'm not sure I'd say that a mention of a flag embedded in a ksh script is "documented" but I expect it is more related to how SMB integrates to Spectrum Scale and perhaps there is some secret sauce leveraged here. We don't do SMB on Scale, but if I we were, this would be an interesting read: https://manpages.debian.org/testing/samba-vfs-modules/vfs_gpfs.8.en.html. Alec On Tue, Aug 30, 2022 at 3:03 AM Helge Hauglin wrote: > Hi Stephen. > > > Also I didn't mention that we also need NFSv4 access and > > native GPFS, this will not be SMB-only. It will actually be mostly > > GPFS native. > > Beware that when writing via SMB, samba default permissions will be applied > to new files and folders, which might not give the permissions your > users need. > > On our CES clusters, the samba default permission is 0755 / 0744 [1]. > We want either 0770 or 0775 by default. This we get by setting these > permissions in NFSv4 ACLs in relevant folders, plus turn on inheritance > for the ACEs to new files and folders. The side effect of having NFSv4 > ACLs with inheritance is that 'umask' in processes writing via GPFS or > NFS is ignored. I have not tried. but I guees it works similarly with > POSIX ACLs. > > [1] > > | # testparm -s -v | grep mask > | Load smb config files from /var/mmfs/ces/smb.conf > | [...] > | create mask = 0744 > | directory mask = 0755 > > > I don't think existing ACLs will be adversely > > affected. In a test filesystem with "-k all" I set some POSIX ACLs and > > converted the filesystem to "-k nfs4" and the result looked > > reasonable. Plus I ran mmgetacl -k nfs4 on numerous files/dirs with > > POSIX ACLs in our production filesystem and the results looked > > promising. > > I would recommend standardizing on one type of ACLs, which will give you > less variants to deal with, simplifying administration. > > -- > Regards, > > Helge Hauglin > > ---------------------------------------------------------------- > Mr. Helge Hauglin, Senior Engineer > System administrator > Center for Information Technology, University of Oslo, Norway > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.taylor at psi.ch Fri Sep 2 10:49:01 2022 From: joshua.taylor at psi.ch (Taylor Joshua George (PSI)) Date: Fri, 2 Sep 2022 09:49:01 +0000 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> Message-ID: <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> Perhaps I should try and describe my use-case a bit: There is a daemon process that gets data and must then write it out into a project directory in GPFS. The project directory is owned by the project owner, and has the setgid bit set on the directory. This is so that members of the group can read/write/etc files put in the directory. The daemon is not a member of the project group. The daemon process used to run as root, however, my goal with ACLs is to enable it to write into project dirs, without needing to change the effective UID/GID (or be a member of the group) - so, in this case, it runs as a regular user. The ACL should allow it to write into the project directory. Essentially, the ACL is to enable the daemon to run as a regular user. Once the data is written, the Daemon will (likely) never access the data again, only the users in the project. So, my goal is to have the daemon write files owned by the project Group (so, with g+rw), as the project users would just use the standard posix permissions to access the data. I suppose I'm trying to blend the use of ACLs and standard posix perms. More importantly, I'm trying to get rid of the execute bit that new files are created with/inherited. Answers to questions below... On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > > > > Hi Everyone, > > I'm trying implement some ACLs, however some of the documentation > > is a > > bit unclear to me. > > > > Using > ?> > https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > > as a reference, I'm trying to understand what to use to achieve > > 0660 > > permissions on files and 2770 on directories. > > > > It's not clear from this whether you are trying to achieve the > equivalent of 0660 and 2770 on files and directories or have an ls > show > the permissions as 0660 and 2770. I'm trying to get the files to be created with 0660 perms (currently, they are created 0770). I've tried playing with the `rwxc` in the first line of the NFS4 ACL definition, with no luck (it seems to ignore that, except for error checking). E.g. I've tried removing the `x`, which passes the syntax check but doesn't seem to change anything. so, daemon uses ACLs to write/create files, users use posix perms to access/modify those files. > > > So far, I've managed to achieve 0000 perms, but user with the ACL > > permission can chmod, or 0770 perms. > > > > Basically neither of the above two options is possible because there > is > no exact mapping between POSIX permissions and NFSv4 ACL's. > > For example you can't get the equivalent of the set group id > permission. > You can however put an inheritable ACL for a group on the directory > that > gives r/w plus say search directory and possibly execute permissions > if > you want those as well. > > A user with ACL permissions can change permissions that is completely > expected. Note that traditional 2770 permissions are only suggestive, > the file or member of the group would be able to change them to > something else. In fact programs often do when you save, and Samba > just > completely ignores them for the most part. At least with NFSv4 ACL's > you > can remove the ACL permission :-) > > How permissions display on an ls/stat is not an exact mapping and > will > tend to go to something like 0000, but actual ability to access etc. > the > file will be based on the ACL not what you see in ls/stat. > > Attached is a txt file with the mmgetacl output, as well as file > > listing on a test file, and finally, the ACL definition I used. > > > > As one can see in the attachment, the ACL requested appears > > differently > > for what it _actually_ applied. > > > > What ACL schematics does the file system have? Is it NFSv4 or both? > It's pure NFSv4 -D nfs4 File locking semantics in effect -k nfs4 ACL semantics in effect > > If you are wedded to POSIX style permissions perhaps change to POSIX > ACL > schematics on the file system? I confess, I was hoping to mix the perms so as not to change our processes too much. (we currently use reg. posix perms to enable group writes [via sgid]) > > > JAB. > -- Joshua Taylor --- Paul Scherrer Institut System Engineer Science IT Infrastructure and Services department (AWI) WHGA/038 Forschungstrasse 111 5232 Villigen PSI Switzerlandd +41 56 310 52 50 From anacreo at gmail.com Fri Sep 2 11:08:25 2022 From: anacreo at gmail.com (Alec) Date: Fri, 2 Sep 2022 03:08:25 -0700 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> Message-ID: Taylor, What I provided would work for your use case 1000%... at the top level you'll need to add an entry for your process or a group that your process is a member of then it would be able to create files that the members of the sgid group have access to... $ mmgetacl . #NFSv4 ACL #owner:someuser #group:somegroup special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rw-c:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:group@:rwx-:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:group@:rw--:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:everyone@:----:allow:FileInherit:DirInherit (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (-)SYNCHRONIZE (-)READ_ACL (-)READ_ATTR (-)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED user:nongroupuser:rwx-:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (-)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED You can decide in that last block how much control to give (or not give) on that file. New files by "nongroupuser" would be owned by nongroupuser, but the group would be the group of the parent directory, not their primary group. Alec On Fri, Sep 2, 2022 at 2:50 AM Taylor Joshua George (PSI) < joshua.taylor at psi.ch> wrote: > Perhaps I should try and describe my use-case a bit: > > There is a daemon process that gets data and must then write it out > into a project directory in GPFS. The project directory is owned by the > project owner, and has the setgid bit set on the directory. This is so > that members of the group can read/write/etc files put in the > directory. The daemon is not a member of the project group. > > The daemon process used to run as root, however, my goal with ACLs is > to enable it to write into project dirs, without needing to change the > effective UID/GID (or be a member of the group) - so, in this case, it > runs as a regular user. The ACL should allow it to write into the > project directory. Essentially, the ACL is to enable the daemon to run > as a regular user. Once the data is written, the Daemon will (likely) > never access the data again, only the users in the project. > > So, my goal is to have the daemon write files owned by the project > Group (so, with g+rw), as the project users would just use the standard > posix permissions to access the data. I suppose I'm trying to blend the > use of ACLs and standard posix perms. More importantly, I'm trying to > get rid of the execute bit that new files are created with/inherited. > > Answers to questions below... > > > On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: > > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > > > > > > > Hi Everyone, > > > I'm trying implement some ACLs, however some of the documentation > > > is a > > > bit unclear to me. > > > > > > Using > > > > > > https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > > > as a reference, I'm trying to understand what to use to achieve > > > 0660 > > > permissions on files and 2770 on directories. > > > > > > > It's not clear from this whether you are trying to achieve the > > equivalent of 0660 and 2770 on files and directories or have an ls > > show > > the permissions as 0660 and 2770. > > I'm trying to get the files to be created with 0660 perms (currently, > they are created 0770). I've tried playing with the `rwxc` in the first > line of the NFS4 ACL definition, with no luck (it seems to ignore that, > except for error checking). E.g. I've tried removing the `x`, which > passes the syntax check but doesn't seem to change anything. > > so, daemon uses ACLs to write/create files, users use posix perms to > access/modify those files. > > > > > > So far, I've managed to achieve 0000 perms, but user with the ACL > > > permission can chmod, or 0770 perms. > > > > > > > Basically neither of the above two options is possible because there > > is > > no exact mapping between POSIX permissions and NFSv4 ACL's. > > > > For example you can't get the equivalent of the set group id > > permission. > > You can however put an inheritable ACL for a group on the directory > > that > > gives r/w plus say search directory and possibly execute permissions > > if > > you want those as well. > > > > A user with ACL permissions can change permissions that is completely > > expected. Note that traditional 2770 permissions are only suggestive, > > the file or member of the group would be able to change them to > > something else. In fact programs often do when you save, and Samba > > just > > completely ignores them for the most part. At least with NFSv4 ACL's > > you > > can remove the ACL permission :-) > > > > How permissions display on an ls/stat is not an exact mapping and > > will > > tend to go to something like 0000, but actual ability to access etc. > > the > > file will be based on the ACL not what you see in ls/stat. > > > Attached is a txt file with the mmgetacl output, as well as file > > > listing on a test file, and finally, the ACL definition I used. > > > > > > As one can see in the attachment, the ACL requested appears > > > differently > > > for what it _actually_ applied. > > > > > > > What ACL schematics does the file system have? Is it NFSv4 or both? > > > > It's pure NFSv4 > -D nfs4 File locking semantics in effect > -k nfs4 ACL semantics in effect > > > > > If you are wedded to POSIX style permissions perhaps change to POSIX > > ACL > > schematics on the file system? > > I confess, I was hoping to mix the perms so as not to change our > processes too much. (we currently use reg. posix perms to enable group > writes [via sgid]) > > > > > > > JAB. > > > > -- > Joshua Taylor > > --- > Paul Scherrer Institut > System Engineer > Science IT Infrastructure and Services department (AWI) > WHGA/038 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerlandd > +41 56 310 52 50 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacreo at gmail.com Fri Sep 2 11:22:15 2022 From: anacreo at gmail.com (Alec) Date: Fri, 2 Sep 2022 03:22:15 -0700 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> Message-ID: Hmm I think you're missing in what I sent that there are TWO ACE's for special owner@: special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rw-c:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED The first one sets up the new DIRECTORIES with exec/SEARCH. The second one sets up the new FILES without EXEC/search. This provides for new files to be made 660 while directories are created 2770. I think you're conflating what the POSIX interface does with regards to a umask when a file is created versus what the ACL interface does when a file is created. In fact the Open man page on Linux specifies that POSIX permissions are (mode & umask) in the ABSENCE of a default ACL. NFS4 doesn't have a default ACL but the inheritance is an effective default ACL as it is copying the parent's ACL and building up from there. Hope That Helps Alec On Fri, Sep 2, 2022 at 3:08 AM Alec wrote: > Taylor, > What I provided would work for your use case 1000%... at the top level > you'll need to add an entry for your process or a group that your process > is a member of then it would be able to create files that the members of > the sgid group have access to... > > $ mmgetacl . > #NFSv4 ACL > #owner:someuser > #group:somegroup > special:owner@:rwxc:allow:DirInherit > (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL > (X)READ_ATTR (X)READ_NAMED > (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL > (X)WRITE_ATTR (X)WRITE_NAMED > > special:owner@:rw-c:allow:FileInherit:InheritOnly > (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL > (X)READ_ATTR (X)READ_NAMED > (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL > (X)WRITE_ATTR (X)WRITE_NAMED > > special:group@:rwx-:allow:DirInherit > (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL > (X)READ_ATTR (X)READ_NAMED > (X)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL > (-)WRITE_ATTR (-)WRITE_NAMED > > special:group@:rw--:allow:FileInherit:InheritOnly > (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL > (X)READ_ATTR (X)READ_NAMED > (X)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL > (-)WRITE_ATTR (-)WRITE_NAMED > > special:everyone@:----:allow:FileInherit:DirInherit > (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (-)SYNCHRONIZE (-)READ_ACL > (-)READ_ATTR (-)READ_NAMED > (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL > (-)WRITE_ATTR (-)WRITE_NAMED > > user:nongroupuser:rwx-:allow:DirInherit > > (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (-)SYNCHRONIZE (X)READ_ACL > (X)READ_ATTR (X)READ_NAMED > > (-)DELETE (-)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL > (-)WRITE_ATTR (-)WRITE_NAMED > > > You can decide in that last block how much control to give (or not give) > on that file. New files by "nongroupuser" would be owned by nongroupuser, > but the group would be the group of the parent directory, not their primary > group. > > Alec > > On Fri, Sep 2, 2022 at 2:50 AM Taylor Joshua George (PSI) < > joshua.taylor at psi.ch> wrote: > >> Perhaps I should try and describe my use-case a bit: >> >> There is a daemon process that gets data and must then write it out >> into a project directory in GPFS. The project directory is owned by the >> project owner, and has the setgid bit set on the directory. This is so >> that members of the group can read/write/etc files put in the >> directory. The daemon is not a member of the project group. >> >> The daemon process used to run as root, however, my goal with ACLs is >> to enable it to write into project dirs, without needing to change the >> effective UID/GID (or be a member of the group) - so, in this case, it >> runs as a regular user. The ACL should allow it to write into the >> project directory. Essentially, the ACL is to enable the daemon to run >> as a regular user. Once the data is written, the Daemon will (likely) >> never access the data again, only the users in the project. >> >> So, my goal is to have the daemon write files owned by the project >> Group (so, with g+rw), as the project users would just use the standard >> posix permissions to access the data. I suppose I'm trying to blend the >> use of ACLs and standard posix perms. More importantly, I'm trying to >> get rid of the execute bit that new files are created with/inherited. >> >> Answers to questions below... >> >> >> On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: >> > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: >> > >> > > >> > > Hi Everyone, >> > > I'm trying implement some ACLs, however some of the documentation >> > > is a >> > > bit unclear to me. >> > > >> > > Using >> > > >> > >> https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists >> > > as a reference, I'm trying to understand what to use to achieve >> > > 0660 >> > > permissions on files and 2770 on directories. >> > > >> > >> > It's not clear from this whether you are trying to achieve the >> > equivalent of 0660 and 2770 on files and directories or have an ls >> > show >> > the permissions as 0660 and 2770. >> >> I'm trying to get the files to be created with 0660 perms (currently, >> they are created 0770). I've tried playing with the `rwxc` in the first >> line of the NFS4 ACL definition, with no luck (it seems to ignore that, >> except for error checking). E.g. I've tried removing the `x`, which >> passes the syntax check but doesn't seem to change anything. >> >> so, daemon uses ACLs to write/create files, users use posix perms to >> access/modify those files. >> >> > >> > > So far, I've managed to achieve 0000 perms, but user with the ACL >> > > permission can chmod, or 0770 perms. >> > > >> > >> > Basically neither of the above two options is possible because there >> > is >> > no exact mapping between POSIX permissions and NFSv4 ACL's. >> > >> > For example you can't get the equivalent of the set group id >> > permission. >> > You can however put an inheritable ACL for a group on the directory >> > that >> > gives r/w plus say search directory and possibly execute permissions >> > if >> > you want those as well. >> > >> > A user with ACL permissions can change permissions that is completely >> > expected. Note that traditional 2770 permissions are only suggestive, >> > the file or member of the group would be able to change them to >> > something else. In fact programs often do when you save, and Samba >> > just >> > completely ignores them for the most part. At least with NFSv4 ACL's >> > you >> > can remove the ACL permission :-) >> > >> > How permissions display on an ls/stat is not an exact mapping and >> > will >> > tend to go to something like 0000, but actual ability to access etc. >> > the >> > file will be based on the ACL not what you see in ls/stat. >> > > Attached is a txt file with the mmgetacl output, as well as file >> > > listing on a test file, and finally, the ACL definition I used. >> > > >> > > As one can see in the attachment, the ACL requested appears >> > > differently >> > > for what it _actually_ applied. >> > > >> > >> > What ACL schematics does the file system have? Is it NFSv4 or both? >> > >> >> It's pure NFSv4 >> -D nfs4 File locking semantics in effect >> -k nfs4 ACL semantics in effect >> >> > >> > If you are wedded to POSIX style permissions perhaps change to POSIX >> > ACL >> > schematics on the file system? >> >> I confess, I was hoping to mix the perms so as not to change our >> processes too much. (we currently use reg. posix perms to enable group >> writes [via sgid]) >> >> > >> > >> > JAB. >> > >> >> -- >> Joshua Taylor >> >> --- >> Paul Scherrer Institut >> System Engineer >> Science IT Infrastructure and Services department (AWI) >> WHGA/038 >> Forschungstrasse 111 >> 5232 Villigen PSI >> Switzerlandd >> +41 56 310 52 50 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scl at virginia.edu Fri Sep 2 11:27:00 2022 From: scl at virginia.edu (Losen, Stephen C (scl)) Date: Fri, 2 Sep 2022 10:27:00 +0000 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> Message-ID: Hi, Set-gid behavior on directories with nfs4 ACLs works as expected. However, you cannot specify it in the nfs4 ACL itself (nor in a posix ACL). You must set it separately with chmod g+s dirname. As expected, the set-gid bit (02000) is inherited by new directories and the group of the parent directory (special:group@) is inherited by new files and directories. So you must also set the group of the parent directory with chgrp (or chown). mkdir dirname chgrp groupname dirname chmod g+s dirname mmputacl dirname < nfs4-acl-file Also beware that chmod destroys (rewrites) the ACL if you specify any rwx permissions or numeric, but g+s is safe. Anyone with WRITE_ACL enabled can also run chmod so that should be restricted to the file owner (special:owner@). And as you have discovered, when setting an ACL with mmputacl the ":rwxc:" permissions on the first line of an ACE are ignored. Only the (X) and (-) specifiers in the second and third lines matter. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 ?On 9/2/22, 5:50 AM, "gpfsug-discuss on behalf of Taylor Joshua George (PSI)" wrote: Perhaps I should try and describe my use-case a bit: There is a daemon process that gets data and must then write it out into a project directory in GPFS. The project directory is owned by the project owner, and has the setgid bit set on the directory. This is so that members of the group can read/write/etc files put in the directory. The daemon is not a member of the project group. The daemon process used to run as root, however, my goal with ACLs is to enable it to write into project dirs, without needing to change the effective UID/GID (or be a member of the group) - so, in this case, it runs as a regular user. The ACL should allow it to write into the project directory. Essentially, the ACL is to enable the daemon to run as a regular user. Once the data is written, the Daemon will (likely) never access the data again, only the users in the project. So, my goal is to have the daemon write files owned by the project Group (so, with g+rw), as the project users would just use the standard posix permissions to access the data. I suppose I'm trying to blend the use of ACLs and standard posix perms. More importantly, I'm trying to get rid of the execute bit that new files are created with/inherited. Answers to questions below... On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > > > > Hi Everyone, > > I'm trying implement some ACLs, however some of the documentation > > is a > > bit unclear to me. > > > > Using > > > https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > > as a reference, I'm trying to understand what to use to achieve > > 0660 > > permissions on files and 2770 on directories. > > > > It's not clear from this whether you are trying to achieve the > equivalent of 0660 and 2770 on files and directories or have an ls > show > the permissions as 0660 and 2770. I'm trying to get the files to be created with 0660 perms (currently, they are created 0770). I've tried playing with the `rwxc` in the first line of the NFS4 ACL definition, with no luck (it seems to ignore that, except for error checking). E.g. I've tried removing the `x`, which passes the syntax check but doesn't seem to change anything. so, daemon uses ACLs to write/create files, users use posix perms to access/modify those files. > > > So far, I've managed to achieve 0000 perms, but user with the ACL > > permission can chmod, or 0770 perms. > > > > Basically neither of the above two options is possible because there > is > no exact mapping between POSIX permissions and NFSv4 ACL's. > > For example you can't get the equivalent of the set group id > permission. > You can however put an inheritable ACL for a group on the directory > that > gives r/w plus say search directory and possibly execute permissions > if > you want those as well. > > A user with ACL permissions can change permissions that is completely > expected. Note that traditional 2770 permissions are only suggestive, > the file or member of the group would be able to change them to > something else. In fact programs often do when you save, and Samba > just > completely ignores them for the most part. At least with NFSv4 ACL's > you > can remove the ACL permission :-) > > How permissions display on an ls/stat is not an exact mapping and > will > tend to go to something like 0000, but actual ability to access etc. > the > file will be based on the ACL not what you see in ls/stat. > > Attached is a txt file with the mmgetacl output, as well as file > > listing on a test file, and finally, the ACL definition I used. > > > > As one can see in the attachment, the ACL requested appears > > differently > > for what it _actually_ applied. > > > > What ACL schematics does the file system have? Is it NFSv4 or both? > It's pure NFSv4 -D nfs4 File locking semantics in effect -k nfs4 ACL semantics in effect > > If you are wedded to POSIX style permissions perhaps change to POSIX > ACL > schematics on the file system? I confess, I was hoping to mix the perms so as not to change our processes too much. (we currently use reg. posix perms to enable group writes [via sgid]) > > > JAB. > -- Joshua Taylor --- Paul Scherrer Institut System Engineer Science IT Infrastructure and Services department (AWI) WHGA/038 Forschungstrasse 111 5232 Villigen PSI Switzerlandd +41 56 310 52 50 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From joshua.taylor at psi.ch Fri Sep 2 11:32:31 2022 From: joshua.taylor at psi.ch (Taylor Joshua George (PSI)) Date: Fri, 2 Sep 2022 10:32:31 +0000 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> Message-ID: <781575e1b00944b672b8f2ccd0e7c5f16f4558a7.camel@psi.ch> I'll give this a try! Thank you! this helps clarify things a bit! best, Josh On Fri, 2022-09-02 at 03:22 -0700, Alec wrote: Hmm I think you're missing in what I sent that there are TWO ACE's for special owner@: special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rw-c:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED The first one sets up the new DIRECTORIES with exec/SEARCH. The second one sets up the new FILES without EXEC/search. This provides for new files to be made 660 while directories are created 2770. I think you're conflating what the POSIX interface does with regards to a umask when a file is created versus what the ACL interface does when a file is created. In fact the Open man page on Linux specifies that POSIX permissions are (mode & umask) in the ABSENCE of a default ACL. NFS4 doesn't have a default ACL but the inheritance is an effective default ACL as it is copying the parent's ACL and building up from there. Hope That Helps Alec On Fri, Sep 2, 2022 at 3:08 AM Alec > wrote: Taylor, What I provided would work for your use case 1000%... at the top level you'll need to add an entry for your process or a group that your process is a member of then it would be able to create files that the members of the sgid group have access to... $ mmgetacl . #NFSv4 ACL #owner:someuser #group:somegroup special:owner@:rwxc:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:owner@:rw-c:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED special:group@:rwx-:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (X)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:group@:rw--:allow:FileInherit:InheritOnly (X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (X)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED special:everyone@:----:allow:FileInherit:DirInherit (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (-)SYNCHRONIZE (-)READ_ACL (-)READ_ATTR (-)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED user:nongroupuser:rwx-:allow:DirInherit (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (-)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED (-)DELETE (-)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED You can decide in that last block how much control to give (or not give) on that file. New files by "nongroupuser" would be owned by nongroupuser, but the group would be the group of the parent directory, not their primary group. Alec On Fri, Sep 2, 2022 at 2:50 AM Taylor Joshua George (PSI) > wrote: Perhaps I should try and describe my use-case a bit: There is a daemon process that gets data and must then write it out into a project directory in GPFS. The project directory is owned by the project owner, and has the setgid bit set on the directory. This is so that members of the group can read/write/etc files put in the directory. The daemon is not a member of the project group. The daemon process used to run as root, however, my goal with ACLs is to enable it to write into project dirs, without needing to change the effective UID/GID (or be a member of the group) - so, in this case, it runs as a regular user. The ACL should allow it to write into the project directory. Essentially, the ACL is to enable the daemon to run as a regular user. Once the data is written, the Daemon will (likely) never access the data again, only the users in the project. So, my goal is to have the daemon write files owned by the project Group (so, with g+rw), as the project users would just use the standard posix permissions to access the data. I suppose I'm trying to blend the use of ACLs and standard posix perms. More importantly, I'm trying to get rid of the execute bit that new files are created with/inherited. Answers to questions below... On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: > > > > > Hi Everyone, > > I'm trying implement some ACLs, however some of the documentation > > is a > > bit unclear to me. > > > > Using > > > https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists > > as a reference, I'm trying to understand what to use to achieve > > 0660 > > permissions on files and 2770 on directories. > > > > It's not clear from this whether you are trying to achieve the > equivalent of 0660 and 2770 on files and directories or have an ls > show > the permissions as 0660 and 2770. I'm trying to get the files to be created with 0660 perms (currently, they are created 0770). I've tried playing with the `rwxc` in the first line of the NFS4 ACL definition, with no luck (it seems to ignore that, except for error checking). E.g. I've tried removing the `x`, which passes the syntax check but doesn't seem to change anything. so, daemon uses ACLs to write/create files, users use posix perms to access/modify those files. > > > So far, I've managed to achieve 0000 perms, but user with the ACL > > permission can chmod, or 0770 perms. > > > > Basically neither of the above two options is possible because there > is > no exact mapping between POSIX permissions and NFSv4 ACL's. > > For example you can't get the equivalent of the set group id > permission. > You can however put an inheritable ACL for a group on the directory > that > gives r/w plus say search directory and possibly execute permissions > if > you want those as well. > > A user with ACL permissions can change permissions that is completely > expected. Note that traditional 2770 permissions are only suggestive, > the file or member of the group would be able to change them to > something else. In fact programs often do when you save, and Samba > just > completely ignores them for the most part. At least with NFSv4 ACL's > you > can remove the ACL permission :-) > > How permissions display on an ls/stat is not an exact mapping and > will > tend to go to something like 0000, but actual ability to access etc. > the > file will be based on the ACL not what you see in ls/stat. > > Attached is a txt file with the mmgetacl output, as well as file > > listing on a test file, and finally, the ACL definition I used. > > > > As one can see in the attachment, the ACL requested appears > > differently > > for what it _actually_ applied. > > > > What ACL schematics does the file system have? Is it NFSv4 or both? > It's pure NFSv4 -D nfs4 File locking semantics in effect -k nfs4 ACL semantics in effect > > If you are wedded to POSIX style permissions perhaps change to POSIX > ACL > schematics on the file system? I confess, I was hoping to mix the perms so as not to change our processes too much. (we currently use reg. posix perms to enable group writes [via sgid]) > > > JAB. > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -- Joshua Taylor --- Paul Scherrer Institut System Engineer Science IT Infrastructure and Services department (AWI) WHGA/038 Forschungstrasse 111 5232 Villigen PSI Switzerlandd +41 56 310 52 50 -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Fri Sep 2 13:38:10 2022 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Fri, 2 Sep 2022 12:38:10 +0000 Subject: [gpfsug-discuss] User Meeting in Cologne (Germany) at October 19+20, 2022 Message-ID: Greetings! IBM is organizing a Spectrum Scale User Meeting in Cologne, Germany. We have an exciting agenda covering user stories, roadmap update, the latest insights into data fabrics, data orchestration and data management architectures, plus access to IBM experts and your peers. We look forward to welcoming you to this event. Please register here: https://www.spectrumscaleug.org/event/german-2022-user-meeting/ Draft Agenda (Most talks will be in German): October 19th, 2022 9:00 10:00 Registration and Networking 10:00 10:15 Welcome 10:15 10:40 Strategy Update 10:40 11:05 Easy Installation using Terraform and Ansible 11:05 11:25 Spectrum Scale on IBM Cloud 11:25 11:50 Global Data Orchestration 11:50 12:00 Meet the devs 12:00 13:00 Lunch and Networking 13:00 13:40 What is new in Spectrum Scale? 13:40 14:10 ESS 3500 14:10 14:30 Customer or partner talk - ESS3500 14:30 15:00 Coffee and Networking 15:00 15:20 AFM - Enhancements and Use Cases 15:20 15:40 Customer or partner talk 15:40 16:00 Customer or partner talk 16:00 16:15 New requirement process 16:15 16:35 Break 16:35 16:50 Update from ESCC (Pre-Sales/Lab Services/Support ) 16:50 17:05 Sneak Preview 17:05 17:25 CNSA & CSI Update 17:25 17:45 Container & Object Storage hints and tips 17:45 18:00 Ask us anything 18:00 20:00 Get Together October 20th, 2022 8:30 9:00 Coffee und Networking Spectrum Scale licensing 9:00 12:00 Break-out Sessions - AFM Deep Dive - Spectrum Scale and Tape - Introduction to Spectrum Fusion - Secured Data Sharing to Applications running in OpenShift - ESS Deep Dive - Spectrum Scale on Z - HDFS hands-on experience with Cloudera CDP - Terraform and Ansible Deep Dive - Introduction to Spectrum Discover 12:00 13:00 Lunch and Networking 13:00 13:20 New S3 Access for AI and Analytics 13:20 13:40 GPUDirect Storage Update 13:40 14:00 HDFS Update 14:00 14:30 Coffee and Networking 14:30 14:50 Customer or partner talk 14:50 15:10 Monitoring and serviceability enhancements 15:10 15:30 Field Update 15:30 15:50 Performance Update 15:50 16:00 Wrap-up Ulf Troppens Senior Technical Staff Member Spectrum Scale Development IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen / Gesch?ftsf?hrung: David Faller Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.taylor at psi.ch Mon Sep 5 14:43:15 2022 From: joshua.taylor at psi.ch (Joshua Taylor) Date: Mon, 5 Sep 2022 15:43:15 +0200 Subject: [gpfsug-discuss] NF4 ACLs In-Reply-To: <781575e1b00944b672b8f2ccd0e7c5f16f4558a7.camel@psi.ch> References: <61d06bd5a42200df014ee25e3197de457b582feb.camel@psi.ch> <9a98a767-2a01-333f-40f9-8155e4a35927@strath.ac.uk> <1e21585e0ee1c847a84fe477d0d2db8abd6ea5d6.camel@psi.ch> <781575e1b00944b672b8f2ccd0e7c5f16f4558a7.camel@psi.ch> Message-ID: <160432cb-7aac-80f0-7458-55f7c5aa9b06@psi.ch> Hi Again - @Alec - thanks! this helped clarify things quite a bit. However, with some experimenting, i was able to understand the "CHOWN" part. The user with the chown permission is not able to give ownership arbitrarily to any other user - they are only able to take ownership of the file. that is: given userX and userY, userY writes a file (is default owner) where file inherits ACL for userX to chown. UserX can only 'chown userX file' - essentially give ownership to themselves. They can not give ownership back to userY. There may be groups and group ACLs involved which complicate this even further, however I wasn't able to test that. This is to say the "CHOWN" permission is not 'global', like root would have. This might be useful for future documentation updates, as would delineating the difference between ACLs for files and ACLs for directories in relation to the standard posix umask file creation (@IBM). Thanks again to everyone for helping with this! Joshua Taylor --- Paul Scherrer Institut System Engineer Science IT Infrastructure and Services department (AWI) WHGA/038 Forschungstrasse 111 5232 Villigen PSI Switzerlandd +41 56 310 52 50 On 9/2/22 12:32, Taylor Joshua George (PSI) wrote: > I'll give this a try! > > Thank you! this helps clarify things a bit! > > best, > Josh > > On Fri, 2022-09-02 at 03:22 -0700, Alec wrote: >> Hmm I think you're missing in what I sent that there are TWO ACE's for >> special owner@: >> special:owner@:rwxc:allow:DirInherit >> ?(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE >> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >> ?(-)DELETE? ? (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH(X)WRITE_ACL >> (X)WRITE_ATTR (X)WRITE_NAMED >> >> special:owner@:rw-c:allow:FileInherit:InheritOnly >> ?(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE >> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >> ?(-)DELETE? ? (X)DELETE_CHILD (X)CHOWN (-)EXEC/SEARCH(X)WRITE_ACL >> (X)WRITE_ATTR (X)WRITE_NAMED >> >> The first one sets up the new DIRECTORIES with exec/SEARCH. >> The second one sets up the new FILES without EXEC/search. >> >> This provides for new files to be made 660 while directories are >> created 2770. >> >> I think you're conflating what the POSIX interface does with?regards >> to a umask when a file is created versus what?the ACL interface does >> when a file is created.? In fact the Open man page on Linux specifies >> that POSIX permissions are (mode & umask) in the ABSENCE of a default >> ACL.? NFS4 doesn't have a default ACL but the inheritance is an >> effective default ACL as it is copying the parent's ACL and building >> up from there. >> >> Hope That Helps >> >> Alec >> >> On Fri, Sep 2, 2022 at 3:08 AM Alec > > wrote: >>> Taylor, >>> ? What I provided would work for your use case 1000%... at the top >>> level you'll need to add an entry for your process or a group that >>> your process is a member of then it would be able to create files >>> that the members of the sgid group have access to... >>> >>> $ mmgetacl . >>> #NFSv4 ACL >>> #owner:someuser >>> #group:somegroup >>> special:owner@:rwxc:allow:DirInherit >>> ?(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE >>> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >>> ?(-)DELETE? ? (X)DELETE_CHILD (X)CHOWN? ? ? ? (X)EXEC/SEARCH >>> (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED >>> >>> special:owner@:rw-c:allow:FileInherit:InheritOnly >>> ?(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE >>> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >>> ?(-)DELETE? ? (X)DELETE_CHILD (X)CHOWN? ? ? ? (-)EXEC/SEARCH >>> (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED >>> >>> special:group@:rwx-:allow:DirInherit >>> ?(X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE >>> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >>> ?(X)DELETE? ? (X)DELETE_CHILD (-)CHOWN? ? ? ? (X)EXEC/SEARCH >>> (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED >>> >>> special:group@:rw--:allow:FileInherit:InheritOnly >>> ?(X)READ/LIST (X)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE >>> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED >>> ?(X)DELETE? ? (-)DELETE_CHILD (-)CHOWN? ? ? ? (-)EXEC/SEARCH >>> (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED >>> >>> special:everyone@:----:allow:FileInherit:DirInherit >>> ?(-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (-)SYNCHRONIZE >>> (-)READ_ACL? (-)READ_ATTR? (-)READ_NAMED >>> ?(-)DELETE? ? (-)DELETE_CHILD (-)CHOWN? ? ? ? (-)EXEC/SEARCH >>> (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED >>> >>> user:nongroupuser:rwx-:allow:DirInherit____ >>> >>> (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (-)SYNCHRONIZE >>> (X)READ_ACL? (X)READ_ATTR? (X)READ_NAMED____ >>> >>> (-)DELETE?? ?(-)DELETE_CHILD (-)CHOWN??????? (X)EXEC/SEARCH >>> (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED >>> >>> >>> You can decide in that last block how much control to give (or not >>> give) on that file.? New files by "nongroupuser" would be owned by >>> nongroupuser, but the group would be the group of the parent >>> directory, not their primary group. >>> >>> Alec >>> >>> On Fri, Sep 2, 2022 at 2:50 AM Taylor Joshua George (PSI) >>> > wrote: >>>> Perhaps I should try and describe my use-case a bit: >>>> >>>> There is a daemon process that gets data and must then write it out >>>> into a project directory in GPFS. The project directory is owned by the >>>> project owner, and has the setgid bit set on the directory. This is so >>>> that members of the group can read/write/etc files put in the >>>> directory. The daemon is not a member of the project group. >>>> >>>> The daemon process used to run as root, however, my goal with ACLs is >>>> to enable it to write into project dirs, without needing to change the >>>> effective UID/GID (or be a member of the group) - so, in this case, it >>>> runs as a regular user. The ACL should allow it to write into the >>>> project directory. Essentially, the ACL is to enable the daemon to run >>>> as a regular user. Once the data is written, the Daemon will (likely) >>>> never access the data again, only the users in the project. >>>> >>>> So, my goal is to have the daemon write files owned by the project >>>> Group (so, with g+rw), as the project users would just use the standard >>>> posix permissions to access the data. I suppose I'm trying to blend the >>>> use of ACLs and standard posix perms. More importantly, I'm trying to >>>> get rid of the execute bit that new files are created with/inherited. >>>> >>>> Answers to questions below... >>>> >>>> >>>> On Fri, 2022-09-02 at 09:23 +0100, Jonathan Buzzard wrote: >>>> > On 01/09/2022 22:18, Taylor Joshua George (PSI) wrote: >>>> > >>>> > > >>>> > > Hi Everyone, >>>> > > I'm trying implement some ACLs, however some of the documentation >>>> > > is a >>>> > > bit unclear to me. >>>> > > >>>> > > Using >>>> > ?> >>>> > >>>> https://www.ibm.com/docs/en/spectrum-scale/5.1.4?topic=administration-setting-nfs-v4-access-control-lists >>>> >>>> > > as a reference, I'm trying to understand what to use to achieve >>>> > > 0660 >>>> > > permissions on files and 2770 on directories. >>>> > > >>>> > >>>> > It's not clear from this whether you are trying to achieve the >>>> > equivalent of 0660 and 2770 on files and directories or have an ls >>>> > show >>>> > the permissions as 0660 and 2770. >>>> >>>> I'm trying to get the files to be created with 0660 perms (currently, >>>> they are created 0770). I've tried playing with the `rwxc` in the first >>>> line of the NFS4 ACL definition, with no luck (it seems to ignore that, >>>> except for error checking). E.g. I've tried removing the `x`, which >>>> passes the syntax check but doesn't seem to change anything. >>>> >>>> so, daemon uses ACLs to write/create files, users use posix perms to >>>> access/modify those files. >>>> >>>> > >>>> > > So far, I've managed to achieve 0000 perms, but user with the ACL >>>> > > permission can chmod, or 0770 perms. >>>> > > >>>> > >>>> > Basically neither of the above two options is possible because there >>>> > is >>>> > no exact mapping between POSIX permissions and NFSv4 ACL's. >>>> > >>>> > For example you can't get the equivalent of the set group id >>>> > permission. >>>> > You can however put an inheritable ACL for a group on the directory >>>> > that >>>> > gives r/w plus say search directory and possibly execute permissions >>>> > if >>>> > you want those as well. >>>> > >>>> > A user with ACL permissions can change permissions that is completely >>>> > expected. Note that traditional 2770 permissions are only suggestive, >>>> > the file or member of the group would be able to change them to >>>> > something else. In fact programs often do when you save, and Samba >>>> > just >>>> > completely ignores them for the most part. At least with NFSv4 ACL's >>>> > you >>>> > can remove the ACL permission :-) >>>> > >>>> > How permissions display on an ls/stat is not an exact mapping and >>>> > will >>>> > tend to go to something like 0000, but actual ability to access etc. >>>> > the >>>> > file will be based on the ACL not what you see in ls/stat. >>>> > > Attached is a txt file with the mmgetacl output, as well as file >>>> > > listing on a test file, and finally, the ACL definition I used. >>>> > > >>>> > > As one can see in the attachment, the ACL requested appears >>>> > > differently >>>> > > for what it _actually_ applied. >>>> > > >>>> > >>>> > What ACL schematics does the file system have? Is it NFSv4 or both? >>>> > >>>> >>>> It's pure NFSv4 >>>> -D? ? ? ?nfs4? ? ? ?File locking semantics in effect >>>> -k? ? ? ?nfs4? ? ? ?ACL semantics in effect >>>> >>>> > >>>> > If you are wedded to POSIX style permissions perhaps change to POSIX >>>> > ACL >>>> > schematics on the file system? >>>> >>>> I confess, I was hoping to mix the perms so as not to change our >>>> processes too much. (we currently use reg. posix perms to enable group >>>> writes [via sgid]) >>>> >>>> > >>>> > >>>> > JAB. >>>> > >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at gpfsug.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org >>>> > > -- > > Joshua Taylor > > --- > Paul Scherrer Institut > System Engineer > Science IT Infrastructure and Services department (AWI) > WHGA/038 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerlandd > +41 56 310 52 50 > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From adao at ibm.com Tue Sep 6 22:48:46 2022 From: adao at ibm.com (Anh Dao) Date: Tue, 6 Sep 2022 21:48:46 +0000 Subject: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) Message-ID: Regarding the behavior with CHOWN in Spectrum Scale, to avoid quota abuse and security exposures, we have restricted that file owners can only chown only to themselves or to a group that they are a member of. This has been noted since Scale 4.2.0: https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=applications-gpfs-exceptions-limitations-nfs-v4-acls ?NFS V4 allows ACL entries that grant users (or groups) permission to change the owner or owning group of the file (for example, with the chown command). For security reasons, GPFS now restricts this so that non-privileged users may only chown such a file to themselves (becoming the owner) or to a group that they are a member of.? Regards, Anh Dao IBM Spectrum Scale Software Developer adao at ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacreo at gmail.com Tue Sep 6 23:03:35 2022 From: anacreo at gmail.com (Alec) Date: Tue, 6 Sep 2022 15:03:35 -0700 Subject: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) In-Reply-To: References: Message-ID: Anh, I was going to call that one out. But there also isn't a reason you couldn't make your own setuid chown wrapper with some logic in it to examine the chown ACL and decide if it will allow the user to give ownership of the file away or not. You could say have it see if users are in the same primary group of the file, and ACL provides chown to allow assignment to someone else in the same primary group.. perhaps. Wouldn't be too hard to write up that wrapper. Alec On Tue, Sep 6, 2022, 2:52 PM Anh Dao wrote: > Regarding the behavior with CHOWN in Spectrum Scale, to avoid quota abuse > and security exposures, we have restricted that file owners can only chown > only to themselves or to a group that they are a member of. This has been > noted since Scale 4.2.0: > > https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=applications-gpfs-exceptions-limitations-nfs-v4-acls > > ?NFS V4 allows ACL entries that grant users (or groups) permission to > change the owner or owning group of the file (for example, with the chown > command). For security reasons, GPFS now restricts this so that > non-privileged users may only chown such a file to themselves (becoming the > owner) or to a group that they are a member of.? > > Regards, > Anh Dao > IBM Spectrum Scale > Software Developer > adao at ibm.com > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Wed Sep 7 01:22:19 2022 From: ulmer at ulmer.org (Stephen Ulmer) Date: Tue, 6 Sep 2022 20:22:19 -0400 Subject: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) In-Reply-To: References: Message-ID: Is there a way to designate a group that Scale would treat membership in as privileged? Like the ?system? group in AIX. That would push the privilege escalation back into Scale, rather than depending on scripts. It might be even better to lean on RBAC for AIX and SELinux for Linux. Maybe a flag that means to request transitions or capabilities, and just punt verification to the OS? -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Sep 6, 2022, at 5:29 PM, Alec wrote: > > ? > Anh, > I was going to call that one out. But there also isn't a reason you couldn't make your own setuid chown wrapper with some logic in it to examine the chown ACL and decide if it will allow the user to give ownership of the file away or not. > > You could say have it see if users are in the same primary group of the file, and ACL provides chown to allow assignment to someone else in the same primary group.. perhaps. Wouldn't be too hard to write up that wrapper. > > Alec > > >> On Tue, Sep 6, 2022, 2:52 PM Anh Dao wrote: >> Regarding the behavior with CHOWN in Spectrum Scale, to avoid quota abuse and security exposures, we have restricted that file owners can only chown only to themselves or to a group that they are a member of. This has been noted since Scale 4.2.0: >> https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=applications-gpfs-exceptions-limitations-nfs-v4-acls >> >> ?NFS V4 allows ACL entries that grant users (or groups) permission to change the owner or owning group of the file (for example, with the chown command). For security reasons, GPFS now restricts this so that non-privileged users may only chown such a file to themselves (becoming the owner) or to a group that they are a member of.? >> >> Regards, >> Anh Dao >> IBM Spectrum Scale >> Software Developer >> adao at ibm.com >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From adao at ibm.com Wed Sep 7 21:44:33 2022 From: adao at ibm.com (Anh Dao) Date: Wed, 7 Sep 2022 20:44:33 +0000 Subject: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) Message-ID: <51A0BB79-3BFC-4AE9-B7AC-03E5966F7A9F@ibm.com> In-Reply-To: CAGhSTwiMcszfSE0JmqAmooLE9yBGbd_v1tHsJAWuan1Rk4CRAA at mail.gmail.com In Linux, chown has the following note: man 2 chown ?Only a privileged process (Linux: one with the CAP_CHOWN capability) may change the owner of a file. The owner of a file may change the group of the file to any group of which that owner is a member. A privileged process (Linux: with CAP_CHOWN) may change the group arbitrarily.? Scale now adds NFSv4 ACLs, and the CHOWN permission is basically an additional restriction on top of what Linux does. Since Scale is only invoked after Linux has perform its checks (chown_ok https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/attr.c?h=v5.19.7), it cannot overcome the restrictions in place in the Linux VFS. Regarding the wrapper mentioned, the admin (root) is certainly able to implement such setuid wrapper, but they should be very careful on the security aspects of doing so. This seems risky for Scale to implement such program. Regards, Anh Dao IBM Spectrum Scale Software Developer adao at ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From scl at virginia.edu Wed Sep 7 22:55:54 2022 From: scl at virginia.edu (Losen, Stephen C (scl)) Date: Wed, 7 Sep 2022 21:55:54 +0000 Subject: [gpfsug-discuss] Nfs4 ACLs and rsync Message-ID: <09A7C470-BE00-4F15-A821-A2059B8E0473@virginia.edu> Hi, Can anyone recommend a file copying utility that lets nfs4 ACEs propagate unchanged? I am trying to use rsync, but it always modifies (or removes) the ACEs that I want to propagate via inheritance on newly created files. In contrast, this is not a problem when I create a new file with output redirect (such as: echo test > testfile) so rsync is doing something different. I have read the rsync man page thoroughly and tried many different options, but I?m afraid that rsync always calls chmod() or something on newly created files, which undermines ACE inheritance. I am migrating files from a NFS mounted Qumulo NAS on a system that is also a GPF cluster node, so it?s a local rsync from the NFS filesystem to the GPFS filesystem. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Thu Sep 8 14:32:25 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 8 Sep 2022 14:32:25 +0100 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings Message-ID: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> Hum, just done the first node in my DSS-G cluster to 4.2a and it looks like all the ConnectX-4 interfaces have super unhelpfully (I am being polite here) been changed from Ethernet to Infiniband!!! Has anyone else seen this behaviour? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From pieter.adams at vrt.be Thu Sep 8 17:08:12 2022 From: pieter.adams at vrt.be (Pieter Adams) Date: Thu, 8 Sep 2022 18:08:12 +0200 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> Message-ID: <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> yes, happens after firmware update of the Mellanox card(s). Kind regards Pieter Adams btw: DSS-G 4.2b is available On 08/09/2022 15:32, Jonathan Buzzard wrote: > > Hum, just done the first node in my DSS-G cluster to 4.2a and it looks > like all the ConnectX-4 interfaces have super unhelpfully (I am being > polite here) been changed from Ethernet to Infiniband!!! > > Has anyone else seen this behaviour? > > > JAB. > -- Disclaimer -- Vlaamse Radio- en Televisieomroeporganisatie Auguste Reyerslaan 52 1043 Brussel nv van publiek recht BTW BE 0244.142.664 RPR Brussel VRT Gebruikersvoorwaarden From mweil at wustl.edu Thu Sep 8 18:00:13 2022 From: mweil at wustl.edu (Weil, Matthew) Date: Thu, 8 Sep 2022 17:00:13 +0000 Subject: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) In-Reply-To: <51A0BB79-3BFC-4AE9-B7AC-03E5966F7A9F@ibm.com> References: <51A0BB79-3BFC-4AE9-B7AC-03E5966F7A9F@ibm.com> Message-ID: Hello all, Sort of on this topic has anyone have a transfer tool like rsync or mmxcp that transfers the NFSv4 ACL?s correctly? Thanks Matt From: gpfsug-discuss on behalf of Anh Dao Date: Wednesday, September 7, 2022 at 3:48 PM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] NF4 ACLs (Joshua Taylor) In-Reply-To: CAGhSTwiMcszfSE0JmqAmooLE9yBGbd_v1tHsJAWuan1Rk4CRAA at mail.gmail.com In Linux, chown has the following note: man 2 chown ?Only a privileged process (Linux: one with the CAP_CHOWN capability) may change the owner of a file. The owner of a file may change the group of the file to any group of which that owner is a member. A privileged process (Linux: with CAP_CHOWN) may change the group arbitrarily.? Scale now adds NFSv4 ACLs, and the CHOWN permission is basically an additional restriction on top of what Linux does. Since Scale is only invoked after Linux has perform its checks (chown_ok https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/attr.c?h=v5.19.7), it cannot overcome the restrictions in place in the Linux VFS. Regarding the wrapper mentioned, the admin (root) is certainly able to implement such setuid wrapper, but they should be very careful on the security aspects of doing so. This seems risky for Scale to implement such program. Regards, Anh Dao IBM Spectrum Scale Software Developer adao at ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Thu Sep 8 19:29:01 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 8 Sep 2022 19:29:01 +0100 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> Message-ID: On 08/09/2022 17:08, Pieter Adams wrote: > > yes, happens after firmware update of the Mellanox card(s). > That's a crock of sh*t not to put too fine a point on it. What's the thought on applying the firmware update to the ConnectX cards after you have shutdown GPFS but before you reinstall? That said I am sure that I have updated firmware on Mellanox cards before without loosing the interface type setting. > > btw: DSS-G 4.2b is available When did that become available? When I log onto https://serviceconnect.lenovo.com/ I apparently now have no service entitlements in the ESD section JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From novosirj at rutgers.edu Thu Sep 8 19:39:28 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Thu, 8 Sep 2022 18:39:28 +0000 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> Message-ID: <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> Thanks for the warning. I don?t believe we?d run into that before, though. Is that new? -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Sep 8, 2022, at 12:08 PM, Pieter Adams > wrote: yes, happens after firmware update of the Mellanox card(s). Kind regards Pieter Adams btw: DSS-G 4.2b is available On 08/09/2022 15:32, Jonathan Buzzard wrote: Hum, just done the first node in my DSS-G cluster to 4.2a and it looks like all the ConnectX-4 interfaces have super unhelpfully (I am being polite here) been changed from Ethernet to Infiniband!!! Has anyone else seen this behaviour? JAB. -- Disclaimer -- Vlaamse Radio- en Televisieomroeporganisatie Auguste Reyerslaan 52 1043 Brussel nv van publiek recht BTW BE 0244.142.664 RPR Brussel VRT Gebruikersvoorwaarden _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Sep 9 00:48:16 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Thu, 8 Sep 2022 23:48:16 +0000 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> Message-ID: I see them in mine: Product NameExtended Product DescriptionVersion Released Last Updated DateView Files * DSS-G Standard * Full installation tarball * 2.10b * 31 Aug 2022 * 30 Aug 2022 * * DSS-G Standard * Full installation tarball * 4.2b * 31 Aug 2022 * 31 Aug 2022 * -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Sep 8, 2022, at 2:29 PM, Jonathan Buzzard > wrote: On 08/09/2022 17:08, Pieter Adams wrote: yes, happens after firmware update of the Mellanox card(s). That's a crock of sh*t not to put too fine a point on it. What's the thought on applying the firmware update to the ConnectX cards after you have shutdown GPFS but before you reinstall? That said I am sure that I have updated firmware on Mellanox cards before without loosing the interface type setting. btw: DSS-G 4.2b is available When did that become available? When I log onto https://serviceconnect.lenovo.com/ I apparently now have no service entitlements in the ESD section JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri Sep 9 08:41:47 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 9 Sep 2022 08:41:47 +0100 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> Message-ID: <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> On 09/09/2022 00:48, Ryan Novosielski wrote: > I see them in mine: > Thanks for the information. Though just to be clear magically since the 30th of August when I last checked to yesterday I no longer have any ESD entitlements and can download *nothing* from the Lenovo Service Connect website. I have no current and no past entitlements!!! Lenovo Technical support are utterly clueless in helping me sort this out. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From novosirj at rutgers.edu Fri Sep 9 14:00:50 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 9 Sep 2022 13:00:50 +0000 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> Message-ID: <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> > On Sep 9, 2022, at 03:43, Jonathan Buzzard wrote: > > ?On 09/09/2022 00:48, Ryan Novosielski wrote: >> I see them in mine: > > Thanks for the information. Though just to be clear magically since the 30th of August when I last checked to yesterday I no longer have any ESD entitlements and can download *nothing* from the Lenovo Service Connect website. I have no current and no past entitlements!!! > > Lenovo Technical support are utterly clueless in helping me sort this out. For us, this is typically been that somebody forgot to renew (usually me, because I handle this most frequently at my organization), or that the entitlements we paid for are under a different customer number than we are registered for in Service Connect. If you have other customer numbers that you?re aware of that are not associated with your account in Service Connect, that would do it. From jonathan.buzzard at strath.ac.uk Fri Sep 9 14:52:08 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 9 Sep 2022 14:52:08 +0100 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> Message-ID: On 09/09/2022 14:00, Ryan Novosielski wrote: [SNIP] > For us, this is typically been that somebody forgot to renew (usually > me, because I handle this most frequently at my organization), or > that the entitlements we paid for are under a different customer > number than we are registered for in Service Connect. If you have > other customer numbers that you?re aware of that are not associated > with your account in Service Connect, that would do it. Expect in our case it was purchased with 5 years support in 2018, so from our perspective it does not run out till late April next year. To put it another way it is impossible to need to renew anything when purchased with 5 years support on an SR-650 based DSS-G because they have not existed for that long. In our situation the DSS-G hardware was on-site for several weeks *before* the first DSS-G software release that supported the hardware was available. My working theory is someone at Lenovo stuffed up when they created the original ESD entitlements and entered the wrong expiry date. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From matthew.robinson02 at gmail.com Fri Sep 9 14:52:58 2022 From: matthew.robinson02 at gmail.com (Matthew Robinson) Date: Fri, 9 Sep 2022 09:52:58 -0400 Subject: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings In-Reply-To: <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> Message-ID: Yes they are as a fact. Lenovo never cared about GPFS solution support. This would be the biggest bread winner in the world. Distributed computing to the biggest bigger compute models of the world. This was a side package to package to the ibm Lenovo buy out. I would have cared if I saw this on this forum. On Fri, Sep 9, 2022, 9:02 AM Ryan Novosielski wrote: > > On Sep 9, 2022, at 03:43, Jonathan Buzzard < > jonathan.buzzard at strath.ac.uk> wrote: > > > > ?On 09/09/2022 00:48, Ryan Novosielski wrote: > >> I see them in mine: > > > > Thanks for the information. Though just to be clear magically since the > 30th of August when I last checked to yesterday I no longer have any ESD > entitlements and can download *nothing* from the Lenovo Service Connect > website. I have no current and no past entitlements!!! > > > > Lenovo Technical support are utterly clueless in helping me sort this > out. > > For us, this is typically been that somebody forgot to renew (usually me, > because I handle this most frequently at my organization), or that the > entitlements we paid for are under a different customer number than we are > registered for in Service Connect. If you have other customer numbers that > you?re aware of that are not associated with your account in Service > Connect, that would do it. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sthompson2 at lenovo.com Fri Sep 9 16:40:13 2022 From: sthompson2 at lenovo.com (Simon Thompson2) Date: Fri, 9 Sep 2022 15:40:13 +0000 Subject: [gpfsug-discuss] [External] Re: DSS-G upgrade and ConnectX-4 settings In-Reply-To: References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <82aebec8-ff98-3d39-ee6c-2f74a320ccbf@strath.ac.uk> <8A35184C-FF21-4389-A9AB-C436047772EC@rutgers.edu> Message-ID: Hi Jonathan, I'm not sure at the moment why you have no entitlement showing up on ESD, but we've picked this up directly with the Lenovo entitlements team to check what is going on with this for you. Could you also email me (off list) with the Lenovo support ticket number you raised and I'll make sure we get this resolved. Simon Simon Thompson He/Him/His Sr. Manager, HPC Storage and Performance Chineham Business Park, Crockford Lane, Basingstoke, Hampshire, RG24 8WQ HPC Customer Solutions Lenovo UK +44 7788 320635 sthompson2 at lenovo.com ? Lenovo.com/uk Twitter?|?Instagram?|?Facebook?|?Linkedin?|?YouTube?|?Privacy -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: 09 September 2022 14:52 To: gpfsug-discuss at gpfsug.org Subject: [External] Re: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings On 09/09/2022 14:00, Ryan Novosielski wrote: [SNIP] > For us, this is typically been that somebody forgot to renew (usually > me, because I handle this most frequently at my organization), or that > the entitlements we paid for are under a different customer number > than we are registered for in Service Connect. If you have other > customer numbers that you're aware of that are not associated with > your account in Service Connect, that would do it. Expect in our case it was purchased with 5 years support in 2018, so from our perspective it does not run out till late April next year. To put it another way it is impossible to need to renew anything when purchased with 5 years support on an SR-650 based DSS-G because they have not existed for that long. In our situation the DSS-G hardware was on-site for several weeks *before* the first DSS-G software release that supported the hardware was available. My working theory is someone at Lenovo stuffed up when they created the original ESD entitlements and entered the wrong expiry date. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Csthompson2%40lenovo.com%7C0142f80c66fc4c9f80bf08da926abd99%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C637983284408181137%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FfQD2S%2BbOwmli4p4ky%2BJ0e6R6%2FPMYpcN%2FwCTogh6l10%3D&reserved=0 From sthompson2 at lenovo.com Fri Sep 9 16:53:24 2022 From: sthompson2 at lenovo.com (Simon Thompson2) Date: Fri, 9 Sep 2022 15:53:24 +0000 Subject: [gpfsug-discuss] [External] Re: DSS-G upgrade and ConnectX-4 settings In-Reply-To: <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> Message-ID: Just to note that I've picked this up directly with our development team and support teams internally. It is not intended behaviour to reset the adapter settings. @JAB, could you let me know the DSS-G release you were moving from and to get to 4.2a? Maybe there is some specific from and to release where this occurs. Thanks Simon ________________________________ Simon Thompson He/Him/His Sr. Manager, HPC Storage and Performance Chineham Business Park, Crockford Lane, Basingstoke, Hampshire, RG24 8WQ HPC Customer Solutions Lenovo UK [Phone]+44 7788 320635 [Email]sthompson2 at lenovo.com Lenovo.com/uk Twitter | Instagram | Facebook | Linkedin | YouTube | Privacy [cid:image003.png at 01D8C46C.ABE14950] From: gpfsug-discuss On Behalf Of Ryan Novosielski Sent: 08 September 2022 19:39 To: gpfsug main discussion list Subject: [External] Re: [gpfsug-discuss] DSS-G upgrade and ConnectX-4 settings Thanks for the warning. I don't believe we'd run into that before, though. Is that new? -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Sep 8, 2022, at 12:08 PM, Pieter Adams > wrote: yes, happens after firmware update of the Mellanox card(s). Kind regards Pieter Adams btw: DSS-G 4.2b is available On 08/09/2022 15:32, Jonathan Buzzard wrote: Hum, just done the first node in my DSS-G cluster to 4.2a and it looks like all the ConnectX-4 interfaces have super unhelpfully (I am being polite here) been changed from Ethernet to Infiniband!!! Has anyone else seen this behaviour? JAB. -- Disclaimer -- Vlaamse Radio- en Televisieomroeporganisatie Auguste Reyerslaan 52 1043 Brussel nv van publiek recht BTW BE 0244.142.664 RPR Brussel VRT Gebruikersvoorwaarden > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 92 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 128 bytes Desc: image002.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 9878 bytes Desc: image003.png URL: From jonathan.buzzard at strath.ac.uk Sat Sep 10 21:37:19 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 10 Sep 2022 20:37:19 +0000 Subject: [gpfsug-discuss] [External] Re: DSS-G upgrade and ConnectX-4 settings In-Reply-To: References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> Message-ID: On Fri, 2022-09-09 at 15:53 +0000, Simon Thompson2 wrote: > > Just to note that I?ve picked this up directly with our development > team and support teams internally. It is not intended behaviour to > reset the adapter settings. > > > @JAB, could you let me know the DSS-G release you were moving from > and to get to 4.2a? Maybe there is some specific from and to release > where this occurs. Lots of experimentation today with repeated reinstalls of 4.2b, now Lenovo's screwup with respect to my ESD entitlements has been sorted. It has nothing to do with firmware updates. If I reinstall the server via xcat the ports get set back to Infiniband, even if the server already has 4.2b installed. If I reset the ports back to Ethernet and do a forced install of the firmware (because I am already at the right level and that's what the DSS-G does see *1) then reboot the machine and it comes back up with the ports still in Ethernet mode. For reference both 4.2a and 4.2b have the same firmware level for the ConnectX-4 PCIe FDR 2-Port QSFP VPI adapter (SN30L27795_Ax) installed in the machine specifically 12.28.2006 However what I did notice is that the install is putting in Mellanox drivers rather than relying on the standard kernel drivers and stack in RHEL 8. This is noticable because the Mellanox drivers cause the device names to change from the form ens5f1 to ens5f1np1. The result is a right mess of the networking setup. Not sure who thought this was a good idea because they clearly didn't check to make sure it didn't make a right mess of things. Anyway I am reasonably confident that it's the installation of the Mellanox drivers that is messing everything up. I just can't see in the several hundred lines of Perl where it changes things back to default yet. I am not impressed with the new xcat method of install that ignores all the postscripts in the postscript table so I have to mess about manaually setting things. Well I don't I am going to edit dssgserver.stanza and add them there thank you very much and you should jolly well be documenting this shenanigins IMHO. Frankly the engineers at Lenovo have been on the crack pipe again. JAB. 1. /install/dss-g-4.2b-standard-5.1/opt/lenovo/dss/bin/dsschfw-ofed -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From novosirj at rutgers.edu Sat Sep 10 21:50:13 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Sat, 10 Sep 2022 20:50:13 +0000 Subject: [gpfsug-discuss] [External] Re: DSS-G upgrade and ConnectX-4 settings In-Reply-To: References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> Message-ID: <3FF13754-2F18-4702-8B00-AB390009EE31@rutgers.edu> > On Sep 10, 2022, at 16:38, Jonathan Buzzard wrote: > > ?On Fri, 2022-09-09 at 15:53 +0000, Simon Thompson2 wrote: >> >> Just to note that I?ve picked this up directly with our development >> team and support teams internally. It is not intended behaviour to >> reset the adapter settings. >> >> >> @JAB, could you let me know the DSS-G release you were moving from >> and to get to 4.2a? Maybe there is some specific from and to release >> where this occurs. > > Lots of experimentation today with repeated reinstalls of 4.2b, > now Lenovo's screwup with respect to my ESD entitlements has been > sorted. > > It has nothing to do with firmware updates. If I reinstall the server > via xcat the ports get set back to Infiniband, even if the server > already has 4.2b installed. > > If I reset the ports back to Ethernet and do a forced install of the > firmware (because I am already at the right level and that's what the > DSS-G does see *1) then reboot the machine and it comes back up with > the ports still in Ethernet mode. > > For reference both 4.2a and 4.2b have the same firmware level for the > ConnectX-4 PCIe FDR 2-Port QSFP VPI adapter (SN30L27795_Ax) installed > in the machine specifically 12.28.2006 > > However what I did notice is that the install is putting in Mellanox > drivers rather than relying on the standard kernel drivers and stack in > RHEL 8. > > This is noticable because the Mellanox drivers cause the device names > to change from the form ens5f1 to ens5f1np1. The result is a right mess > of the networking setup. Not sure who thought this was a good idea because they clearly didn't check to make sure it didn't make a right mess of things. > > Anyway I am reasonably confident that it's the installation of the > Mellanox drivers that is messing everything up. I just can't see in the > several hundred lines of Perl where it changes things back to default > yet. > > I am not impressed with the new xcat method of install that ignores all > the postscripts in the postscript table so I have to mess about > manaually setting things. Well I don't I am going to edit > dssgserver.stanza and add them there thank you very much and you should > jolly well be documenting this shenanigins IMHO. Frankly the engineers > at Lenovo have been on the crack pipe again. > > > JAB. > > 1. /install/dss-g-4.2b-standard-5.1/opt/lenovo/dss/bin/dsschfw-ofed It?s useful to hear these experiences before our site does this this month. What sort of DSS-G hardware do you have? As far as the change of the networking interfaces, but believe that is something that happened in a relatively recent version of OFED, 5.2 to 5.4 maybe? It?s in the release notes anyhow. That bit me once on a different system, and I rolled back until I could plan for that better. However, DSS-G installs have always used OFED, and GSS before them, not the RHEL-supplied drivers. I am as close to 100% confident about that as I am with anything. From ivano.talamo at psi.ch Tue Sep 13 13:34:35 2022 From: ivano.talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Tue, 13 Sep 2022 12:34:35 +0000 Subject: [gpfsug-discuss] Release date for wide links support Message-ID: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> Hi all, I have a question that's probably more for the IBM people in this group. A few months ago I added the idea [1] in the IBM portal to introduce again the wide links support in the samba shares and it is currently in the "Planned for future release" status. Considering that one of our cluster is somehow dependent on this feature, is there any way to know in which release/date it will be implemented? Regards, Ivano [1] https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-817 __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 11 E-Mail: ivano.talamo at psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivano.talamo at psi.ch Tue Sep 13 13:34:35 2022 From: ivano.talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Tue, 13 Sep 2022 12:34:35 +0000 Subject: [gpfsug-discuss] Release date for wide links support Message-ID: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> Hi all, I have a question that's probably more for the IBM people in this group. A few months ago I added the idea [1] in the IBM portal to introduce again the wide links support in the samba shares and it is currently in the "Planned for future release" status. Considering that one of our cluster is somehow dependent on this feature, is there any way to know in which release/date it will be implemented? Regards, Ivano [1] https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-817 __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 11 E-Mail: ivano.talamo at psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Sep 13 14:45:08 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 13 Sep 2022 13:45:08 +0000 Subject: [gpfsug-discuss] [External] Re: DSS-G upgrade and ConnectX-4 settings In-Reply-To: <3FF13754-2F18-4702-8B00-AB390009EE31@rutgers.edu> References: <505e1ab8-3374-db98-1b36-be2902671c87@strath.ac.uk> <713604d8-17de-f2ae-078e-ae221cff89ae@vrt.be> <2523702B-6D11-49AA-85A7-400B61387CC5@rutgers.edu> <3FF13754-2F18-4702-8B00-AB390009EE31@rutgers.edu> Message-ID: On Sat, 2022-09-10 at 20:50 +0000, Ryan Novosielski wrote: [SNIP] > It?s useful to hear these experiences before our site does this this > month. What sort of DSS-G hardware do you have? > We have a version 2 DSS-G system based on SR650's. It's a G2x0 style system so two SR650's with ConnectX-4 cards and D3284 shelves. > As far as the change of the networking interfaces, but believe that > is something that happened in a relatively recent version of OFED, > 5.2 to 5.4 maybe? It?s in the release notes anyhow. That bit me once > on a different system, and I rolled back until I could plan for that > better. > > However, DSS-G installs have always used OFED, and GSS before them, > not the RHEL-supplied drivers. I am as close to 100% confident about > that as I am with anything. You are correct DSS-G have always used OFED drivers. However the interface name change between stock drivers and OFED drivers that now exists throws a massive spanner in the works. Basically it means you can't have xcat (or anything else for that matter) setup the networking at install time as you could previously which sucks. I spent much of Sunday developing a shell script that nukes the network configuration from on top and sets it all up using nmcli. However it can't be run till after the install, and even then I have to put the ports back into ethernet mode and reboot first. As time passes I grow nostaglic for the days when everything was ethX. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Tue Sep 13 15:23:23 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 13 Sep 2022 14:23:23 +0000 Subject: [gpfsug-discuss] Release date for wide links support In-Reply-To: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> References: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> Message-ID: <852d1dfc8c6b9ecd1765531d9aabf5b860f0e04a.camel@strath.ac.uk> On Tue, 2022-09-13 at 12:34 +0000, Talamo Ivano Giuseppe (PSI) wrote: > Hi all, > > I have a question that's probably more for the IBM people in this > group. > A few months ago I added the idea [1] in the IBM portal to introduce > again the wide links support in the samba shares and it is > currently in the "Planned for future release" status. > Considering that one of our cluster is somehow dependent on this > feature, is there any way to know in which release/date it will be > implemented? > If wide links is enabled in Samba you can kiss good by too any security you had. I was under the impression that the Samba guys had actually removed suppor for it for this reason. I would be surprised if it was being added back in. It would certainly a retrograde step. TL;DR if you are using wide links in Samba you need to come up with an alternative solution that is actual secure ASAP. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Tue Sep 13 15:23:23 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 13 Sep 2022 14:23:23 +0000 Subject: [gpfsug-discuss] Release date for wide links support In-Reply-To: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> References: <95c3bf65c0e64d3ea25f4dac71856b05@psi.ch> Message-ID: <852d1dfc8c6b9ecd1765531d9aabf5b860f0e04a.camel@strath.ac.uk> On Tue, 2022-09-13 at 12:34 +0000, Talamo Ivano Giuseppe (PSI) wrote: > Hi all, > > I have a question that's probably more for the IBM people in this > group. > A few months ago I added the idea [1] in the IBM portal to introduce > again the wide links support in the samba shares and it is > currently in the "Planned for future release" status. > Considering that one of our cluster is somehow dependent on this > feature, is there any way to know in which release/date it will be > implemented? > If wide links is enabled in Samba you can kiss good by too any security you had. I was under the impression that the Samba guys had actually removed suppor for it for this reason. I would be surprised if it was being added back in. It would certainly a retrograde step. TL;DR if you are using wide links in Samba you need to come up with an alternative solution that is actual secure ASAP. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Tue Sep 13 19:23:01 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 13 Sep 2022 19:23:01 +0100 Subject: [gpfsug-discuss] RHEL9? Message-ID: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> Is there any word on when we might see RHEL9 support? I noticed that it is not in 5.1.5.1 Also reading the 5.1.5.1 notices, it seems to suggest support for RHEL8.5 was dropped and only 8.4 and 8.6 supported? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From christof.schmitt at us.ibm.com Wed Sep 14 17:41:12 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 14 Sep 2022 16:41:12 +0000 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> Message-ID: <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> On Tue, 2022-09-13 at 19:23 +0100, Jonathan Buzzard wrote: > Is there any word on when we might see RHEL9 support? I noticed that > it > is not in 5.1.5.1 We are working on RHEL9 support, but that hit an unexpected problem with scp, which first requires a fix from Redhat. Work is still ongoing, with a tentative target of fourth quarter. If anything else comes up, that can also change again. > Also reading the 5.1.5.1 notices, it seems to suggest support for > RHEL8.5 was dropped and only 8.4 and 8.6 supported? The general strategy is that Scale does not support any distro releases that are out of support by the Linux vendor. If there is a distro related problem, we need to have a way to get Linux support involved. Since the release of RHEL 8.6 ended support for RHEL 8.5, 8.5 is dropped from Scale. RHEL 8.4 and 8.6 receive EUS updates, so those are supported longer: https://access.redhat.com/support/policy/updates/errata#RHEL8_and_9_Life_Cycle Christof From novosirj at rutgers.edu Wed Sep 14 18:18:54 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Wed, 14 Sep 2022 17:18:54 +0000 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> Message-ID: <438E68BB-3C59-4510-B438-1926F71197C4@rutgers.edu> > On Sep 14, 2022, at 12:41 PM, Christof Schmitt wrote: > >> Also reading the 5.1.5.1 notices, it seems to suggest support for >> RHEL8.5 was dropped and only 8.4 and 8.6 supported? > > The general strategy is that Scale does not support any distro releases > that are out of support by the Linux vendor. If there is a distro > related problem, we need to have a way to get Linux support involved. > Since the release of RHEL 8.6 ended support for RHEL 8.5, 8.5 is > dropped from Scale. RHEL 8.4 and 8.6 receive EUS updates, so those are > supported longer: > > https://access.redhat.com/support/policy/updates/errata#RHEL8_and_9_Life_Cycle Just while we?re talking about this, if I can offer a slight tangent, I?d appreciate it if IBM made sure to support the final point release of a RedHat major version, regardless of whether it has EUS status or not (I can?t speak to why it wouldn?t, or if that?s an unusual case). I don?t remember the specifics surrounding what happened with RHEL 7.8 and 7.9 at this point with Spectrum Scale 4.2.3.x (and it was also an unusual situation where DDN equipment got dead-ended, making moving to a later version much more complicated than it ordinarily would be), but I remember that I was stuck in a position of choosing between security patches or Spectrum Scale support. It?s understandable if the OS itself is no longer supported that we?d need to purchase EUS, but that wasn?t the case with RHEL 7.9. -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' From christof.schmitt at us.ibm.com Wed Sep 14 18:28:01 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 14 Sep 2022 17:28:01 +0000 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: <438E68BB-3C59-4510-B438-1926F71197C4@rutgers.edu> References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> <438E68BB-3C59-4510-B438-1926F71197C4@rutgers.edu> Message-ID: On Wed, 2022-09-14 at 17:18 +0000, Ryan Novosielski wrote: > Just while we?re talking about this, if I can offer a slight tangent, > I?d appreciate it if IBM made sure to support the final point release > of a RedHat major version, regardless of whether it has EUS status or > not (I can?t speak to why it wouldn?t, or if that?s an unusual case). > I don?t remember the specifics surrounding what happened with RHEL > 7.8 and 7.9 at this point with Spectrum Scale 4.2.3.x (and it was > also an unusual situation where DDN equipment got dead-ended, making > moving to a later version much more complicated than it ordinarily > would be), but I remember that I was stuck in a position of choosing > between security patches or Spectrum Scale support. It?s > understandable if the OS itself is no longer supported that we?d need > to purchase EUS, but that wasn?t the case with RHEL 7.9. I have no insight into the specific situation with 4.2.3 and DDN. Looking at the current Spectrum Scale support: Both supported release streams, 5.1.5 and 5.1.2 do support RHEL 7.9. Christof From jonathan.buzzard at strath.ac.uk Thu Sep 15 15:11:03 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 15 Sep 2022 15:11:03 +0100 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> Message-ID: On 14/09/2022 17:41, Christof Schmitt wrote: > CAUTION: This email originated outside the University. Check before clicking links or attachments. > > On Tue, 2022-09-13 at 19:23 +0100, Jonathan Buzzard wrote: >> Is there any word on when we might see RHEL9 support? I noticed that >> it >> is not in 5.1.5.1 > > We are working on RHEL9 support, but that hit an unexpected problem > with scp, which first requires a fix from Redhat. Work is still > ongoing, with a tentative target of fourth quarter. If anything else > comes up, that can also change again. > I presume that it is related to the switch to sftp? https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know >> Also reading the 5.1.5.1 notices, it seems to suggest support for >> RHEL8.5 was dropped and only 8.4 and 8.6 supported? > > The general strategy is that Scale does not support any distro releases > that are out of support by the Linux vendor. If there is a distro > related problem, we need to have a way to get Linux support involved. > Since the release of RHEL 8.6 ended support for RHEL 8.5, 8.5 is > dropped from Scale. RHEL 8.4 and 8.6 receive EUS updates, so those are > supported longer: Fair enough. I guess I need to get my homebrew EUS 8.5 onto 8.6 now we have finished the switch from RHEL7 to RHEL8. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From p.ward at nhm.ac.uk Thu Sep 15 16:53:03 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Thu, 15 Sep 2022 15:53:03 +0000 Subject: [gpfsug-discuss] Supported samba options Message-ID: I am aware that force user and force group have not been implemented in 5.0.5, not sure for 5.1.1. I want to make user of Force directory mode, and force create mode IS there a document stating what samba options have been implemented? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 5356 bytes Desc: image001.jpg URL: From christof.schmitt at us.ibm.com Thu Sep 15 16:55:31 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 15 Sep 2022 15:55:31 +0000 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> Message-ID: <370158f317fe98d8c3d09730198b448bc81f0184.camel@us.ibm.com> On Thu, 2022-09-15 at 15:11 +0100, Jonathan Buzzard wrote: > On 14/09/2022 17:41, Christof Schmitt wrote: > > CAUTION: This email originated outside the University. Check before > > clicking links or attachments. > > > > On Tue, 2022-09-13 at 19:23 +0100, Jonathan Buzzard wrote: > > > Is there any word on when we might see RHEL9 support? I noticed > > > that > > > it > > > is not in 5.1.5.1 > > > > We are working on RHEL9 support, but that hit an unexpected problem > > with scp, which first requires a fix from Redhat. Work is still > > ongoing, with a tentative target of fourth quarter. If anything > > else > > comes up, that can also change again. > > > > I presume that it is related to the switch to sftp? > > https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know? Yes, related. That protocol change also brought a change in behavior when overwriting the same file: https://access.redhat.com/solutions/6956768 https://bugzilla.redhat.com/show_bug.cgi?id=2056884 Getting that addressed is a pre-requisite for RHEL9 support, otherwise significant parts of the mm cli would have to be changed. Christof From andersnb at ucar.edu Thu Sep 15 16:59:44 2022 From: andersnb at ucar.edu (Bill Anderson) Date: Thu, 15 Sep 2022 09:59:44 -0600 Subject: [gpfsug-discuss] question about using compression and snapshots Message-ID: Hi All, We're trying to use both compression and snapshots on our filesystem, but find that we can't delete snapshots when compression is running. Since compression can take days or longer to run on our large filesystem (80+ PB), it's making it impractical to use both on our filesystem. Has anyone had experience with using both of these features and have any tips on their coexistence? Thanks! Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From christof.schmitt at us.ibm.com Thu Sep 15 17:13:59 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 15 Sep 2022 16:13:59 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: Message-ID: On Thu, 2022-09-15 at 15:53 +0000, Paul Ward wrote: > I am aware that force user and force group have not been implemented > in 5.0.5, not sure for 5.1.1. > I want to make user of > Force directory mode, and force create mode > > IS there a document stating what samba options have been implemented? Options that are available through the mmsmb CLI or GUI are officially supported: https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=reference-mmsmb-command Christof From anacreo at gmail.com Thu Sep 15 17:26:18 2022 From: anacreo at gmail.com (Alec) Date: Thu, 15 Sep 2022 09:26:18 -0700 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: <370158f317fe98d8c3d09730198b448bc81f0184.camel@us.ibm.com> References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> <370158f317fe98d8c3d09730198b448bc81f0184.camel@us.ibm.com> Message-ID: Did you guys happen to read the patch? It doesn't just silently fix the behaviour.. now if it reaches that point I believe you'll get back an scp error. I don't think you're out of the woods. So now no data loss, but the scp will come back with an error.. may take code changes to handle that still. Thanks for sharing the thread on this one. Now, how do I get IBM Bug Proxy to insist that rsync copy operations fallback to a non-compare method when working on local... I don't think users expect rsync to go as slow as it does when used on local to local copies. Apparently they don't care about my bug reports as much as IBMs. Alec On Thu, Sep 15, 2022, 8:59 AM Christof Schmitt wrote: > On Thu, 2022-09-15 at 15:11 +0100, Jonathan Buzzard wrote: > > On 14/09/2022 17:41, Christof Schmitt wrote: > > > CAUTION: This email originated outside the University. Check before > > > clicking links or attachments. > > > > > > On Tue, 2022-09-13 at 19:23 +0100, Jonathan Buzzard wrote: > > > > Is there any word on when we might see RHEL9 support? I noticed > > > > that > > > > it > > > > is not in 5.1.5.1 > > > > > > We are working on RHEL9 support, but that hit an unexpected problem > > > with scp, which first requires a fix from Redhat. Work is still > > > ongoing, with a tentative target of fourth quarter. If anything > > > else > > > comes up, that can also change again. > > > > > > > I presume that it is related to the switch to sftp? > > > > > https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know > > > Yes, related. That protocol change also brought a change in behavior > when overwriting the same file: > https://access.redhat.com/solutions/6956768 > https://bugzilla.redhat.com/show_bug.cgi?id=2056884 > > Getting that addressed is a pre-requisite for RHEL9 support, otherwise > significant parts of the mm cli would have to be changed. > > Christof > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christof.schmitt at us.ibm.com Thu Sep 15 18:39:59 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 15 Sep 2022 17:39:59 +0000 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> <370158f317fe98d8c3d09730198b448bc81f0184.camel@us.ibm.com> Message-ID: <58f70579c3bb4611b0efd9785b1533fd61ff1e21.camel@us.ibm.com> On Thu, 2022-09-15 at 09:26 -0700, Alec wrote: > Did you guys happen to read the patch? It doesn't just silently fix > the behaviour.. now if it reaches that point I believe you'll get > back an scp error. I don't think you're out of the woods. So now no > data loss, but the scp will come back with an error.. may take code > changes to handle that still. Thanks for sharing the thread on this > one. The code just omits the O_TRUC flag. And the result is as expected, scp from and to the same file no longer truncates the file. The fix is already available in the latest CentOS9 stream updates, in case you want to test this. > Now, how do I get IBM Bug Proxy to insist that rsync copy operations > fallback to a non-compare method when working on local... I don't > think users expect rsync to go as slow as it does when used on local > to local copies. Apparently they don't care about my bug reports as > much as IBMs. Reporting the problem to the rsync project could be another option. Christof From jonathan.buzzard at strath.ac.uk Thu Sep 15 19:03:05 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 15 Sep 2022 19:03:05 +0100 Subject: [gpfsug-discuss] RHEL9? In-Reply-To: References: <08ed8e6f-0166-0c41-cf40-f41f71f227a8@strath.ac.uk> <1717729cbb67283fe1dbe8dcf670aec6ceec1fa6.camel@us.ibm.com> <370158f317fe98d8c3d09730198b448bc81f0184.camel@us.ibm.com> Message-ID: On 15/09/2022 17:26, Alec wrote: [SNIP] > Apparently they don't care about my bug reports as much as IBMs. > Well IBM do now own RedHat, so one would expect them to care more :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Thu Sep 15 19:21:43 2022 From: anacreo at gmail.com (Alec) Date: Thu, 15 Sep 2022 11:21:43 -0700 Subject: [gpfsug-discuss] question about using compression and snapshots In-Reply-To: References: Message-ID: I don't know if this would help you but I use a mod function on the inode of the file to break down the workload into smaller batches across the file system when doing compression and encryption options. In the policy.. WHERE MOD(INODE,10)<=shares If you ran your encryption as an mmyapplypolicy -M shares=1 sleep 300 mmyapplypolicy -M shares=2 You'd break your work down into 10 steps with 5 minute rests. And the shares would be cumulative so you won't miss any files when you get to 10 it's really doing 1-10.. number of shares can be any arbitrary number. Hope that helps. Alec On Thu, Sep 15, 2022, 9:01 AM Bill Anderson wrote: > > Hi All, > > We're trying to use both compression and snapshots on our filesystem, > but find that we can't delete snapshots when compression is running. Since > compression can take days or longer to run on our large filesystem (80+ > PB), it's making it impractical to use both on our filesystem. > > Has anyone had experience with using both of these features and have > any tips on their coexistence? > > Thanks! > > Bill > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.ward at nhm.ac.uk Fri Sep 16 11:02:51 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Fri, 16 Sep 2022 10:02:51 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: Message-ID: Thanks Christof, But we are already using 'hosts deny', 'hosts allow' and 'valid users' which appear to have been implemented. Is there a document showing what is implemented, rather than just supported. If there are supported commands, that replace the three I have mentioned (and force user/ force group) please let me know. We have shares where we want to restrict access to one of more servers, no password required. And shares where we want to restrict access to multiple AD users, currently not specified in AD groups, although that is an option. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Christof Schmitt Sent: 15 September 2022 17:14 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On Thu, 2022-09-15 at 15:53 +0000, Paul Ward wrote: > I am aware that force user and force group have not been implemented > in 5.0.5, not sure for 5.1.1. > I want to make user of > Force directory mode, and force create mode > > IS there a document stating what samba options have been implemented? Options that are available through the mmsmb CLI or GUI are officially supported: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.1%3Ftopic%3Dreference-mmsmb-command&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C361dcd1a12d2480f64fe08da9735dd81%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637988554860514676%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CeERUYv0f5XdKTyAOusEVZJiWt6YDOxb0GHqCXM5veU%3D&reserved=0 Christof _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C361dcd1a12d2480f64fe08da9735dd81%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637988554860514676%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5XSEaOi7Z7ttpv0NBHUxeBvaqo38oBaCMngaqnMNJC4%3D&reserved=0 From christof.schmitt at us.ibm.com Fri Sep 16 17:16:23 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Fri, 16 Sep 2022 16:16:23 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: Message-ID: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> On Fri, 2022-09-16 at 10:02 +0000, Paul Ward wrote: > But we are already using 'hosts deny', 'hosts allow' and 'valid > users' which appear to have been implemented. > Is there a document showing what is implemented, rather than just > supported. Samba has a vast list of config options, that can be seen in the smb.conf manpage (man smb.conf). Testing all possible combinations for Scale is not feasible, and some features also do not interact well with the clustered SMB server usecase on the CES nodes. So for Scale the answer is: Only the SMB options exposed through mmsmb and the GUI are tested and supported. You can try others, but do not expect support. The official way to get more options supported (through mmsmb) is to request this through an RFE. > If there are supported commands, that replace the three I have > mentioned (and force user/ force group) please let me know. "force user" and "force group" have been added in 5.1.3: https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-mmsmb-command https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=summary-changes > We have shares where we want to restrict access to one of more > servers, no password required. > And shares where we want to restrict access to multiple AD users, > currently not specified in AD groups, although that is an option. Restricting access to a SMB share can be done with SMB share ACLs. That is essentially a second layer of ACLs, specific to SMB: https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=shares-creating-smb-share-acls Regards, Christof From jonathan.buzzard at strath.ac.uk Fri Sep 16 22:40:42 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 16 Sep 2022 22:40:42 +0100 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: Message-ID: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> On 16/09/2022 11:02, Paul Ward wrote: > > Thanks Christof, > > But we are already using 'hosts deny', 'hosts allow' and 'valid users' which appear to have been implemented. > Is there a document showing what is implemented, rather than just supported. > > If there are supported commands, that replace the three I have mentioned (and force user/ force group) please let me know. > > We have shares where we want to restrict access to one of more servers, no password required. > And shares where we want to restrict access to multiple AD users, currently not specified in AD groups, although that is an option. > In my experience, though this was all many years ago, as I have not run Samba on GPFS for over a decade now (it's about to change as I am in the process of setting up some protocol nodes) the force user, etc. etc. did not work well. The "right" solution is or certainly was to use NFSv4 ACL's and the vfs_gpfs module to make it all work as near as possible to a Windows server. I of course had the realization a couple of days ago that I am going to have to put NFSv4 ACL's on everything in the file system which means backing it all up again :-( JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From enrico.tagliavini at fmi.ch Mon Sep 19 08:41:58 2022 From: enrico.tagliavini at fmi.ch (Tagliavini, Enrico) Date: Mon, 19 Sep 2022 07:41:58 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> References: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> Message-ID: We run stock Samba (not CES), with vfs_gpfs module enabled and we use POSIX ACL, it works well. Granted Windows permission do not map 1:1 to the POSIX ACL, some options will do nothing, but that's acceptable for us and avoid the use of the NFSv4 ACL, which are not supported by pretty much any common tool (e.g. rsync). -- Enrico Tagliavini Systems / Software Engineer enrico.tagliavini at fmi.ch Friedrich Miescher Institute for Biomedical Research Informatics Maulbeerstrasse 66 4058 Basel Switzerland On Fri, 2022-09-16 at 22:40 +0100, Jonathan Buzzard wrote: > On 16/09/2022 11:02, Paul Ward wrote: > > > > Thanks Christof, > > > > But we are already using 'hosts deny', 'hosts allow' and 'valid users' which appear to have been implemented. > > Is there a document showing what is implemented, rather than just supported. > > > > If there are supported commands, that replace the three I have mentioned (and force user/ force group) please let me know. > > > > We have shares where we want to restrict access to one of more servers, no password required. > > And shares where we want to restrict access to multiple AD users, currently not specified in AD groups, although that is an option. > > > > In my experience, though this was all many years ago, as I have not run > Samba on GPFS for over a decade now (it's about to change as I am in the > process of setting up some protocol nodes) the force user, etc. etc. did > not work well. > > The "right" solution is or certainly was to use NFSv4 ACL's and the > vfs_gpfs module to make it all work as near as possible to a Windows server. > > I of course had the realization a couple of days ago that I am going to > have to put NFSv4 ACL's on everything in the file system which means > backing it all up again :-( > > > JAB. > From ivano.talamo at psi.ch Tue Sep 20 14:20:43 2022 From: ivano.talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Tue, 20 Sep 2022 13:20:43 +0000 Subject: [gpfsug-discuss] Release date for wide links support Message-ID: <8335c25891ef41b798242848b9129ea2@psi.ch> Hi all, I have a question that's probably more for the IBM people in this group. A few months ago I added the idea [1] in the IBM portal to introduce again the wide links support in the samba shares and that's currently in the "Planned for future release" status. Considering that some of our clusters are somehow dependent on this feature, is there any way to know in which release/date it will be implemented? Regards, Ivano [1] https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-817 __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 11 E-Mail: ivano.talamo at psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivano.talamo at psi.ch Tue Sep 20 15:47:14 2022 From: ivano.talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Tue, 20 Sep 2022 14:47:14 +0000 Subject: [gpfsug-discuss] Release date for wide links support In-Reply-To: <8335c25891ef41b798242848b9129ea2@psi.ch> References: <8335c25891ef41b798242848b9129ea2@psi.ch> Message-ID: <59ffad872e4d40afb6574bf59e48db83@psi.ch> Sorry for the duplicate post ________________________________ From: gpfsug-discuss on behalf of Talamo Ivano Giuseppe (PSI) Sent: Tuesday, September 20, 2022 3:20 PM To: gpfsug-discuss at gpfsug.org Subject: [gpfsug-discuss] Release date for wide links support Hi all, I have a question that's probably more for the IBM people in this group. A few months ago I added the idea [1] in the IBM portal to introduce again the wide links support in the samba shares and that's currently in the "Planned for future release" status. Considering that some of our clusters are somehow dependent on this feature, is there any way to know in which release/date it will be implemented? Regards, Ivano [1] https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-817 __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 11 E-Mail: ivano.talamo at psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From christof.schmitt at us.ibm.com Tue Sep 20 16:38:02 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 20 Sep 2022 15:38:02 +0000 Subject: [gpfsug-discuss] Release date for wide links support In-Reply-To: <8335c25891ef41b798242848b9129ea2@psi.ch> References: <8335c25891ef41b798242848b9129ea2@psi.ch> Message-ID: <28b29f87da81d6e802347e1882ebfac1830f7d50.camel@us.ibm.com> On Tue, 2022-09-20 at 13:20 +0000, Talamo Ivano Giuseppe (PSI) wrote: > This Message Is From an External Sender > This message came from outside your organization. > Hi all, > > I have a question that's probably more for the IBM people in this > group. > A few months ago I added the idea [1] in the IBM portal to introduce > again the wide links support in the samba shares and that's > currently in the "Planned for future release" status. > Considering that some of our clusters are somehow dependent on this > feature, is there any way to know in which release/date it will be > implemented? I cannot comment on exact plans, but i can give some background. First, "wide links" were never a supported feature in Scale. That option was never added to the supported options in mmsmb or the GUI. Probably the assumption on the user side was that the availability of the option in the Samba code would be sufficient. Then that assumption broke with code changes in Samba 4.13. But again, that was never considered a supported feature in Scale. The public Samba 4.13 release notes explain the view of the Samba project: https://www.samba.org/samba/history/samba-4.13.0.html |Please note that the Samba developers recommend changing any Samba |installations that currently use "wide links = yes" to use bind mounts |as soon as possible, as "wide links = yes" is an inherently insecure |configuration which we would like to remove from Samba. Moving the |feature into a VFS module allows this to be done in a cleaner way |in future. Given that this is an inherently insecure feature, on a track to eventually disappear, it is unlikely that this would be added as a supported feature in Scale. Regards, Christof From p.ward at nhm.ac.uk Wed Sep 21 11:26:21 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Wed, 21 Sep 2022 10:26:21 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: Thank you Christof, We are on 5.1.1. When our system was setup our IBM setup engineer spoke with dev, and he setup the 'net use' features we are currently using. Very glad to hear not only are 'force user' and 'force group' now in mmsmb, but so are 'host allow' and 'host deny'. Looks like a further delay while we upgrade to 5.1.3... Thank you very much for this info. Btw, with the exportacl command it mentions 'user, group and system' I can't see it mention anywhere the acceptable uses of 'system'. Is it just the AD name of a server, or can it be IP address? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Christof Schmitt Sent: 16 September 2022 17:16 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On Fri, 2022-09-16 at 10:02 +0000, Paul Ward wrote: > But we are already using 'hosts deny', 'hosts allow' and 'valid users' > which appear to have been implemented. > Is there a document showing what is implemented, rather than just > supported. Samba has a vast list of config options, that can be seen in the smb.conf manpage (man smb.conf). Testing all possible combinations for Scale is not feasible, and some features also do not interact well with the clustered SMB server usecase on the CES nodes. So for Scale the answer is: Only the SMB options exposed through mmsmb and the GUI are tested and supported. You can try others, but do not expect support. The official way to get more options supported (through mmsmb) is to request this through an RFE. > If there are supported commands, that replace the three I have > mentioned (and force user/ force group) please let me know. "force user" and "force group" have been added in 5.1.3: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dreference-mmsmb-command&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C1438ba12c059478e1af808da97ff60a8%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637989420333181309%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=11zzcZ4J7q1MYHjNZkzxB91%2BXR1OXsfXDnQt4PZ%2FoZ8%3D&reserved=0 https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dsummary-changes&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C1438ba12c059478e1af808da97ff60a8%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637989420333181309%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1fS1uYsI3Q2BqY72RRPrKGSOJvzn9IeoAhVcfgifqvw%3D&reserved=0 > We have shares where we want to restrict access to one of more > servers, no password required. > And shares where we want to restrict access to multiple AD users, > currently not specified in AD groups, although that is an option. Restricting access to a SMB share can be done with SMB share ACLs. That is essentially a second layer of ACLs, specific to SMB: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dshares-creating-smb-share-acls&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C1438ba12c059478e1af808da97ff60a8%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637989420333181309%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=C040Ar8cvc%2BOWLZ12jHNedczxrzmYsAPdW%2FbXWq60No%3D&reserved=0 Regards, Christof _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C1438ba12c059478e1af808da97ff60a8%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637989420333181309%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CwbjVuElOqctr8beWl28UxkrfhHttX9jyJjQOmB2qnQ%3D&reserved=0 From p.ward at nhm.ac.uk Wed Sep 21 11:29:45 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Wed, 21 Sep 2022 10:29:45 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> References: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> Message-ID: We have been using force user/group on our older GPFS cluster for 15+ years. As it only needs to force everything to one user and once group for all samba shares it worked well. Fingers crossed it works as smoothly his time. >From my experience of changing the ACLS of over 120M files on our existing system, it only causes a metadata backup change, not the whole file. IF... you do it via the CLI, not via windows. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: 16 September 2022 22:41 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On 16/09/2022 11:02, Paul Ward wrote: > > Thanks Christof, > > But we are already using 'hosts deny', 'hosts allow' and 'valid users' which appear to have been implemented. > Is there a document showing what is implemented, rather than just supported. > > If there are supported commands, that replace the three I have mentioned (and force user/ force group) please let me know. > > We have shares where we want to restrict access to one of more servers, no password required. > And shares where we want to restrict access to multiple AD users, currently not specified in AD groups, although that is an option. > In my experience, though this was all many years ago, as I have not run Samba on GPFS for over a decade now (it's about to change as I am in the process of setting up some protocol nodes) the force user, etc. etc. did not work well. The "right" solution is or certainly was to use NFSv4 ACL's and the vfs_gpfs module to make it all work as near as possible to a Windows server. I of course had the realization a couple of days ago that I am going to have to put NFSv4 ACL's on everything in the file system which means backing it all up again :-( JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C5083afebb60c4f3dd9fd08da982c5bfd%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637989613533088822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XcMoFZmAVNSyNpLS7B2eqS8UiunJTmVMagBEh5lsgTM%3D&reserved=0 From p.ward at nhm.ac.uk Wed Sep 21 11:45:35 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Wed, 21 Sep 2022 10:45:35 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> Message-ID: For our HPC shares, we just implement one group per folder, essentially POSIX permissions. Few groups want smb access. Nearly all other shares are just SMB. Of course its the most important one that uses smb and NFS! Surely there must a be a guidance document on setting up dual protocol shares. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Tagliavini, Enrico Sent: 19 September 2022 08:42 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options We run stock Samba (not CES), with vfs_gpfs module enabled and we use POSIX ACL, it works well. Granted Windows permission do not map 1:1 to the POSIX ACL, some options will do nothing, but that's acceptable for us and avoid the use of the NFSv4 ACL, which are not supported by pretty much any common tool (e.g. rsync). -- Enrico Tagliavini Systems / Software Engineer enrico.tagliavini at fmi.ch Friedrich Miescher Institute for Biomedical Research Informatics Maulbeerstrasse 66 4058 Basel Switzerland On Fri, 2022-09-16 at 22:40 +0100, Jonathan Buzzard wrote: > On 16/09/2022 11:02, Paul Ward wrote: > > > > Thanks Christof, > > > > But we are already using 'hosts deny', 'hosts allow' and 'valid users' which appear to have been implemented. > > Is there a document showing what is implemented, rather than just supported. > > > > If there are supported commands, that replace the three I have mentioned (and force user/ force group) please let me know. > > > > We have shares where we want to restrict access to one of more servers, no password required. > > And shares where we want to restrict access to multiple AD users, currently not specified in AD groups, although that is an option. > > > > In my experience, though this was all many years ago, as I have not > run Samba on GPFS for over a decade now (it's about to change as I am > in the process of setting up some protocol nodes) the force user, etc. > etc. did not work well. > > The "right" solution is or certainly was to use NFSv4 ACL's and the > vfs_gpfs module to make it all work as near as possible to a Windows server. > > I of course had the realization a couple of days ago that I am going > to have to put NFSv4 ACL's on everything in the file system which > means backing it all up again :-( > > > JAB. > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cd80e103e060641ee84d308da9a12af35%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637991702285798537%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=39Sy9QokhwXaFnobEWiyI7XAgzGEEluwUx6T%2FradXyg%3D&reserved=0 From jonathan.buzzard at strath.ac.uk Wed Sep 21 13:36:56 2022 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 21 Sep 2022 13:36:56 +0100 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <14176972-1af6-450a-226f-574b756b539d@strath.ac.uk> Message-ID: <5590c6e0-dde3-2e30-5b84-4ea79d89b748@strath.ac.uk> On 21/09/2022 11:45, Paul Ward wrote: > > For our HPC shares, we just implement one group per folder, essentially POSIX permissions. > Few groups want smb access. > > Nearly all other shares are just SMB. > > Of course its the most important one that uses smb and NFS! > > Surely there must a be a guidance document on setting up dual protocol shares. > Don't, there in lies a pit of woe. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From christof.schmitt at us.ibm.com Wed Sep 21 16:20:39 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 21 Sep 2022 15:20:39 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: On Wed, 2022-09-21 at 10:26 +0000, Paul Ward wrote: > Btw, with the exportacl command it mentions 'user, group and system' > I can't see it mention anywhere the acceptable uses of 'system'. > Is it just the AD name of a server, or can it be IP address? I have not tested that, but it should be just the "machine account name" or "computer object", or the SID of the object in AD. It is not the IP address. Christof From p.ward at nhm.ac.uk Mon Sep 26 11:02:00 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Mon, 26 Sep 2022 10:02:00 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: Hi Christof, I can't find any permutation of computer name that works. If we upgrade to 5.1.3, where we are "valid user", "host allow", "Host deny" via the net config command, do you have any idea what will happen to these values? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Christof Schmitt Sent: 21 September 2022 16:21 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On Wed, 2022-09-21 at 10:26 +0000, Paul Ward wrote: > Btw, with the exportacl command it mentions 'user, group and system' > I can't see it mention anywhere the acceptable uses of 'system'. > Is it just the AD name of a server, or can it be IP address? I have not tested that, but it should be just the "machine account name" or "computer object", or the SID of the object in AD. It is not the IP address. Christof _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C8810fee5b7294324229f08da9be56f69%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637993706961037070%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LGeZ%2BpQTlGes5GEIuGF9E%2Fn968PUD3R1t6LdJoEi%2BiU%3D&reserved=0 From christof.schmitt at us.ibm.com Mon Sep 26 15:27:28 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Mon, 26 Sep 2022 14:27:28 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: On Mon, 2022-09-26 at 10:02 +0000, Paul Ward wrote: > I can't find any permutation of computer name that works. If there is a problem, maybe best to track this through a support ticket. > If we upgrade to 5.1.3, where we are "valid user", "host allow", > "Host deny" via the net config command, do you have any idea what > will happen to these values? The config database where these are set will be kept. So not additional steps should be required. Regards, Christof From p.ward at nhm.ac.uk Mon Sep 26 16:40:08 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Mon, 26 Sep 2022 15:40:08 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: So the 5.1.3+ implementation uses the values in the SMB registry for "valid user", "host allow", "Host deny" If we have force user/group set, via net config, will it just pick that up? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Christof Schmitt Sent: 26 September 2022 15:27 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On Mon, 2022-09-26 at 10:02 +0000, Paul Ward wrote: > I can't find any permutation of computer name that works. If there is a problem, maybe best to track this through a support ticket. > If we upgrade to 5.1.3, where we are "valid user", "host allow", "Host > deny" via the net config command, do you have any idea what will > happen to these values? The config database where these are set will be kept. So not additional steps should be required. Regards, Christof _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7C9269b4f81b3f4dd01c5108da9fcbd02d%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637997994965821459%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QhnJoFdlK9WU9WiM62joq2rC57%2FzK2EBa%2BWUFnI45wc%3D&reserved=0 From christof.schmitt at us.ibm.com Mon Sep 26 18:50:28 2022 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Mon, 26 Sep 2022 17:50:28 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> Message-ID: <853e883c665280454637f87095d05b3f635ec17d.camel@us.ibm.com> On Mon, 2022-09-26 at 15:40 +0000, Paul Ward wrote: > So the 5.1.3+ implementation uses the values in the SMB registry for > "valid user", "host allow", "Host deny" > If we have force user/group set, via net config, will it just pick > that up? Yes, whatever is set will be kept. The settings will then also be visible in mmsmb. mmsmb is just a wrapper that accesses the same config as "net conf" internally in the background. Regards, Christof From p.ward at nhm.ac.uk Thu Sep 29 16:41:10 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Thu, 29 Sep 2022 15:41:10 +0000 Subject: [gpfsug-discuss] Supported samba options In-Reply-To: <853e883c665280454637f87095d05b3f635ec17d.camel@us.ibm.com> References: <11aa0d4a4bf9566dd7b0e71f746ec9bb3c881c92.camel@us.ibm.com> <853e883c665280454637f87095d05b3f635ec17d.camel@us.ibm.com> Message-ID: Thank you for all your help. While doing some tests today I found I had access to read/write to folders that I shouldn't have as the user I was connected as. I then discovered force user and force group had been set for that share and were working in 5.1.1. Further tests confirmed this when set by: net conf setparm [share] "force user" "" I'm now very curious what tests I did months ago, that failed! As I understand it, these setting will be used and available via mmsmb in 5.1.3 upwards. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of Christof Schmitt Sent: 26 September 2022 18:50 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] Supported samba options On Mon, 2022-09-26 at 15:40 +0000, Paul Ward wrote: > So the 5.1.3+ implementation uses the values in the SMB registry for > "valid user", "host allow", "Host deny" > If we have force user/group set, via net config, will it just pick > that up? Yes, whatever is set will be kept. The settings will then also be visible in mmsmb. mmsmb is just a wrapper that accesses the same config as "net conf" internally in the background. Regards, Christof _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Ce7f740c8249f418db0c208da9fe82b0e%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637998116782749229%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hpvlzn%2BDFx3zR8mzJHPiT7ZOhiT6R7gfxP3LZDayl8o%3D&reserved=0 From scl at virginia.edu Thu Sep 29 20:14:56 2022 From: scl at virginia.edu (Losen, Stephen C (scl)) Date: Thu, 29 Sep 2022 19:14:56 +0000 Subject: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Message-ID: Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs to mmputacl, it only accepts nfs4 ACLs. You cannot run setfacl, it fails. If you run mmgetacl it shows the ACL in nfs4 format. But if you use mmgetacl -k native it shows you the ?real? ACL, which may be a posix ACL. If you have a default posix ACL set on a directory, new files inherit from the posix ACL and they themselves end up with a posix ACL. The behavior of chmod is different. If a file has a nfs4 ACL then chmod destroys it and replaces it with a nfs4 ACL that essentially mimics the permissions set by the chmod command. In particular, the new ACL only has ACEs for special:owner@, special:group@, and special:everyone at . Any other ACEs are lost. However, if the file has a posix ACL, then chmod works as expected for a posix ACL. It does not completely replace the ACL, but it may change the mask:: entry or the user:: entry or the other:: entry. If you set a nfs4 ACL on a file with a posix ACL, then it converts to a nfs4 ACL (mmgetacl -k native outputs the nfs4 ACL). Needless to say this is all rather confusing, but we had to run mmchfs -k nfs4 in order to enable SMB access, which we need. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Thu Sep 29 20:55:38 2022 From: stockf at us.ibm.com (Frederick Stock) Date: Thu, 29 Sep 2022 19:55:38 +0000 Subject: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs In-Reply-To: References: Message-ID: There is a setting at the fileset level (mmcrfileset/mmchfilest), --allow-permission-change, that allows you to control how ACLs and permission bits interact, including having both on a file. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Losen, Stephen C (scl) Date: Thursday, September 29, 2022 at 3:16 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs to mmputacl, it only accepts nfs4 ACLs. You cannot run setfacl, it fails. If you run mmgetacl it shows the ACL in nfs4 format. But if you use mmgetacl -k native it shows you the ?real? ACL, which may be a posix ACL. If you have a default posix ACL set on a directory, new files inherit from the posix ACL and they themselves end up with a posix ACL. The behavior of chmod is different. If a file has a nfs4 ACL then chmod destroys it and replaces it with a nfs4 ACL that essentially mimics the permissions set by the chmod command. In particular, the new ACL only has ACEs for special:owner@, special:group@, and special:everyone at . Any other ACEs are lost. However, if the file has a posix ACL, then chmod works as expected for a posix ACL. It does not completely replace the ACL, but it may change the mask:: entry or the user:: entry or the other:: entry. If you set a nfs4 ACL on a file with a posix ACL, then it converts to a nfs4 ACL (mmgetacl -k native outputs the nfs4 ACL). Needless to say this is all rather confusing, but we had to run mmchfs -k nfs4 in order to enable SMB access, which we need. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scl at virginia.edu Thu Sep 29 22:45:01 2022 From: scl at virginia.edu (Losen, Stephen C (scl)) Date: Thu, 29 Sep 2022 21:45:01 +0000 Subject: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Message-ID: Thanks Fred, Yes I have played with ?allow-permission-change. It basically allows permissions to be changed by 1) chmod only or 2) set ACL only, or 3) either. So if you allow either, then chmod replaces any nfs4 ACL with the traditional Unix permission bits. I played with ?setaclonly? and it disables the C library chmod() call so it returns an error code. So the chmod command fails with an error. Depending on its options rsync prints errors, in particular ?rsync -a? which tries to preserve permissions. cp -r works fine. Apparently SS supports three styles of permisisons: classic Unix mode bits, posix ACLs, or nfs4 ACLs. (Classic may just be a subset of posix ACLs) If you have a file with a nfs4 ACL and call chmod() on it, then that converts the nfs4 ACL to classic Unix mode bits. If you run mmgetacl -k native you see what looks like a posix ACL but it only has entries for user::, group::, and other::. And the nfs4 representation is analogous with special:owner@, special:group@, and special:everyone at . If you start with a posix ACL and call chmod() then you get the expected posix behavior. Chmod may modify the user::, mask::, and other:: entries but it leaves any other posix ACL entries intact. (Of course the mask:: may effectively remove permissions from some ACL entries.) Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 From: gpfsug-discuss on behalf of Frederick Stock Reply-To: gpfsug main discussion list Date: Thursday, September 29, 2022 at 3:59 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs There is a setting at the fileset level (mmcrfileset/mmchfilest), --allow-permission-change, that allows you to control how ACLs and permission bits interact, including having both on a file. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Losen, Stephen C (scl) Date: Thursday, September 29, 2022 at 3:16 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs to mmputacl, it only accepts nfs4 ACLs. You cannot run setfacl, it fails. If you run mmgetacl it shows the ACL in nfs4 format. But if you use mmgetacl -k native it shows you the ?real? ACL, which may be a posix ACL. If you have a default posix ACL set on a directory, new files inherit from the posix ACL and they themselves end up with a posix ACL. The behavior of chmod is different. If a file has a nfs4 ACL then chmod destroys it and replaces it with a nfs4 ACL that essentially mimics the permissions set by the chmod command. In particular, the new ACL only has ACEs for special:owner@, special:group@, and special:everyone at . Any other ACEs are lost. However, if the file has a posix ACL, then chmod works as expected for a posix ACL. It does not completely replace the ACL, but it may change the mask:: entry or the user:: entry or the other:: entry. If you set a nfs4 ACL on a file with a posix ACL, then it converts to a nfs4 ACL (mmgetacl -k native outputs the nfs4 ACL). Needless to say this is all rather confusing, but we had to run mmchfs -k nfs4 in order to enable SMB access, which we need. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Fri Sep 30 14:23:44 2022 From: stockf at us.ibm.com (Frederick Stock) Date: Fri, 30 Sep 2022 13:23:44 +0000 Subject: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs In-Reply-To: References: Message-ID: I am assuming you want ACLs and permission bits to coexist. If that is true are you saying that setting ?allow-permission-change to the value chmodAndUpdateAcl does not meet your needs? Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Losen, Stephen C (scl) Date: Thursday, September 29, 2022 at 5:46 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Thanks Fred, Yes I have played with ?allow-permission-change. It basically allows permissions to be changed by 1) chmod only or 2) set ACL only, or 3) either. So if you allow either, then chmod replaces any nfs4 ACL with the traditional Unix ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks Fred, Yes I have played with ?allow-permission-change. It basically allows permissions to be changed by 1) chmod only or 2) set ACL only, or 3) either. So if you allow either, then chmod replaces any nfs4 ACL with the traditional Unix permission bits. I played with ?setaclonly? and it disables the C library chmod() call so it returns an error code. So the chmod command fails with an error. Depending on its options rsync prints errors, in particular ?rsync -a? which tries to preserve permissions. cp -r works fine. Apparently SS supports three styles of permisisons: classic Unix mode bits, posix ACLs, or nfs4 ACLs. (Classic may just be a subset of posix ACLs) If you have a file with a nfs4 ACL and call chmod() on it, then that converts the nfs4 ACL to classic Unix mode bits. If you run mmgetacl -k native you see what looks like a posix ACL but it only has entries for user::, group::, and other::. And the nfs4 representation is analogous with special:owner@, special:group@, and special:everyone at . If you start with a posix ACL and call chmod() then you get the expected posix behavior. Chmod may modify the user::, mask::, and other:: entries but it leaves any other posix ACL entries intact. (Of course the mask:: may effectively remove permissions from some ACL entries.) Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 From: gpfsug-discuss on behalf of Frederick Stock Reply-To: gpfsug main discussion list Date: Thursday, September 29, 2022 at 3:59 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs There is a setting at the fileset level (mmcrfileset/mmchfilest), --allow-permission-change, that allows you to control how ACLs and permission bits interact, including having both on a file. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Losen, Stephen C (scl) Date: Thursday, September 29, 2022 at 3:16 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs to mmputacl, it only accepts nfs4 ACLs. You cannot run setfacl, it fails. If you run mmgetacl it shows the ACL in nfs4 format. But if you use mmgetacl -k native it shows you the ?real? ACL, which may be a posix ACL. If you have a default posix ACL set on a directory, new files inherit from the posix ACL and they themselves end up with a posix ACL. The behavior of chmod is different. If a file has a nfs4 ACL then chmod destroys it and replaces it with a nfs4 ACL that essentially mimics the permissions set by the chmod command. In particular, the new ACL only has ACEs for special:owner@, special:group@, and special:everyone at . Any other ACEs are lost. However, if the file has a posix ACL, then chmod works as expected for a posix ACL. It does not completely replace the ACL, but it may change the mask:: entry or the user:: entry or the other:: entry. If you set a nfs4 ACL on a file with a posix ACL, then it converts to a nfs4 ACL (mmgetacl -k native outputs the nfs4 ACL). Needless to say this is all rather confusing, but we had to run mmchfs -k nfs4 in order to enable SMB access, which we need. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Sep 30 15:58:48 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 30 Sep 2022 14:58:48 +0000 Subject: [gpfsug-discuss] Changing filesystem from -k all to -k nfs4 with mmchfs In-Reply-To: References: Message-ID: <9EAFF59E-7DD5-4517-8769-10466A855360@rutgers.edu> I don?t need this information now, but I can imagine it would be very helpful if I found myself where you were, so I just wanted to take a moment and thank you for taking the time! -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Sep 29, 2022, at 3:14 PM, Losen, Stephen C (scl) > wrote: Hi folks, Recently I asked what happens when you use ?mmchfs -k nfs4? when you already have numerous files (we have millions) with posix ACLs. I have discovered the answer ? NOTHING. No existing ACLs change. However, you cannot feed posix ACLs to mmputacl, it only accepts nfs4 ACLs. You cannot run setfacl, it fails. If you run mmgetacl it shows the ACL in nfs4 format. But if you use mmgetacl -k native it shows you the ?real? ACL, which may be a posix ACL. If you have a default posix ACL set on a directory, new files inherit from the posix ACL and they themselves end up with a posix ACL. The behavior of chmod is different. If a file has a nfs4 ACL then chmod destroys it and replaces it with a nfs4 ACL that essentially mimics the permissions set by the chmod command. In particular, the new ACL only has ACEs for special:owner@, special:group@, and special:everyone at . Any other ACEs are lost. However, if the file has a posix ACL, then chmod works as expected for a posix ACL. It does not completely replace the ACL, but it may change the mask:: entry or the user:: entry or the other:: entry. If you set a nfs4 ACL on a file with a posix ACL, then it converts to a nfs4 ACL (mmgetacl -k native outputs the nfs4 ACL). Needless to say this is all rather confusing, but we had to run mmchfs -k nfs4 in order to enable SMB access, which we need. Steve Losen Research Computing University of Virginia scl at virginia.edu 434-924-0640 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: