From amjadcsu at gmail.com  Mon Feb  5 17:37:43 2024
From: amjadcsu at gmail.com (Amjad Syed)
Date: Mon, 5 Feb 2024 17:37:43 +0000
Subject: [gpfsug-discuss] Rsync from home directory to AFM directory rename
 permission error.
Message-ID: <CAJWdRQhgBCFGLF4Wkxv=sPp_k9HAjygjwPGDui0SUyhWrgNKng@mail.gmail.com>

Hello

I am using rsync 3.1.2-10 on centos 7.

I have an afm share /gpfs/afm/XXX-share/Backup

The ACL on this share are as follows


 mmgetacl /gpfs/afm/marcela-archive/Backup_ADA_2feb24/
#NFSv4 ACL
#owner:pbj20xqu(me)
#group:bio
group:Authorised_XXX_Share_Users:r-x-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL
(X)READ_ATTR  (X)READ_NAMED
(-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL
(-)WRITE_ATTR (-)WRITE_NAMED



group:Research_XXX-Archive_RWD_Share_Users:rwx-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL
(X)READ_ATTR  (X)READ_NAMED
(X)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL
(X)WRITE_ATTR (X)WRITE_NAMED



>From my home directory  in /gpfs/home . I have a test file testrysnc


getfacl testrysnc

# file: testrysnc

# owner: pbj20xqu

# group: bio

user::rw-

group::rw-

other::rw-


pwd is my home directory


pwd

/gpfs/home/pbj20xqu



When i try to rsync i get the following error


rsync  -Xav testrysnc  /gpfs/afm/XXX/Backup_ADA_2feb24/

sending incremental file list

testrysnc

rsync: rename
"/gpfs/afm/marcela-archive/Backup_ADA_2feb24/.testrysnc.bZ3DsR" ->
"testrysnc": Permission denied (13)


sent 108 bytes  received 156 bytes  528.00 bytes/sec

total size is 0  speedup is 0.00

rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1179) [sender=3.1.2]



Does AFM support rsync.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240205/579de346/attachment.htm>

From scl at virginia.edu  Mon Feb  5 18:55:54 2024
From: scl at virginia.edu (Losen, Stephen C (scl))
Date: Mon, 5 Feb 2024 18:55:54 +0000
Subject: [gpfsug-discuss] Rsync from home directory to AFM directory
 rename permission error.
In-Reply-To: <CAJWdRQhgBCFGLF4Wkxv=sPp_k9HAjygjwPGDui0SUyhWrgNKng@mail.gmail.com>
References: <CAJWdRQhgBCFGLF4Wkxv=sPp_k9HAjygjwPGDui0SUyhWrgNKng@mail.gmail.com>
Message-ID: <840E49C1-59A9-42E9-9AA5-1563CFC2F25F@virginia.edu>

Hi,
Try enabling DELETE_CHILD on Backup_ADA_2feb24

Steve Losen
University of Virginia
Research Computing

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Amjad Syed <amjadcsu at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: Monday, February 5, 2024 at 12:39 PM
To: "gpfsug-discuss at gpfsug.org" <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] Rsync from home directory to AFM directory rename permission error.

Hello

I am using rsync 3.1.2-10 on centos 7.

I have an afm share /gpfs/afm/XXX-share/Backup

The ACL on this share are as follows



 mmgetacl /gpfs/afm/marcela-archive/Backup_ADA_2feb24/
#NFSv4 ACL
#owner:pbj20xqu(me)
#group:bio
group:Authorised_XXX_Share_Users:r-x-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED



group:Research_XXX-Archive_RWD_Share_Users:rwx-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(X)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED





From my home directory  in /gpfs/home . I have a test file testrysnc



getfacl testrysnc

# file: testrysnc

# owner: pbj20xqu

# group: bio

user::rw-

group::rw-

other::rw-



pwd is my home directory



pwd

/gpfs/home/pbj20xqu





When i try to rsync i get the following error



rsync  -Xav testrysnc  /gpfs/afm/XXX/Backup_ADA_2feb24/

sending incremental file list

testrysnc

rsync: rename "/gpfs/afm/marcela-archive/Backup_ADA_2feb24/.testrysnc.bZ3DsR" -> "testrysnc": Permission denied (13)



sent 108 bytes  received 156 bytes  528.00 bytes/sec

total size is 0  speedup is 0.00

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]





Does AFM support rsync.?




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240205/1b4ade27/attachment.htm>

From Michal.Hruska at mcomputers.cz  Wed Feb  7 13:06:03 2024
From: Michal.Hruska at mcomputers.cz (=?iso-8859-2?Q?Michal_Hru=B9ka?=)
Date: Wed, 7 Feb 2024 13:06:03 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance tuning
Message-ID: <0b1d64a68c7f458a86ec89610837ca2f@mcomputers.cz>

Dear gpfsUserGroup,

we are dealing with new gpfs cluster (Storage Scale 5.1.9 - RHEL 9.3) [3 FE servers and one storage system] and some performance issues.
We were able to tune underlying storage system to get to performance ~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.
Once we installed the gpfs - one FS accross all 8 RAID groups and we observed performance drop down to ~3300 MiB/s using the same fio test.
All tests were performed from one front-end node connected directly to the storage system via FibreChannel (4 paths each path is 32GFC).

Storage systems RAID groups are sized to fit 2MB data blocks to utilize full-stripe writes as the RAID geometry is 8+2 using 256KB block size -> 8*256KB = 2MB.
I/O pattern on FC is optimized too.
GPFS metadata were moved to NVMe SSDs on different storage system.
We already tried some obvious troubleshooting on gpfs side like: maxMBpS, scatter/cluster, different block sizes and some other parameters but there is no performance gain.

We were advised that gpfs might not perform pure sequential writes towards the storage system and therefore the storage system is performing more random I/O than sequential.

Could you please share with us some thougths about how to make gpfs as much sequential as possible? The goal is to reach at least 4000 MiB/s for sequential writes/reads.

best regards,
Michal Hru?ka



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240207/d10aaa59/attachment.htm>

From aaron.knister at gmail.com  Wed Feb  7 19:30:29 2024
From: aaron.knister at gmail.com (Aaron Knister)
Date: Wed, 7 Feb 2024 14:30:29 -0500
Subject: [gpfsug-discuss] sequential I/O write - performance tuning
In-Reply-To: <0b1d64a68c7f458a86ec89610837ca2f@mcomputers.cz>
References: <0b1d64a68c7f458a86ec89610837ca2f@mcomputers.cz>
Message-ID: <66364C6F-F44C-4D2A-93C3-E3DA3B455972@gmail.com>

What does iostat output look like when you?re running the tests on GPFS? It would be good to confirm that GPFS is successfully submitting 2MB I/o requests. 

Sent from my iPhone

> On Feb 7, 2024, at 08:08, Michal Hru?ka <Michal.Hruska at mcomputers.cz> wrote:
> 
> ?
> Dear gpfsUserGroup,
>  
> we are dealing with new gpfs cluster (Storage Scale 5.1.9 ? RHEL 9.3) [3 FE servers and one storage system] and some performance issues.
> We were able to tune underlying storage system to get to performance ~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.
> Once we installed the gpfs ? one FS accross all 8 RAID groups and we observed performance drop down to ~3300 MiB/s using the same fio test.
> All tests were performed from one front-end node connected directly to the storage system via FibreChannel (4 paths each path is 32GFC).
>  
> Storage systems RAID groups are sized to fit 2MB data blocks to utilize full-stripe writes as the RAID geometry is 8+2 using 256KB block size -> 8*256KB = 2MB.
> I/O pattern on FC is optimized too.
> GPFS metadata were moved to NVMe SSDs on different storage system.
> We already tried some obvious troubleshooting on gpfs side like: maxMBpS, scatter/cluster, different block sizes and some other parameters but there is no performance gain.
>  
> We were advised that gpfs might not perform pure sequential writes towards the storage system and therefore the storage system is performing more random I/O than sequential.
>  
> Could you please share with us some thougths about how to make gpfs as much sequential as possible? The goal is to reach at least 4000 MiB/s for sequential writes/reads.
>  
> best regards,
> Michal Hru?ka
> 
> 
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240207/2d9bec90/attachment.htm>

From janfrode at tanso.net  Wed Feb  7 23:00:01 2024
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Thu, 8 Feb 2024 00:00:01 +0100
Subject: [gpfsug-discuss] sequential I/O write - performance tuning
In-Reply-To: <66364C6F-F44C-4D2A-93C3-E3DA3B455972@gmail.com>
References: <0b1d64a68c7f458a86ec89610837ca2f@mcomputers.cz>
 <66364C6F-F44C-4D2A-93C3-E3DA3B455972@gmail.com>
Message-ID: <CAHwPathA=D88852-RSrs+VoNxSq67ha+fgfFOLiMr1_fiNO+KA@mail.gmail.com>

Also, please show mmlsconfig output.

  -jf

ons. 7. feb. 2024 kl. 20:32 skrev Aaron Knister <aaron.knister at gmail.com>:

> What does iostat output look like when you?re running the tests on GPFS?
> It would be good to confirm that GPFS is successfully submitting 2MB I/o
> requests.
>
> Sent from my iPhone
>
> On Feb 7, 2024, at 08:08, Michal Hru?ka <Michal.Hruska at mcomputers.cz>
> wrote:
>
> ?
>
> Dear gpfsUserGroup,
>
>
>
> we are dealing with new gpfs cluster (Storage Scale 5.1.9 ? RHEL 9.3) [3
> FE servers and one storage system] and some performance issues.
>
> We were able to tune underlying storage system to get to performance ~4500
> MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and
> parallel fio test.
>
> Once we installed the gpfs ? one FS accross all 8 RAID groups and we
> observed performance drop down to ~3300 MiB/s using the same fio test.
>
> All tests were performed from one front-end node connected directly to the
> storage system via FibreChannel (4 paths each path is 32GFC).
>
>
>
> Storage systems RAID groups are sized to fit 2MB data blocks to utilize
> full-stripe writes as the RAID geometry is 8+2 using 256KB block size ->
> 8*256KB = 2MB.
>
> I/O pattern on FC is optimized too.
>
> GPFS metadata were moved to NVMe SSDs on different storage system.
>
> We already tried some obvious troubleshooting on gpfs side like: maxMBpS,
> scatter/cluster, different block sizes and some other parameters but there
> is no performance gain.
>
>
>
> We were advised that gpfs might not perform pure sequential writes towards
> the storage system and therefore the storage system is performing more
> random I/O than sequential.
>
>
>
> Could you please share with us some thougths about how to make gpfs as
> much sequential as possible? The goal is to reach at least 4000 MiB/s for
> sequential writes/reads.
>
>
>
> best regards,
>
> *Michal Hru?ka*
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/a94aad7f/attachment.htm>

From p.ward at nhm.ac.uk  Thu Feb  8 12:52:34 2024
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Thu, 8 Feb 2024 12:52:34 +0000
Subject: [gpfsug-discuss] Rsync from home directory to AFM directory
 rename permission error.
In-Reply-To: <840E49C1-59A9-42E9-9AA5-1563CFC2F25F@virginia.edu>
References: <CAJWdRQhgBCFGLF4Wkxv=sPp_k9HAjygjwPGDui0SUyhWrgNKng@mail.gmail.com>
 <840E49C1-59A9-42E9-9AA5-1563CFC2F25F@virginia.edu>
Message-ID: <LO0P265MB5215E8AE5A8200C2467E0380D3442@LO0P265MB5215.GBRP265.PROD.OUTLOOK.COM>

It doesn?t sound like the same situation we had, but:
Try rsync with
--inplace

I believe default behaviour is to copy to a temp file, then rename.
--inplace writes to the file with the correct name straight off.

Kindest regards,
Paul

Paul Ward
TS Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>
[cid:image001.png at 01DA5A8D.89C21EE0]

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Losen, Stephen C (scl)
Sent: Monday, February 5, 2024 6:56 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Rsync from home directory to AFM directory rename permission error.

Hi,
Try enabling DELETE_CHILD on Backup_ADA_2feb24

Steve Losen
University of Virginia
Research Computing

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Amjad Syed <amjadcsu at gmail.com<mailto:amjadcsu at gmail.com>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Monday, February 5, 2024 at 12:39 PM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [gpfsug-discuss] Rsync from home directory to AFM directory rename permission error.

Hello

I am using rsync 3.1.2-10 on centos 7.

I have an afm share /gpfs/afm/XXX-share/Backup

The ACL on this share are as follows



 mmgetacl /gpfs/afm/marcela-archive/Backup_ADA_2feb24/
#NFSv4 ACL
#owner:pbj20xqu(me)
#group:bio
group:Authorised_XXX_Share_Users:r-x-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED



group:Research_XXX-Archive_RWD_Share_Users:rwx-:allow:FileInherit:DirInherit:Inherited
(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(X)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED





From my home directory  in /gpfs/home . I have a test file testrysnc



getfacl testrysnc

# file: testrysnc

# owner: pbj20xqu

# group: bio

user::rw-

group::rw-

other::rw-



pwd is my home directory



pwd

/gpfs/home/pbj20xqu





When i try to rsync i get the following error



rsync  -Xav testrysnc  /gpfs/afm/XXX/Backup_ADA_2feb24/

sending incremental file list

testrysnc

rsync: rename "/gpfs/afm/marcela-archive/Backup_ADA_2feb24/.testrysnc.bZ3DsR" -> "testrysnc": Permission denied (13)



sent 108 bytes  received 156 bytes  528.00 bytes/sec

total size is 0  speedup is 0.00

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]





Does AFM support rsync.?




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/0860b162/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12974 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/0860b162/attachment.png>

From Michal.Hruska at mcomputers.cz  Thu Feb  8 14:59:15 2024
From: Michal.Hruska at mcomputers.cz (=?iso-8859-2?Q?Michal_Hru=B9ka?=)
Date: Thu, 8 Feb 2024 14:59:15 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
Message-ID: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>

@Aaron
Yes, I can confirm that 2MB blocks are transfered over.

@ Jan-Frode
We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.

#mmlsconfig
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.1.9.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
pagepool 64G
maxblocksize 16384K
maxMBpS 40000
maxReceiverThreads 32
nsdMaxWorkerThreads 512
nsdMinWorkerThreads 8
nsdMultiQueue 256
nsdSmallThreadRatio 0
nsdThreadsPerQueue 3
prefetchAggressiveness 2
adminMode central

/dev/fs0

@Uwe
Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the "problematic" dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.

Best,
Michal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/e3abefe9/attachment.htm>

From novosirj at rutgers.edu  Thu Feb  8 16:44:26 2024
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 8 Feb 2024 16:44:26 +0000
Subject: [gpfsug-discuss] Key Management for Storage Scale Data Management
 Edition
Message-ID: <31F7BB83-E99F-4DFE-961A-0CC56DEC3ABE@rutgers.edu>

Hi al,

I asked this question on Slack as well, so it's a repeat for some people, but:

Anyone using the Data Management edition with a key manager that isn?t IBM?s, or have any info to share on their experiences/an alternative to IBM, etc.

We?re in the quoting phase for this, don?t have any experience with key management as required by Storage Scale, and have been warned that IBM licensing can be expensive (we also have no direct relationship with IBM and Lenovo is not sure they can resell this anymore).

--
#BlackLivesMatter
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/287982cd/attachment.htm>

From gcorneau at us.ibm.com  Thu Feb  8 18:50:08 2024
From: gcorneau at us.ibm.com (Glen Corneau)
Date: Thu, 8 Feb 2024 18:50:08 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
Message-ID: <DM6PR15MB34505E92B187A44399367F26FD442@DM6PR15MB3450.namprd15.prod.outlook.com>

Just a few thoughts:

We've often increased the seqDiscardThreshold in SAS on AIX GPFS implementations to much larger values ( a lot of large file sequential I/O).  With a pagepool of 64G, maybe set to 4 or 8GB?

Sequential writes from the application when spread across mutliple LUNs on the storage+Scale often do not resemble sequential writes from the storage POV.

If overrunning a disk device representation in the OS, the device queue depth is one parameter that might be expanded to offer some relief.
---
Glen Corneau
Senior, Power Partner Technical Specialist (PTS-P)
IBM Technology, North America
Email: gcorneau at us.ibm.com
Cell: 512-420-7988
________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Michal Hru?ka <Michal.Hruska at mcomputers.cz>
Sent: Thursday, February 8, 2024 08:59
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] sequential I/O write - performance

@Aaron Yes, I can confirm that 2MB blocks are transfered over. @ Jan-Frode We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know. #mmlsconfig autoload no dmapiFileHandleSize 32 minReleaseLevel
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2e-hQ74-QNlQ0KX70gHlDsWODTjDLcgSEK8lhkIKcCimb6m97aqUeYtL-5luaJl8omKubeXu1XqHpskHhQfv3H_4lomxcIKH$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd

@Aaron

Yes, I can confirm that 2MB blocks are transfered over.


@ Jan-Frode

We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.



#mmlsconfig

autoload no

dmapiFileHandleSize 32

minReleaseLevel 5.1.9.0

tscCmdAllowRemoteConnections no

ccrEnabled yes

cipherList AUTHONLY

sdrNotifyAuthEnabled yes

pagepool 64G

maxblocksize 16384K

maxMBpS 40000

maxReceiverThreads 32

nsdMaxWorkerThreads 512

nsdMinWorkerThreads 8

nsdMultiQueue 256

nsdSmallThreadRatio 0

nsdThreadsPerQueue 3

prefetchAggressiveness 2

adminMode central



/dev/fs0


@Uwe

Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.



Best,
Michal


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/857706dc/attachment.htm>

From janfrode at tanso.net  Thu Feb  8 19:08:08 2024
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Thu, 8 Feb 2024 20:08:08 +0100
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
Message-ID: <CAHwPatjo3G7a3BKiHcebBYb70Y4_nNC-dv+3kdJC9_U9MKEjvQ@mail.gmail.com>

You?re missing a few standard configs that might be relevant.. I would
suggest:


workerThreads=512 (or 1024)
ignoreprefetchluncount=yes
numamemoryinterleave=yes (and make sure numactl is installed)

ignoreprefetchluncount is important to tell the system that you have
multiple HDDs backing each LUN, otherwise it thinks there a single spindle,
and won?t schedule much readahead/writebehind.



  -jf


tor. 8. feb. 2024 kl. 16:00 skrev Michal Hru?ka <Michal.Hruska at mcomputers.cz
>:

> @Aaron
>
> Yes, I can confirm that 2MB blocks are transfered over.
>
> @ Jan-Frode
>
> We tried to change multiple parameters, but if you know the best
> combination for sequential IO, please let me know.
>
>
>
> #mmlsconfig
>
> autoload no
>
> dmapiFileHandleSize 32
>
> minReleaseLevel 5.1.9.0
>
> tscCmdAllowRemoteConnections no
>
> ccrEnabled yes
>
> cipherList AUTHONLY
>
> sdrNotifyAuthEnabled yes
>
> pagepool 64G
>
> maxblocksize 16384K
>
> maxMBpS 40000
>
> maxReceiverThreads 32
>
> nsdMaxWorkerThreads 512
>
> nsdMinWorkerThreads 8
>
> nsdMultiQueue 256
>
> nsdSmallThreadRatio 0
>
> nsdThreadsPerQueue 3
>
> prefetchAggressiveness 2
>
> adminMode central
>
>
>
> /dev/fs0
>
> @Uwe
>
> Using iohist we found out that gpfs is overloading one dm-device (it took
> about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we
> have enough drives to play with) for new one but the overloading issue just
> jumped to another dm-device.
> We believe that this behaviour is caused by the gpfs but we are unable to
> locate the root cause of it.
>
>
>
> Best,
> Michal
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/38252def/attachment.htm>

From anacreo at gmail.com  Thu Feb  8 19:30:58 2024
From: anacreo at gmail.com (Alec)
Date: Thu, 8 Feb 2024 11:30:58 -0800
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
Message-ID: <CAGhSTwigARnTJm2-HGZTmzV5ASrOO6Q=OcVEBd4WFBeixc2AUA@mail.gmail.com>

This won't affect your current issue but if you're doing a lot of large
sequential IO you may want to consider setting prefetchPct to 40 to 60
percent instead of default 20%.  In our environment that has measurable
impact, but we have a lot less ram than you do in the pagepool (8g versus
64g).

Also do you have a dedicated meta pool?  If not that could be a source of
contention.  Highly recommend a small pinnable LUN or two as a dedicated
meta pool.

Alec

On Thu, Feb 8, 2024, 7:01?AM Michal Hru?ka <Michal.Hruska at mcomputers.cz>
wrote:

> @Aaron
>
> Yes, I can confirm that 2MB blocks are transfered over.
>
> @ Jan-Frode
>
> We tried to change multiple parameters, but if you know the best
> combination for sequential IO, please let me know.
>
>
>
> #mmlsconfig
>
> autoload no
>
> dmapiFileHandleSize 32
>
> minReleaseLevel 5.1.9.0
>
> tscCmdAllowRemoteConnections no
>
> ccrEnabled yes
>
> cipherList AUTHONLY
>
> sdrNotifyAuthEnabled yes
>
> pagepool 64G
>
> maxblocksize 16384K
>
> maxMBpS 40000
>
> maxReceiverThreads 32
>
> nsdMaxWorkerThreads 512
>
> nsdMinWorkerThreads 8
>
> nsdMultiQueue 256
>
> nsdSmallThreadRatio 0
>
> nsdThreadsPerQueue 3
>
> prefetchAggressiveness 2
>
> adminMode central
>
>
>
> /dev/fs0
>
> @Uwe
>
> Using iohist we found out that gpfs is overloading one dm-device (it took
> about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we
> have enough drives to play with) for new one but the overloading issue just
> jumped to another dm-device.
> We believe that this behaviour is caused by the gpfs but we are unable to
> locate the root cause of it.
>
>
>
> Best,
> Michal
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240208/302c5b64/attachment.htm>

From macthev at gmail.com  Thu Feb  8 21:32:17 2024
From: macthev at gmail.com (dale mac)
Date: Fri, 9 Feb 2024 08:32:17 +1100
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <CAGhSTwigARnTJm2-HGZTmzV5ASrOO6Q=OcVEBd4WFBeixc2AUA@mail.gmail.com>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
 <CAGhSTwigARnTJm2-HGZTmzV5ASrOO6Q=OcVEBd4WFBeixc2AUA@mail.gmail.com>
Message-ID: <A441927B-9A7D-43E8-9BD4-2329183D1B28@gmail.com>

Michal,

I think you need to revise your testing method. Let me explain.

Based on my understandings:

3 FE servers and one storage system

~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.
 one FS accross all 8 RAID groups and we observed performance drop down to ~3300

The test you are running is a non-clustered fs versus a clustered fs.

XFS,

8 XFS filesystems.
Each FS has it own Array and independent Meta, not shared between nodes
Array will see sequential IO for each array and will be able to aggregate IO?s and prefetch on write.
No lock traffic between nodes
Didn?t mention for the FIO runs is this one node or the three nodes with fs?s spread across?

Clusterd,

1 Filesystem (fs0) In this case
Parallel Filsystem with shared Meta and access
Lock and Meta traffic across nodes
GPFS Stripes across NSD, 8 in this case.  
Each FIO stream will in a less sequential stream at the array level
The LBA will be spread causing the array to work harder
Array logics will not see this a sequential and delivery a much lower performance from a sequential point of view.
 


What to do,

Try 
8 FS with your FIO test like XFS test
1 FS 1 Array and matching 1 FIO ( then x8 result)



PS: You haven?t mention the type of array used? Sometimes the following is important.

Disable prefetch at the array.  This causes the array to sometimes over work it backend due to incorrectly fetching data that is never used causing xtra io and cache displacement.  Ie GPFS aggressively prefetches which triggers the array to do further prefetch and both are not used.

Dale

> On 9 Feb 2024, at 6:30 am, Alec <anacreo at gmail.com> wrote:
> 
> This won't affect your current issue but if you're doing a lot of large sequential IO you may want to consider setting prefetchPct to 40 to 60 percent instead of default 20%.  In our environment that has measurable impact, but we have a lot less ram than you do in the pagepool (8g versus 64g).
> 
> Also do you have a dedicated meta pool?  If not that could be a source of contention.  Highly recommend a small pinnable LUN or two as a dedicated meta pool.
> 
> Alec
> 
> On Thu, Feb 8, 2024, 7:01?AM Michal Hru?ka <Michal.Hruska at mcomputers.cz <mailto:Michal.Hruska at mcomputers.cz>> wrote:
>> @Aaron
>> 
>> Yes, I can confirm that 2MB blocks are transfered over.
>> 
>> 
>> @ Jan-Frode
>> 
>> We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.
>> 
>>  
>> 
>> #mmlsconfig
>> 
>> autoload no
>> 
>> dmapiFileHandleSize 32
>> 
>> minReleaseLevel 5.1.9.0
>> 
>> tscCmdAllowRemoteConnections no
>> 
>> ccrEnabled yes
>> 
>> cipherList AUTHONLY
>> 
>> sdrNotifyAuthEnabled yes
>> 
>> pagepool 64G
>> 
>> maxblocksize 16384K
>> 
>> maxMBpS 40000
>> 
>> maxReceiverThreads 32
>> 
>> nsdMaxWorkerThreads 512
>> 
>> nsdMinWorkerThreads 8
>> 
>> nsdMultiQueue 256
>> 
>> nsdSmallThreadRatio 0
>> 
>> nsdThreadsPerQueue 3
>> 
>> prefetchAggressiveness 2
>> 
>> adminMode central
>> 
>>  
>> 
>> /dev/fs0
>> 
>> 
>> @Uwe
>> 
>> Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
>> We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.
>> 
>>  
>> 
>> Best,
>> Michal
>> 
>>  
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org <http://gpfsug.org/>
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240209/fd4dde29/attachment.htm>

From spectrumscale at kiranghag.com  Fri Feb  9 02:04:09 2024
From: spectrumscale at kiranghag.com (KG)
Date: Fri, 9 Feb 2024 07:34:09 +0530
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <A441927B-9A7D-43E8-9BD4-2329183D1B28@gmail.com>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
 <CAGhSTwigARnTJm2-HGZTmzV5ASrOO6Q=OcVEBd4WFBeixc2AUA@mail.gmail.com>
 <A441927B-9A7D-43E8-9BD4-2329183D1B28@gmail.com>
Message-ID: <CAA-1hNbCA_ahrDPpwZFxZk-N3ZMS+EQaRyZsLUTbHWSBEC2bVg@mail.gmail.com>

Can you check the value of workerThreads? It seems to be default and that
is 48 without protocols.

On Fri, Feb 9, 2024 at 3:04?AM dale mac <macthev at gmail.com> wrote:

> Michal,
>
> I think you need to revise your testing method. Let me explain.
>
> Based on my understandings:
>
> 3 FE servers and one storage system
>
>
>
>    - ~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID
>    group) and parallel fio test.
>    -  one FS accross all 8 RAID groups and we observed performance drop
>    down to ~3300
>
>
> The test you are running is a non-clustered fs versus a clustered fs.
>
>
> XFS,
>
>
>
>    - 8 XFS filesystems.
>    - Each FS has it own Array and independent Meta, not shared between
>    nodes
>    - Array will see sequential IO for each array and will be able to
>    aggregate IO?s and prefetch on write.
>    - No lock traffic between nodes
>    - Didn?t mention for the FIO runs is this one node or the three nodes
>    with fs?s spread across?
>
>
> Clusterd,
>
>
>    - 1 Filesystem (fs0) In this case
>    - Parallel Filsystem with shared Meta and access
>    - Lock and Meta traffic across nodes
>    - GPFS Stripes across NSD, 8 in this case.
>    - Each FIO stream will in a less sequential stream at the array level
>    - The LBA will be spread causing the array to work harder
>    - Array logics will not see this a sequential and delivery a much
>    lower performance from a sequential point of view.
>
>
>
>
>
> What to do,
>
>
> Try
>
>
>    1. 8 FS with your FIO test like XFS test
>    2. 1 FS 1 Array and matching 1 FIO ( then x8 result)
>
>
>
>
> PS: You haven?t mention the type of array used? Sometimes the following is
> important.
>
>
>
>    - Disable prefetch at the array.  This causes the array to sometimes
>    over work it backend due to incorrectly fetching data that is never used
>    causing xtra io and cache displacement.  Ie GPFS aggressively prefetches
>    which triggers the array to do further prefetch and both are not used.
>
>
> Dale
>
> On 9 Feb 2024, at 6:30 am, Alec <anacreo at gmail.com> wrote:
>
> This won't affect your current issue but if you're doing a lot of large
> sequential IO you may want to consider setting prefetchPct to 40 to 60
> percent instead of default 20%.  In our environment that has measurable
> impact, but we have a lot less ram than you do in the pagepool (8g versus
> 64g).
>
> Also do you have a dedicated meta pool?  If not that could be a source of
> contention.  Highly recommend a small pinnable LUN or two as a dedicated
> meta pool.
>
> Alec
>
> On Thu, Feb 8, 2024, 7:01?AM Michal Hru?ka <Michal.Hruska at mcomputers.cz>
> wrote:
>
>> @Aaron
>>
>> Yes, I can confirm that 2MB blocks are transfered over.
>>
>> @ Jan-Frode
>>
>> We tried to change multiple parameters, but if you know the best
>> combination for sequential IO, please let me know.
>>
>>
>>
>> #mmlsconfig
>>
>> autoload no
>>
>> dmapiFileHandleSize 32
>>
>> minReleaseLevel 5.1.9.0
>>
>> tscCmdAllowRemoteConnections no
>>
>> ccrEnabled yes
>>
>> cipherList AUTHONLY
>>
>> sdrNotifyAuthEnabled yes
>>
>> pagepool 64G
>>
>> maxblocksize 16384K
>>
>> maxMBpS 40000
>>
>> maxReceiverThreads 32
>>
>> nsdMaxWorkerThreads 512
>>
>> nsdMinWorkerThreads 8
>>
>> nsdMultiQueue 256
>>
>> nsdSmallThreadRatio 0
>>
>> nsdThreadsPerQueue 3
>>
>> prefetchAggressiveness 2
>>
>> adminMode central
>>
>>
>>
>> /dev/fs0
>>
>> @Uwe
>>
>> Using iohist we found out that gpfs is overloading one dm-device (it took
>> about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we
>> have enough drives to play with) for new one but the overloading issue just
>> jumped to another dm-device.
>> We believe that this behaviour is caused by the gpfs but we are unable to
>> locate the root cause of it.
>>
>>
>>
>> Best,
>> Michal
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240209/97b9f34e/attachment.htm>

From macthev at gmail.com  Fri Feb  9 02:59:56 2024
From: macthev at gmail.com (dale mac)
Date: Fri, 9 Feb 2024 13:59:56 +1100
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <CAA-1hNbCA_ahrDPpwZFxZk-N3ZMS+EQaRyZsLUTbHWSBEC2bVg@mail.gmail.com>
References: <CAA-1hNbCA_ahrDPpwZFxZk-N3ZMS+EQaRyZsLUTbHWSBEC2bVg@mail.gmail.com>
Message-ID: <F6A3636E-6D36-4296-B1E0-EC2F542A1C18@gmail.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240209/180ac696/attachment.htm>

From salvet at ics.muni.cz  Fri Feb  9 08:19:54 2024
From: salvet at ics.muni.cz (Zdenek Salvet)
Date: Fri, 9 Feb 2024 09:19:54 +0100
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
Message-ID: <20240209081954.GM10934@horn.ics.muni.cz>

On Thu, Feb 08, 2024 at 02:59:15PM +0000, Michal Hru?ka wrote:
> @Uwe
> Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the "problematic" dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
> We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.

Hello,
this behaviour could be caused by an assymmetry in data paths 
of your storage, relatively small imbalance can make request queue
of a slightly slower disk grow seemingly unproportionally.

In general, I think you need to scale your GPFS parameters down, not up,
in order to force better write clustering and achieve top speed
of rotational disks unless array controllers use huge cache memory.
If you can change your benchmark workload, try synchronous writes
(dd oflag=dsync ...).

Best regards,
Zdenek Salvet                                              salvet at ics.muni.cz 
Institute of Computer Science of Masaryk University, Brno, Czech Republic
and CESNET, z.s.p.o., Prague, Czech Republic
Phone: ++420-549 49 6534                           Fax: ++420-541 212 747
----------------------------------------------------------------------------
      Teamwork is essential -- it allows you to blame someone else.




From anacreo at gmail.com  Sat Feb 10 19:14:22 2024
From: anacreo at gmail.com (Alec)
Date: Sat, 10 Feb 2024 11:14:22 -0800
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <20240209081954.GM10934@horn.ics.muni.cz>
References: <4828c6cd20d54740a94e8a4b2addc811@mcomputers.cz>
 <20240209081954.GM10934@horn.ics.muni.cz>
Message-ID: <CAGhSTwj+m9FA-fBw09AoEEpSbwQh=xeZmsrBDKZ4ofcP-CYBZg@mail.gmail.com>

You know I hadn't seen the original message when I commented earlier, sorry
about that.

It may sound dumb but have you tried tuning downwards as Zdenek says?  Try
the storage on a single RAID group and a 1MB block size, and then move up
from there.  We're on practically ancient infrastructure compared to what
you have and get throughput equivalent.


   - Try returning the page pool to 8GB maybe you're prefetching TOO much
   data (this effect happens on Oracle when it is given too much memory
   everything slows to a crawl as it tries to keep every DB in memory).
   - Define your FS on a single RAID group, see what your max performance
   is (are you sure you're not wrapping any disk/controller access with so
   many platters being engaged adding latency?)
   - Tune your block size to 1MB. (the large block size could add latency
   depending on how the storage is handling that chunk)

Once you've got a baseline on a more normal configuration you can scale up
one variable at a time and see what gives better or worse performance.  At
IBM labs we were able to achieve 2.5GB/s on a single thread and 18GB/s to
an ESS on a single Linux VM and our GPFS configuration was more or less
baseline and with 1MB block size if I remember right.

I'm building on Zdenek's answer as it's likely the step I would take in
this instance, look for areas where the scale of your configuration could
be introducing latencies.

Alec

On Fri, Feb 9, 2024 at 12:21?AM Zdenek Salvet <salvet at ics.muni.cz> wrote:

> On Thu, Feb 08, 2024 at 02:59:15PM +0000, Michal Hru?ka wrote:
> > @Uwe
> > Using iohist we found out that gpfs is overloading one dm-device (it
> took about 500ms to finish IOs). We replaced the "problematic" dm-device
> (as we have enough drives to play with) for new one but the overloading
> issue just jumped to another dm-device.
> > We believe that this behaviour is caused by the gpfs but we are unable
> to locate the root cause of it.
>
> Hello,
> this behaviour could be caused by an assymmetry in data paths
> of your storage, relatively small imbalance can make request queue
> of a slightly slower disk grow seemingly unproportionally.
>
> In general, I think you need to scale your GPFS parameters down, not up,
> in order to force better write clustering and achieve top speed
> of rotational disks unless array controllers use huge cache memory.
> If you can change your benchmark workload, try synchronous writes
> (dd oflag=dsync ...).
>
> Best regards,
> Zdenek Salvet
> salvet at ics.muni.cz
> Institute of Computer Science of Masaryk University, Brno, Czech Republic
> and CESNET, z.s.p.o., Prague, Czech Republic
> Phone: ++420-549 49 6534                           Fax: ++420-541 212 747
>
> ----------------------------------------------------------------------------
>       Teamwork is essential -- it allows you to blame someone else.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240210/594a2a71/attachment.htm>

From macthev at gmail.com  Sat Feb 10 19:39:11 2024
From: macthev at gmail.com (dale mac)
Date: Sun, 11 Feb 2024 06:39:11 +1100
Subject: [gpfsug-discuss] sequential I/O write - performance
Message-ID: <83FDFC1E-2A28-4459-A3E4-4B419F5559CE@gmail.com>

Michal,

I think you need to revise your testing method. Let me explain.

Based on my understandings:

3 FE servers and one storage system


~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.
one FS across all 8 RAID groups and we observed performance drop down to ~3300


The test you are running is a non-clustered fs versus a clustered fs.

XFS,

8 XFS filesystems.
Each FS has it own Array and independent Meta, not shared between nodes
Array will see sequential IO for each array and will be able to aggregate IO?s and prefetch on read.
No lock traffic between nodes
Didn?t mention for the FIO runs is this one node or the three nodes with 8 XFS fs?s spread across?

GPFS Clustered Filesystem

1 GPFS Filesystem (fs0) In this case
Parallel Filesystem with shared Meta and access
Lock and Meta traffic across nodes
GPFS Stripes across NSD, 8 in this case.  So each fio stream will appear random at the storage  when combined. ( this very different from your 8x XFS test)
Array logic will not see this as sequential and delivery a much lower performance from a sequential point of view as each stream is intermixed.

 
What to do,

Try 

8 individual gpfs FS with your FIO test like XFS test. Ie do like for like.  8x XFS versus 8x GPFS.  From an array perspective same io pattern.
1 gpfs FS and 1 Array and matching 1 FIO ( then x8 result)


PS: You haven?t mention the type of array used? Sometimes the following is important.


Disable prefetch at the array.  This causes the array to sometimes over work the backend due to incorrectly fetching data that is never used causing extra io and cache displacement.  Ie GPFS aggressively prefetches which triggers the array to do further prefetch and both are not used.


Dale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240211/48fbd9bf/attachment.htm>

From jonathan.buzzard at strath.ac.uk  Mon Feb 12 14:54:21 2024
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Mon, 12 Feb 2024 14:54:21 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <83FDFC1E-2A28-4459-A3E4-4B419F5559CE@gmail.com>
References: <83FDFC1E-2A28-4459-A3E4-4B419F5559CE@gmail.com>
Message-ID: <036a299d-fe8c-4a3b-9561-ec4b224fe8c9@strath.ac.uk>

On 10/02/2024 19:39, dale mac wrote:

[SNIP]

> *What to do,*
> 
> **
> 
> *Try *
> 
> 
>  1. 8 individual gpfs FS with your FIO test like XFS test. Ie do like
>     for like. *8x XFS versus 8x GPFS*. ?From an array perspective same
>     io pattern.
>  2. 1 gpfs FS and 1 Array and matching 1 FIO ( then x8 result)
> 

Pull in the eight devices, using LVM create PV's on them, and then 
combine the eight PV's into a single VG, make a single LV taking up all 
the VG, and then create a single XFS file system on the LV.

You want to use "-i 8" when creating the LV and experiment with the -I 
option to set the size of the stripe.

You now have an XFS file system that matches the GPFS file system. Rerun 
your tests comparing apples to apples rather than oranges.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From Michal.Hruska at mcomputers.cz  Wed Feb 14 18:29:24 2024
From: Michal.Hruska at mcomputers.cz (=?utf-8?B?TWljaGFsIEhydcWha2E=?=)
Date: Wed, 14 Feb 2024 18:29:24 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
Message-ID: <e10a57cb56ed4aa88a776306c2c32379@mcomputers.cz>

Dear friends,

Thank you all for your time and thoughts/ideas!

The main goal for sharing our test results comparing XFS and GPFS was to show, that the storage subsystem is able to do better if the I/O is provided in different way. We were not trying to compare XFS and GPFS directly, we understand that there will be some performance drop using GPFS (compared to ?raw? performance) but we are just surprised by the ~20-25% performance drop.

We tried to change multiple suggested parameters but we got no performance gain. As there was no change we tried to do more troubleshooting using different configurations.

To better understand what we tried I have to describe our environment a bit more:
Our underlying storage system is IBM FS7300 (each controller has 384 GB cache). There are 8 DRAIDs (8+2+1). Each DRAID has its own pool and each pool has one Volume (LUN). Every FE server (we have 3 of them) is connected directly to this storage using two 32 GFC connections. 3 client servers and FE servers are connected to LAN switch using 100GbE connection.

Testing results (metadata are located on NVMe SSD DRAID):

  1.  We used second - identical storage to test the performance but we are getting almost the same results compared to first storage. In iohist we can see that one LUN (dm-device) is probably overloaded as IO time is high ? from 300 to 500 ms.
  2.  Using both storage systems together in one big FS (GPFS): always is only one dm-device slow (according to iohist output) but the ?problematic? dm-device changes in time.
  3.  During out tests we also tried synchronous fio test but we observed significant performance drop.
  4.  We tried to compare single LUN performance GPFS against XFS: GPFS 435MB/s compared to XFS 485MB/s. From single server. The drop is not so significant but when we added more LUNs to the comparison the performance drop was more painful.
For this testing ?session? we were able to gather data by Storage Insights to check storage performance:

  1.  There is no problematic HDD ? the worst latency seen is 42ms from all 176 drives in two storage systems. Average latency is 15ms.
  2.  CPU usage was at 25% max.
  3.  ?Problematic? DRAID latency ? average is 16ms the worst is 430ms. I can not tell if there was the same peak in latency during XFS tests but I think that no (or not so bad) ? as the XFS is able to perform better than GPFS.
  4.  During our tests the write cache for all pools was fully allocated. Both for XFS and GPFS tests. Which is expected state as the cache is much faster than HDDs and it should help organize writes before they are forwarded to RAID groups.

Do you see some other possible problems we missed?
I do not want to leave it behind ?unfinished? but I am out of ideas. ?

Best,
Michal

From: Michal Hru?ka
Sent: Thursday, February 8, 2024 3:59 PM
To: 'gpfsug-discuss at gpfsug.org' <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] sequential I/O write - performance

@Aaron
Yes, I can confirm that 2MB blocks are transfered over.
@ Jan-Frode
We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.

#mmlsconfig
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.1.9.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
pagepool 64G
maxblocksize 16384K
maxMBpS 40000
maxReceiverThreads 32
nsdMaxWorkerThreads 512
nsdMinWorkerThreads 8
nsdMultiQueue 256
nsdSmallThreadRatio 0
nsdThreadsPerQueue 3
prefetchAggressiveness 2
adminMode central

/dev/fs0
@Uwe
Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.

Best,
Michal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240214/5c2c448b/attachment.htm>

From YARD at il.ibm.com  Wed Feb 14 18:47:20 2024
From: YARD at il.ibm.com (YARON DANIEL)
Date: Wed, 14 Feb 2024 18:47:20 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <e10a57cb56ed4aa88a776306c2c32379@mcomputers.cz>
References: <e10a57cb56ed4aa88a776306c2c32379@mcomputers.cz>
Message-ID: <BYAPR15MB41208FDDCA943EF3D50FBFDBF64E2@BYAPR15MB4120.namprd15.prod.outlook.com>

Hi Michal,

How are you ?

Can you also tell:

  1.  How much luns are allocate for the GPFS File-System from Storage (minimum is 16) ?
  2.  What is the Block-size defining in the GPFS FS ?
  3.  How much pools you have in the FS ?
  4.  Does all tests are run on single server ?
Regards



Yaron Daniel
94 Em Ha'Moshavot Rd
[cid:image001.png at 01DA5F86.FF6F94D0]
Storage and Cloud Consultant
Petach Tiqva, 49527
Technology Services
IBM Technology Lifecycle Service
Israel


Phone:
+972-3-916-5672

Fax:
+972-3-916-5672


Mobile:
+972-52-8395593


e-mail:
yard at il.ibm.com<mailto:yard at il.ibm.com>


Webex:            https://ibm.webex.com/meet/yard<webex:%20%20%20%20%20%20%20%20%20%20%20%20https://ibm.webex.com/meet/yard>
IBM Israel<webex:%20%20%20%20%20%20%20%20%20%20%20%20%20https://ibm.webex.com/meet/yard%0dIBM%20Israel>



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Michal Hru?ka
Sent: Wednesday, 14 February 2024 20:29
To: gpfsug-discuss at gpfsug.org
Subject: [EXTERNAL] Re: [gpfsug-discuss] sequential I/O write - performance

Dear friends, Thank you all for your time and thoughts/ideas! The main goal for sharing our test results comparing XFS and GPFS was to show, that the storage subsystem is able to do better if the I/O is provided in different way. We were not
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2e-rYp51ahlw0GUbEivFBBHwoZM0xRLp7zx9uoYh3vwIvX_PsvdQxWgLetD5R1ZaeiVSdoVnUTPHHILYFoplm1IzLjvECA$>   ?
ZjQcmQRYFpfptBannerEnd
Dear friends,

Thank you all for your time and thoughts/ideas!
The main goal for sharing our test results comparing XFS and GPFS was to show, that the storage subsystem is able to do better if the I/O is provided in different way. We were not trying to compare XFS and GPFS directly, we understand that there will be some performance drop using GPFS (compared to ?raw? performance) but we are just surprised by the ~20-25% performance drop.

We tried to change multiple suggested parameters but we got no performance gain. As there was no change we tried to do more troubleshooting using different configurations.
To better understand what we tried I have to describe our environment a bit more:
Our underlying storage system is IBM FS7300 (each controller has 384 GB cache). There are 8 DRAIDs (8+2+1). Each DRAID has its own pool and each pool has one Volume (LUN). Every FE server (we have 3 of them) is connected directly to this storage using two 32 GFC connections. 3 client servers and FE servers are connected to LAN switch using 100GbE connection.
Testing results (metadata are located on NVMe SSD DRAID):

  1.  We used second - identical storage to test the performance but we are getting almost the same results compared to first storage. In iohist we can see that one LUN (dm-device) is probably overloaded as IO time is high ? from 300 to 500 ms.
  2.  Using both storage systems together in one big FS (GPFS): always is only one dm-device slow (according to iohist output) but the ?problematic? dm-device changes in time.
  3.  During out tests we also tried synchronous fio test but we observed significant performance drop.
  4.  We tried to compare single LUN performance GPFS against XFS: GPFS 435MB/s compared to XFS 485MB/s. From single server. The drop is not so significant but when we added more LUNs to the comparison the performance drop was more painful.
For this testing ?session? we were able to gather data by Storage Insights to check storage performance:

  1.  There is no problematic HDD ? the worst latency seen is 42ms from all 176 drives in two storage systems. Average latency is 15ms.
  2.  CPU usage was at 25% max.
  3.  ?Problematic? DRAID latency ? average is 16ms the worst is 430ms. I can not tell if there was the same peak in latency during XFS tests but I think that no (or not so bad) ? as the XFS is able to perform better than GPFS.
  4.  During our tests the write cache for all pools was fully allocated. Both for XFS and GPFS tests. Which is expected state as the cache is much faster than HDDs and it should help organize writes before they are forwarded to RAID groups.

Do you see some other possible problems we missed?
I do not want to leave it behind ?unfinished? but I am out of ideas. ?

Best,
Michal
From: Michal Hru?ka
Sent: Thursday, February 8, 2024 3:59 PM
To: 'gpfsug-discuss at gpfsug.org' <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] sequential I/O write - performance

@Aaron
Yes, I can confirm that 2MB blocks are transfered over.
@ Jan-Frode
We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.

#mmlsconfig
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.1.9.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
pagepool 64G
maxblocksize 16384K
maxMBpS 40000
maxReceiverThreads 32
nsdMaxWorkerThreads 512
nsdMinWorkerThreads 8
nsdMultiQueue 256
nsdSmallThreadRatio 0
nsdThreadsPerQueue 3
prefetchAggressiveness 2
adminMode central

/dev/fs0
@Uwe
Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device.
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.

Best,
Michal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240214/5e488c59/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1049 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240214/5e488c59/attachment.png>

From janfrode at tanso.net  Wed Feb 14 21:44:20 2024
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 14 Feb 2024 22:44:20 +0100
Subject: [gpfsug-discuss] sequential I/O write - performance
In-Reply-To: <e10a57cb56ed4aa88a776306c2c32379@mcomputers.cz>
References: <e10a57cb56ed4aa88a776306c2c32379@mcomputers.cz>
Message-ID: <CAHwPathyeiNiEhVvZq5HFaS8qqjrbbOmUD25rLezGxLOWZryZQ@mail.gmail.com>

Have disabled prefetching on the FS7200?

chsystem -cache_prefetch off


  -jf

ons. 14. feb. 2024 kl. 19:31 skrev Michal Hru?ka <
Michal.Hruska at mcomputers.cz>:

> Dear friends,
>
>
>
> Thank you all for your time and thoughts/ideas!
>
> The main goal for sharing our test results comparing XFS and GPFS was to
> show, that the storage subsystem is able to do better if the I/O is
> provided in different way. We were not trying to compare XFS and GPFS
> directly, we understand that there will be some performance drop using GPFS
> (compared to ?raw? performance) but we are just surprised by the ~20-25%
> performance drop.
>
>
>
> We tried to change multiple suggested parameters but we got no performance
> gain. As there was no change we tried to do more troubleshooting using
> different configurations.
>
> To better understand what we tried I have to describe our environment a
> bit more:
>
> Our underlying storage system is IBM FS7300 (each controller has 384 GB
> cache). There are 8 DRAIDs (8+2+1). Each DRAID has its own pool and each
> pool has one Volume (LUN). Every FE server (we have 3 of them) is connected
> directly to this storage using two 32 GFC connections. 3 client servers and
> FE servers are connected to LAN switch using 100GbE connection.
>
> Testing results (metadata are located on NVMe SSD DRAID):
>
>    1. We used second - identical storage to test the performance but we
>    are getting almost the same results compared to first storage. In iohist we
>    can see that one LUN (dm-device) is probably overloaded as IO time is high
>    ? from 300 to 500 ms.
>    2. Using both storage systems together in one big FS (GPFS): always is
>    only one dm-device slow (according to iohist output) but the ?problematic?
>    dm-device changes in time.
>    3. During out tests we also tried synchronous fio test but we observed
>    significant performance drop.
>    4. We tried to compare single LUN performance GPFS against XFS: GPFS
>    435MB/s compared to XFS 485MB/s. From single server. The drop is not so
>    significant but when we added more LUNs to the comparison the performance
>    drop was more painful.
>
> For this testing ?session? we were able to gather data by Storage Insights
> to check storage performance:
>
>    1. There is no problematic HDD ? the worst latency seen is 42ms from
>    all 176 drives in two storage systems. Average latency is 15ms.
>    2. CPU usage was at 25% max.
>    3. ?Problematic? DRAID latency ? average is 16ms the worst is 430ms. I
>    can not tell if there was the same peak in latency during XFS tests but I
>    think that no (or not so bad) ? as the XFS is able to perform better than
>    GPFS.
>    4. During our tests the write cache for all pools was fully allocated.
>    Both for XFS and GPFS tests. Which is expected state as the cache is much
>    faster than HDDs and it should help organize writes before they are
>    forwarded to RAID groups.
>
>
>
> Do you see some other possible problems we missed?
>
> I do not want to leave it behind ?unfinished? but I am out of ideas. ?
>
>
>
> Best,
> Michal
>
> *From:* Michal Hru?ka
> *Sent:* Thursday, February 8, 2024 3:59 PM
> *To:* 'gpfsug-discuss at gpfsug.org' <gpfsug-discuss at gpfsug.org>
> *Subject:* Re: [gpfsug-discuss] sequential I/O write - performance
>
>
>
> @Aaron
>
> Yes, I can confirm that 2MB blocks are transfered over.
>
> @ Jan-Frode
>
> We tried to change multiple parameters, but if you know the best
> combination for sequential IO, please let me know.
>
>
>
> #mmlsconfig
>
> autoload no
>
> dmapiFileHandleSize 32
>
> minReleaseLevel 5.1.9.0
>
> tscCmdAllowRemoteConnections no
>
> ccrEnabled yes
>
> cipherList AUTHONLY
>
> sdrNotifyAuthEnabled yes
>
> pagepool 64G
>
> maxblocksize 16384K
>
> maxMBpS 40000
>
> maxReceiverThreads 32
>
> nsdMaxWorkerThreads 512
>
> nsdMinWorkerThreads 8
>
> nsdMultiQueue 256
>
> nsdSmallThreadRatio 0
>
> nsdThreadsPerQueue 3
>
> prefetchAggressiveness 2
>
> adminMode central
>
>
>
> /dev/fs0
>
> @Uwe
>
> Using iohist we found out that gpfs is overloading one dm-device (it took
> about 500ms to finish IOs). We replaced the ?problematic? dm-device (as we
> have enough drives to play with) for new one but the overloading issue just
> jumped to another dm-device.
> We believe that this behaviour is caused by the gpfs but we are unable to
> locate the root cause of it.
>
>
>
> Best,
> Michal
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240214/bdbc3fae/attachment.htm>

From jonathan.buzzard at strath.ac.uk  Mon Feb 19 13:35:43 2024
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Mon, 19 Feb 2024 13:35:43 +0000
Subject: [gpfsug-discuss] [External] GUI followup
In-Reply-To: <SEYPR03MB66508A7C47B0450A9D483B82B17D2@SEYPR03MB6650.apcprd03.prod.outlook.com>
References: <70acbe2f-5160-4473-a932-4f7bb33126b4@strath.ac.uk>
 <SEYPR03MB66508A7C47B0450A9D483B82B17D2@SEYPR03MB6650.apcprd03.prod.outlook.com>
Message-ID: <7b895851-fe16-45b2-aa89-406221bc88d5@strath.ac.uk>

On 30/01/2024 18:06, Nicolas CALIMET wrote:
> 
> Hi,
> 
> Try the following command on the Confluent management server:
> 
> osdeploy initialize -l
> 
> The last option is dash-lowercase-L.
> See meaning of the multiple initialize options with "osdeploy initialize -h".
> 

Getting back to it, as it was an xcat 4.4a deployment that was not an 
option. However making it so that the two GUI nodes could SSH 
passwordlessly back into the xcat server fixed the problem. This 
requirement would appear to be undocumented as far as I can tell.

However the next road block is when trying to setup the GUI server, I get

   Discovery of servers and storage enclosures failed. Run
   mmdiscovercomp -N GUI_RG_SERVERS command in the CLI to see the
   problem.

Running that in the command line, I can see it identifying the servers 
and enclosures but then I get this

   Identifying component groups ...

   ERROR: local variable 'enc_class' referenced before assignment

Oh boy, this GUI thing seems as robust as a piece of wet toilet paper.

Could this be related to the fact the two DSS-G servers are called 
gpfs[1,2]? This created lots of problems when setting the system up in 
the past with scripts bombing out to the point where I had to create the 
file system by hand.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From Michal.Hruska at mcomputers.cz  Mon Feb 19 16:00:23 2024
From: Michal.Hruska at mcomputers.cz (=?iso-8859-2?Q?Michal_Hru=B9ka?=)
Date: Mon, 19 Feb 2024 16:00:23 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
Message-ID: <5af06327a3fd412ab88cccef1716c8c8@mcomputers.cz>

Hello,

@Jan-Frode
Yes, we tried to disable cache prefetch on storage but this is not the best way for gaining performance on sequential writes.

@Yaron
1.) There are 8 LUNs exported from single storage system. Each LUN has its own pool and its own mdisk (8+2+1 drives). When we doubled LUNs per pool the performance dropped a bit. When we used bigger mdisks - with more drives(76) we have seen performance gain using multiple LUNs per pool.
2.) Block-size was set to multiple values - 2MB, 4MB, 16MB. The best was to use 4MB. Even when the storage systems writes 2MB blocks on one mdisk as stripe size is fixed at 256KB and there is 8 data drives in one mdisk.
3.) As there is only one type of LUNs (data) exported from storage system we used only one pool in GPFS FS and it was the system pool.
4.) Not always - we tried to use multiple clients (up to 3) and multiple servers (up to 3) but we have not seen performance gain. We were able to get the same performance from clients communicating with servers over 100GbE LAN using native client connection.

Best,
Michal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240219/39fbf59e/attachment.htm>

From INDULISB at uk.ibm.com  Mon Feb 26 16:58:24 2024
From: INDULISB at uk.ibm.com (Indulis Bernsteins1)
Date: Mon, 26 Feb 2024 16:58:24 +0000
Subject: [gpfsug-discuss] sequential I/O write - performance
Message-ID: <PH0PR15MB50872F5530F522D70B54F6BFE15A2@PH0PR15MB5087.namprd15.prod.outlook.com>

> Using iohist we found out that gpfs is overloading one dm-device (it

> took about 500ms to finish IOs). We replaced the "problematic"

> dm-device (as we have enough drives to play with) for new one but the

> overloading issue just jumped to another dm-device.

> > We believe that this behaviour is caused by the gpfs but we are

> > unable

> to locate the root cause of it.

>

> Hello,

> this behaviour could be caused by an assymmetry in data paths of your

> storage, relatively small imbalance can make request queue of a

> slightly slower disk grow seemingly unproportionally.

This problem is a real "blast from the past" for me. I saw similar behaviour a LONG time ago, and I think it is very possible that you might have a "preferred paths" issue with your NSD servers to your target drives. If you are using Scale talking to a Storage System which has multiple paths to the device, and multiple Scale NSD servers can see the same LUN (which is correct from availability) then you can in some cases get exactly this sort of behaviour.

I am guessing you are running "LAN free" architecture, with many servers doing direct I/O to the NSDs/LUNs. Not doing Scale Client -> Scale NSD server -> NSD/LUN

I'll bet you see low I/O rates and long latencies to the "problem" NSD/LUN/drive.

The 500ms I/O delay can be the target NSD/LUN being switched from being "owned" by one of the controllers in the storage system to the other.

I can't see how Scale can do anything to make a device take 500 ms to complete an I/O when tracked by IOHIST at the OS level - because you are clearly not able to drive a lot of throughput to the devices, so it can't bethat  device overloading is causing a long queue on the device. There is something else happening. Not at Scale, not at the device, it is somewhere in whatever network or SAN is between the Scale NSD server and the NSD device. Something is trying to do recovery.

Say your Scale NSD servers sends an I/O to a target NSD and it goes to Storage controller A. Then another Scale NSD server sends an I/O to the same target NSD and instead of it going via a path that leads it to Storage system  controller A it goes to Storage controller B. At that point the storage system says "Oh it looks like future I/O will be coming into Storage controller B, let's switch the internal ownership to B. OK, we need to flush write caches and do some other things. That will take about 500 ms."

Then an I/O goes to Storage System controller A, and you get another switch back of the LUN from B to A. Another 500 ms.

The drive is being "ping pong-ed" from one Storage System controller to another. Because there are I/Os randomly coming on to the drive to one or other Storage Controller.

You need to make sure that all NSD servers access each LUN using the same default path to the same Storage System controller. There is a way to do this, to choose the "preferred path" unless that path is down. Could be that some servers can't use the "preferred path"?

This will probably only happen if you have something running on the actual Scale NSD servers that is accessing the filesystem, otherwise Scale Clients will always go across the Scale network to the current Primary NSD server to get to an NSD.

Or there is some other problem causing a "ping pong" effect. But it sounds like a "ping pong" to me, especially because when you replaced the dm device the problem moved elsewhere.

Regards,

Indulis Bernsteins
Storage Architect, IBM Worldwide Technical Sales








Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/9bbff522/attachment.htm>

From robert.horton at icr.ac.uk  Mon Feb 26 17:26:18 2024
From: robert.horton at icr.ac.uk (Robert Horton)
Date: Mon, 26 Feb 2024 17:26:18 +0000
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
Message-ID: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>

Hello,

We?ve been using nfs4_get/setfacl to manage nfs4 acls as an alternative to the mm commands for a few months. It mostly works pretty well ? in particular it?s simple to set permissions with inheritance flags recursively on a directory structure and it?s easy to add an ace for a user/group without affecting existing permissions.

We?ve noticed though that when permissions have been changed via SMB they get these DACL ACL flags set:

# mmgetacl NewTest
#NFSv4 ACL
#owner:ICR\rhorton
#group:ICR\infotech
#ACL flags:
#  DACL_PRESENT
#  DACL_AUTO_INHERITED
#  SACL_AUTO_INHERITED
#  NULL_SACL
user:ICR\rhorton:r-x-:allow:FileInherit:DirInherit
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

?which upsets nfs4_getfacl:

# nfs4_getfacl NewTest
# file: NewTest
A::bin:C
Bad Ace Type:768
Error while printing ACE.

It gets worse if someone does a nfs4_getfacl -R -a <new ace> as it sometimes ends up replacing the existing ACEs with one for a ?bin? user.

I?m guessing that with ACL flags present the presented system.nfs4_acl extended attribute isn?t in a form that nfs4-acl-tools expects..?

So?

  1.  What do the DACL flags actually do? Where are they stored? Is there a way to check for / modify them? I can see them referenced in https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=users-authorizing-file-protocol but still not entirely clear. Is it possible to get the Samba gpfs module to not set them? I?m guessing not from https://www.samba.org/samba/docs/current/man-html/vfs_gpfs.8.html
  2.  Am I right in thinking this is a Scale bug? If so is it possible to get it fixed ? ?
  3.  How do other people manage ACLs?

Background: It?s an ESS5000. Client access is via a remote mount to an HPC system, a large number of Windows/Mac/few Linux SMB clients and a handful of NFS (4) clients via CES. We?re on 5.1.9, although the cluster and fs versions are a bit behind. -k is nfs4.

Any thoughts appreciated!

Thanks,
Rob

Robert Horton | Scientific Computing Infrastructure Lead
The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
T +44 (0) 20 7153 5350 | E robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk> | W www.icr.ac.uk<http://www.icr.ac.uk/> | Twitter @ICR_London<https://twitter.com/ICR_London>
Facebook www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch>
Making the discoveries that defeat cancer
[ICR Logo]<http://www.icr.ac.uk/>


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/c6334fb2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3162 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/c6334fb2/attachment.gif>

From janfrode at tanso.net  Mon Feb 26 18:16:47 2024
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Mon, 26 Feb 2024 19:16:47 +0100
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
In-Reply-To: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
References: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
Message-ID: <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>

It?s not just the nfs4_setfacl tool. Also cp and rsync will fail to cooy
such ACLs.

I have an RFE for this :

https://ideas.ibm.com/ideas/GPFS-I-986



  -jf

man. 26. feb. 2024 kl. 18:27 skrev Robert Horton <robert.horton at icr.ac.uk>:

> Hello,
>
>
>
> We?ve been using nfs4_get/setfacl to manage nfs4 acls as an alternative to
> the mm commands for a few months. It mostly works pretty well ? in
> particular it?s simple to set permissions with inheritance flags
> recursively on a directory structure and it?s easy to add an ace for a
> user/group without affecting existing permissions.
>
>
>
> We?ve noticed though that when permissions have been changed via SMB they
> get these DACL ACL flags set:
>
>
>
> # mmgetacl NewTest
>
> #NFSv4 ACL
>
> #owner:ICR\rhorton
>
> #group:ICR\infotech
>
> #ACL flags:
>
> #  DACL_PRESENT
>
> #  DACL_AUTO_INHERITED
>
> #  SACL_AUTO_INHERITED
>
> #  NULL_SACL
>
> user:ICR\rhorton:r-x-:allow:FileInherit:DirInherit
>
> (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL
> (X)READ_ATTR  (X)READ_NAMED
>
> (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL
> (-)WRITE_ATTR (-)WRITE_NAMED
>
>
>
> ?which upsets nfs4_getfacl:
>
>
>
> # nfs4_getfacl NewTest
>
> # file: NewTest
>
> A::bin:C
>
> Bad Ace Type:768
>
> Error while printing ACE.
>
>
>
> It gets worse if someone does a nfs4_getfacl -R -a <new ace> as it
> sometimes ends up replacing the existing ACEs with one for a ?bin? user.
>
>
>
> I?m guessing that with ACL flags present the presented system.nfs4_acl
> extended attribute isn?t in a form that nfs4-acl-tools expects..?
>
>
>
> So?
>
>    1. What do the DACL flags actually do? Where are they stored? Is there
>    a way to check for / modify them? I can see them referenced in
>    https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=users-authorizing-file-protocol
>    but still not entirely clear. Is it possible to get the Samba gpfs module
>    to not set them? I?m guessing not from
>    https://www.samba.org/samba/docs/current/man-html/vfs_gpfs.8.html
>    2. Am I right in thinking this is a Scale bug? If so is it possible to
>    get it fixed ? ?
>    3. How do other people manage ACLs?
>
>
>
> Background: It?s an ESS5000. Client access is via a remote mount to an HPC
> system, a large number of Windows/Mac/few Linux SMB clients and a handful
> of NFS (4) clients via CES. We?re on 5.1.9, although the cluster and fs
> versions are a bit behind. -k is nfs4.
>
>
>
> Any thoughts appreciated!
>
>
>
> Thanks,
>
> Rob
>
>
>
> *Robert Horton* | Scientific Computing Infrastructure Lead
>
> The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
> <https://www.google.com/maps/search/237+Fulham+Road,+London,+SW3+6JB?entry=gmail&source=g>
>
> *T* +44 (0) 20 7153 5350 | *E* robert.horton at icr.ac.uk | *W* www.icr.ac.uk
> | *Twitter* @ICR_London <https://twitter.com/ICR_London>
>
> *Facebook* www.facebook.com/theinstituteofcancerresearch
>
> *Making the discoveries that defeat cancer*
>
> [image: ICR Logo] <http://www.icr.ac.uk/>
>
>
>
> The Institute of Cancer Research: Royal Cancer Hospital, a charitable
> Company Limited by Guarantee, Registered in England under Company No.
> 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP
> <https://www.google.com/maps/search/123+Old+Brompton+Road,+London+SW7+3RP?entry=gmail&source=g>.
>
>
> This e-mail message is confidential and for use by the addressee only. If
> the message is received by anyone other than the addressee, please return
> the message to the sender by replying to it and then delete the message
> from your computer and network.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/671ac45a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3162 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/671ac45a/attachment.gif>

From scale at us.ibm.com  Mon Feb 26 20:37:52 2024
From: scale at us.ibm.com (scale)
Date: Mon, 26 Feb 2024 20:37:52 +0000
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
In-Reply-To: <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
References: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
Message-ID: <BL1PR15MB536175176CF8D15A1E740003835A2@BL1PR15MB5361.namprd15.prod.outlook.com>

When ACLs contain the DACL/SACL flags, nfs4_getfacl producing incorrect/unexpected output is a known problem.

This is because the nfs4-acl-tools manage DACL and SACL via the extended attributes system.nfs4_dacl and system.nfs4_sacl, respectively.

The system.nfs4_acl attribute, which Scale currently supports, only expect NFSv4 ACLs without such flags. There are two issues in Scale with system.nfs4_acl: mmxcp accesses the system.nfs4_acl attribute, which does not expose the ACL flags. On the other hand, the nfs4-acl-tools access the system.nfs4_acl attribute and that does not present ACL with the ACL flags set correctly.

In Scale, the ACL flags are stored as another field in the ACL data structure that results in a different ACL header length compared to a regular NFSv4 ACL without the flags. When modifying ACLs via Samba/SMB/Windows, the ACL flags are always set. If one wants to clear those ACL flags, mmputacl or nfs4_setfacl -S can be performed. A future Scale release is under consideration to address this gap with the ACL flags. If there are mixed OS environments (e.g Windows and Linux) and data is exported for SMB clients, it is recommended to manage ACLs from a Windows client.

https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=users-working-acls

Anh Dao
Storage Scale Developer
adao at ibm.com
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Jan-Frode Myklebust <janfrode at tanso.net>
Date: Monday, February 26, 2024 at 11:19?AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] DACLs and nfs4-acl-tools
It?s not just the nfs4_setfacl tool. Also cp and rsync will fail to cooy such ACLs. I have an RFE for this : https:?//ideas.?ibm.?com/ideas/GPFS-I-986 -jf man. 26. feb. 2024 kl. 18:?27 skrev Robert Horton <robert.?horton@?icr.?ac.?uk>: Hello,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!12-vrJF3prOyMQvwNA4NTvuZV-UwL8nUzX1Xd2FqZ4Uaz85QpnrLfsaZTjPihs_gRnNQnINY4kE-qNWudUbUrVBZ1Ob-_FxzjMMWueeQN_MQDmN6nZqtdTrQ$>


ZjQcmQRYFpfptBannerEnd
It?s not just the nfs4_setfacl tool. Also cp and rsync will fail to cooy such ACLs.

I have an RFE for this :

https://ideas.ibm.com/ideas/GPFS-I-986



  -jf

man. 26. feb. 2024 kl. 18:27 skrev Robert Horton <robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk>>:
Hello,

We?ve been using nfs4_get/setfacl to manage nfs4 acls as an alternative to the mm commands for a few months. It mostly works pretty well ? in particular it?s simple to set permissions with inheritance flags recursively on a directory structure and it?s easy to add an ace for a user/group without affecting existing permissions.

We?ve noticed though that when permissions have been changed via SMB they get these DACL ACL flags set:

# mmgetacl NewTest
#NFSv4 ACL
#owner:ICR\rhorton
#group:ICR\infotech
#ACL flags:
#  DACL_PRESENT
#  DACL_AUTO_INHERITED
#  SACL_AUTO_INHERITED
#  NULL_SACL
user:ICR\rhorton:r-x-:allow:FileInherit:DirInherit
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
(-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

?which upsets nfs4_getfacl:

# nfs4_getfacl NewTest
# file: NewTest
A::bin:C
Bad Ace Type:768
Error while printing ACE.

It gets worse if someone does a nfs4_getfacl -R -a <new ace> as it sometimes ends up replacing the existing ACEs with one for a ?bin? user.

I?m guessing that with ACL flags present the presented system.nfs4_acl extended attribute isn?t in a form that nfs4-acl-tools expects..?

So?

  1.  What do the DACL flags actually do? Where are they stored? Is there a way to check for / modify them? I can see them referenced in https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=users-authorizing-file-protocol but still not entirely clear. Is it possible to get the Samba gpfs module to not set them? I?m guessing not from https://www.samba.org/samba/docs/current/man-html/vfs_gpfs.8.html<https://www.samba.org/samba/docs/current/man-html/vfs_gpfs.8.html>
  2.  Am I right in thinking this is a Scale bug? If so is it possible to get it fixed ? ?
  3.  How do other people manage ACLs?

Background: It?s an ESS5000. Client access is via a remote mount to an HPC system, a large number of Windows/Mac/few Linux SMB clients and a handful of NFS (4) clients via CES. We?re on 5.1.9, although the cluster and fs versions are a bit behind. -k is nfs4.

Any thoughts appreciated!

Thanks,
Rob

Robert Horton | Scientific Computing Infrastructure Lead
The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB<https://www.google.com/maps/search/237+Fulham+Road,+London,+SW3+6JB?entry=gmail&source=g>
T +44 (0) 20 7153 5350 | E robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk> | W www.icr.ac.uk<http://www.icr.ac.uk/> | Twitter @ICR_London<https://twitter.com/ICR_London>
Facebook www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch>
Making the discoveries that defeat cancer
[ICR Logo]<http://www.icr.ac.uk/>


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP<https://www.google.com/maps/search/123+Old+Brompton+Road,+London+SW7+3RP?entry=gmail&source=g>.

This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/7ac180b5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3162 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240226/7ac180b5/attachment.gif>

From jonathan.buzzard at strath.ac.uk  Mon Feb 26 22:21:05 2024
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Mon, 26 Feb 2024 22:21:05 +0000
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
In-Reply-To: <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
References: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
Message-ID: <92d9b3ad-7b46-446a-a753-28b89832b81e@strath.ac.uk>

On 26/02/2024 18:16, Jan-Frode Myklebust wrote:

> It?s not just the nfs4_setfacl tool. Also cp and rsync will fail to cooy 
> such ACLs.
> 
> I have an RFE for this :
> 
> https://ideas.ibm.com/ideas/GPFS-I-986 
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fideas.ibm.com%2Fideas%2FGPFS-I-986&data=05%7C02%7Cjonathan.buzzard%40strath.ac.uk%7C13dbdc56c38c4236cfe508dc36f789c4%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C638445684027108564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=lu5eSEI751NFgfUM9BaK4Vd4ao1ZasMTynvxXsB%2FLlU%3D&reserved=0>
> 
> 

At this point in time requests for being able to recursively set NFSv4 
ACL's on GPFS are over 12 years old. At least that would be about the 
time frame for requests for the feature made by myself.

Unfortunately I think the chances of IBM doing anything about it is 
somewhere around ?

Shame really as it would not be particularly hard to do. I have an old 
proof of concept that uses the FreeBSD tool (mainly of licensing 
reasons). Basically it transforms the GPFS ACL to the storage format 
used by the FreeBSD tool. Just needs refactoring to use the GPFS ACL 
storage format throughout. Probably about a weeks developer effort.

There are some poorly publicly documented features of the GPFS ACL 
format that make me reluctant to release my code in case it chews 
someone's filesystem.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From jonathan.buzzard at strath.ac.uk  Mon Feb 26 23:15:34 2024
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Mon, 26 Feb 2024 23:15:34 +0000
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
In-Reply-To: <92d9b3ad-7b46-446a-a753-28b89832b81e@strath.ac.uk>
References: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
 <92d9b3ad-7b46-446a-a753-28b89832b81e@strath.ac.uk>
Message-ID: <5d5fe5b6-a195-44d3-bde3-17dba1ee5b78@strath.ac.uk>

On 26/02/2024 22:21, Jonathan Buzzard wrote:
> CAUTION: This email originated outside the University. Check before 
> clicking links or attachments.
> 
> On 26/02/2024 18:16, Jan-Frode Myklebust wrote:
> 
>> It?s not just the nfs4_setfacl tool. Also cp and rsync will fail to cooy
>> such ACLs.
>>
>> I have an RFE for this :
>>
>> https://ideas.ibm.com/ideas/GPFS-I-986
>> <https://ideas.ibm.com/ideas/GPFS-I-986>
>>
>>
> 
> At this point in time requests for being able to recursively set NFSv4
> ACL's on GPFS are over 12 years old. At least that would be about the
> time frame for requests for the feature made by myself.
> 
> Unfortunately I think the chances of IBM doing anything about it is
> somewhere around ?
> 
> Shame really as it would not be particularly hard to do. I have an old
> proof of concept that uses the FreeBSD tool (mainly of licensing
> reasons). Basically it transforms the GPFS ACL to the storage format
> used by the FreeBSD tool. Just needs refactoring to use the GPFS ACL
> storage format throughout. Probably about a weeks developer effort.
> 
> There are some poorly publicly documented features of the GPFS ACL
> format that make me reluctant to release my code in case it chews
> someone's filesystem.
> 

Looks like IBM might have actually delivered it as a feature

   https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-695

In summary

   The version of chmod linked below fully supports the manipulation of
   NFSv4 ACLs. This is the version that Isilon OneFS uses and it's
   fantastic. I can add/remove Active Directory users and groups, modify
   permissions, add ACL control flags, etc., all without ever touching a
   Windows computer.

 
https://www.freebsd.org/cgi/man.cgi?query=chmod&apropos=0&sektion=0&manpath=Darwin+8.0.1%2Fppc&format=html

   Can this work on GPFS?

It is marked as delivered in Scale 5.1.7 in September of last year. 
Though I have not the foggiest how you use it and can't find any mention 
in the command reference for 5.1.9


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From christof.schmitt at us.ibm.com  Mon Feb 26 23:27:06 2024
From: christof.schmitt at us.ibm.com (Christof Schmitt)
Date: Mon, 26 Feb 2024 23:27:06 +0000
Subject: [gpfsug-discuss] DACLs and nfs4-acl-tools
In-Reply-To: <5d5fe5b6-a195-44d3-bde3-17dba1ee5b78@strath.ac.uk>
References: <LO4P265MB4050222D71DF25945B511C60AC5A2@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <CAHwPatiBoTJv3BV1+wAhGBggPnFFXC2oJzyy2Ypc+6GKw1Qn8A@mail.gmail.com>
 <92d9b3ad-7b46-446a-a753-28b89832b81e@strath.ac.uk>
 <5d5fe5b6-a195-44d3-bde3-17dba1ee5b78@strath.ac.uk>
Message-ID: <9df83579da653eba04a52721840ae17c69fdf7dd.camel@us.ibm.com>

On Mon, 2024-02-26 at 23:15 +0000, Jonathan Buzzard wrote:
> 
> Looks like IBM might have actually delivered it as a feature
> 
> ?? https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-695

Scale 5.1.7 delivered the ability to use the Linux nfs4-acl-tools for
managing ACLs:

https://www.ibm.com/docs/en/storage-scale/5.1.7?topic=summary-changes
        "Support for setting the extended system.nfs4_acl attribute"

I assume that since that servces the same goal of modifying NFSv4 ACLs,
the RFE has been marked as delivered with that release.

Regards,

Christof

From martin at uni-mainz.de  Wed Feb 28 15:50:20 2024
From: martin at uni-mainz.de (Christoph Martin)
Date: Wed, 28 Feb 2024 16:50:20 +0100
Subject: [gpfsug-discuss] storage-scale-object
Message-ID: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>

Hi folks,

https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=upgrading-object-packages 
says:

> CES Swift Object protocol feature is not supported from IBM Storage Scale 5.1.9 onwards.
> IBM Storage Scale 5.1.8 is the last release that has CES Swift Object protocol.

> Please contact IBM for further details and migration planning.

Has anybody some insights what the future of object is and how we can 
migrate our installation to a supported form for our object customers?

Regards
Christoph

-- 
Christoph Martin
Zentrum f?r Datenverarbeitung (ZDV)
Leiter Unix & Cloud

Johannes Gutenberg-Universit?t Mainz
Anselm Franz von Bentzel-Weg 12, 55128 Mainz
Tel: +49 6131 39 26337
martin at uni-mainz.de
www.zdv.uni-mainz.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240228/49803cef/attachment.sig>

From daniel.kidger at hpe.com  Wed Feb 28 16:06:05 2024
From: daniel.kidger at hpe.com (Kidger, Daniel)
Date: Wed, 28 Feb 2024 16:06:05 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
Message-ID: <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>

Cristoph,

I would assume it is because most customers want to use the S3 Interface rather than the Swift one?

Daniel

Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kidger at hpe.com

+44 (0)7818 522266??

hpe.com




-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Christoph Martin
Sent: Wednesday, February 28, 2024 3:50 PM
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] storage-scale-object

Hi folks,

https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=upgrading-object-packages
says:

> CES Swift Object protocol feature is not supported from IBM Storage Scale 5.1.9 onwards.
> IBM Storage Scale 5.1.8 is the last release that has CES Swift Object protocol.

> Please contact IBM for further details and migration planning.

Has anybody some insights what the future of object is and how we can migrate our installation to a supported form for our object customers?

Regards
Christoph

--
Christoph Martin
Zentrum f?r Datenverarbeitung (ZDV)
Leiter Unix & Cloud

Johannes Gutenberg-Universit?t Mainz
Anselm Franz von Bentzel-Weg 12, 55128 Mainz
Tel: +49 6131 39 26337
martin at uni-mainz.de
www.zdv.uni-mainz.de

From martin at uni-mainz.de  Wed Feb 28 16:23:36 2024
From: martin at uni-mainz.de (Christoph Martin)
Date: Wed, 28 Feb 2024 17:23:36 +0100
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
Message-ID: <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>

Hi Daniel,

Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
> I would assume it is because most customers want to use the S3 Interface rather than the Swift one?

And Openstack/Swift is very complex. So I can understand why IBM gets 
rid of it.
But 5.1.9.1 has no object support at all. There is no S3 interface.

Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240228/3852e216/attachment.sig>

From Renar.Grunenberg at huk-coburg.de  Wed Feb 28 16:37:16 2024
From: Renar.Grunenberg at huk-coburg.de (Grunenberg, Renar)
Date: Wed, 28 Feb 2024 16:37:16 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
Message-ID: <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>

Hallo Daniel, Christoph,
currently IBM make a not good job on that, but it's light on that topic.
Currently you can use Swift/s3 on 5.1.9.1 with a backlevel scale (swift) version. But i don't recommend this.
Since 5.1.3.1 you can use IBM Spectrum Scale DAS S3 -Service (containerized s3 endpoints on noobaa-based operator called Openshift Data Foundation)
But this will be after 5.1.9.1 DAS also deprecated. (We use this Version today).
In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version of these s3-Endpoints. And this is promissing already for us.



Renar Grunenberg
Abteilung Informatik - Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:  09561 96-44110
Telefax:  09561 96-44104
E-Mail:   Renar.Grunenberg at huk-coburg.de
Internet: www.huk.de
=======================================================================
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel Thomas.
=======================================================================
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
=======================================================================

-----Urspr?ngliche Nachricht-----
Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Christoph Martin
Gesendet: Mittwoch, 28. Februar 2024 17:24
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Betreff: Re: [gpfsug-discuss] storage-scale-object

Hi Daniel,

Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
> I would assume it is because most customers want to use the S3 Interface rather than the Swift one?

And Openstack/Swift is very complex. So I can understand why IBM gets
rid of it.
But 5.1.9.1 has no object support at all. There is no S3 interface.

Christoph

From st.graf at fz-juelich.de  Wed Feb 28 17:02:14 2024
From: st.graf at fz-juelich.de (Stephan Graf)
Date: Wed, 28 Feb 2024 18:02:14 +0100
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
Message-ID: <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>

Hi Christoph,

we are running CES-Object in production and are facing the same issue 
that this solution is deprecated. Our security division sending us scan 
reports that the server are running an python version which is not in 
support any more (and we should upgrade).
So we are looking for another solution which looks to be minIO (on top 
of a scale file system).

Stephan

On 2/28/24 17:37, Grunenberg, Renar wrote:
> Hallo Daniel, Christoph,
> currently IBM make a not good job on that, but it's light on that topic.
> Currently you can use Swift/s3 on 5.1.9.1 with a backlevel scale (swift) version. But i don't recommend this.
> Since 5.1.3.1 you can use IBM Spectrum Scale DAS S3 -Service (containerized s3 endpoints on noobaa-based operator called Openshift Data Foundation)
> But this will be after 5.1.9.1 DAS also deprecated. (We use this Version today).
> In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version of these s3-Endpoints. And this is promissing already for us.
> 
> 
> 
> Renar Grunenberg
> Abteilung Informatik - Betrieb
> 
> HUK-COBURG
> Bahnhofsplatz
> 96444 Coburg
> Telefon:  09561 96-44110
> Telefax:  09561 96-44104
> E-Mail:   Renar.Grunenberg at huk-coburg.de
> Internet: www.huk.de
> =======================================================================
> HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
> Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel Thomas.
> =======================================================================
> Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
> informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.
> 
> This information may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this information in error) please notify the
> sender immediately and destroy this information.
> Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
> =======================================================================
> 
> -----Urspr?ngliche Nachricht-----
> Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Christoph Martin
> Gesendet: Mittwoch, 28. Februar 2024 17:24
> An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Betreff: Re: [gpfsug-discuss] storage-scale-object
> 
> Hi Daniel,
> 
> Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
>> I would assume it is because most customers want to use the S3 Interface rather than the Swift one?
> 
> And Openstack/Swift is very complex. So I can understand why IBM gets
> rid of it.
> But 5.1.9.1 has no object support at all. There is no S3 interface.
> 
> Christoph
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-- 
Stephan Graf
HPC, Cloud and Data Systems and Services
Juelich Supercomputing Centre

Phone:  +49-2461-61-6578
Fax:    +49-2461-61-6656
E-mail: st.graf at fz-juelich.de
WWW:    http://www.fz-juelich.de/jsc/
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Stefan M?ller
Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende),
Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Peter Jansens
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5938 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240228/ffe9ca75/attachment.bin>

From douglasof at us.ibm.com  Wed Feb 28 17:20:09 2024
From: douglasof at us.ibm.com (DOUGLAS O'FLAHERTY)
Date: Wed, 28 Feb 2024 17:20:09 +0000
Subject: [gpfsug-discuss] storage-scale-object
Message-ID: <F02EDAE3-AB6E-4591-AA27-FB686655CFE3@contoso.com>

IBM has highighting object evolution to Nooba and containerized ?high performance object? in various user group presentations.

This is the link to Chris Maestas? presentation from SC23.
https://www.spectrumscaleug.org/wp-content/uploads/2023/11/SSUG23SC-Whats-new-in-Storage-Scale.pdf

Slide is about half-way through the deck and includes links to a post about it.
https://community.ibm.com/community/user/storage/blogs/pratibha-joshi/2023/09/29/using-nsfs-standalone-noobaa-with-ibm-storage-scal?CommunityKey=1142f81e-95e4-4381-95d0-7977f20d53fa

Though posting from my IBM account, this isn?t an IBM official statement.

Douglas O'Flaherty

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240228/f8e65f88/attachment.htm>

From brnelson at us.ibm.com  Wed Feb 28 17:59:16 2024
From: brnelson at us.ibm.com (Brian Nelson)
Date: Wed, 28 Feb 2024 17:59:16 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <mailman.8.1709140999.3529453.gpfsug-discuss_gpfsug.org@gpfsug.org>
References: <mailman.8.1709140999.3529453.gpfsug-discuss_gpfsug.org@gpfsug.org>
Message-ID: <DM6PR15MB24114B4607147518B9B014B5FE582@DM6PR15MB2411.namprd15.prod.outlook.com>

The migration of objects stored in the existing Swift-based Object protocol will depend on whether they're stored as traditional Swift objects or as regular files (Unified File and Object mode). As you may have seen, the objects uploaded to Object protocol will end up on disk in one of two naming formats:

Traditional Swift (hashed path): 8406/605/835811b46b065bf86d4b58601e0f5605/1709142045.35693.data
Unified File and Object (regular path): AUTH_12345/buildlogs/2024/jan/20240123.log

Objects which are stored in a storage policy defined as Unfiied File and Object mode are stored as regular files and directories.  Since they are regular files, the directory which represents the container can be configured as a bucket in the DAS/CES S3 environment. Objects which are stored in the traditional Swift format will need to be downloaded out of Swift and saved as regular files using a cloud migration tool  (such as rclone). Once the objects have been downloaded into files, the resulting directory can be set as a bucket in DAS/CES S3.

Keep in mind that the DAS/CES S3 interface is S3 only. Applications which use the Swift protocol will need to be converted to use the S3 protocol.

-Brian

===================================
Brian Nelson
IBM Spectrum Scale
brnelson at us.ibm.com




From arc at b4restore.com  Thu Feb 29 06:46:48 2024
From: arc at b4restore.com (=?utf-8?B?QW5kaSBOw7hyIENocmlzdGlhbnNlbg==?=)
Date: Thu, 29 Feb 2024 06:46:48 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
 <9f784775-ccc9-448e-ac30-ed2049e3448e.06856278-b9b1-4fa9-8c7b-33136e95fc8a.6c39db14-906e-4479-8ef1-de215fd2a019@emailsignatures365.codetwo.com>
Message-ID: <DB9P194MB14022C0F1A9D79997E41EC84FB5F2@DB9P194MB1402.EURP194.PROD.OUTLOOK.COM>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b4r-logo200px_462d7941-ffe2-4f69-8c53-aade89528399.jpg
Type: image/jpeg
Size: 4781 bytes
Desc: b4r-logo200px_462d7941-ffe2-4f69-8c53-aade89528399.jpg
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linkedin_a3dd4789-eec6-4d7c-8d20-670718e518a6.png
Type: image/png
Size: 1386 bytes
Desc: linkedin_a3dd4789-eec6-4d7c-8d20-670718e518a6.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: twitter_14d6b75a-f4b7-45ec-ba88-75a90e22a7be.png
Type: image/png
Size: 1724 bytes
Desc: twitter_14d6b75a-f4b7-45ec-ba88-75a90e22a7be.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: facebook_915c5beb-e98f-46ec-bd13-f86d9235df9b.png
Type: image/png
Size: 1193 bytes
Desc: facebook_915c5beb-e98f-46ec-bd13-f86d9235df9b.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: email_d9cf36a4-1bfe-4032-9d73-17a7eb3d40d3.png
Type: image/png
Size: 1970 bytes
Desc: email_d9cf36a4-1bfe-4032-9d73-17a7eb3d40d3.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/bb1b29bd/attachment-0003.png>

From Alexander.Saupp at de.ibm.com  Thu Feb 29 07:27:46 2024
From: Alexander.Saupp at de.ibm.com (Alexander Saupp)
Date: Thu, 29 Feb 2024 07:27:46 +0000
Subject: [gpfsug-discuss] storage-scale-object - summary
Message-ID: <DM6PR15MB39424C85E8CDB8C2DB6937A8D95F2@DM6PR15MB3942.namprd15.prod.outlook.com>

Hi all,

there was a couple of very good concerns, questions and statements already. I?d like to summarize my very personal understanding ? not necessarily officially speaking for my employer IBM.
Be invited to reach out to IBM Client Engineering (a presales invest by IBM) if you have a need to discuss, demo or evaluate in connection to an active Opportunity ( i know, but that?s the rules of engagement )


  *   Swift S3 is complex to maintain, as said by Christoph Martin. IBM supports multiple stacks for multiple products. That along with currency the main reason to move away



  *   For alternatives a focus on ?unified access via file and s3? was set, so the feature set required is something (for you as a customer) to evaluate.


  *   You can find references on our future architecture publicly available, its based on the same noobaa stack that is used in RedHat ODF MCG.
I would like to recommend the following blog of my IBM CE peer Nils Haustein:

https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2024/02/21/s3-tiering-to-tape-with-noobaa-part-1-introduction

https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2024/02/21/s3-tiering-to-tape-with-noobaa-part-2-deployment

https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2024/02/26/s3-tiering-to-tape-with-noobaa-part-3-tiering



  *   Some might consider release timing, .. But here is the plan, thanks for already outlining, Renar!
https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=summary-changes

     *   IBM Storage Scale 5.1.8 is the last release that has CES Swift Object protocol.
     *   IBM Storage Scale 5.1.9 [EUS release] will tolerate the update of a CES node from IBM Storage Scale 5.1.8 - so if you have it, you can keep it
     *   I?m expecting TechPreview and GA within the next two releases ? technical details as per above blog.

I hope this helps to clarify IBM?s plan of record. I?d like to reinvite you to reach out to IBM (via IBM Sales / directly) if you?d like to follow-up.


Mit freundlichen Gr??en / Kind regards

Alexander Saupp
Senior Technology Engineer | IBM Client Engineering EMEA | Software Defined Storage
+49 172 725 1072 Mobile
alexander.saupp at de.ibm.com<mailto:alexander.saupp at de.ibm.com>

IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>
IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Gesch?ftsf?hrung: Wolfgang Wendt (Vorsitzender), Dr. Andreas Buchelt, Dr. Frank Kohls, Christine Rupp
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/f5dcc0be/attachment.htm>

From patryk.belzak at pwr.edu.pl  Thu Feb 29 08:11:11 2024
From: patryk.belzak at pwr.edu.pl (Patryk =?utf-8?Q?Be=C5=82zak?=)
Date: Thu, 29 Feb 2024 09:11:11 +0100
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <DB9P194MB14022C0F1A9D79997E41EC84FB5F2@DB9P194MB1402.EURP194.PROD.OUTLOOK.COM>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
 <9f784775-ccc9-448e-ac30-ed2049e3448e.06856278-b9b1-4fa9-8c7b-33136e95fc8a.6c39db14-906e-4479-8ef1-de215fd2a019@emailsignatures365.codetwo.com>
 <DB9P194MB14022C0F1A9D79997E41EC84FB5F2@DB9P194MB1402.EURP194.PROD.OUTLOOK.COM>
Message-ID: <a66pmbasrbfknj7th3ftuxijktt63ndsfbhri4xcwckcgsu7on@d77ljdzdkeg4>

Hello,
I find this information very interesting:
> In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version of these s3-Endpoints. And this is promissing already for us.

any ideas when that release will be available? I heard something about Q1/2 2024, and I am truly looking forward for some serious S3 on gpfs solution.

Best regards,
Patryk Belzak.

HPC sysadmin,
Wroclaw Centre for Networking and Supercomputing.

On 24/02/29 06:46AM, Andi N?r Christiansen wrote:
>    MinIO is not supported on GPFS, just an fyi. Had a long discussion with
>    them about that as we also had CES S3 that we wanted to move away from.
> 
>    We went in the same direction as Renar.
> 
>    Best Regards
> 
>    Andi N?r Christiansen
>    IT Solution Specialist
> 
>    Mobile +45 23 89 59 75   
>    E-mail arc at b4restore.com 
>    Web    www.b4restore.com 
> 
>    [IMG] [IMG] [IMG] [IMG]
> 
>    -----Original Message-----
>    From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of
>    Stephan Graf
>    Sent: 28. februar 2024 18:02
>    To: gpfsug-discuss at gpfsug.org
>    Subject: Re: [gpfsug-discuss] storage-scale-object
> 
>    Hi Christoph,
> 
>    we are running CES-Object in production and are facing the same issue that
>    this solution is deprecated. Our security division sending us scan reports
>    that the server are running an python version which is not in support any
>    more (and we should upgrade).
>    So we are looking for another solution which looks to be minIO (on top of
>    a scale file system).
> 
>    Stephan
> 
>    On 2/28/24 17:37, Grunenberg, Renar wrote:
>    > Hallo Daniel, Christoph,
>    > currently IBM make a not good job on that, but it's light on that topic.
>    > Currently you can use Swift/s3 on 5.1.9.1 with a backlevel scale (swift)
>    version. But i don't recommend this.
>    > Since 5.1.3.1 you can use IBM Spectrum Scale DAS S3 -Service
>    > (containerized s3 endpoints on noobaa-based operator called Openshift
>    Data Foundation) But this will be after 5.1.9.1 DAS also deprecated. (We
>    use this Version today).
>    > In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version
>    of these s3-Endpoints. And this is promissing already for us.
>    >
>    >
>    >
>    > Renar Grunenberg
>    > Abteilung Informatik - Betrieb
>    >
>    > HUK-COBURG
>    > Bahnhofsplatz
>    > 96444 Coburg
>    > Telefon:  09561 96-44110
>    > Telefax:  09561 96-44104
>    > E-Mail:   Renar.Grunenberg at huk-coburg.de
>    > Internet: www.huk.de
>    > ======================================================================
>    > = HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter
>    > Deutschlands a. G. in Coburg Reg.-Gericht Coburg HRB 100; St.-Nr.
>    > 9212/101/00021 Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
>    > Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
>    > Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans
>    Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel
>    Thomas.
>    > ======================================================================
>    > = Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte
>    > Informationen.
>    > Wenn Sie nicht der richtige Adressat sind oder diese Nachricht
>    > irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und
>    vernichten Sie diese Nachricht.
>    > Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
>    ist nicht gestattet.
>    >
>    > This information may contain confidential and/or privileged information.
>    > If you are not the intended recipient (or have received this
>    > information in error) please notify the sender immediately and destroy
>    this information.
>    > Any unauthorized copying, disclosure or distribution of the material in
>    this information is strictly forbidden.
>    > ======================================================================
>    > =
>    >
>    > -----Urspr?ngliche Nachricht-----
>    > Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von
>    > Christoph Martin
>    > Gesendet: Mittwoch, 28. Februar 2024 17:24
>    > An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
>    > Betreff: Re: [gpfsug-discuss] storage-scale-object
>    >
>    > Hi Daniel,
>    >
>    > Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
>    >> I would assume it is because most customers want to use the S3
>    Interface rather than the Swift one?
>    >
>    > And Openstack/Swift is very complex. So I can understand why IBM gets
>    > rid of it.
>    > But 5.1.9.1 has no object support at all. There is no S3 interface.
>    >
>    > Christoph
>    > _______________________________________________
>    > gpfsug-discuss mailing list
>    > gpfsug-discuss at gpfsug.org
>    > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
> 
>    --
>    Stephan Graf
>    HPC, Cloud and Data Systems and Services Juelich Supercomputing Centre
> 
>    Phone:  +49-2461-61-6578
>    Fax:    +49-2461-61-6656
>    E-mail: st.graf at fz-juelich.de
>    WWW:    http://www.fz-juelich.de/jsc/
>    ---------------------------------------------------------------------------------------------
>    ---------------------------------------------------------------------------------------------
>    Forschungszentrum Juelich GmbH
>    52425 Juelich
>    Sitz der Gesellschaft: Juelich
>    Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>    Vorsitzender des Aufsichtsrats: MinDir Stefan M?ller
>    Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten
>    Beneke (stellv. Vorsitzender), Dr. Ir. Peter Jansens
>    ---------------------------------------------------------------------------------------------
>    ---------------------------------------------------------------------------------------------







> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5887 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/582c730a/attachment.bin>

From Renar.Grunenberg at huk-coburg.de  Thu Feb 29 08:17:16 2024
From: Renar.Grunenberg at huk-coburg.de (Grunenberg, Renar)
Date: Thu, 29 Feb 2024 08:17:16 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <a66pmbasrbfknj7th3ftuxijktt63ndsfbhri4xcwckcgsu7on@d77ljdzdkeg4>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
 <9f784775-ccc9-448e-ac30-ed2049e3448e.06856278-b9b1-4fa9-8c7b-33136e95fc8a.6c39db14-906e-4479-8ef1-de215fd2a019@emailsignatures365.codetwo.com>
 <DB9P194MB14022C0F1A9D79997E41EC84FB5F2@DB9P194MB1402.EURP194.PROD.OUTLOOK.COM>
 <a66pmbasrbfknj7th3ftuxijktt63ndsfbhri4xcwckcgsu7on@d77ljdzdkeg4>
Message-ID: <5d4e250fbc784d7a83b1b508b0609558@huk-coburg.de>

Hallo Patryk,
what i know with GA 5.2.0.0 we will see a TechPreview in April and GA 5.2.1 will be in July.




Renar Grunenberg
Abteilung Informatik - Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:  09561 96-44110
Telefax:  09561 96-44104
E-Mail:   Renar.Grunenberg at huk-coburg.de
Internet: www.huk.de
=======================================================================
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel Thomas.
=======================================================================
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
=======================================================================

-----Urspr?ngliche Nachricht-----
Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Patryk Belzak
Gesendet: Donnerstag, 29. Februar 2024 09:11
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Betreff: Re: [gpfsug-discuss] storage-scale-object

Hello,
I find this information very interesting:
> In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version of these s3-Endpoints. And this is promissing already for us.

any ideas when that release will be available? I heard something about Q1/2 2024, and I am truly looking forward for some serious S3 on gpfs solution.

Best regards,
Patryk Belzak.

HPC sysadmin,
Wroclaw Centre for Networking and Supercomputing.

On 24/02/29 06:46AM, Andi N?r Christiansen wrote:
>    MinIO is not supported on GPFS, just an fyi. Had a long discussion with
>    them about that as we also had CES S3 that we wanted to move away from.
>
>    We went in the same direction as Renar.
>
>    Best Regards
>
>    Andi N?r Christiansen
>    IT Solution Specialist
>
>    Mobile +45 23 89 59 75
>    E-mail arc at b4restore.com
>    Web    www.b4restore.com
>
>    [IMG] [IMG] [IMG] [IMG]
>
>    -----Original Message-----
>    From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of
>    Stephan Graf
>    Sent: 28. februar 2024 18:02
>    To: gpfsug-discuss at gpfsug.org
>    Subject: Re: [gpfsug-discuss] storage-scale-object
>
>    Hi Christoph,
>
>    we are running CES-Object in production and are facing the same issue that
>    this solution is deprecated. Our security division sending us scan reports
>    that the server are running an python version which is not in support any
>    more (and we should upgrade).
>    So we are looking for another solution which looks to be minIO (on top of
>    a scale file system).
>
>    Stephan
>
>    On 2/28/24 17:37, Grunenberg, Renar wrote:
>    > Hallo Daniel, Christoph,
>    > currently IBM make a not good job on that, but it's light on that topic.
>    > Currently you can use Swift/s3 on 5.1.9.1 with a backlevel scale (swift)
>    version. But i don't recommend this.
>    > Since 5.1.3.1 you can use IBM Spectrum Scale DAS S3 -Service
>    > (containerized s3 endpoints on noobaa-based operator called Openshift
>    Data Foundation) But this will be after 5.1.9.1 DAS also deprecated. (We
>    use this Version today).
>    > In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version
>    of these s3-Endpoints. And this is promissing already for us.
>    >
>    >
>    >
>    > Renar Grunenberg
>    > Abteilung Informatik - Betrieb
>    >
>    > HUK-COBURG
>    > Bahnhofsplatz
>    > 96444 Coburg
>    > Telefon:  09561 96-44110
>    > Telefax:  09561 96-44104
>    > E-Mail:   Renar.Grunenberg at huk-coburg.de
>    > Internet: www.huk.de
>    > ======================================================================
>    > = HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter
>    > Deutschlands a. G. in Coburg Reg.-Gericht Coburg HRB 100; St.-Nr.
>    > 9212/101/00021 Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
>    > Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
>    > Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans
>    Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel
>    Thomas.
>    > ======================================================================
>    > = Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte
>    > Informationen.
>    > Wenn Sie nicht der richtige Adressat sind oder diese Nachricht
>    > irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und
>    vernichten Sie diese Nachricht.
>    > Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
>    ist nicht gestattet.
>    >
>    > This information may contain confidential and/or privileged information.
>    > If you are not the intended recipient (or have received this
>    > information in error) please notify the sender immediately and destroy
>    this information.
>    > Any unauthorized copying, disclosure or distribution of the material in
>    this information is strictly forbidden.
>    > ======================================================================
>    > =
>    >
>    > -----Urspr?ngliche Nachricht-----
>    > Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von
>    > Christoph Martin
>    > Gesendet: Mittwoch, 28. Februar 2024 17:24
>    > An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
>    > Betreff: Re: [gpfsug-discuss] storage-scale-object
>    >
>    > Hi Daniel,
>    >
>    > Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
>    >> I would assume it is because most customers want to use the S3
>    Interface rather than the Swift one?
>    >
>    > And Openstack/Swift is very complex. So I can understand why IBM gets
>    > rid of it.
>    > But 5.1.9.1 has no object support at all. There is no S3 interface.
>    >
>    > Christoph
>    > _______________________________________________
>    > gpfsug-discuss mailing list
>    > gpfsug-discuss at gpfsug.org
>    > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>    --
>    Stephan Graf
>    HPC, Cloud and Data Systems and Services Juelich Supercomputing Centre
>
>    Phone:  +49-2461-61-6578
>    Fax:    +49-2461-61-6656
>    E-mail: st.graf at fz-juelich.de
>    WWW:    http://www.fz-juelich.de/jsc/
>    ---------------------------------------------------------------------------------------------
>    ---------------------------------------------------------------------------------------------
>    Forschungszentrum Juelich GmbH
>    52425 Juelich
>    Sitz der Gesellschaft: Juelich
>    Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>    Vorsitzender des Aufsichtsrats: MinDir Stefan M?ller
>    Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten
>    Beneke (stellv. Vorsitzender), Dr. Ir. Peter Jansens
>    ---------------------------------------------------------------------------------------------
>    ---------------------------------------------------------------------------------------------







> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org


From abeattie at au1.ibm.com  Thu Feb 29 08:57:31 2024
From: abeattie at au1.ibm.com (ANDREW BEATTIE)
Date: Thu, 29 Feb 2024 08:57:31 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <5d4e250fbc784d7a83b1b508b0609558@huk-coburg.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
 <9f784775-ccc9-448e-ac30-ed2049e3448e.06856278-b9b1-4fa9-8c7b-33136e95fc8a.6c39db14-906e-4479-8ef1-de215fd2a019@emailsignatures365.codetwo.com>
 <DB9P194MB14022C0F1A9D79997E41EC84FB5F2@DB9P194MB1402.EURP194.PROD.OUTLOOK.COM>
 <a66pmbasrbfknj7th3ftuxijktt63ndsfbhri4xcwckcgsu7on@d77ljdzdkeg4>
 <5d4e250fbc784d7a83b1b508b0609558@huk-coburg.de>
Message-ID: <00E10AE8-D3CA-4B98-962C-55FB48CB3B31@au1.ibm.com>

IBM Storage Scale 5.2.0 will have a tech preview for CES-S3
IBM Storage Scale 5.2.1 will include GA for CES-S3 


Sent from my iPhone

> On 29 Feb 2024, at 18:19, Grunenberg, Renar <Renar.Grunenberg at huk-coburg.de> wrote:
> 
> ?Hallo Patryk,
> what i know with GA 5.2.0.0 we will see a TechPreview in April and GA 5.2.1 will be in July.
> 
> 
> 
> 
> Renar Grunenberg
> Abteilung Informatik - Betrieb
> 
> HUK-COBURG
> Bahnhofsplatz
> 96444 Coburg
> Telefon:  09561 96-44110
> Telefax:  09561 96-44104
> E-Mail:   Renar.Grunenberg at huk-coburg.de
> Internet: www.huk.de
> =======================================================================
> HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
> Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel Thomas.
> =======================================================================
> Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
> informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.
> 
> This information may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this information in error) please notify the
> sender immediately and destroy this information.
> Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
> =======================================================================
> 
> -----Urspr?ngliche Nachricht-----
> Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Patryk Belzak
> Gesendet: Donnerstag, 29. Februar 2024 09:11
> An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Betreff: Re: [gpfsug-discuss] storage-scale-object
> 
> Hello,
> I find this information very interesting:
>> In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version of these s3-Endpoints. And this is promissing already for us.
> 
> any ideas when that release will be available? I heard something about Q1/2 2024, and I am truly looking forward for some serious S3 on gpfs solution.
> 
> Best regards,
> Patryk Belzak.
> 
> HPC sysadmin,
> Wroclaw Centre for Networking and Supercomputing.
> 
>> On 24/02/29 06:46AM, Andi N?r Christiansen wrote:
>>   MinIO is not supported on GPFS, just an fyi. Had a long discussion with
>>   them about that as we also had CES S3 that we wanted to move away from.
>> 
>>   We went in the same direction as Renar.
>> 
>>   Best Regards
>> 
>>   Andi N?r Christiansen
>>   IT Solution Specialist
>> 
>>   Mobile +45 23 89 59 75
>>   E-mail arc at b4restore.com
>>   Web    www.b4restore.com
>> 
>>   [IMG] [IMG] [IMG] [IMG]
>> 
>>   -----Original Message-----
>>   From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of
>>   Stephan Graf
>>   Sent: 28. februar 2024 18:02
>>   To: gpfsug-discuss at gpfsug.org
>>   Subject: Re: [gpfsug-discuss] storage-scale-object
>> 
>>   Hi Christoph,
>> 
>>   we are running CES-Object in production and are facing the same issue that
>>   this solution is deprecated. Our security division sending us scan reports
>>   that the server are running an python version which is not in support any
>>   more (and we should upgrade).
>>   So we are looking for another solution which looks to be minIO (on top of
>>   a scale file system).
>> 
>>   Stephan
>> 
>>>   On 2/28/24 17:37, Grunenberg, Renar wrote:
>>> Hallo Daniel, Christoph,
>>> currently IBM make a not good job on that, but it's light on that topic.
>>> Currently you can use Swift/s3 on 5.1.9.1 with a backlevel scale (swift)
>>   version. But i don't recommend this.
>>> Since 5.1.3.1 you can use IBM Spectrum Scale DAS S3 -Service
>>> (containerized s3 endpoints on noobaa-based operator called Openshift
>>   Data Foundation) But this will be after 5.1.9.1 DAS also deprecated. (We
>>   use this Version today).
>>> In The Upcoming Scale Release there will be a CES-S3 (Baremetal) Version
>>   of these s3-Endpoints. And this is promissing already for us.
>>> 
>>> 
>>> 
>>> Renar Grunenberg
>>> Abteilung Informatik - Betrieb
>>> 
>>> HUK-COBURG
>>> Bahnhofsplatz
>>> 96444 Coburg
>>> Telefon:  09561 96-44110
>>> Telefax:  09561 96-44104
>>> E-Mail:   Renar.Grunenberg at huk-coburg.de
>>> Internet: www.huk.de
>>> ======================================================================
>>> = HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter
>>> Deutschlands a. G. in Coburg Reg.-Gericht Coburg HRB 100; St.-Nr.
>>> 9212/101/00021 Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
>>> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
>>> Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans
>>   Olav Her?y, Dr. Helen Reck, Dr. J?rg Rheinl?nder, Thomas Sehn, Daniel
>>   Thomas.
>>> ======================================================================
>>> = Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte
>>> Informationen.
>>> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht
>>> irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und
>>   vernichten Sie diese Nachricht.
>>> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
>>   ist nicht gestattet.
>>> 
>>> This information may contain confidential and/or privileged information.
>>> If you are not the intended recipient (or have received this
>>> information in error) please notify the sender immediately and destroy
>>   this information.
>>> Any unauthorized copying, disclosure or distribution of the material in
>>   this information is strictly forbidden.
>>> ======================================================================
>>> =
>>> 
>>> -----Urspr?ngliche Nachricht-----
>>> Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von
>>> Christoph Martin
>>> Gesendet: Mittwoch, 28. Februar 2024 17:24
>>> An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
>>> Betreff: Re: [gpfsug-discuss] storage-scale-object
>>> 
>>> Hi Daniel,
>>> 
>>> Am 28.02.24 um 17:06 schrieb Kidger, Daniel:
>>>> I would assume it is because most customers want to use the S3
>>   Interface rather than the Swift one?
>>> 
>>> And Openstack/Swift is very complex. So I can understand why IBM gets
>>> rid of it.
>>> But 5.1.9.1 has no object support at all. There is no S3 interface.
>>> 
>>> Christoph
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at gpfsug.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>> 
>>   --
>>   Stephan Graf
>>   HPC, Cloud and Data Systems and Services Juelich Supercomputing Centre
>> 
>>   Phone:  +49-2461-61-6578
>>   Fax:    +49-2461-61-6656
>>   E-mail: st.graf at fz-juelich.de
>>   WWW:    http://www.fz-juelich.de/jsc/
>>   ---------------------------------------------------------------------------------------------
>>   ---------------------------------------------------------------------------------------------
>>   Forschungszentrum Juelich GmbH
>>   52425 Juelich
>>   Sitz der Gesellschaft: Juelich
>>   Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>   Vorsitzender des Aufsichtsrats: MinDir Stefan M?ller
>>   Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten
>>   Beneke (stellv. Vorsitzender), Dr. Ir. Peter Jansens
>>   ---------------------------------------------------------------------------------------------
>>   ---------------------------------------------------------------------------------------------
> 
> 
> 
> 
> 
> 
> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

From janfrode at tanso.net  Thu Feb 29 09:47:46 2024
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Thu, 29 Feb 2024 10:47:46 +0100
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
Message-ID: <CAHwPatgfoHjPE6_4XjJqE5gje9Ksr0YmGrtvi-6KTxmF21y3WQ@mail.gmail.com>

On Wed, Feb 28, 2024 at 6:03?PM Stephan Graf <st.graf at fz-juelich.de> wrote:

>
> So we are looking for another solution which looks to be minIO (on top
> of a scale file system).
>
>
You're going to like CES-S3.

It's basically the same as minio on top of Scale. A simple systemd-service
running noobaa-standalone. No databases. No cross-node synchronization.
Using cesSharedRoot to store a shared config between all protocol nodes.



  -jf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/a6d01d69/attachment.htm>

From luis.bolinches at fi.ibm.com  Thu Feb 29 10:22:23 2024
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Thu, 29 Feb 2024 10:22:23 +0000
Subject: [gpfsug-discuss] storage-scale-object
In-Reply-To: <CAHwPatgfoHjPE6_4XjJqE5gje9Ksr0YmGrtvi-6KTxmF21y3WQ@mail.gmail.com>
References: <761deae1-8a8e-4a6f-90dc-23e9b5159ebc@uni-mainz.de>
 <PH7PR84MB17013DBC8726ACCE14C066B9FB582@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <6211dd9b-268a-4b5f-918b-b1c383cefd8d@uni-mainz.de>
 <99d4afbf5603470494fdd5fa671133a6@huk-coburg.de>
 <14c58436-f625-4525-b330-02f3c542be6a@fz-juelich.de>
 <CAHwPatgfoHjPE6_4XjJqE5gje9Ksr0YmGrtvi-6KTxmF21y3WQ@mail.gmail.com>
Message-ID: <SA1PR15MB5061292354C700300E60B0E3A45F2@SA1PR15MB5061.namprd15.prod.outlook.com>

We did some presentations last week on SCA at Sydney were the architecture and migration were discussed and presented. I hope the slides get updated to the UG web ?soon?

--
Yst?v?llisin terveisin/Regards/Saludos/Salutations/Salutacions
Luis Bolinches
Executive IT Specialist
IBM Storage Scale Server (formerly ESS) developer
Phone: +358503112585

Ab IBM Finland Oy
Toinen linja 7
00530 Helsinki
Uusimaa - Finland

Visitors entrance: Siltasaarenkatu 22

"If you always give you will always have" --  Anonymous

https://www.credly.com/users/luis-bolinches/badges

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Jan-Frode Myklebust
Sent: Thursday, 29 February 2024 11.48
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] storage-scale-object

On Wed, Feb 28, 2024 at 6:?03 PM Stephan Graf <st.?graf@?fz-juelich.?de> wrote: So we are looking for another solution which looks to be minIO (on top of a scale file system). You're going to like CES-S3. It's basically the same as
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!12-vrJF3prOyMM9wN4oJgH7357z9u7z0suC_49Gc2mNMIaMHGjT0tMhh6hmLERinUGujjrHVWeibFxBLLrpB4A8aB8p5EIYy-mFoiOU8nqpoZKxMgkdgBN77nhGSfI5V$>   ?
ZjQcmQRYFpfptBannerEnd

On Wed, Feb 28, 2024 at 6:03?PM Stephan Graf <st.graf at fz-juelich.de<mailto:st.graf at fz-juelich.de>> wrote:

So we are looking for another solution which looks to be minIO (on top
of a scale file system).

You're going to like CES-S3.

It's basically the same as minio on top of Scale. A simple systemd-service running noobaa-standalone. No databases. No cross-node synchronization. Using cesSharedRoot to store a shared config between all protocol nodes.



  -jf


Unless otherwise stated above:

Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240229/c848c6ad/attachment.htm>