From chair at gpfsug.org  Wed Mar  1 19:08:19 2023
From: chair at gpfsug.org (chair at gpfsug.org)
Date: Wed, 01 Mar 2023 19:08:19 +0000
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday
 22nd June 2023
Message-ID: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>

Hi all,

Just a reminder that the next UK User Group meeting will be taking place 
in London (IBM York Road) on Wednesday 21st and Thursday 22nd June

Now is your opportunity to help shape the agenda, please feel free to 
send me ideas for talks we could ask IBM for (I don't promise we'll get 
them on the agenda mind!).

Also if you would like to do a user talk, please get in touch with me 
and let me know. It could be a large scale deployment or even just a 
couple of nodes, it's your opportunity to showcase how you use Spectrum 
Scale and what for. Every year people tell us how valuable they find the 
user talks, but this needs YOU, so please do think about if you are able 
to offer a talk!

As in the past we are looking for sponsorship to enable us to run an 
evening networking event. I've sent out details to those who have 
sponsored in the past and to those who have asked us directly about 
sponsorship opportunities. If you either haven't received this or are 
interested in becoming a sponsor, please email me directly.

Thanks

Paul



From knop at us.ibm.com  Thu Mar  2 03:33:19 2023
From: knop at us.ibm.com (Felipe Knop)
Date: Thu, 2 Mar 2023 03:33:19 +0000
Subject: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded
Message-ID: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>

Walter,

Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will be in the upcoming 5.1.7.0 release.

Thanks for opening the case and working with the support team on this one.

  Felipe

----
Felipe Knop        knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: Wednesday, February 22, 2023 at 5:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi ; sorry for the delay
Our case is TS012184140
They are still analizing
As soon as I get feedback I will update you




Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Ryan Novosielski
Gesendet: Friday, February 17, 2023 11:52 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

I talked about it a lot in TS011616986.

Part of the problem is we?re having a lot of strange problems at the same time, and so the different issues we?re having often come together (like one cause shows two symptoms).

I can?t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'



On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>> wrote:

Hi Ryan and Felipe!
Could you eventually tell me the case number if you remember it?
I opened the case and would reference to your case ID
Or shall I send you mine ?


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: Freitag, 17. Februar 2023 06:43
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Thanks for this, Felipe.

We?ve started seeing intermittent overdue leases in large numbers and don?t otherwise have an explanation for it, other than ?look at your network,? which actually does show occasional signs of strange behavior/higher-than-normal RTO values, but we?re not necessarily seeing those things happen at the same times as the lease issues. We?ve also seen ?GPFS Critical Thread Watchdog? recently.

We had a case open about it, but didn?t draw any real conclusions. If any of our data might be helpful/if there?s a case we could reference to see if we?re also running into that, we could provide a gpfs.snap.

FWIW, we are running 5.1.3-1 on the storage side (except one system that?s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///utgers/>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'




On Feb 16, 2023, at 12:02, Felipe Knop <knop at us.ibm.com<mailto:knop at us.ibm.com>> wrote:

Walter,

Thanks for the detailed description.

I don?t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.

I?d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to ?loop? for a while.  That might cause the critical thread watchdog to flag the lease thread as taking too long to ?check in?.

Capturing gpfs.snap is important, since we?d be looking into all the

[W] ------------------[GPFS Critical Thread Watchdog]------------------

instances.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Thursday, February 16, 2023 at 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy ? I did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Felipe!
Once again me. Thank you very much for the hint
I did not open a PMR yet because I fear they will ask me/us if we are cracy ?
I did not tell the full story yet
We have a 3 node cluster, 2 NSD servers  o1,o2 (same site ) and g1 (different site). (rhel 8.7)
All of them are Vmware VMs
O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD ,  for a filesystem fs4vm (m,r=2 )


[root at ogpfs1 ras]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         edvdesign-cluster.local
  GPFS cluster id:           12147978822727803186
  GPFS UID domain:           edvdesign-cluster.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name     IP address     Admin node name      Designation
----------------------------------------------------------------------------
   1   ogpfs1-hs.local      10.20.30.1     ogpfs1-hs.local      quorum-manager-perfmon
   2   ogpfs2-hs.local      10.20.30.2     ogpfs2-hs.local      quorum-manager-perfmon
   3   ggpfsq.mgmt.cloudia  xxxx.other.net<http://xxxx.other.net/> ggpfsq.mgmt. a  quorum-perfmon

[root at ogpfs1 ras]# mmlsconfig
Configuration data for cluster edvdesign-cluster.local:
-------------------------------------------------------
clusterName edvdesign-cluster.local
clusterId 12147978822727803186
autoload yes
profile gpfsProtocolRandomIO
dmapiFileHandleSize 32
minReleaseLevel 5.1.6.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
[srv]
verbsPorts mlx5_0/1 mlx5_1/1
[common]
cesSharedRoot /fs4vmware/cesSharedRoot
[srv]
maxFilesToCache 10000
maxStatCache 20000
[common]
verbsRdma enable
[ggpfsq]
verbsRdma disable
[common]
verbsRdmaSend yes
[ggpfsq]
verbsRdmaSend no
[common]
verbsRdmaCm enable
[ggpfsq]
verbsRdmaCm disable
[srv]
pagepool 32G
[common]
adminMode central

File systems in cluster edvdesign-cluster.local:
------------------------------------------------
/dev/fs4vm


[root at ogpfs1 ras]# mmlsdisk fs4vm -L
disk         driver   sector     failure holds    holds                                    storage
name         type       size       group metadata data  status        availability disk id pool         remarks
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------
ogpfs1_1     nsd         512           1 yes      yes   ready         up                 1 system        desc
ogpfs2_1     nsd         512           2 yes      yes   ready         up                 2 system        desc
ggpfsq_qdisk nsd         512          -1 no       no    ready         up                 3 system        desc
Number of quorum disks: 3
Read quorum value:      2
Write quorum value:     2

And the two nodes o1 and o2 export the filesystem via CES NFS  functions ( for VMware)
I think this isn?supported , that a NSD Server is also a CES Node?

And finally the RDMA Network :
The both NSD servers also have a Mellanox ConnectX-6 Lx  dual port 25Gb adapter also via passthrough
And these interfaces we configured for rdma (RoCE) ,
Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )

RDMA Connections between nodes:
  Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     0   Y  RTS   (Y)256  478202 (0  ) 12728       67024       8864789(0  ) 22776     4643      0             0
  Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     1   Y  RTS   (Y)256  477659 (0  ) 12489       67034       8864773(0  ) 22794     4639      0             0
[root at ogpfs1 ras]#


You mentioned that it might be a cpu contention :  Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)
And also the direct attached RDMA ( +DAEMON) network is also not good?

Do you think IBM would say no to check such a configuration ?


Best regards
Walter




From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Thanks for the details.

The stack trace below captures the lease thread in the middle of sending the ?lease? RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace ?does not show? whether there was anything blocking the thread prior to the point where the RPCs are sent.

At a first glance:

2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)

I believe nivcsw: 10 means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there is some CPU contention going on.

Could you open a case to get debug data collected?  If the problem can be recreated, I think we?ll need a recreate of the problem with traces enabled.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 15, 2023 at 4:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! This is a ?full? sequence in mmfs.?log.?latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19:?43:?51.?474+0100: [N] Disk lease period expired 0.?030 seconds ago in cluster
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi!
This is a ?full? sequence in mmfs.log.latest
Fortunately this was also the last event until now (yesterday evening)

Maybe you can have a look?
2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)
2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):
2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi
gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0
2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u
nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0
2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0
2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig
uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0
2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,
TscReply*), void*, void**, int, ...) + 0x1DA at ??:0
2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0
2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0
2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0
2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0
2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0
2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0
2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0
2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.

Thank you very much!
Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 00:06
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

All,

These messages like

[W] ------------------[GPFS Critical Thread Watchdog]------------------

indicate that a ?critical thread?, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.

Do you have other samples of the message, with a more complete stack trace?   Or was the instance below the only one?

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Tuesday, February 14, 2023 at 10:49 AM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! I started with 5.?1.?6.?0 and now am at [root@?ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5.?1.?6.?1 ". the messages started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@?gpfsug.?org> On
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi!
I started with 5.1.6.0 and now am at [root at ogpfs1 ~]# mmfsadm dump version
Dump level: verbose
Build branch "5.1.6.1 ".

the messages started  from the beginning



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Christian Vieser
Sent: Dienstag, 14. Februar 2023 15:34
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded


What version of Spectrum Scale is running there? Do these errors appear since your last version update?
Am 14.02.23 um 14:09 schrieb Walter Sklenka:
Dear Collegues!
May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?
Is this a ?local node? Problem or a network problem ?
I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load



Following is an excerpt from mmfs.log.latest

2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_12:06:53.600+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)
2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):
2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0
2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0
2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive
2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0
2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/8a9cb4bc/attachment.htm>

From daniel.kidger at hpe.com  Thu Mar  2 08:31:33 2023
From: daniel.kidger at hpe.com (Kidger, Daniel)
Date: Thu, 2 Mar 2023 08:31:33 +0000
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday
 22nd June 2023
In-Reply-To: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
References: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
Message-ID: <PH7PR84MB170121AAB576A06020E72768FBB29@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>



" it's your opportunity to showcase how you use Spectrum Scale and what for."



Spectrum Scale?  I haven?t heard the software being called that in a long time?   ?



#StorageScale

Daniel



Daniel Kidger

HPC Storage Solutions Architect, EMEA

daniel.kidger at hpe.com



+44 (0)7818 522266



hpe.com









-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of chair at gpfsug.org
Sent: 01 March 2023 19:08
To: Gpfsug Discuss <gpfsug-discuss at spectrumscaleug.org>
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday 22nd June 2023



Hi all,



Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on Wednesday 21st and Thursday 22nd June



Now is your opportunity to help shape the agenda, please feel free to send me ideas for talks we could ask IBM for (I don't promise we'll get them on the agenda mind!).



Also if you would like to do a user talk, please get in touch with me and let me know. It could be a large scale deployment or even just a couple of nodes, it's your opportunity to showcase how you use Spectrum Scale and what for. Every year people tell us how valuable they find the user talks, but this needs YOU, so please do think about if you are able to offer a talk!



As in the past we are looking for sponsorship to enable us to run an evening networking event. I've sent out details to those who have sponsored in the past and to those who have asked us directly about sponsorship opportunities. If you either haven't received this or are interested in becoming a sponsor, please email me directly.



Thanks



Paul



_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at gpfsug.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/1f1df20b/attachment.htm>

From daniel.kidger at hpe.com  Thu Mar  2 08:31:33 2023
From: daniel.kidger at hpe.com (Kidger, Daniel)
Date: Thu, 2 Mar 2023 08:31:33 +0000
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday
 22nd June 2023
In-Reply-To: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
References: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
Message-ID: <PH7PR84MB170121AAB576A06020E72768FBB29@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>



" it's your opportunity to showcase how you use Spectrum Scale and what for."



Spectrum Scale?  I haven?t heard the software being called that in a long time?   ?



#StorageScale

Daniel



Daniel Kidger

HPC Storage Solutions Architect, EMEA

daniel.kidger at hpe.com



+44 (0)7818 522266



hpe.com









-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of chair at gpfsug.org
Sent: 01 March 2023 19:08
To: Gpfsug Discuss <gpfsug-discuss at spectrumscaleug.org>
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday 22nd June 2023



Hi all,



Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on Wednesday 21st and Thursday 22nd June



Now is your opportunity to help shape the agenda, please feel free to send me ideas for talks we could ask IBM for (I don't promise we'll get them on the agenda mind!).



Also if you would like to do a user talk, please get in touch with me and let me know. It could be a large scale deployment or even just a couple of nodes, it's your opportunity to showcase how you use Spectrum Scale and what for. Every year people tell us how valuable they find the user talks, but this needs YOU, so please do think about if you are able to offer a talk!



As in the past we are looking for sponsorship to enable us to run an evening networking event. I've sent out details to those who have sponsored in the past and to those who have asked us directly about sponsorship opportunities. If you either haven't received this or are interested in becoming a sponsor, please email me directly.



Thanks



Paul



_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at gpfsug.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/1f1df20b/attachment-0001.htm>

From WPeters at ATPCO.NET  Thu Mar  2 16:26:02 2023
From: WPeters at ATPCO.NET (Bill Peters)
Date: Thu, 2 Mar 2023 16:26:02 +0000
Subject: [gpfsug-discuss] spanning datacenter to AWS
In-Reply-To: <PH0PR15MB506974F922E20BC7057D1055FDAA9@PH0PR15MB5069.namprd15.prod.outlook.com>
References: <MN2PR20MB3055CC4ED0B73FFE36C5CC3DB0AA9@MN2PR20MB3055.namprd20.prod.outlook.com>
 <PH0PR15MB506974F922E20BC7057D1055FDAA9@PH0PR15MB5069.namprd15.prod.outlook.com>
Message-ID: <MN2PR20MB30552FDA9DB74E7438E6B4C3B0B29@MN2PR20MB3055.namprd20.prod.outlook.com>

Thanks Fred,
It took a while, but it looks like you are correct. I will need to upgrade to 5.1.2.7 for a fix.

Thanks,
-Bill

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Frederick Stock
Sent: Wednesday, February 22, 2023 2:01 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>; gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] spanning datacenter to AWS

Bill, if my memory serves me correctly, there was a fix done in later versions of Scale (there may be an efix available) for the situation you described.  Notably, Scale was not properly propagating information about files created through NFS.  I suggest you contact Scale support to see if they can provide more details, as well as options for obtaining the fix, assuming my mind has not completely failed me on this issue ?

Fred

Fred Stock, Spectrum Scale Development Advocacy
stockf at us.ibm.com<mailto:stockf at us.ibm.com> | 720-430-8821



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Bill Peters <WPeters at ATPCO.NET<mailto:WPeters at ATPCO.NET>>
Date: Wednesday, February 22, 2023 at 12:53 PM
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] [gpfsug-discuss] spanning datacenter to AWS
Hello all, I?ve been on the mailing list for a few years but have not been active except my introduction email. We are having an issue I?d like to run past everyone and see if anyone has experience that may help. Currently using
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hello all,
I?ve been on the mailing list for a few years but have not been active except my introduction email.

We are having an issue I?d like to run past everyone and see if anyone has experience that may help.

Currently using Spectrum Scale Data Management Edition 5.1.1.0

Our Spectrum Scale cluster is running on Linux VMs on IBM z/VM. We have one application that
cannot support the z/VM architecture so we used to have those servers running on VMware in our
datacenter and those servers were client nodes in the Spectrum Scale cluster. This configuration
worked great. We recently retired VMWare and moved all that workload to AWS. Because this
was no longer on our LAN we thought it would be a good idea (IBM support also recommended it) to
use CES NFS rather than adding the AWS instances to the cluster. Since doing this we have seen
problems under high IO. Some NFS clients will try to access files that don?t seem to be there
resulting in file not found errors. We know the files have been created but the NFS clients can?t
see them. The read process runs successfully shortly after. We are not saturating our AWS
connection.

I haven?t seen any NFS tuning that looks like it would help, but that is an option I would be willing to try.
The other option I?m thinking about is just adding the NFS clients to the cluster. Has anyone spanned
datacenters like this?

Thanks, any help is appreciated.

-Bill



Bill Peters
Senior Platform Engineer
703-475-3386
wpeters at atpco.net<mailto:wpeters at atpco.net>

atpco.net
45005 Aviation Drive
Dulles, VA 20166

[A close up of a signDescription automatically generated]<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbit.ly%2F35vlWu3&data=05%7C01%7Cwpeters%40atpco.net%7Cf65e182b0f6d4afe981b08db1507a5ca%7C25a0fe74b1cb46ee81f58cbb7b5ca144%7C0%7C1%7C638126894859278168%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tYlSjbuuC9m40rAABTSi7kA0HfLGmjI1os%2FgKo8lIGs%3D&reserved=0>
[Title: Facebook - Description: Facebook icon]<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fatpconews%2F&data=05%7C01%7Cwpeters%40atpco.net%7Cf65e182b0f6d4afe981b08db1507a5ca%7C25a0fe74b1cb46ee81f58cbb7b5ca144%7C0%7C1%7C638126894859278168%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8s60Nga9hlKC2AFoT1ufKIN0DtNUTJTrX29noBuNMCg%3D&reserved=0>  [Title: Twitter - Description: Twitter icon] <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FATPCOnews&data=05%7C01%7Cwpeters%40atpco.net%7Cf65e182b0f6d4afe981b08db1507a5ca%7C25a0fe74b1cb46ee81f58cbb7b5ca144%7C0%7C1%7C638126894859278168%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UIQ%2FsbYIt2P6ESIXanjZ55cm6bXnRxUQQg7p29FxI%2Bw%3D&reserved=0>   [Title: LinkedIn - Description: LinkedIn icon] <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fatpco%2F&data=05%7C01%7Cwpeters%40atpco.net%7Cf65e182b0f6d4afe981b08db1507a5ca%7C25a0fe74b1cb46ee81f58cbb7b5ca144%7C0%7C1%7C638126894859278168%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4izKdWZdYHdJp6oqR5gwYsfVCYCxiWfUHPxWoeffo60%3D&reserved=0>








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/c66056da/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17692 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/c66056da/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1266 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/c66056da/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1329 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/c66056da/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1378 bytes
Desc: image004.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230302/c66056da/attachment-0003.png>

From chair at gpfsug.org  Fri Mar  3 11:43:49 2023
From: chair at gpfsug.org (chair at gpfsug.org)
Date: Fri, 03 Mar 2023 11:43:49 +0000
Subject: [gpfsug-discuss] GPFS UK Meeting Wednesday 21st June - Thursday
 22nd June 2023 - Postponed
In-Reply-To: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
References: <b104eef220a0ae251af31c730fc416d0@gpfsug.org>
Message-ID: <852d6fde37de92b95917092171e1a008@gpfsug.org>

Dear All,

Due to an issue with the IBM Centre, we have had to postpone the planned 
meeting in June and will update you with a new date in the near future

Sorry for this

Regards

Paul



From dmagda+gpfs at ee.torontomu.ca  Fri Mar  3 19:29:11 2023
From: dmagda+gpfs at ee.torontomu.ca (David Magda)
Date: Fri, 3 Mar 2023 14:29:11 -0500
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS, cron,
 other?
Message-ID: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>

Hello,

I am a new user of GPFS (Spectrum Scale) and would like to know if there is a ?best practice? on handling kernel updates on HPC clients. We are running Ubuntu 18.04 and 20.04 clients with 5.1.x, talking to RHEL storage servers, and would like to know how to handle re-compiling the client-side kernel modules.

There is of course the ?mmbuildgpl? utility:

	https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-mmbuildgpl-command

but how do folks invoke it? Manually, via cron at night on or reboot, via some kind of apt (dpkg-trigger(1)) / RPM hook?

We have the ?unattended-upgrades? package enabled, which only installs security-tagged updates by default, but sometimes this does include kernel updates, which may become active on the next reboot:

	https://packages.ubuntu.com/search?keywords=unattended-upgrades

So is there a best practice? Has someone invented this wheel that I could leverage, or will I have to invent it myself?

Thanks for any info.

-- 
David Magda


From christof.schmitt at us.ibm.com  Fri Mar  3 19:59:40 2023
From: christof.schmitt at us.ibm.com (Christof Schmitt)
Date: Fri, 3 Mar 2023 19:59:40 +0000
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS,
 cron, other?
In-Reply-To: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
References: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
Message-ID: <1bff90bbe0bb5052192a5d8769fef2c31aec9c21.camel@us.ibm.com>

On Fri, 2023-03-03 at 14:29 -0500, David Magda wrote:
> Hello,
> 
> I am a new user of GPFS (Spectrum Scale) and would like to know if
> there is a ?best practice? on handling kernel updates on HPC clients.
> We are running Ubuntu 18.04 and 20.04 clients with 5.1.x, talking to
> RHEL storage servers, and would like to know how to handle re-
> compiling the client-side kernel modules.
> 
> There is of course the ?mmbuildgpl? utility:
> 
> 	
> https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-mmbuildgpl-command
> 
> but how do folks invoke it? Manually, via cron at night on or reboot,
> via some kind of apt (dpkg-trigger(1)) / RPM hook?

There is the Scale option:
mmchconfig autoBuildGPL=yes

When Scale starts, it checks whether the kernel modules are available
for the current kernel; and if not, mmbuildgpl is run before acctually
starting the daemon. That should be one way to solve this.

See 
https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=reference-mmchconfig-command

Regards,

Christof

From jonathan.buzzard at strath.ac.uk  Fri Mar  3 20:05:30 2023
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 3 Mar 2023 20:05:30 +0000
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS,
 cron, other?
In-Reply-To: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
References: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
Message-ID: <bd4d1665-7271-0819-0493-f55697e41d0a@strath.ac.uk>

On 03/03/2023 19:29, David Magda wrote:

> 
> Hello,
> 
> I am a new user of GPFS (Spectrum Scale) and would like to know if
> there is a ?best practice? on handling kernel updates on HPC clients.
> We are running Ubuntu 18.04 and 20.04 clients with 5.1.x, talking to
> RHEL storage servers, and would like to know how to handle
> re-compiling the client-side kernel modules.
> 
> There is of course the ?mmbuildgpl? utility:
> 
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dreference-mmbuildgpl-command&data=05%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C441bff633acd491f1d5d08db1c1e178d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C638134687790209306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZQS%2BCAgmIhByy6mrDx9UudY%2BjkRGsaudBE8LUI%2Ft9Mg%3D&reserved=0
>
>  but how do folks invoke it? Manually, via cron at night on or
> reboot, via some kind of apt (dpkg-trigger(1)) / RPM hook?
> 
> We have the ?unattended-upgrades? package enabled, which only
> installs security-tagged updates by default, but sometimes this does
> include kernel updates, which may become active on the next reboot:
> 
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpackages.ubuntu.com%2Fsearch%3Fkeywords%3Dunattended-upgrades&data=05%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C441bff633acd491f1d5d08db1c1e178d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C638134687790209306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tfzFmrS5zdNlFz75newnxLxDPj5AzUWnF%2BX1NK6GoHE%3D&reserved=0

I would suggest that you disable any automatic upgrading of the kernel. 
Kernel upgrades should *only* be done *after* you have verified that it 
will work.

If you don't it is only a matter of time before a security update breaks 
GPFS. There was at least one instance of that happing in the last five 
years.

>
>  So is there a best practice? Has someone invented this wheel that I
> could leverage, or will I have to invent it myself?
> 

I use a RPM package called gpfs-helper that I created. It installs a 
local "helper" to the systemd gpfs unit file

/etc/systemd/system/gpfs.service.d/install-module.conf

[Service]
ExecStartPre=-/usr/bin/dnf --assumeyes --enablerepo dssg install 
gpfs.gplbin-%v

This causes the correct gpfs.gplbin RPM to be installed when starting 
GPFS. If the correct one is already installed it does nothing, otherwise 
it downloads the correct RPM and installs it before attempting to start 
GPFS. I am pretty sure you could do the same with apt. The %v is the 
magic bit which basically matches the running kernel.

So after testing that GPFS works on the new kernel, I build the RPM, put 
it in the local repo and winner winner chicken dinner.


JAB.

--
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From Marcy.D.Cortes at wellsfargo.com  Fri Mar  3 21:10:02 2023
From: Marcy.D.Cortes at wellsfargo.com (Marcy.D.Cortes at wellsfargo.com)
Date: Fri, 3 Mar 2023 21:10:02 +0000
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS,
 cron, other?
In-Reply-To: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
References: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
Message-ID: <411ca83df2a74e53a8ceef846e3dd04d@wellsfargo.com>

We build the rpm and it gets installed if needed in a systemd script that puts it on and then restarts GPFS.  Apps are dependent on a health check startup script. To use that auto build you have to have a compiler installed and we avoid that on production servers


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of: David Magda <dmagda+gpfs at ee.torontomu.ca<mailto:dmagda+gpfs at ee.torontomu.ca>>
Date: Friday, Mar 03, 2023 at 2:32 PM
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS, cron, other?

Hello,

I am a new user of GPFS (Spectrum Scale) and would like to know if there is a ?best practice? on handling kernel updates on HPC clients. We are running Ubuntu 18.04 and 20.04 clients with 5.1.x, talking to RHEL storage servers, and would like to know how to handle re-compiling the client-side kernel modules.

There is of course the ?mmbuildgpl? utility:

        https://urldefense.com/v3/__https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-mmbuildgpl-command__;!!F9svGWnIaVPGSwU!tqT64CJ2SAvhvbhPiYeVKH9b1XztNG7r2d8dAVOj48Zej07X8wGOjK_lrIA3R9Y-lHfOSLaUjPtEWB5YIlBHO_nwEGE-eT0mkfQ$

but how do folks invoke it? Manually, via cron at night on or reboot, via some kind of apt (dpkg-trigger(1)) / RPM hook?

We have the ?unattended-upgrades? package enabled, which only installs security-tagged updates by default, but sometimes this does include kernel updates, which may become active on the next reboot:

        https://urldefense.com/v3/__https://packages.ubuntu.com/search?keywords=unattended-upgrades__;!!F9svGWnIaVPGSwU!tqT64CJ2SAvhvbhPiYeVKH9b1XztNG7r2d8dAVOj48Zej07X8wGOjK_lrIA3R9Y-lHfOSLaUjPtEWB5YIlBHO_nwEGE-Mb-QG4A$

So is there a best practice? Has someone invented this wheel that I could leverage, or will I have to invent it myself?

Thanks for any info.

--
David Magda
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!F9svGWnIaVPGSwU!tqT64CJ2SAvhvbhPiYeVKH9b1XztNG7r2d8dAVOj48Zej07X8wGOjK_lrIA3R9Y-lHfOSLaUjPtEWB5YIlBHO_nwEGE-G6wiNEE$
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230303/3963b05a/attachment.htm>

From Walter.Sklenka at EDV-Design.at  Sun Mar  5 15:10:17 2023
From: Walter.Sklenka at EDV-Design.at (Walter Sklenka)
Date: Sun, 5 Mar 2023 15:10:17 +0000
Subject: [gpfsug-discuss] detectIpPairAggressiveness RE: Reasons for
 DiskLeaseThread Overloaded
In-Reply-To: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>
References: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>
Message-ID: <c4a86e4a219e490aab688d00a559d658@Mail.EDVDesign.cloudia>

Hi Felipe!
Yes . I am very sorry that I answer with such a delay!!
This was the response from the Support
?
The IP pair connectivity detection is to address this issue:
If there are more than one IP pair between a pair of nodes, if one of the IP pair has some problems, and disk lease request or reply
happens be sent on this IP pair, disk lease overdue could happen since TCP has very long retransmit timeout, but since the other IP
pairs are in good condition, we should avoid the node expel and send disk lease via other good IP pairs. So, when sending disk
lease and reply, we will detect the connectivity of the IP pair, if it?s in good condition, disk lease and reply will be sent, otherwise, we
will try other IP pairs for sending.
detectIpPairAggressiveness (undocumented configuration parameter) control whether we do detection, here there is only ONE IP
pair, actually, we don't need to check the IP pair connectivity. But we still need ping to work since we have other places to do
ping check, like when disk lease overdue happens.
---------------------------

until the 5.1.7 code will be available early march detectIpPairAggressiveness may can be disabled.
echo 999 | mmchconfig detectIpPairAggressiveness=0 -i


THANK YOU VERY MUCH, Felipe!!!

Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Felipe Knop
Sent: Donnerstag, 2. M?rz 2023 04:33
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will be in the upcoming 5.1.7.0 release.

Thanks for opening the case and working with the support team on this one.

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 22, 2023 at 5:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi ; sorry for the delay
Our case is TS012184140
They are still analizing
As soon as I get feedback I will update you




Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> Im Auftrag von Ryan Novosielski
Gesendet: Friday, February 17, 2023 11:52 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

I talked about it a lot in TS011616986.

Part of the problem is we?re having a lot of strange problems at the same time, and so the different issues we?re having often come together (like one cause shows two symptoms).

I can?t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'


On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>> wrote:

Hi Ryan and Felipe!
Could you eventually tell me the case number if you remember it?
I opened the case and would reference to your case ID
Or shall I send you mine ?


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: Freitag, 17. Februar 2023 06:43
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Thanks for this, Felipe.

We?ve started seeing intermittent overdue leases in large numbers and don?t otherwise have an explanation for it, other than ?look at your network,? which actually does show occasional signs of strange behavior/higher-than-normal RTO values, but we?re not necessarily seeing those things happen at the same times as the lease issues. We?ve also seen ?GPFS Critical Thread Watchdog? recently.

We had a case open about it, but didn?t draw any real conclusions. If any of our data might be helpful/if there?s a case we could reference to see if we?re also running into that, we could provide a gpfs.snap.

FWIW, we are running 5.1.3-1 on the storage side (except one system that?s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///utgers/>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'



On Feb 16, 2023, at 12:02, Felipe Knop <knop at us.ibm.com<mailto:knop at us.ibm.com>> wrote:

Walter,

Thanks for the detailed description.

I don?t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.

I?d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to ?loop? for a while.  That might cause the critical thread watchdog to flag the lease thread as taking too long to ?check in?.

Capturing gpfs.snap is important, since we?d be looking into all the

[W] ------------------[GPFS Critical Thread Watchdog]------------------

instances.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Thursday, February 16, 2023 at 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy ? I did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi Felipe!
Once again me. Thank you very much for the hint
I did not open a PMR yet because I fear they will ask me/us if we are cracy ?
I did not tell the full story yet
We have a 3 node cluster, 2 NSD servers  o1,o2 (same site ) and g1 (different site). (rhel 8.7)
All of them are Vmware VMs
O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD ,  for a filesystem fs4vm (m,r=2 )


[root at ogpfs1 ras]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         edvdesign-cluster.local
  GPFS cluster id:           12147978822727803186
  GPFS UID domain:           edvdesign-cluster.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name     IP address     Admin node name      Designation
----------------------------------------------------------------------------
   1   ogpfs1-hs.local      10.20.30.1     ogpfs1-hs.local      quorum-manager-perfmon
   2   ogpfs2-hs.local      10.20.30.2     ogpfs2-hs.local      quorum-manager-perfmon
   3   ggpfsq.mgmt.cloudia  xxxx.other.net<http://xxxx.other.net/> ggpfsq.mgmt. a  quorum-perfmon

[root at ogpfs1 ras]# mmlsconfig
Configuration data for cluster edvdesign-cluster.local:
-------------------------------------------------------
clusterName edvdesign-cluster.local
clusterId 12147978822727803186
autoload yes
profile gpfsProtocolRandomIO
dmapiFileHandleSize 32
minReleaseLevel 5.1.6.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
[srv]
verbsPorts mlx5_0/1 mlx5_1/1
[common]
cesSharedRoot /fs4vmware/cesSharedRoot
[srv]
maxFilesToCache 10000
maxStatCache 20000
[common]
verbsRdma enable
[ggpfsq]
verbsRdma disable
[common]
verbsRdmaSend yes
[ggpfsq]
verbsRdmaSend no
[common]
verbsRdmaCm enable
[ggpfsq]
verbsRdmaCm disable
[srv]
pagepool 32G
[common]
adminMode central

File systems in cluster edvdesign-cluster.local:
------------------------------------------------
/dev/fs4vm


[root at ogpfs1 ras]# mmlsdisk fs4vm -L
disk         driver   sector     failure holds    holds                                    storage
name         type       size       group metadata data  status        availability disk id pool         remarks
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------
ogpfs1_1     nsd         512           1 yes      yes   ready         up                 1 system        desc
ogpfs2_1     nsd         512           2 yes      yes   ready         up                 2 system        desc
ggpfsq_qdisk nsd         512          -1 no       no    ready         up                 3 system        desc
Number of quorum disks: 3
Read quorum value:      2
Write quorum value:     2

And the two nodes o1 and o2 export the filesystem via CES NFS  functions ( for VMware)
I think this isn?supported , that a NSD Server is also a CES Node?

And finally the RDMA Network :
The both NSD servers also have a Mellanox ConnectX-6 Lx  dual port 25Gb adapter also via passthrough
And these interfaces we configured for rdma (RoCE) ,
Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )

RDMA Connections between nodes:
  Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     0   Y  RTS   (Y)256  478202 (0  ) 12728       67024       8864789(0  ) 22776     4643      0             0
  Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     1   Y  RTS   (Y)256  477659 (0  ) 12489       67034       8864773(0  ) 22794     4639      0             0
[root at ogpfs1 ras]#


You mentioned that it might be a cpu contention :  Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)
And also the direct attached RDMA ( +DAEMON) network is also not good?

Do you think IBM would say no to check such a configuration ?


Best regards
Walter




From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Thanks for the details.

The stack trace below captures the lease thread in the middle of sending the ?lease? RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace ?does not show? whether there was anything blocking the thread prior to the point where the RPCs are sent.

At a first glance:

2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)

I believe nivcsw: 10 means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there is some CPU contention going on.

Could you open a case to get debug data collected?  If the problem can be recreated, I think we?ll need a recreate of the problem with traces enabled.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 15, 2023 at 4:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! This is a ?full? sequence in mmfs.?log.?latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19:?43:?51.?474+0100: [N] Disk lease period expired 0.?030 seconds ago in cluster
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
This is a ?full? sequence in mmfs.log.latest
Fortunately this was also the last event until now (yesterday evening)

Maybe you can have a look?
2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)
2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):
2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi
gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0
2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u
nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0
2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0
2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig
uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0
2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,
TscReply*), void*, void**, int, ...) + 0x1DA at ??:0
2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0
2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0
2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0
2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0
2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0
2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0
2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0
2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.

Thank you very much!
Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 00:06
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

All,

These messages like

[W] ------------------[GPFS Critical Thread Watchdog]------------------

indicate that a ?critical thread?, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.

Do you have other samples of the message, with a more complete stack trace?   Or was the instance below the only one?

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Tuesday, February 14, 2023 at 10:49 AM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! I started with 5.?1.?6.?0 and now am at [root@?ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5.?1.?6.?1 ". the messages started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@?gpfsug.?org> On
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
I started with 5.1.6.0 and now am at [root at ogpfs1 ~]# mmfsadm dump version
Dump level: verbose
Build branch "5.1.6.1 ".

the messages started  from the beginning



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Christian Vieser
Sent: Dienstag, 14. Februar 2023 15:34
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded


What version of Spectrum Scale is running there? Do these errors appear since your last version update?
Am 14.02.23 um 14:09 schrieb Walter Sklenka:
Dear Collegues!
May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?
Is this a ?local node? Problem or a network problem ?
I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load



Following is an excerpt from mmfs.log.latest

2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_12:06:53.600+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)
2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):
2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0
2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0
2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive
2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0
2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230305/bd6a17b4/attachment.htm>

From Walter.Sklenka at EDV-Design.at  Sun Mar  5 15:11:38 2023
From: Walter.Sklenka at EDV-Design.at (Walter Sklenka)
Date: Sun, 5 Mar 2023 15:11:38 +0000
Subject: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded //
 detectIpPairAggressiveness
In-Reply-To: <c4a86e4a219e490aab688d00a559d658@Mail.EDVDesign.cloudia>
References: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>
 <c4a86e4a219e490aab688d00a559d658@Mail.EDVDesign.cloudia>
Message-ID: <2febe7ed49c240c0b458961e48b87904@Mail.EDVDesign.cloudia>


Hi Felipe!
Yes . I am very sorry that I answer with such a delay!!
This was the response from the Support
?
The IP pair connectivity detection is to address this issue:
If there are more than one IP pair between a pair of nodes, if one of the IP pair has some problems, and disk lease request or reply
happens be sent on this IP pair, disk lease overdue could happen since TCP has very long retransmit timeout, but since the other IP
pairs are in good condition, we should avoid the node expel and send disk lease via other good IP pairs. So, when sending disk
lease and reply, we will detect the connectivity of the IP pair, if it?s in good condition, disk lease and reply will be sent, otherwise, we
will try other IP pairs for sending.
detectIpPairAggressiveness (undocumented configuration parameter) control whether we do detection, here there is only ONE IP
pair, actually, we don't need to check the IP pair connectivity. But we still need ping to work since we have other places to do
ping check, like when disk lease overdue happens.
---------------------------

until the 5.1.7 code will be available early march detectIpPairAggressiveness may can be disabled.
echo 999 | mmchconfig detectIpPairAggressiveness=0 -i


THANK YOU VERY MUCH, Felipe!!!

Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Donnerstag, 2. M?rz 2023 04:33
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will be in the upcoming 5.1.7.0 release.

Thanks for opening the case and working with the support team on this one.

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 22, 2023 at 5:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi ; sorry for the delay
Our case is TS012184140
They are still analizing
As soon as I get feedback I will update you




Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> Im Auftrag von Ryan Novosielski
Gesendet: Friday, February 17, 2023 11:52 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

I talked about it a lot in TS011616986.

Part of the problem is we?re having a lot of strange problems at the same time, and so the different issues we?re having often come together (like one cause shows two symptoms).

I can?t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>> wrote:

Hi Ryan and Felipe!
Could you eventually tell me the case number if you remember it?
I opened the case and would reference to your case ID
Or shall I send you mine ?


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: Freitag, 17. Februar 2023 06:43
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Thanks for this, Felipe.

We?ve started seeing intermittent overdue leases in large numbers and don?t otherwise have an explanation for it, other than ?look at your network,? which actually does show occasional signs of strange behavior/higher-than-normal RTO values, but we?re not necessarily seeing those things happen at the same times as the lease issues. We?ve also seen ?GPFS Critical Thread Watchdog? recently.

We had a case open about it, but didn?t draw any real conclusions. If any of our data might be helpful/if there?s a case we could reference to see if we?re also running into that, we could provide a gpfs.snap.

FWIW, we are running 5.1.3-1 on the storage side (except one system that?s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///utgers/>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'


On Feb 16, 2023, at 12:02, Felipe Knop <knop at us.ibm.com<mailto:knop at us.ibm.com>> wrote:

Walter,

Thanks for the detailed description.

I don?t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.

I?d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to ?loop? for a while.  That might cause the critical thread watchdog to flag the lease thread as taking too long to ?check in?.

Capturing gpfs.snap is important, since we?d be looking into all the

[W] ------------------[GPFS Critical Thread Watchdog]------------------

instances.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Thursday, February 16, 2023 at 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy ? I did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi Felipe!
Once again me. Thank you very much for the hint
I did not open a PMR yet because I fear they will ask me/us if we are cracy ?
I did not tell the full story yet
We have a 3 node cluster, 2 NSD servers  o1,o2 (same site ) and g1 (different site). (rhel 8.7)
All of them are Vmware VMs
O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD ,  for a filesystem fs4vm (m,r=2 )


[root at ogpfs1 ras]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         edvdesign-cluster.local
  GPFS cluster id:           12147978822727803186
  GPFS UID domain:           edvdesign-cluster.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name     IP address     Admin node name      Designation
----------------------------------------------------------------------------
   1   ogpfs1-hs.local      10.20.30.1     ogpfs1-hs.local      quorum-manager-perfmon
   2   ogpfs2-hs.local      10.20.30.2     ogpfs2-hs.local      quorum-manager-perfmon
   3   ggpfsq.mgmt.cloudia  xxxx.other.net<http://xxxx.other.net/> ggpfsq.mgmt. a  quorum-perfmon

[root at ogpfs1 ras]# mmlsconfig
Configuration data for cluster edvdesign-cluster.local:
-------------------------------------------------------
clusterName edvdesign-cluster.local
clusterId 12147978822727803186
autoload yes
profile gpfsProtocolRandomIO
dmapiFileHandleSize 32
minReleaseLevel 5.1.6.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
[srv]
verbsPorts mlx5_0/1 mlx5_1/1
[common]
cesSharedRoot /fs4vmware/cesSharedRoot
[srv]
maxFilesToCache 10000
maxStatCache 20000
[common]
verbsRdma enable
[ggpfsq]
verbsRdma disable
[common]
verbsRdmaSend yes
[ggpfsq]
verbsRdmaSend no
[common]
verbsRdmaCm enable
[ggpfsq]
verbsRdmaCm disable
[srv]
pagepool 32G
[common]
adminMode central

File systems in cluster edvdesign-cluster.local:
------------------------------------------------
/dev/fs4vm


[root at ogpfs1 ras]# mmlsdisk fs4vm -L
disk         driver   sector     failure holds    holds                                    storage
name         type       size       group metadata data  status        availability disk id pool         remarks
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------
ogpfs1_1     nsd         512           1 yes      yes   ready         up                 1 system        desc
ogpfs2_1     nsd         512           2 yes      yes   ready         up                 2 system        desc
ggpfsq_qdisk nsd         512          -1 no       no    ready         up                 3 system        desc
Number of quorum disks: 3
Read quorum value:      2
Write quorum value:     2

And the two nodes o1 and o2 export the filesystem via CES NFS  functions ( for VMware)
I think this isn?supported , that a NSD Server is also a CES Node?

And finally the RDMA Network :
The both NSD servers also have a Mellanox ConnectX-6 Lx  dual port 25Gb adapter also via passthrough
And these interfaces we configured for rdma (RoCE) ,
Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )

RDMA Connections between nodes:
  Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     0   Y  RTS   (Y)256  478202 (0  ) 12728       67024       8864789(0  ) 22776     4643      0             0
  Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     1   Y  RTS   (Y)256  477659 (0  ) 12489       67034       8864773(0  ) 22794     4639      0             0
[root at ogpfs1 ras]#


You mentioned that it might be a cpu contention :  Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)
And also the direct attached RDMA ( +DAEMON) network is also not good?

Do you think IBM would say no to check such a configuration ?


Best regards
Walter




From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Thanks for the details.

The stack trace below captures the lease thread in the middle of sending the ?lease? RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace ?does not show? whether there was anything blocking the thread prior to the point where the RPCs are sent.

At a first glance:

2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)

I believe nivcsw: 10 means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there is some CPU contention going on.

Could you open a case to get debug data collected?  If the problem can be recreated, I think we?ll need a recreate of the problem with traces enabled.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 15, 2023 at 4:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! This is a ?full? sequence in mmfs.?log.?latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19:?43:?51.?474+0100: [N] Disk lease period expired 0.?030 seconds ago in cluster
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
This is a ?full? sequence in mmfs.log.latest
Fortunately this was also the last event until now (yesterday evening)

Maybe you can have a look?
2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)
2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):
2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi
gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0
2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u
nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0
2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0
2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig
uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0
2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,
TscReply*), void*, void**, int, ...) + 0x1DA at ??:0
2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0
2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0
2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0
2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0
2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0
2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0
2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0
2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.

Thank you very much!
Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 00:06
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

All,

These messages like

[W] ------------------[GPFS Critical Thread Watchdog]------------------

indicate that a ?critical thread?, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.

Do you have other samples of the message, with a more complete stack trace?   Or was the instance below the only one?

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Tuesday, February 14, 2023 at 10:49 AM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! I started with 5.?1.?6.?0 and now am at [root@?ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5.?1.?6.?1 ". the messages started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@?gpfsug.?org> On
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
I started with 5.1.6.0 and now am at [root at ogpfs1 ~]# mmfsadm dump version
Dump level: verbose
Build branch "5.1.6.1 ".

the messages started  from the beginning



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Christian Vieser
Sent: Dienstag, 14. Februar 2023 15:34
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded


What version of Spectrum Scale is running there? Do these errors appear since your last version update?
Am 14.02.23 um 14:09 schrieb Walter Sklenka:
Dear Collegues!
May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?
Is this a ?local node? Problem or a network problem ?
I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load



Following is an excerpt from mmfs.log.latest

2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_12:06:53.600+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)
2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):
2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0
2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0
2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive
2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0
2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230305/221dbe03/attachment.htm>

From Walter.Sklenka at EDV-Design.at  Mon Mar  6 11:04:21 2023
From: Walter.Sklenka at EDV-Design.at (Walter Sklenka)
Date: Mon, 6 Mar 2023 11:04:21 +0000
Subject: [gpfsug-discuss] flash NVME / pool system and data together or
 separated
In-Reply-To: <2febe7ed49c240c0b458961e48b87904@Mail.EDVDesign.cloudia>
References: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>
 <c4a86e4a219e490aab688d00a559d658@Mail.EDVDesign.cloudia>
 <2febe7ed49c240c0b458961e48b87904@Mail.EDVDesign.cloudia>
Message-ID: <286790f1c9544bd6a27e9fdd61758f42@Mail.EDVDesign.cloudia>

Hi!
What would you suggest when using
If we have a large flash storage: Does it make sense to split metdata from data?
Are there any disadvantages when putty meta and data into the same disk pool (fastes technology)


Best regards
Walter



Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Walter Sklenka
Gesendet: Sunday, March 5, 2023 4:12 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded // detectIpPairAggressiveness


Hi Felipe!
Yes . I am very sorry that I answer with such a delay!!
This was the response from the Support
?
The IP pair connectivity detection is to address this issue:
If there are more than one IP pair between a pair of nodes, if one of the IP pair has some problems, and disk lease request or reply
happens be sent on this IP pair, disk lease overdue could happen since TCP has very long retransmit timeout, but since the other IP
pairs are in good condition, we should avoid the node expel and send disk lease via other good IP pairs. So, when sending disk
lease and reply, we will detect the connectivity of the IP pair, if it?s in good condition, disk lease and reply will be sent, otherwise, we
will try other IP pairs for sending.
detectIpPairAggressiveness (undocumented configuration parameter) control whether we do detection, here there is only ONE IP
pair, actually, we don't need to check the IP pair connectivity. But we still need ping to work since we have other places to do
ping check, like when disk lease overdue happens.
---------------------------

until the 5.1.7 code will be available early march detectIpPairAggressiveness may can be disabled.
echo 999 | mmchconfig detectIpPairAggressiveness=0 -i


THANK YOU VERY MUCH, Felipe!!!

Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Donnerstag, 2. M?rz 2023 04:33
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will be in the upcoming 5.1.7.0 release.

Thanks for opening the case and working with the support team on this one.

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 22, 2023 at 5:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi ; sorry for the delay
Our case is TS012184140
They are still analizing
As soon as I get feedback I will update you




Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> Im Auftrag von Ryan Novosielski
Gesendet: Friday, February 17, 2023 11:52 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

I talked about it a lot in TS011616986.

Part of the problem is we?re having a lot of strange problems at the same time, and so the different issues we?re having often come together (like one cause shows two symptoms).

I can?t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>> wrote:

Hi Ryan and Felipe!
Could you eventually tell me the case number if you remember it?
I opened the case and would reference to your case ID
Or shall I send you mine ?


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: Freitag, 17. Februar 2023 06:43
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Thanks for this, Felipe.

We?ve started seeing intermittent overdue leases in large numbers and don?t otherwise have an explanation for it, other than ?look at your network,? which actually does show occasional signs of strange behavior/higher-than-normal RTO values, but we?re not necessarily seeing those things happen at the same times as the lease issues. We?ve also seen ?GPFS Critical Thread Watchdog? recently.

We had a case open about it, but didn?t draw any real conclusions. If any of our data might be helpful/if there?s a case we could reference to see if we?re also running into that, we could provide a gpfs.snap.

FWIW, we are running 5.1.3-1 on the storage side (except one system that?s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///utgers/>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

On Feb 16, 2023, at 12:02, Felipe Knop <knop at us.ibm.com<mailto:knop at us.ibm.com>> wrote:

Walter,

Thanks for the detailed description.

I don?t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.

I?d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to ?loop? for a while.  That might cause the critical thread watchdog to flag the lease thread as taking too long to ?check in?.

Capturing gpfs.snap is important, since we?d be looking into all the

[W] ------------------[GPFS Critical Thread Watchdog]------------------

instances.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Thursday, February 16, 2023 at 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy ? I did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi Felipe!
Once again me. Thank you very much for the hint
I did not open a PMR yet because I fear they will ask me/us if we are cracy ?
I did not tell the full story yet
We have a 3 node cluster, 2 NSD servers  o1,o2 (same site ) and g1 (different site). (rhel 8.7)
All of them are Vmware VMs
O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD ,  for a filesystem fs4vm (m,r=2 )


[root at ogpfs1 ras]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         edvdesign-cluster.local
  GPFS cluster id:           12147978822727803186
  GPFS UID domain:           edvdesign-cluster.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name     IP address     Admin node name      Designation
----------------------------------------------------------------------------
   1   ogpfs1-hs.local      10.20.30.1     ogpfs1-hs.local      quorum-manager-perfmon
   2   ogpfs2-hs.local      10.20.30.2     ogpfs2-hs.local      quorum-manager-perfmon
   3   ggpfsq.mgmt.cloudia  xxxx.other.net<http://xxxx.other.net/> ggpfsq.mgmt. a  quorum-perfmon

[root at ogpfs1 ras]# mmlsconfig
Configuration data for cluster edvdesign-cluster.local:
-------------------------------------------------------
clusterName edvdesign-cluster.local
clusterId 12147978822727803186
autoload yes
profile gpfsProtocolRandomIO
dmapiFileHandleSize 32
minReleaseLevel 5.1.6.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
[srv]
verbsPorts mlx5_0/1 mlx5_1/1
[common]
cesSharedRoot /fs4vmware/cesSharedRoot
[srv]
maxFilesToCache 10000
maxStatCache 20000
[common]
verbsRdma enable
[ggpfsq]
verbsRdma disable
[common]
verbsRdmaSend yes
[ggpfsq]
verbsRdmaSend no
[common]
verbsRdmaCm enable
[ggpfsq]
verbsRdmaCm disable
[srv]
pagepool 32G
[common]
adminMode central

File systems in cluster edvdesign-cluster.local:
------------------------------------------------
/dev/fs4vm


[root at ogpfs1 ras]# mmlsdisk fs4vm -L
disk         driver   sector     failure holds    holds                                    storage
name         type       size       group metadata data  status        availability disk id pool         remarks
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------
ogpfs1_1     nsd         512           1 yes      yes   ready         up                 1 system        desc
ogpfs2_1     nsd         512           2 yes      yes   ready         up                 2 system        desc
ggpfsq_qdisk nsd         512          -1 no       no    ready         up                 3 system        desc
Number of quorum disks: 3
Read quorum value:      2
Write quorum value:     2

And the two nodes o1 and o2 export the filesystem via CES NFS  functions ( for VMware)
I think this isn?supported , that a NSD Server is also a CES Node?

And finally the RDMA Network :
The both NSD servers also have a Mellanox ConnectX-6 Lx  dual port 25Gb adapter also via passthrough
And these interfaces we configured for rdma (RoCE) ,
Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )

RDMA Connections between nodes:
  Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     0   Y  RTS   (Y)256  478202 (0  ) 12728       67024       8864789(0  ) 22776     4643      0             0
  Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     1   Y  RTS   (Y)256  477659 (0  ) 12489       67034       8864773(0  ) 22794     4639      0             0
[root at ogpfs1 ras]#


You mentioned that it might be a cpu contention :  Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)
And also the direct attached RDMA ( +DAEMON) network is also not good?

Do you think IBM would say no to check such a configuration ?


Best regards
Walter




From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Thanks for the details.

The stack trace below captures the lease thread in the middle of sending the ?lease? RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace ?does not show? whether there was anything blocking the thread prior to the point where the RPCs are sent.

At a first glance:

2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)

I believe nivcsw: 10 means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there is some CPU contention going on.

Could you open a case to get debug data collected?  If the problem can be recreated, I think we?ll need a recreate of the problem with traces enabled.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 15, 2023 at 4:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! This is a ?full? sequence in mmfs.?log.?latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19:?43:?51.?474+0100: [N] Disk lease period expired 0.?030 seconds ago in cluster
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
This is a ?full? sequence in mmfs.log.latest
Fortunately this was also the last event until now (yesterday evening)

Maybe you can have a look?
2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)
2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):
2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi
gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0
2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u
nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0
2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0
2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig
uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0
2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,
TscReply*), void*, void**, int, ...) + 0x1DA at ??:0
2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0
2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0
2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0
2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0
2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0
2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0
2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0
2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.

Thank you very much!
Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 00:06
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

All,

These messages like

[W] ------------------[GPFS Critical Thread Watchdog]------------------

indicate that a ?critical thread?, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.

Do you have other samples of the message, with a more complete stack trace?   Or was the instance below the only one?

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Tuesday, February 14, 2023 at 10:49 AM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! I started with 5.?1.?6.?0 and now am at [root@?ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5.?1.?6.?1 ". the messages started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@?gpfsug.?org> On
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender

This message came from outside your organization.



ZjQcmQRYFpfptBannerEnd
Hi!
I started with 5.1.6.0 and now am at [root at ogpfs1 ~]# mmfsadm dump version
Dump level: verbose
Build branch "5.1.6.1 ".

the messages started  from the beginning



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Christian Vieser
Sent: Dienstag, 14. Februar 2023 15:34
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded


What version of Spectrum Scale is running there? Do these errors appear since your last version update?
Am 14.02.23 um 14:09 schrieb Walter Sklenka:
Dear Collegues!
May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?
Is this a ?local node? Problem or a network problem ?
I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load



Following is an excerpt from mmfs.log.latest

2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_12:06:53.600+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)
2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):
2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0
2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0
2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive
2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0
2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230306/9b598799/attachment.htm>

From novosirj at rutgers.edu  Mon Mar  6 16:28:25 2023
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 6 Mar 2023 16:28:25 +0000
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS,
 cron, other?
In-Reply-To: <bd4d1665-7271-0819-0493-f55697e41d0a@strath.ac.uk>
References: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
 <bd4d1665-7271-0819-0493-f55697e41d0a@strath.ac.uk>
Message-ID: <6F466A3D-420B-4B66-9595-58B7846922D2@rutgers.edu>

On Mar 3, 2023, at 15:05, Jonathan Buzzard <jonathan.buzzard at strath.ac.uk> wrote:

I would suggest that you disable any automatic upgrading of the kernel. Kernel upgrades should *only* be done *after* you have verified that it will work.

If you don't it is only a matter of time before a security update breaks GPFS. There was at least one instance of that happing in the last five years.

Definitely more than once, and technically you should not upgrade at all unless the kernel is specifically listed as supported on this page, otherwise you may get to be the one that finds the non-obvious compatibility bug:

https://www.ibm.com/docs/en/spectrum-scale?topic=STXKQY/gpfsclustersfaq.html

As a result, we typically don?t bother with any automatic updating of GPFS, because if we?re going to upgrade either GPFS or the kernel, we already know about it and do it on purpose. We also build the RPMs on one system, since the vast majority of our equipment does not have the full compiler set installed.

--
#BlackLivesMatter
____
|| \\UTGERS,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230306/4145eb8a/attachment.htm>

From skylar2 at uw.edu  Mon Mar  6 16:38:58 2023
From: skylar2 at uw.edu (Skylar Thompson)
Date: Mon, 6 Mar 2023 08:38:58 -0800
Subject: [gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS,
 cron, other?
In-Reply-To: <6F466A3D-420B-4B66-9595-58B7846922D2@rutgers.edu>
References: <0BB7CBDB-1DFF-4EE1-9306-2C6C3C64D66C@ee.torontomu.ca>
 <bd4d1665-7271-0819-0493-f55697e41d0a@strath.ac.uk>
 <6F466A3D-420B-4B66-9595-58B7846922D2@rutgers.edu>
Message-ID: <20230306163858.ubwp5lkwmqoqy6db@thargelion>

Yep, this is our strategy too. We version-lock GPFS and the kernel in yum
and define the target versions in our configuration management (Puppet),
with the version-specific gplbin packages stored in our local yum repo.
When we want to upgrade, we schedule an outage, build a new package, put in
the repo, then bump the target versions in Puppet and reboot.

On Mon, Mar 06, 2023 at 04:28:25PM +0000, Ryan Novosielski wrote:
> On Mar 3, 2023, at 15:05, Jonathan Buzzard <jonathan.buzzard at strath.ac.uk> wrote:
> 
> I would suggest that you disable any automatic upgrading of the kernel. Kernel upgrades should *only* be done *after* you have verified that it will work.
> 
> If you don't it is only a matter of time before a security update breaks GPFS. There was at least one instance of that happing in the last five years.
> 
> Definitely more than once, and technically you should not upgrade at all unless the kernel is specifically listed as supported on this page, otherwise you may get to be the one that finds the non-obvious compatibility bug:
> 
> https://urldefense.com/v3/__https://www.ibm.com/docs/en/spectrum-scale?topic=STXKQY*gpfsclustersfaq.html__;Lw!!K-Hz7m0Vt54!m8GZ2APARwPJKv-TGg_fh2W4XPf_2ZHMVCXR97Wdf-lcpQ9X7zGpaUeKCI0KsEX_Vrt8__J-OROZ9CEPjiGG$ 
> 
> As a result, we typically don???t bother with any automatic updating of GPFS, because if we???re going to upgrade either GPFS or the kernel, we already know about it and do it on purpose. We also build the RPMs on one system, since the vast majority of our equipment does not have the full compiler set installed.
> 
> --
> #BlackLivesMatter
> ____
> || \\UTGERS,    |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
>      `'
> 

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!K-Hz7m0Vt54!m8GZ2APARwPJKv-TGg_fh2W4XPf_2ZHMVCXR97Wdf-lcpQ9X7zGpaUeKCI0KsEX_Vrt8__J-OROZ9Ny2WL5J$ 


-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His



From dmagda+gpfs at ee.torontomu.ca  Mon Mar  6 21:36:09 2023
From: dmagda+gpfs at ee.torontomu.ca (David Magda)
Date: Mon, 6 Mar 2023 16:36:09 -0500
Subject: [gpfsug-discuss] Mailman interface HTTP 500
Message-ID: <CD9AB56C-B632-4CF0-994A-04AE5FBA5815@ee.torontomu.ca>

To whomever runs the web server:

Apache (?) seems to be throwing errors when I go to Mailman web interface:

	https://gpfsug.org/mailman/listinfo/
	https://www.gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
	https://www.spectrumscaleug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

The archives are fine (since they?re probably just flat HTML files and not CGI):

	https://www.gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/
	https://www.spectrumscaleug.org/pipermail/gpfsug-discuss_gpfsug.org/

-- 
David Magda


From systems at gpfsug.org  Tue Mar  7 10:22:11 2023
From: systems at gpfsug.org (systems at gpfsug.org)
Date: Tue, 7 Mar 2023 10:22:11 -0000
Subject: [gpfsug-discuss] Mailman interface HTTP 500
In-Reply-To: <CD9AB56C-B632-4CF0-994A-04AE5FBA5815@ee.torontomu.ca>
References: <CD9AB56C-B632-4CF0-994A-04AE5FBA5815@ee.torontomu.ca>
Message-ID: <095301d950de$ae1e8da0$0a5ba8e0$@gpfsug.org>

Hi David,

Thanks for letting us know, this has been reported on slack too.

There seems to be a bug in one of the scripts that is causing EGID and RGID errors; it's been reported to the software vendor and we're awaiting feedback.

-- systems

-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of David Magda
Sent: 06 March 2023 21:36
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] Mailman interface HTTP 500

To whomever runs the web server:

Apache (?) seems to be throwing errors when I go to Mailman web interface:

	https://gpfsug.org/mailman/listinfo/
	https://www.gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
	https://www.spectrumscaleug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

The archives are fine (since they?re probably just flat HTML files and not CGI):

	https://www.gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/
	https://www.spectrumscaleug.org/pipermail/gpfsug-discuss_gpfsug.org/

-- 
David Magda
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org




From jpoling at us.ibm.com  Wed Mar  8 17:14:32 2023
From: jpoling at us.ibm.com (Jenny Poling)
Date: Wed, 8 Mar 2023 17:14:32 +0000
Subject: [gpfsug-discuss] mmchconfig ESS error
Message-ID: <BLAPR15MB392333B1090CF399E7A12B3282B49@BLAPR15MB3923.namprd15.prod.outlook.com>

We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1
The mmbuildgpl  completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.  Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.

mmchconfig release=LATEST
mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.
mmchconfig: Command failed. Examine previous error messages to determine cause.

Here are the rpm packages installed
gpfs.license.adv-5.1.5-1.ppc64le
gpfs.base-5.1.5-1.ppc64le
gpfs.docs-5.1.5-1.noarch
gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le
gpfs.adv-5.1.5-1.ppc64le
gpfs.gpl-5.1.5-1.noarch
gpfs.compression-5.1.5-1.ppc64le
gpfs.gskit-8.0.55-19.1.ppc64le
gpfs.msg.en_US-5.1.5-1.noarch

RH OS level is 8.7
Architect: ppc64le


Jenny Poling
IT Specialist  / HPSS Test Team
IGNITE/Test Innovation
Global Business Services - Public Service
713.582.7690(c)
jpoling at us.ibm.com<mailto:jpoling at us.ibm.com>; www.hpss-collaboration.org<http://www.hpss-collaboration.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/6ecc5c3d/attachment.htm>

From knop at us.ibm.com  Wed Mar  8 18:15:29 2023
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 8 Mar 2023 18:15:29 +0000
Subject: [gpfsug-discuss] mmchconfig ESS error
Message-ID: <8CA50659-9D82-4112-88EF-77555A372FA9@us.ibm.com>

Jenny,

Please open a customer case for this one, as you may need help cleaning up some ?sql? statements in the mmsdrfs file. It appears to me that gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le should not be installed if the node is not an ESS server.

Regards,

  Felipe

----
Felipe Knop        knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Jenny Poling <jpoling at us.ibm.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: Wednesday, March 8, 2023 at 12:19 PM
To: "gpfsug-discuss at gpfsug.org" <gpfsug-discuss at gpfsug.org>
Cc: Harry Yuen <hyuen at us.ibm.com>
Subject: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error

We are trying to manually upgrade Spectrum Scale from 5.?1.?2.?1 to 5.?1.?5.?1 The mmbuildgpl completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1
The mmbuildgpl  completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.  Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.

mmchconfig release=LATEST
mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.
mmchconfig: Command failed. Examine previous error messages to determine cause.

Here are the rpm packages installed
gpfs.license.adv-5.1.5-1.ppc64le
gpfs.base-5.1.5-1.ppc64le
gpfs.docs-5.1.5-1.noarch
gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le
gpfs.adv-5.1.5-1.ppc64le
gpfs.gpl-5.1.5-1.noarch
gpfs.compression-5.1.5-1.ppc64le
gpfs.gskit-8.0.55-19.1.ppc64le
gpfs.msg.en_US-5.1.5-1.noarch

RH OS level is 8.7
Architect: ppc64le


Jenny Poling
IT Specialist  / HPSS Test Team
IGNITE/Test Innovation
Global Business Services - Public Service
713.582.7690(c)
jpoling at us.ibm.com<mailto:jpoling at us.ibm.com>; www.hpss-collaboration.org<http://www.hpss-collaboration.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/513fd007/attachment.htm>

From knop at us.ibm.com  Wed Mar  8 19:08:39 2023
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 8 Mar 2023 19:08:39 +0000
Subject: [gpfsug-discuss] mmchconfig ESS error
In-Reply-To: <8CA50659-9D82-4112-88EF-77555A372FA9@us.ibm.com>
References: <8CA50659-9D82-4112-88EF-77555A372FA9@us.ibm.com>
Message-ID: <29CA0112-D2BE-46FE-815B-0CBE3BF0609F@us.ibm.com>

All,

Apologies. My

that gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le should not be installed if the node is not an ESS server.

statement was incorrect, as that RPM is installed even in non-ESS-server nodes.

  Felipe

----
Felipe Knop        knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: Felipe Knop <knop at us.ibm.com>
Date: Wednesday, March 8, 2023 at 1:15 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Cc: Harry Yuen <hyuen at us.ibm.com>
Subject: Re: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error

Jenny,

Please open a customer case for this one, as you may need help cleaning up some ?sql? statements in the mmsdrfs file. It appears to me that gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le should not be installed if the node is not an ESS server.

Regards,

  Felipe

----
Felipe Knop        knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Jenny Poling <jpoling at us.ibm.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: Wednesday, March 8, 2023 at 12:19 PM
To: "gpfsug-discuss at gpfsug.org" <gpfsug-discuss at gpfsug.org>
Cc: Harry Yuen <hyuen at us.ibm.com>
Subject: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error

We are trying to manually upgrade Spectrum Scale from 5.?1.?2.?1 to 5.?1.?5.?1 The mmbuildgpl completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1
The mmbuildgpl  completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.  Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.

mmchconfig release=LATEST
mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.
mmchconfig: Command failed. Examine previous error messages to determine cause.

Here are the rpm packages installed
gpfs.license.adv-5.1.5-1.ppc64le
gpfs.base-5.1.5-1.ppc64le
gpfs.docs-5.1.5-1.noarch
gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le
gpfs.adv-5.1.5-1.ppc64le
gpfs.gpl-5.1.5-1.noarch
gpfs.compression-5.1.5-1.ppc64le
gpfs.gskit-8.0.55-19.1.ppc64le
gpfs.msg.en_US-5.1.5-1.noarch

RH OS level is 8.7
Architect: ppc64le


Jenny Poling
IT Specialist  / HPSS Test Team
IGNITE/Test Innovation
Global Business Services - Public Service
713.582.7690(c)
jpoling at us.ibm.com<mailto:jpoling at us.ibm.com>; www.hpss-collaboration.org<http://www.hpss-collaboration.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/ea2b5f68/attachment.htm>

From anobre at br.ibm.com  Wed Mar  8 20:47:34 2023
From: anobre at br.ibm.com (Anderson Ferreira Nobre)
Date: Wed, 8 Mar 2023 20:47:34 +0000
Subject: [gpfsug-discuss] mmchconfig ESS error
In-Reply-To: <BLAPR15MB392333B1090CF399E7A12B3282B49@BLAPR15MB3923.namprd15.prod.outlook.com>
References: <BLAPR15MB392333B1090CF399E7A12B3282B49@BLAPR15MB3923.namprd15.prod.outlook.com>
Message-ID: <DM6PR15MB274605D11856BC61EB12E0B2F8B49@DM6PR15MB2746.namprd15.prod.outlook.com>

Hi Jenny,

Try to run the upgrade with spectrumscale command. Perhaps it gives you more insight about what?s missing.

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Jenny Poling
Sent: Wednesday, March 8, 2023 2:15 PM
To: gpfsug-discuss at gpfsug.org
Cc: Harry Yuen <hyuen at us.ibm.com>
Subject: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error

We are trying to manually upgrade Spectrum Scale from 5.?1.?2.?1 to 5.?1.?5.?1 The mmbuildgpl completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1
The mmbuildgpl  completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.  Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.

mmchconfig release=LATEST
mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.
mmchconfig: Command failed. Examine previous error messages to determine cause.

Here are the rpm packages installed
gpfs.license.adv-5.1.5-1.ppc64le
gpfs.base-5.1.5-1.ppc64le
gpfs.docs-5.1.5-1.noarch
gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le
gpfs.adv-5.1.5-1.ppc64le
gpfs.gpl-5.1.5-1.noarch
gpfs.compression-5.1.5-1.ppc64le
gpfs.gskit-8.0.55-19.1.ppc64le
gpfs.msg.en_US-5.1.5-1.noarch

RH OS level is 8.7
Architect: ppc64le


Jenny Poling
IT Specialist  / HPSS Test Team
IGNITE/Test Innovation
Global Business Services - Public Service
713.582.7690(c)
jpoling at us.ibm.com<mailto:jpoling at us.ibm.com>; www.hpss-collaboration.org<http://www.hpss-collaboration.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/9ef5b7d4/attachment.htm>

From jjdoherty at yahoo.com  Wed Mar  8 20:59:24 2023
From: jjdoherty at yahoo.com (Jim Doherty)
Date: Wed, 8 Mar 2023 20:59:24 +0000 (UTC)
Subject: [gpfsug-discuss] mmchconfig ESS error
In-Reply-To: <DM6PR15MB274605D11856BC61EB12E0B2F8B49@DM6PR15MB2746.namprd15.prod.outlook.com>
References: <BLAPR15MB392333B1090CF399E7A12B3282B49@BLAPR15MB3923.namprd15.prod.outlook.com>
 <DM6PR15MB274605D11856BC61EB12E0B2F8B49@DM6PR15MB2746.namprd15.prod.outlook.com>
Message-ID: <542905356.658906.1678309164263@mail.yahoo.com>

 Try running? the following:?
# script mmchconfig.log# DEBUG=1? ; mmchconfig??release=LATEST# exit
Then you can open a ticket to the Spectrum Scale queue? ?and upload the mmchconfig.log along with a gpfs.snap from the node.

    On Wednesday, March 8, 2023 at 03:52:43 PM EST, Anderson Ferreira Nobre <anobre at br.ibm.com> wrote:  
 
 <!--#yiv9601048786 filtered {}#yiv9601048786 filtered {}#yiv9601048786 filtered {}#yiv9601048786 filtered {}#yiv9601048786 p.yiv9601048786MsoNormal, #yiv9601048786 li.yiv9601048786MsoNormal, #yiv9601048786 div.yiv9601048786MsoNormal {margin:0in;font-size:10.0pt;font-family:"Calibri", sans-serif;}#yiv9601048786 a:link, #yiv9601048786 span.yiv9601048786MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv9601048786 span.yiv9601048786EmailStyle19 {font-family:"IBM Plex Sans", sans-serif;color:windowtext;}#yiv9601048786 .yiv9601048786MsoChpDefault {font-size:10.0pt;}#yiv9601048786 filtered {}#yiv9601048786 div.yiv9601048786WordSection1 {}-->
Hi Jenny,
 
 ?
 
Try to run the upgrade with spectrumscale command. Perhaps it gives you more insight about what?s missing.
 
 ?
 
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org>On Behalf Of Jenny Poling
Sent: Wednesday, March 8, 2023 2:15 PM
To: gpfsug-discuss at gpfsug.org
Cc: Harry Yuen <hyuen at us.ibm.com>
Subject: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error
 
 ?
 
We are trying to manually upgrade Spectrum Scale from 5.?1.?2.?1 to 5.?1.?5.?1 The mmbuildgpl completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package
 
ZjQcmQRYFpfptBannerStart
 
| 
| 
| 
This Message Is From an External Sender
  |
| 
This message came from outside your organization.
  |

 |

 |


ZjQcmQRYFpfptBannerEnd
 
We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1
 
The mmbuildgpl ?completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.? Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.
 
?
 
mmchconfig release=LATEST
 
mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.
 
mmchconfig: Command failed. Examine previous error messages to determine cause.
 
 ?
 
Here are the rpm packages installed
 
gpfs.license.adv-5.1.5-1.ppc64le
 
gpfs.base-5.1.5-1.ppc64le
 
gpfs.docs-5.1.5-1.noarch
 
gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le
 
gpfs.adv-5.1.5-1.ppc64le
 
gpfs.gpl-5.1.5-1.noarch
 
gpfs.compression-5.1.5-1.ppc64le
 
gpfs.gskit-8.0.55-19.1.ppc64le
 
gpfs.msg.en_US-5.1.5-1.noarch
 
 ?
 
RH OS level is 8.7
 
Architect: ppc64le
 
 ?
 
 ?
 
Jenny Poling
 
IT Specialist? / HPSS Test Team
 
IGNITE/Test Innovation
 
Global Business Services - Public Service
 
713.582.7690(c)
 
jpoling at us.ibm.com;www.hpss-collaboration.org
 
 ?
 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/08b6afb8/attachment.htm>

From Paul.Sanchez at deshaw.com  Wed Mar  8 21:08:00 2023
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 8 Mar 2023 21:08:00 +0000
Subject: [gpfsug-discuss] mmchconfig ESS error
In-Reply-To: <542905356.658906.1678309164263@mail.yahoo.com>
References: <BLAPR15MB392333B1090CF399E7A12B3282B49@BLAPR15MB3923.namprd15.prod.outlook.com>
 <DM6PR15MB274605D11856BC61EB12E0B2F8B49@DM6PR15MB2746.namprd15.prod.outlook.com>
 <542905356.658906.1678309164263@mail.yahoo.com>
Message-ID: <0fbab32a4a0b491695671d2609488cbe@deshaw.com>

You?re probably missing these?

gpfs.gnr-5.1.5-1.ppc64le
gpfs.gnr.base-1.0.0-0.ppc64le

They?re hard to get from fixcentral unless you have an Erasure Code Edition license. IIRC, they?re in the fixcentral ESS upgrade bundle though which you can unpack and pluck the RPMs out of.  The only catch is that while you can download any version of your entitled editions/architectures of GPFS from fixcentral? if the only way you?re getting these for ESS on ppc64le is via the IBM tested and released upgrade bundles then you don?t have access to arbitrary GPFS versions but rather just the ones which made it into a periodic ESS release.

-Paul

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Jim Doherty
Sent: Wednesday, March 8, 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] mmchconfig ESS error


This message was sent by an external party.

Try running  the following:

# script mmchconfig.log
# DEBUG=1  ; mmchconfig  release=LATEST
# exit

Then you can open a ticket to the Spectrum Scale queue   and upload the mmchconfig.log along with a gpfs.snap from the node.


On Wednesday, March 8, 2023 at 03:52:43 PM EST, Anderson Ferreira Nobre <anobre at br.ibm.com<mailto:anobre at br.ibm.com>> wrote:



Hi Jenny,



Try to run the upgrade with spectrumscale command. Perhaps it gives you more insight about what?s missing.



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Jenny Poling
Sent: Wednesday, March 8, 2023 2:15 PM
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Cc: Harry Yuen <hyuen at us.ibm.com<mailto:hyuen at us.ibm.com>>
Subject: [EXTERNAL] [gpfsug-discuss] mmchconfig ESS error



We are trying to manually upgrade Spectrum Scale from 5.?1.?2.?1 to 5.?1.?5.?1 The mmbuildgpl completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender


This message came from outside your organization.




ZjQcmQRYFpfptBannerEnd

We are trying to manually upgrade Spectrum Scale from 5.1.2.1 to 5.1.5.1

The mmbuildgpl  completes successfully however when we run mmchconfig to bring the software up to the latest we get complaints about Elastic Storage Server and a package that is required for ESS.  Our box has no installation of ESS. Is there a way to bypass or has anyone else encountered a situation like this? Any suggestions will be greatly appreciated. We have combed google, look through logs, but there is not much information.



mmchconfig release=LATEST

mmchconfig: Verify you are executing the command on an ESS/GSS node and that the gpfs.gnr package is properly installed.

mmchconfig: Command failed. Examine previous error messages to determine cause.



Here are the rpm packages installed

gpfs.license.adv-5.1.5-1.ppc64le

gpfs.base-5.1.5-1.ppc64le

gpfs.docs-5.1.5-1.noarch

gpfs.gss.pmsensors-5.1.5-1.el8.ppc64le

gpfs.adv-5.1.5-1.ppc64le

gpfs.gpl-5.1.5-1.noarch

gpfs.compression-5.1.5-1.ppc64le

gpfs.gskit-8.0.55-19.1.ppc64le

gpfs.msg.en_US-5.1.5-1.noarch



RH OS level is 8.7

Architect: ppc64le





Jenny Poling

IT Specialist  / HPSS Test Team

IGNITE/Test Innovation

Global Business Services - Public Service

713.582.7690(c)

jpoling at us.ibm.com<mailto:jpoling at us.ibm.com>; www.hpss-collaboration.org<http://www.hpss-collaboration.org>


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230308/07de614d/attachment.htm>

From daniel.kidger at hpe.com  Thu Mar  9 15:24:22 2023
From: daniel.kidger at hpe.com (Kidger, Daniel)
Date: Thu, 9 Mar 2023 15:24:22 +0000
Subject: [gpfsug-discuss] flash NVME / pool system and data together or
 separated
In-Reply-To: <286790f1c9544bd6a27e9fdd61758f42@Mail.EDVDesign.cloudia>
References: <9ED44F39-4C0B-4640-9F7F-9D6DDBC6DB49@us.ibm.com>
 <c4a86e4a219e490aab688d00a559d658@Mail.EDVDesign.cloudia>
 <2febe7ed49c240c0b458961e48b87904@Mail.EDVDesign.cloudia>
 <286790f1c9544bd6a27e9fdd61758f42@Mail.EDVDesign.cloudia>
Message-ID: <PH7PR84MB1701009302F13162BBCB0D49FBB59@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>

Walter,

In general it is fine, but you do not say for example what the underlying provisioning or RAID is that you are using?
Nor if this is say ECE, ESS, or perhaps NVMeoF ?

For performance reasons, you might want your metadata to be 3-way replicated, or RAID1, if you do a lot of small metadata updates.

Daniel


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at>
Date: Monday, 6 March 2023 at 12:10
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] flash NVME / pool system and data together or separated
Hi!
What would you suggest when using
If we have a large flash storage: Does it make sense to split metdata from data?
Are there any disadvantages when putty meta and data into the same disk pool (fastes technology)


Best regards
Walter



Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> Im Auftrag von Walter Sklenka
Gesendet: Sunday, March 5, 2023 4:12 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded // detectIpPairAggressiveness


Hi Felipe!
Yes . I am very sorry that I answer with such a delay!!
This was the response from the Support
?
The IP pair connectivity detection is to address this issue:
If there are more than one IP pair between a pair of nodes, if one of the IP pair has some problems, and disk lease request or reply
happens be sent on this IP pair, disk lease overdue could happen since TCP has very long retransmit timeout, but since the other IP
pairs are in good condition, we should avoid the node expel and send disk lease via other good IP pairs. So, when sending disk
lease and reply, we will detect the connectivity of the IP pair, if it?s in good condition, disk lease and reply will be sent, otherwise, we
will try other IP pairs for sending.
detectIpPairAggressiveness (undocumented configuration parameter) control whether we do detection, here there is only ONE IP
pair, actually, we don't need to check the IP pair connectivity. But we still need ping to work since we have other places to do
ping check, like when disk lease overdue happens.
---------------------------

until the 5.1.7 code will be available early march detectIpPairAggressiveness may can be disabled.
echo 999 | mmchconfig detectIpPairAggressiveness=0 -i


THANK YOU VERY MUCH, Felipe!!!

Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Donnerstag, 2. M?rz 2023 04:33
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will be in the upcoming 5.1.7.0 release.

Thanks for opening the case and working with the support team on this one.

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 22, 2023 at 5:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210

Hi ; sorry for the delay
Our case is TS012184140
They are still analizing
As soon as I get feedback I will update you




Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: sklenka at edv-design.at<mailto:sklenka at edv-design.at>
Internet: www.edv-design.at<http://www.edv-design.at/>

Von: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> Im Auftrag von Ryan Novosielski
Gesendet: Friday, February 17, 2023 11:52 PM
An: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Betreff: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

I talked about it a lot in TS011616986.

Part of the problem is we?re having a lot of strange problems at the same time, and so the different issues we?re having often come together (like one cause shows two symptoms).

I can?t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>> wrote:

Hi Ryan and Felipe!
Could you eventually tell me the case number if you remember it?
I opened the case and would reference to your case ID
Or shall I send you mine ?


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: Freitag, 17. Februar 2023 06:43
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Thanks for this, Felipe.

We?ve started seeing intermittent overdue leases in large numbers and don?t otherwise have an explanation for it, other than ?look at your network,? which actually does show occasional signs of strange behavior/higher-than-normal RTO values, but we?re not necessarily seeing those things happen at the same times as the lease issues. We?ve also seen ?GPFS Critical Thread Watchdog? recently.

We had a case open about it, but didn?t draw any real conclusions. If any of our data might be helpful/if there?s a case we could reference to see if we?re also running into that, we could provide a gpfs.snap.

FWIW, we are running 5.1.3-1 on the storage side (except one system that?s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.

--
#BlackLivesMatter
____
|| \\UTGERS<file:///utgers/>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

On Feb 16, 2023, at 12:02, Felipe Knop <knop at us.ibm.com<mailto:knop at us.ibm.com>> wrote:

Walter,

Thanks for the detailed description.

I don?t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.

I?d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to ?loop? for a while.  That might cause the critical thread watchdog to flag the lease thread as taking too long to ?check in?.

Capturing gpfs.snap is important, since we?d be looking into all the

[W] ------------------[GPFS Critical Thread Watchdog]------------------

instances.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Thursday, February 16, 2023 at 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy ? I did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different

Hi Felipe!
Once again me. Thank you very much for the hint
I did not open a PMR yet because I fear they will ask me/us if we are cracy ?
I did not tell the full story yet
We have a 3 node cluster, 2 NSD servers  o1,o2 (same site ) and g1 (different site). (rhel 8.7)
All of them are Vmware VMs
O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD ,  for a filesystem fs4vm (m,r=2 )


[root at ogpfs1 ras]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         edvdesign-cluster.local
  GPFS cluster id:           12147978822727803186
  GPFS UID domain:           edvdesign-cluster.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name     IP address     Admin node name      Designation
----------------------------------------------------------------------------
   1   ogpfs1-hs.local      10.20.30.1     ogpfs1-hs.local      quorum-manager-perfmon
   2   ogpfs2-hs.local      10.20.30.2     ogpfs2-hs.local      quorum-manager-perfmon
   3   ggpfsq.mgmt.cloudia  xxxx.other.net<http://xxxx.other.net/> ggpfsq.mgmt. a  quorum-perfmon

[root at ogpfs1 ras]# mmlsconfig
Configuration data for cluster edvdesign-cluster.local:
-------------------------------------------------------
clusterName edvdesign-cluster.local
clusterId 12147978822727803186
autoload yes
profile gpfsProtocolRandomIO
dmapiFileHandleSize 32
minReleaseLevel 5.1.6.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
cipherList AUTHONLY
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
[srv]
verbsPorts mlx5_0/1 mlx5_1/1
[common]
cesSharedRoot /fs4vmware/cesSharedRoot
[srv]
maxFilesToCache 10000
maxStatCache 20000
[common]
verbsRdma enable
[ggpfsq]
verbsRdma disable
[common]
verbsRdmaSend yes
[ggpfsq]
verbsRdmaSend no
[common]
verbsRdmaCm enable
[ggpfsq]
verbsRdmaCm disable
[srv]
pagepool 32G
[common]
adminMode central

File systems in cluster edvdesign-cluster.local:
------------------------------------------------
/dev/fs4vm


[root at ogpfs1 ras]# mmlsdisk fs4vm -L
disk         driver   sector     failure holds    holds                                    storage
name         type       size       group metadata data  status        availability disk id pool         remarks
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------
ogpfs1_1     nsd         512           1 yes      yes   ready         up                 1 system        desc
ogpfs2_1     nsd         512           2 yes      yes   ready         up                 2 system        desc
ggpfsq_qdisk nsd         512          -1 no       no    ready         up                 3 system        desc
Number of quorum disks: 3
Read quorum value:      2
Write quorum value:     2

And the two nodes o1 and o2 export the filesystem via CES NFS  functions ( for VMware)
I think this isn?supported , that a NSD Server is also a CES Node?

And finally the RDMA Network :
The both NSD servers also have a Mellanox ConnectX-6 Lx  dual port 25Gb adapter also via passthrough
And these interfaces we configured for rdma (RoCE) ,
Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )

RDMA Connections between nodes:
  Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     0   Y  RTS   (Y)256  478202 (0  ) 12728       67024       8864789(0  ) 22776     4643      0             0
  Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0
    hostname                            idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT
    ogpfs2-hs.local                     1   Y  RTS   (Y)256  477659 (0  ) 12489       67034       8864773(0  ) 22794     4639      0             0
[root at ogpfs1 ras]#


You mentioned that it might be a cpu contention :  Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)
And also the direct attached RDMA ( +DAEMON) network is also not good?

Do you think IBM would say no to check such a configuration ?


Best regards
Walter




From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 15:59
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Walter,

Thanks for the details.

The stack trace below captures the lease thread in the middle of sending the ?lease? RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace ?does not show? whether there was anything blocking the thread prior to the point where the RPCs are sent.

At a first glance:

2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)

I believe nivcsw: 10 means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there is some CPU contention going on.

Could you open a case to get debug data collected?  If the problem can be recreated, I think we?ll need a recreate of the problem with traces enabled.

Thanks,

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Wednesday, February 15, 2023 at 4:23 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! This is a ?full? sequence in mmfs.?log.?latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19:?43:?51.?474+0100: [N] Disk lease period expired 0.?030 seconds ago in cluster

Hi!
This is a ?full? sequence in mmfs.log.latest
Fortunately this was also the last event until now (yesterday evening)

Maybe you can have a look?
2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_19:44:07.430+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)
2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):
2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi
gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0
2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u
nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0
2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0
2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig
uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0
2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,
TscReply*), void*, void**, int, ...) + 0x1DA at ??:0
2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0
2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0
2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0
2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0
2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0
2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0
2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0
2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.

Thank you very much!
Best regards
Walter

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Felipe Knop
Sent: Mittwoch, 15. Februar 2023 00:06
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

All,

These messages like

[W] ------------------[GPFS Critical Thread Watchdog]------------------

indicate that a ?critical thread?, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.

Do you have other samples of the message, with a more complete stack trace?   Or was the instance below the only one?

  Felipe

----
Felipe Knop        knop at us.ibm.com<mailto:knop at us.ibm.com>
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Walter Sklenka <Walter.Sklenka at EDV-Design.at<mailto:Walter.Sklenka at EDV-Design.at>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: Tuesday, February 14, 2023 at 10:49 AM
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded

Hi! I started with 5.?1.?6.?0 and now am at [root@?ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5.?1.?6.?1 ". the messages started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@?gpfsug.?org> On

Hi!
I started with 5.1.6.0 and now am at [root at ogpfs1 ~]# mmfsadm dump version
Dump level: verbose
Build branch "5.1.6.1 ".

the messages started  from the beginning



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Christian Vieser
Sent: Dienstag, 14. Februar 2023 15:34
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded


What version of Spectrum Scale is running there? Do these errors appear since your last version update?
Am 14.02.23 um 14:09 schrieb Walter Sklenka:
Dear Collegues!
May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?
Is this a ?local node? Problem or a network problem ?
I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load



Following is an excerpt from mmfs.log.latest

2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------
2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds
2023-02-14_12:06:53.600+0100: [W]  counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)
2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):
2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0
2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0
2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive
2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0
2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.
2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.
2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y
2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y
Mit freundlichen Gr??en
Walter Sklenka
Technical Consultant



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230309/e8e9c766/attachment.htm>

From robert.horton at icr.ac.uk  Thu Mar  9 15:44:39 2023
From: robert.horton at icr.ac.uk (Robert Horton)
Date: Thu, 9 Mar 2023 15:44:39 +0000
Subject: [gpfsug-discuss] mmbackup vs SOBAR
Message-ID: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>

Hi Folks,

I'm setting up a filesystem for "archive" data which will be aggressively tiered to tape using the Spectrum Protect (or whatever it's called today) Space Management. I would like to have two copies on tape for a) reading back the data on demand b) recovering accidentally deleted files etc c) disaster recovery of the whole filesystem if necessary.

My understanding is:

  1.  Backup and Migration are completely separate things to Spectrum Protect. You can't "restore" from a migrated file nor do a DMAPI read from a backup.
  2.  A SOBAR backup would enable the namespace to be restored if the filesystem were lost but needs all files to be (pre-)migrated and needs the filesystem blocksize etc to match.
  3.  A SOBAR backup isn't much help for restoring individual (deleted) files. There is a dsmmigundelete utility that restores individual stubs but doesn't restore directories etc so you really want a separate backup.

My thinking is to do backups to one (non-replicated) tape pool and migrate to another and run mmimgbackup regularly. I'd then have a path to do a full restore if either set of tapes were lost although it seems rather messy and it's a bit of a pain that SP needs to read everything twice.

So... have I understood that correctly and does anyone have any better / alternative suggestions?

Thanks,
Rob

Robert Horton | Scientific Computing Infrastructure Lead
The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
T +44 (0) 20 7153 5350 | E robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk> | W www.icr.ac.uk<http://www.icr.ac.uk/> | Twitter @ICR_London<https://twitter.com/ICR_London>
Facebook www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch>
Making the discoveries that defeat cancer
[ICR Logo]<http://www.icr.ac.uk/>


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230309/399b0029/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3162 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230309/399b0029/attachment.gif>

From p.childs at qmul.ac.uk  Thu Mar  9 16:07:43 2023
From: p.childs at qmul.ac.uk (Peter Childs)
Date: Thu, 9 Mar 2023 16:07:43 +0000
Subject: [gpfsug-discuss] [EXTERNAL]  mmbackup vs SOBAR
In-Reply-To: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
References: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
Message-ID: <DU0PR07MB85133C76E540B16DC4D2D59DA4B59@DU0PR07MB8513.eurprd07.prod.outlook.com>

I've been told "you really should be using SOBAR" a few times, but never really understood how to do so and the steps involved. I feel sure it should have some kind of white paper, so far I've been thinking to setup some kind of test system, but get a little lost on where to start (and lack of time).

We currently use mmbackup to x2 servers using `--tsm-servers TSMServer1, TSMServer2` to have two independant backups and this works nicely until you lose a tape when restoring that tape is going to be a nightmare. (read rebuild the whole shadow database)

We started with a copy pool until we filled our tape library up, and then swapped to Protect replication until we found this really did not work very well (really slow and missing files), and IBM surgested we use mmbackup with 2 servers and have two independ backups, which is working very well for us now.

I think if I was going to implement SOBAR I'd want to run mmbackup as well as SOBAR will not give you point in time recovery or partial recovery and is really only a disarster solution. I'd also probably want 3 copies on tape, 1 in SOBAR, and 2x via mmbackup via two backups or via a copy pool

I'm currently thinking to play with HSM and SOBAR on a test system, but have not started yet...... Maybe a talk at the next UG would be helpful on backups, I'm not sure if I want to do one, or if we can find an "expert"

Peter Childs



________________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Robert Horton <robert.horton at icr.ac.uk>
Sent: Thursday, March 9, 2023 3:44 PM
To: gpfsug-discuss at gpfsug.org
Subject: [EXTERNAL] [gpfsug-discuss] mmbackup vs SOBAR

CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe.

Hi Folks,

I?m setting up a filesystem for ?archive? data which will be aggressively tiered to tape using the Spectrum Protect (or whatever it?s called today) Space Management. I would like to have two copies on tape for a) reading back the data on demand b) recovering accidentally deleted files etc c) disaster recovery of the whole filesystem if necessary.

My understanding is:

  1.  Backup and Migration are completely separate things to Spectrum Protect. You can?t ?restore? from a migrated file nor do a DMAPI read from a backup.
  2.  A SOBAR backup would enable the namespace to be restored if the filesystem were lost but needs all files to be (pre-)migrated and needs the filesystem blocksize etc to match.
  3.  A SOBAR backup isn?t much help for restoring individual (deleted) files. There is a dsmmigundelete utility that restores individual stubs but doesn?t restore directories etc so you really want a separate backup.

My thinking is to do backups to one (non-replicated) tape pool and migrate to another and run mmimgbackup regularly. I?d then have a path to do a full restore if either set of tapes were lost although it seems rather messy and it?s a bit of a pain that SP needs to read everything twice.

So? have I understood that correctly and does anyone have any better / alternative suggestions?

Thanks,
Rob

Robert Horton | Scientific Computing Infrastructure Lead
The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
T +44 (0) 20 7153 5350 | E robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk> | W www.icr.ac.uk<http://www.icr.ac.uk/> | Twitter @ICR_London<https://twitter.com/ICR_London>
Facebook www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch>
Making the discoveries that defeat cancer
[ICR Logo]<http://www.icr.ac.uk/>


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.



From lgayne at us.ibm.com  Thu Mar  9 16:17:34 2023
From: lgayne at us.ibm.com (Lyle Gayne)
Date: Thu, 9 Mar 2023 16:17:34 +0000
Subject: [gpfsug-discuss] mmbackup vs SOBAR
In-Reply-To: <DU0PR07MB85133C76E540B16DC4D2D59DA4B59@DU0PR07MB8513.eurprd07.prod.outlook.com>
References: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <DU0PR07MB85133C76E540B16DC4D2D59DA4B59@DU0PR07MB8513.eurprd07.prod.outlook.com>
Message-ID: <DM5PR15MB183430F308402EA5DA093F69E3B59@DM5PR15MB1834.namprd15.prod.outlook.com>

Here's a reference for SOBaR use:

https://www.ibm.com/docs/en/spectrum-scale/4.2.1?topic=sobar-backup-procedure

The advantage of SOBaR is that it can be used to backup the configuration and metadata, after which data restore can proceed selectively (based on need), and the data already restored can then be accessed while further data restore hasn't been completed.


________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Peter Childs <p.childs at qmul.ac.uk>
Sent: Thursday, March 9, 2023 11:07 AM
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] [EXTERNAL] mmbackup vs SOBAR

I've been told "you really should be using SOBAR" a few times, but never really understood how to do so and the steps involved. I feel sure it should have some kind of white paper, so far I've been thinking to setup some kind of test system, but get a little lost on where to start (and lack of time).

We currently use mmbackup to x2 servers using `--tsm-servers TSMServer1, TSMServer2` to have two independant backups and this works nicely until you lose a tape when restoring that tape is going to be a nightmare. (read rebuild the whole shadow database)

We started with a copy pool until we filled our tape library up, and then swapped to Protect replication until we found this really did not work very well (really slow and missing files), and IBM surgested we use mmbackup with 2 servers and have two independ backups, which is working very well for us now.

I think if I was going to implement SOBAR I'd want to run mmbackup as well as SOBAR will not give you point in time recovery or partial recovery and is really only a disarster solution. I'd also probably want 3 copies on tape, 1 in SOBAR, and 2x via mmbackup via two backups or via a copy pool

I'm currently thinking to play with HSM and SOBAR on a test system, but have not started yet...... Maybe a talk at the next UG would be helpful on backups, I'm not sure if I want to do one, or if we can find an "expert"

Peter Childs



________________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Robert Horton <robert.horton at icr.ac.uk>
Sent: Thursday, March 9, 2023 3:44 PM
To: gpfsug-discuss at gpfsug.org
Subject: [EXTERNAL] [gpfsug-discuss] mmbackup vs SOBAR

CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe.

Hi Folks,

I?m setting up a filesystem for ?archive? data which will be aggressively tiered to tape using the Spectrum Protect (or whatever it?s called today) Space Management. I would like to have two copies on tape for a) reading back the data on demand b) recovering accidentally deleted files etc c) disaster recovery of the whole filesystem if necessary.

My understanding is:

  1.  Backup and Migration are completely separate things to Spectrum Protect. You can?t ?restore? from a migrated file nor do a DMAPI read from a backup.
  2.  A SOBAR backup would enable the namespace to be restored if the filesystem were lost but needs all files to be (pre-)migrated and needs the filesystem blocksize etc to match.
  3.  A SOBAR backup isn?t much help for restoring individual (deleted) files. There is a dsmmigundelete utility that restores individual stubs but doesn?t restore directories etc so you really want a separate backup.

My thinking is to do backups to one (non-replicated) tape pool and migrate to another and run mmimgbackup regularly. I?d then have a path to do a full restore if either set of tapes were lost although it seems rather messy and it?s a bit of a pain that SP needs to read everything twice.

So? have I understood that correctly and does anyone have any better / alternative suggestions?

Thanks,
Rob

Robert Horton | Scientific Computing Infrastructure Lead
The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
T +44 (0) 20 7153 5350 | E robert.horton at icr.ac.uk<mailto:robert.horton at icr.ac.uk> | W www.icr.ac.uk<http://www.icr.ac.uk/ > | Twitter @ICR_London<https://twitter.com/ICR_London >
Facebook www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch<http://www.facebook.com/theinstituteofcancerresearch> >
Making the discoveries that defeat cancer
[ICR Logo]<http://www.icr.ac.uk/ >


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230309/5c7f6961/attachment.htm>

From st.graf at fz-juelich.de  Thu Mar  9 18:22:35 2023
From: st.graf at fz-juelich.de (Stephan Graf)
Date: Thu, 9 Mar 2023 19:22:35 +0100
Subject: [gpfsug-discuss] mmbackup vs SOBAR
In-Reply-To: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
References: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
Message-ID: <f096355f-214c-6931-5c7e-5e5a164e511c@fz-juelich.de>

Hi Rob,

we in J?lich have a long history and experience using GPFS & HSM.
We don't use SOBAR (which is for disaster recovery) but mmbackup (single 
file restore).
In principle it is working fine. But there are two problems.

The first one is if the user rename the file/directory. If renamed, the 
file(s) must be backed up again and will trigger a recall(inline tape 
copy). The same happens if ACLs are modified on files.
This is working, but if we are scaling up (1K ore more files are 
effected, we are storing >40PB in these file systems), the inline tape 
copy for backup is not tape optimized and will take a long time.

If you want more details, you can contact me directly.

Stephan

On 3/9/23 16:44, Robert Horton wrote:
> Hi Folks,
> 
> I?m setting up a filesystem for ?archive? data which will be 
> aggressively tiered to tape using the Spectrum Protect (or whatever it?s 
> called today) Space Management. I would like to have two copies on tape 
> for a) reading back the data on demand b) recovering accidentally 
> deleted files etc c) disaster recovery of the whole filesystem if necessary.
> 
> My understanding is:
> 
>  1. Backup and Migration are completely separate things to Spectrum
>     Protect. You can?t ?restore? from a migrated file nor do a DMAPI
>     read from a backup.
>  2. A SOBAR backup would enable the namespace to be restored if the
>     filesystem were lost but needs all files to be (pre-)migrated and
>     needs the filesystem blocksize etc to match.
>  3. A SOBAR backup isn?t much help for restoring individual (deleted)
>     files. There is a dsmmigundelete utility that restores individual
>     stubs but doesn?t restore directories etc so you really want a
>     separate backup.
> 
> My thinking is to do backups to one (non-replicated) tape pool and 
> migrate to another and run mmimgbackup regularly. I?d then have a path 
> to do a full restore if either set of tapes were lost although it seems 
> rather messy and it?s a bit of a pain that SP needs to read everything 
> twice.
> 
> So? have I understood that correctly and does anyone have any better / 
> alternative suggestions?
> 
> Thanks,
> 
> Rob
> 
> *Robert Horton*| Scientific Computing Infrastructure Lead
> 
> The Institute of Cancer Research | 237 Fulham Road, London, SW3 6JB
> 
> *T*+44 (0) 20 7153 5350 | *E* robert.horton at icr.ac.uk 
> <mailto:robert.horton at icr.ac.uk> | *W* www.icr.ac.uk 
> <http://www.icr.ac.uk/> | *Twitter* @ICR_London 
> <https://twitter.com/ICR_London>
> 
> *Facebook*www.facebook.com/theinstituteofcancerresearch 
> <http://www.facebook.com/theinstituteofcancerresearch>
> 
> *Making the discoveries that defeat cancer*
> 
> ICR Logo <http://www.icr.ac.uk/>
> 
> The Institute of Cancer Research: Royal Cancer Hospital, a charitable 
> Company Limited by Guarantee, Registered in England under Company No. 
> 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.
> 
> This e-mail message is confidential and for use by the addressee only. 
> If the message is received by anyone other than the addressee, please 
> return the message to the sender by replying to it and then delete the 
> message from your computer and network.
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-- 
Stephan Graf
Juelich Supercomputing Centre

Phone:  +49-2461-61-6578
Fax:    +49-2461-61-6656
E-mail: st.graf at fz-juelich.de
WWW:    http://www.fz-juelich.de/jsc/
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
Prof. Dr. Frauke Melchior
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5938 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230309/b786e0b5/attachment.bin>

From jonathan.buzzard at strath.ac.uk  Mon Mar 13 08:13:33 2023
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Mon, 13 Mar 2023 08:13:33 +0000
Subject: [gpfsug-discuss] mmbackup vs SOBAR
In-Reply-To: <f096355f-214c-6931-5c7e-5e5a164e511c@fz-juelich.de>
References: <LO4P265MB4050C9CC3B577FBFA7B1CAFDACB59@LO4P265MB4050.GBRP265.PROD.OUTLOOK.COM>
 <f096355f-214c-6931-5c7e-5e5a164e511c@fz-juelich.de>
Message-ID: <37175a8e-444b-8b35-f187-3ec2b8eb0f1a@strath.ac.uk>

On 09/03/2023 18:22, Stephan Graf wrote:

> Hi Rob,
> 
> we in J?lich have a long history and experience using GPFS & HSM.
> We don't use SOBAR (which is for disaster recovery) but mmbackup (single
> file restore).
> In principle it is working fine. But there are two problems.
> 
> The first one is if the user rename the file/directory. If renamed, the
> file(s) must be backed up again and will trigger a recall(inline tape
> copy). The same happens if ACLs are modified on files.
> This is working, but if we are scaling up (1K ore more files are
> effected, we are storing >40PB in these file systems), the inline tape
> copy for backup is not tape optimized and will take a long time.
> 

In the context of an "archive" file system as per Robert's original post 
I think that it reasonable and even a good idea to scan the file system 
and have all files over a couple of weeks old set immutable. That stops 
users renaming/moving files and triggering mass recalls. It's supposed 
to be an archive after all so you should not be doing that anyway.

I also think that in context of an "archive" files system a reasonable 
approach is also to set very low quotas on the number of files a user is 
allowed. The idea is to "encourage" them to put the archived data in 
things like a zip or tar file. Again it's an "archive" not a general 
purpose file system.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From dlmcnabb at gmail.com  Thu Mar 23 04:52:28 2023
From: dlmcnabb at gmail.com (Daniel McNabb)
Date: Wed, 22 Mar 2023 21:52:28 -0700
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <DM6PR15MB3450F56115EF023F9AA78162FD879@DM6PR15MB3450.namprd15.prod.outlook.com>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM6PR15MB3450F56115EF023F9AA78162FD879@DM6PR15MB3450.namprd15.prod.outlook.com>
Message-ID: <CANvmN+0Bm+sY3b6HwoYE_mdSYxdxYXGkkJ2drN3MgNhTUmVcAQ@mail.gmail.com>

The earliest name was Shark because sharks cannot stop swimming. It
developed wide striping across many disks and became TigerShark (hey,
striped), .

When the first SP systems came out it became MMFS (MultiMedia FS) and went
into a few trials with Tokyo Island and one of the baby Bells to stream
many videos.

VideoCharger was a quick attempt to make some hardware for things like
small hotels to stream digital content to rooms, but I think it got killed
right before launch.

After those attempts it moved from being just a research project to being a
real product and got named GPFS (General Parallel FS). This launched us
into the High Performance Computing sphere for some of the largest SP
systems (on AIX) going into some national labs and other weather and
research systems in 1998. This is the first time it has a full Posix
interface instead of a specialized video streaming interface. It also
became popular with things like banking systems for all its replication
possibilities. It supported Oracle for many years until they worked out
their own redundancy capabilities.  The ESS hardware systems were sold as
hardware embodiments using the GPFS software. The addition of a Linux
implementation expanded the systems GPFS could run on. There were many
innovations added to GPFS over the years for handling AFM replication, NFS
exporting, and distributed striping for fast reconstruction of lost disks.
(There are several other things I cannot remember at the moment.)

After many years of having no marketing division (i.e only word of mouth),
IBM storage decided to get into the game and marketed it as IBM Spectrum
Scale which has just been changed to IBM Storage Scale.

Cheers all,
Daniel McNabb.


On Wed, Mar 22, 2023 at 6:19?PM Glen Corneau <gcorneau at us.ibm.com> wrote:

> I think that product was Videocharger, but it used the multimedia file
> system striped across multiple SCSI-1 disks for streaming video performance
> IIRC!
>
> Sure we're not talking about Tigershark?  ?
>
> (and can't forget the cousin, PIOFS or Parallel I/O File System)
> ---
> Glen Corneau
> Senior, Power Partner Technical Specialist (PTS-P)
> IBM Technology, North America
> Email: gcorneau at us.ibm.com
> Cell: 512-420-7988
>
> ------------------------------
> *From:* gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of
> Ryan Novosielski <novosirj at rutgers.edu>
> *Sent:* Wednesday, March 22, 2023 19:12
> *To:* gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> *Subject:* [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively
> connected to a given NFS export (CES)
>
> The product formerly known as MMFS? -- #BlackLivesMatter ____ || \\UTGERS,
> |---------------------------*O*--------------------------- ||_// the State
> | Ryan Novosielski - novosirj@ rutgers. edu || \\ University | Sr.
> Technologist - 973/972. 0922
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> The product formerly known as MMFS?
>
> --
> #BlackLivesMatter
> ____
> || \\UTGERS,    |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB
> A555B, Newark
>      `'
>
> On Mar 22, 2023, at 17:30, Alec <anacreo at gmail.com> wrote:
>
> Thanks for correction.. been using GPFS so long I forgot my basic NFS
> command.
>
> Or that its now IBM Storage Scale and no longer Spectrum Scale or GPFS...
>
> As a note that info is a little unreliable, but if you take a daily
> snapshots and throw it all together it should give you something.
>
> Alternatively you can have most nfsd daemons log mounts and then scan the
> logs for a more reliable method.
>
> Alec
>
> On Wed, Mar 22, 2023, 2:00 PM Markus Stoeber <
> M.Stoeber at rz.uni-frankfurt.de> wrote:
>
> Am 22.03.2023 um 21:45 schrieb Beckman, Daniel D:
>
> Hi,
>
> showmount -a should do the trick, however the manpage notes that:
>
> -a or --all
>               List both the client hostname or IP address and mounted
> directory in host:dir format. This info should not be considered reliable.
> See the notes on rmtab in rpc.mountd(8).
>
> showmount -d could also be an option:
>
> -d or --directories
>               List only the directories mounted by some client.
>
> Best regards,
>
> Markus
>
> Thanks, but that shows the export list: a list of shares and the hosts /
> networks that have access. It does not show which of those clients are
> currently connected to a given share, as in have it mounted.
>
>
> *From:* gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org>
> <gpfsug-discuss-bounces at gpfsug.org> *On Behalf Of *Alec
> *Sent:* Wednesday, March 22, 2023 4:23 PM
> *To:* gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> <gpfsug-discuss at gpfsug.org>
> *Subject:* Re: [gpfsug-discuss] How to: Show clients actively connected
> to a given NFS export (CES)
>
>
> *CAUTION:* This email message has been received from an external source.
> Please use caution when opening attachments, or clicking on links.
>
> showmount -e nfsserver
>
>
> Normal way to see that for an nfs server.
>
>
> On Wed, Mar 22, 2023, 1:13 PM Beckman, Daniel D <dbec at loc.gov> wrote:
>
> This is probably a dumb question, but I could not find it in the
> documentation. We have certain NFS exports that we suspect are no longer
> used or needed but before removing we?d like to first check if any clients
> are currently mounting them.  (Not just what has access to those exports.)
> Are there commands that will show this?
>
>
> Thanks,
>
> Daniel
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
> --
> Markus Stoeber
> Systemmanagement AIX, Linux / Storagemanagement / Plattformintegration
> Hochschulrechenzentrum der Johann Wolfgang Goethe-Universitaet
> Abt. Basisdienste
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230322/4275bf3c/attachment.htm>

From lgayne at us.ibm.com  Thu Mar 23 12:29:22 2023
From: lgayne at us.ibm.com (Lyle Gayne)
Date: Thu, 23 Mar 2023 12:29:22 +0000
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
Message-ID: <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>

Yes, MMFS was the internal name for an offering that I believe was to be called VideoCharger/MediaStreamer, and was an ahead of its time/market offering.  When MMFS was cancelled, all four of its developers of the time joined the GPFS effort, and we had collaborated with them to sequence changes in the same code base for a few years by that point and knew them all well.
________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Ryan Novosielski <novosirj at rutgers.edu>
Sent: Wednesday, March 22, 2023 8:12 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

The product formerly known as MMFS? -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj@?rutgers.?edu || \\ University | Sr. Technologist - 973/972.?0922
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
The product formerly known as MMFS?

--
#BlackLivesMatter
____
|| \\UTGERS,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'

On Mar 22, 2023, at 17:30, Alec <anacreo at gmail.com> wrote:

Thanks for correction.. been using GPFS so long I forgot my basic NFS command.

Or that its now IBM Storage Scale and no longer Spectrum Scale or GPFS...

As a note that info is a little unreliable, but if you take a daily snapshots and throw it all together it should give you something.

Alternatively you can have most nfsd daemons log mounts and then scan the logs for a more reliable method.

Alec

On Wed, Mar 22, 2023, 2:00 PM Markus Stoeber <M.Stoeber at rz.uni-frankfurt.de<mailto:M.Stoeber at rz.uni-frankfurt.de>> wrote:
Am 22.03.2023 um 21:45 schrieb Beckman, Daniel D:

Hi,

showmount -a should do the trick, however the manpage notes that:

-a or --all
              List both the client hostname or IP address and mounted directory in host:dir format. This info should not be considered reliable. See the notes on rmtab in rpc.mountd(8).

showmount -d could also be an option:

-d or --directories
              List only the directories mounted by some client.

Best regards,

Markus


Thanks, but that shows the export list: a list of shares and the hosts / networks that have access. It does not show which of those clients are currently connected to a given share, as in have it mounted.



From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org><mailto:gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Alec
Sent: Wednesday, March 22, 2023 4:23 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org><mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)



CAUTION: This email message has been received from an external source. Please use caution when opening attachments, or clicking on links.

showmount -e nfsserver



Normal way to see that for an nfs server.



On Wed, Mar 22, 2023, 1:13 PM Beckman, Daniel D <dbec at loc.gov<mailto:dbec at loc.gov>> wrote:

This is probably a dumb question, but I could not find it in the documentation. We have certain NFS exports that we suspect are no longer used or needed but before removing we?d like to first check if any clients are currently mounting them.  (Not just what has access to those exports.) Are there commands that will show this?



Thanks,

Daniel




_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>



--
Markus Stoeber
Systemmanagement AIX, Linux / Storagemanagement / Plattformintegration
Hochschulrechenzentrum der Johann Wolfgang Goethe-Universitaet
Abt. Basisdienste

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/8809d79a/attachment.htm>

From daniel.kidger at hpe.com  Thu Mar 23 13:20:06 2023
From: daniel.kidger at hpe.com (Kidger, Daniel)
Date: Thu, 23 Mar 2023 13:20:06 +0000
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
Message-ID: <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>

It is simple really?.

The product is called ?Storage Scale?.

It used to be called ?Spectrum Scale? but that being abbreviated to ?SS? was a problem, hence the recent change. :-)
As an appliance it is called ESS = ?Elastic Storage System? since before being called Spectrum Scale, it was briefly called ?Elastic Storage?
All the software RPMs though still start with ?gpfs? and that is what every techie person still calls the software.
The software unpacks to /usr/lpp/mmfs/bin since the product was once the ?MultiMedia FileSystem?
The ?lpp? comes from AIX. It stands for ?Licensed Programming Product ?. This term is lost on everyone who comes from a Linux background.
Likewise no one from the Linux world know what ?adm? is, as in /var/adm/ras  (or ras?)

There are c. 333 ?mm? commands like say ?mmdeldisk?. None are actual binaries ? all are Korn shell scripts (Korn was a shell language that Bash eventually replaced)
*(actually a few mm commands are written in Python not Korn.)
Most mm command call underlying binaries which all start with ?ts?  eg ?tsdeldisk?. This is, as we all know is become the product was known as TigerShark but no one could be bothered renaming stuff when ?TigerShark? as a name was dropped 25 years ago.

Simple?




Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kidger at hpe.com<mailto:daniel.kidger at hpe.com>

+44 (0)7818 522266

hpe.com<http://www.hpe.com/>


[cid:image001.png at 01D95D88.0723C360]

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Lyle Gayne
Sent: 23 March 2023 12:29
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

Yes, MMFS was the internal name for an offering that I believe was to be called VideoCharger/MediaStreamer, and was an ahead of its time/market offering.  When MMFS was cancelled, all four of its developers of the time joined the GPFS effort, and we had collaborated with them to sequence changes in the same code base for a few years by that point and knew them all well.
________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Ryan Novosielski <novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>>
Sent: Wednesday, March 22, 2023 8:12 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

The product formerly known as MMFS? -- #BlackLivesMatter ____ || \\UTGERS<file://UTGERS>, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj@?rutgers.?edu || \\ University | Sr. Technologist - 973/972.?0922

The product formerly known as MMFS?

--
#BlackLivesMatter
____
|| \\UTGERS<file://UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'


On Mar 22, 2023, at 17:30, Alec <anacreo at gmail.com<mailto:anacreo at gmail.com>> wrote:

Thanks for correction.. been using GPFS so long I forgot my basic NFS command.

Or that its now IBM Storage Scale and no longer Spectrum Scale or GPFS...

As a note that info is a little unreliable, but if you take a daily snapshots and throw it all together it should give you something.

Alternatively you can have most nfsd daemons log mounts and then scan the logs for a more reliable method.

Alec

On Wed, Mar 22, 2023, 2:00 PM Markus Stoeber <M.Stoeber at rz.uni-frankfurt.de<mailto:M.Stoeber at rz.uni-frankfurt.de>> wrote:
Am 22.03.2023 um 21:45 schrieb Beckman, Daniel D:

Hi,

showmount -a should do the trick, however the manpage notes that:

-a or --all
              List both the client hostname or IP address and mounted directory in host:dir format. This info should not be considered reliable. See the notes on rmtab in rpc.mountd(8).

showmount -d could also be an option:

-d or --directories
              List only the directories mounted by some client.

Best regards,

Markus


Thanks, but that shows the export list: a list of shares and the hosts / networks that have access. It does not show which of those clients are currently connected to a given share, as in have it mounted.


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org><mailto:gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Alec
Sent: Wednesday, March 22, 2023 4:23 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org><mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)


CAUTION: This email message has been received from an external source. Please use caution when opening attachments, or clicking on links.

showmount -e nfsserver


Normal way to see that for an nfs server.


On Wed, Mar 22, 2023, 1:13 PM Beckman, Daniel D <dbec at loc.gov<mailto:dbec at loc.gov>> wrote:

This is probably a dumb question, but I could not find it in the documentation. We have certain NFS exports that we suspect are no longer used or needed but before removing we?d like to first check if any clients are currently mounting them.  (Not just what has access to those exports.) Are there commands that will show this?


Thanks,

Daniel



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>



_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at gpfsug.org<http://gpfsug.org/>

http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>



--

Markus Stoeber

Systemmanagement AIX, Linux / Storagemanagement / Plattformintegration

Hochschulrechenzentrum der Johann Wolfgang Goethe-Universitaet

Abt. Basisdienste
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/685e71dd/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2541 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/685e71dd/attachment.png>

From novosirj at rutgers.edu  Thu Mar 23 13:30:27 2023
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 23 Mar 2023 13:30:27 +0000
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
 <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
Message-ID: <1A5AB110-3A98-4D9F-8168-29E979009842@rutgers.edu>

I had basically /just/ started calling it Spectrum Scale. ? So I guess I?m at least partially responsible for the name change.

Sent from my iPhone

On Mar 23, 2023, at 09:25, Kidger, Daniel <daniel.kidger at hpe.com> wrote:

?
It is simple really?.

The product is called ?Storage Scale?.

It used to be called ?Spectrum Scale? but that being abbreviated to ?SS? was a problem, hence the recent change. :-)
As an appliance it is called ESS = ?Elastic Storage System? since before being called Spectrum Scale, it was briefly called ?Elastic Storage?
All the software RPMs though still start with ?gpfs? and that is what every techie person still calls the software.
The software unpacks to /usr/lpp/mmfs/bin since the product was once the ?MultiMedia FileSystem?
The ?lpp? comes from AIX. It stands for ?Licensed Programming Product ?. This term is lost on everyone who comes from a Linux background.
Likewise no one from the Linux world know what ?adm? is, as in /var/adm/ras  (or ras?)

There are c. 333 ?mm? commands like say ?mmdeldisk?. None are actual binaries ? all are Korn shell scripts (Korn was a shell language that Bash eventually replaced)
*(actually a few mm commands are written in Python not Korn.)
Most mm command call underlying binaries which all start with ?ts?  eg ?tsdeldisk?. This is, as we all know is become the product was known as TigerShark but no one could be bothered renaming stuff when ?TigerShark? as a name was dropped 25 years ago.

Simple?




Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kidger at hpe.com<mailto:daniel.kidger at hpe.com>

+44 (0)7818 522266

hpe.com<http://www.hpe.com/>


<image001.png>

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Lyle Gayne
Sent: 23 March 2023 12:29
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

Yes, MMFS was the internal name for an offering that I believe was to be called VideoCharger/MediaStreamer, and was an ahead of its time/market offering.  When MMFS was cancelled, all four of its developers of the time joined the GPFS effort, and we had collaborated with them to sequence changes in the same code base for a few years by that point and knew them all well.
________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> on behalf of Ryan Novosielski <novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>>
Sent: Wednesday, March 22, 2023 8:12 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

The product formerly known as MMFS? -- #BlackLivesMatter ____ || \\UTGERS<file://UTGERS>, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj@?rutgers.?edu || \\ University | Sr. Technologist - 973/972.?0922
The product formerly known as MMFS?

--
#BlackLivesMatter
____
|| \\UTGERS<file://UTGERS>,    |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'


On Mar 22, 2023, at 17:30, Alec <anacreo at gmail.com<mailto:anacreo at gmail.com>> wrote:

Thanks for correction.. been using GPFS so long I forgot my basic NFS command.

Or that its now IBM Storage Scale and no longer Spectrum Scale or GPFS...

As a note that info is a little unreliable, but if you take a daily snapshots and throw it all together it should give you something.

Alternatively you can have most nfsd daemons log mounts and then scan the logs for a more reliable method.

Alec

On Wed, Mar 22, 2023, 2:00 PM Markus Stoeber <M.Stoeber at rz.uni-frankfurt.de<mailto:M.Stoeber at rz.uni-frankfurt.de>> wrote:
Am 22.03.2023 um 21:45 schrieb Beckman, Daniel D:

Hi,

showmount -a should do the trick, however the manpage notes that:

-a or --all
              List both the client hostname or IP address and mounted directory in host:dir format. This info should not be considered reliable. See the notes on rmtab in rpc.mountd(8).

showmount -d could also be an option:

-d or --directories
              List only the directories mounted by some client.

Best regards,

Markus


Thanks, but that shows the export list: a list of shares and the hosts / networks that have access. It does not show which of those clients are currently connected to a given share, as in have it mounted.


From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org><mailto:gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Alec
Sent: Wednesday, March 22, 2023 4:23 PM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org><mailto:gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)


CAUTION: This email message has been received from an external source. Please use caution when opening attachments, or clicking on links.

showmount -e nfsserver


Normal way to see that for an nfs server.


On Wed, Mar 22, 2023, 1:13 PM Beckman, Daniel D <dbec at loc.gov<mailto:dbec at loc.gov>> wrote:

This is probably a dumb question, but I could not find it in the documentation. We have certain NFS exports that we suspect are no longer used or needed but before removing we?d like to first check if any clients are currently mounting them.  (Not just what has access to those exports.) Are there commands that will show this?


Thanks,

Daniel



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org



_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at gpfsug.org<http://gpfsug.org/>

http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org



--

Markus Stoeber

Systemmanagement AIX, Linux / Storagemanagement / Plattformintegration

Hochschulrechenzentrum der Johann Wolfgang Goethe-Universitaet

Abt. Basisdienste
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/29c008de/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2541 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/29c008de/attachment.png>

From jonathan.buzzard at strath.ac.uk  Thu Mar 23 14:05:38 2023
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Thu, 23 Mar 2023 14:05:38 +0000
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
 <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
Message-ID: <bb2a2bd2-e54f-faa9-43c1-121b0a7606ca@strath.ac.uk>

On 23/03/2023 13:20, Kidger, Daniel wrote:

[SNIP]

> The ?lpp? comes from AIX. It stands for ?Licensed Programming Product??. 
> This term is lost on everyone who comes from a Linux background.
> 
> Likewise no one from the Linux world know what ?adm? is, as in 
> /var/adm/ras? (or ras?)

I knew what lpp is despite coming from a Linux background as I have AIX 
exposure :-)

I believe that adm is just short for administration but don't quote me 
on that.

However "ras" is I believe short for "reliability, accessibility, and 
serviceability", yeah whatever.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG




From luis.bolinches at fi.ibm.com  Thu Mar 23 14:30:33 2023
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Thu, 23 Mar 2023 14:30:33 +0000
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <bb2a2bd2-e54f-faa9-43c1-121b0a7606ca@strath.ac.uk>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
 <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
 <bb2a2bd2-e54f-faa9-43c1-121b0a7606ca@strath.ac.uk>
Message-ID: <SA1PR15MB506187AD18E2CE3EBC25FDC9A4879@SA1PR15MB5061.namprd15.prod.outlook.com>

Well

I guess I add to the retro fest going on, which is the best mails I will get this week, so thanks for that.

There is a command 'spectrumscale', the toolkit, so we have that one covered too. Soon maybe a 'storagescale' one

--
Yst?v?llisin terveisin/Regards/Saludos/Salutations/Salutacions
Luis Bolinches
Executive IT Specialist
IBM Storage Scale development
Phone: +358503112585
 	
Ab IBM Finland Oy
Laajalahdentie 23
00330 Helsinki
Uusimaa - Finland

"If you always give you will always have" --  Anonymous

https://www.credly.com/users/luis-bolinches/badges

-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Jonathan Buzzard
Sent: Thursday, 23 March 2023 16.06
To: gpfsug-discuss at gpfsug.org
Subject: [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively connected to a given NFS export (CES)

On 23/03/2023 13:20, Kidger, Daniel wrote:

[SNIP]

> The ?lpp? comes from AIX. It stands for ?Licensed Programming Product??. 
> This term is lost on everyone who comes from a Linux background.
> 
> Likewise no one from the Linux world know what ?adm? is, as in 
> /var/adm/ras? (or ras?)

I knew what lpp is despite coming from a Linux background as I have AIX exposure :-)

I believe that adm is just short for administration but don't quote me on that.

However "ras" is I believe short for "reliability, accessibility, and serviceability", yeah whatever.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org 

Unless otherwise stated above:

Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland

From anacreo at gmail.com  Thu Mar 23 16:47:48 2023
From: anacreo at gmail.com (Alec)
Date: Thu, 23 Mar 2023 09:47:48 -0700
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
References: <1f66c46d44af44d68959646812179090@loc.gov>
 <CAGhSTwjQ9GGh5YmMN8V1x6xhO3NxwZBCDB5aQ8KTtK0Lb6f+DQ@mail.gmail.com>
 <0c8e6e3032334ef6891195c13d284b56@loc.gov>
 <d2852ad3-aeec-a06d-64ab-2c03f033cd40@rz.uni-frankfurt.de>
 <CAGhSTwiLw+w-Cr=DXR62FMZg+pEwSD4YW5FdSiYanumQQGSvxA@mail.gmail.com>
 <F3AB413F-1CDE-495D-AE02-87D28A65D1F5@rutgers.edu>
 <DM5PR15MB18341CEB1388E31B2EADC559E3879@DM5PR15MB1834.namprd15.prod.outlook.com>
 <PH7PR84MB17019665EF1ABBF45D2538DDFB879@PH7PR84MB1701.NAMPRD84.PROD.OUTLOOK.COM>
Message-ID: <CAGhSTwiPn-KwWQ9eqqDrh6uR=p99SA2bpwyR3r0qwanm9O0rzg@mail.gmail.com>

Whoa whoa whoa... Easy on the fighting words...

Korn shell was a shell language eventually replaced by bash???

ksh is alive and well and I believe (ksh93) superior to bash in many ways...

On Thu, Mar 23, 2023, 6:26 AM Kidger, Daniel <daniel.kidger at hpe.com> wrote:

> It is simple really?.
>
>
>
> The product is called ?Storage Scale?.
>
>
>
> It used to be called ?Spectrum Scale? but that being abbreviated to ?SS?
> was a problem, hence the recent change. :-)
>
> As an appliance it is called ESS = ?Elastic Storage System? since before
> being called Spectrum Scale, it was briefly called ?Elastic Storage?
>
> All the software RPMs though still start with ?gpfs? and that is what
> every techie person still calls the software.
>
> The software unpacks to /usr/lpp/mmfs/bin since the product was once the
> ?MultiMedia FileSystem?
>
> The ?lpp? comes from AIX. It stands for ?Licensed Programming Product ?.
> This term is lost on everyone who comes from a Linux background.
>
> Likewise no one from the Linux world know what ?adm? is, as in
> /var/adm/ras  (or ras?)
>
>
>
> There are c. 333 ?mm? commands like say ?mmdeldisk?. None are actual
> binaries ? all are Korn shell scripts (Korn was a shell language that Bash
> eventually replaced)
>
> *(actually a few mm commands are written in Python not Korn.)
> Most mm command call underlying binaries which all start with ?ts?  eg
> ?tsdeldisk?. This is, as we all know is become the product was known as
> TigerShark but no one could be bothered renaming stuff when ?TigerShark? as
> a name was dropped 25 years ago.
>
>
>
> Simple?
>
>
>
>
>
>
>
>
>
>
> *Daniel Kidger *HPC Storage Solutions Architect, EMEA
> daniel.kidger at hpe.com
>
> +44 (0)7818 522266
>
> *hpe.com* <http://www.hpe.com/>
>
>
>
>
> *From:* gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> *On Behalf Of *Lyle
> Gayne
> *Sent:* 23 March 2023 12:29
> *To:* gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> *Subject:* Re: [gpfsug-discuss] How to: Show clients actively connected
> to a given NFS export (CES)
>
>
>
> Yes, MMFS was the internal name for an offering that I believe was to be
> called VideoCharger/MediaStreamer, and was an ahead of its time/market
> offering.  When MMFS was cancelled, all four of its developers of the time
> joined the GPFS effort, and we had collaborated with them to sequence
> changes in the same code base for a few years by that point and knew them
> all well.
> ------------------------------
>
> *From:* gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of
> Ryan Novosielski <novosirj at rutgers.edu>
> *Sent:* Wednesday, March 22, 2023 8:12 PM
> *To:* gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> *Subject:* [EXTERNAL] Re: [gpfsug-discuss] How to: Show clients actively
> connected to a given NFS export (CES)
>
>
>
> The product formerly known as MMFS? -- #BlackLivesMatter ____ || \\UTGERS,
> |---------------------------*O*--------------------------- ||_// the State
> | Ryan Novosielski - novosirj@ rutgers. edu || \\ University | Sr.
> Technologist - 973/972. 0922
>
> The product formerly known as MMFS?
>
>
>
> --
> #BlackLivesMatter
>
> ____
> || \\UTGERS,    |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB
> A555B, Newark
>      `'
>
>
>
> On Mar 22, 2023, at 17:30, Alec <anacreo at gmail.com> wrote:
>
>
>
> Thanks for correction.. been using GPFS so long I forgot my basic NFS
> command.
>
>
>
> Or that its now IBM Storage Scale and no longer Spectrum Scale or GPFS...
>
>
>
> As a note that info is a little unreliable, but if you take a daily
> snapshots and throw it all together it should give you something.
>
>
>
> Alternatively you can have most nfsd daemons log mounts and then scan the
> logs for a more reliable method.
>
>
>
> Alec
>
>
>
> On Wed, Mar 22, 2023, 2:00 PM Markus Stoeber <
> M.Stoeber at rz.uni-frankfurt.de> wrote:
>
> Am 22.03.2023 um 21:45 schrieb Beckman, Daniel D:
>
>
>
> Hi,
>
>
>
> showmount -a should do the trick, however the manpage notes that:
>
>
>
> -a or --all
>
>               List both the client hostname or IP address and mounted
> directory in host:dir format. This info should not be considered reliable.
> See the notes on rmtab in rpc.mountd(8).
>
>
>
> showmount -d could also be an option:
>
>
>
> -d or --directories
>               List only the directories mounted by some client.
>
>
>
> Best regards,
>
>
>
> Markus
>
>
>
> Thanks, but that shows the export list: a list of shares and the hosts /
> networks that have access. It does not show which of those clients are
> currently connected to a given share, as in have it mounted.
>
>
>
> *From:* gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org>
> <gpfsug-discuss-bounces at gpfsug.org> *On Behalf Of *Alec
> *Sent:* Wednesday, March 22, 2023 4:23 PM
> *To:* gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> <gpfsug-discuss at gpfsug.org>
> *Subject:* Re: [gpfsug-discuss] How to: Show clients actively connected
> to a given NFS export (CES)
>
>
>
> *CAUTION:* This email message has been received from an external source.
> Please use caution when opening attachments, or clicking on links.
>
> showmount -e nfsserver
>
>
>
> Normal way to see that for an nfs server.
>
>
>
> On Wed, Mar 22, 2023, 1:13 PM Beckman, Daniel D <dbec at loc.gov> wrote:
>
> This is probably a dumb question, but I could not find it in the
> documentation. We have certain NFS exports that we suspect are no longer
> used or needed but before removing we?d like to first check if any clients
> are currently mounting them.  (Not just what has access to those exports.)
> Are there commands that will show this?
>
>
>
> Thanks,
>
> Daniel
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
>
> _______________________________________________
>
> gpfsug-discuss mailing list
>
> gpfsug-discuss at gpfsug.org
>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
>
> --
>
> Markus Stoeber
>
> Systemmanagement AIX, Linux / Storagemanagement / Plattformintegration
>
> Hochschulrechenzentrum der Johann Wolfgang Goethe-Universitaet
>
> Abt. Basisdienste
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/79570296/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2541 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/79570296/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2541 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230323/79570296/attachment-0001.png>

From st.graf at fz-juelich.de  Fri Mar 24 07:53:43 2023
From: st.graf at fz-juelich.de (Stephan Graf)
Date: Fri, 24 Mar 2023 08:53:43 +0100
Subject: [gpfsug-discuss] How to: Show clients actively connected to a
 given NFS export (CES)
In-Reply-To: <1f66c46d44af44d68959646812179090@loc.gov>
References: <1f66c46d44af44d68959646812179090@loc.gov>
Message-ID: <5e8d1c02-b349-2302-b810-b6441915923b@fz-juelich.de>

Hi Daniel,

don't know if this will help you but may be it gives you some hints (and 
the tool has a couple of options...)
run on the CES-NFS server:

ganesha_stats list_clients

This will list the clients which are connected or were connected in the 
past.
But to be honest, I have not used it heavily, yet.

Stephan

On 3/22/23 21:08, Beckman, Daniel D wrote:
> This is probably a dumb question, but I could not find it in the 
> documentation. We have certain NFS exports that we suspect are no longer 
> used or needed but before removing we?d like to first check if any 
> clients are currently mounting them. ?(Not just what has access to those 
> exports.) Are there commands that will show this?
> 
> Thanks,
> 
> Daniel
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-- 
Stephan Graf
Juelich Supercomputing Centre

Phone:  +49-2461-61-6578
Fax:    +49-2461-61-6656
E-mail: st.graf at fz-juelich.de
WWW:    http://www.fz-juelich.de/jsc/
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
Prof. Dr. Frauke Melchior
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5938 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230324/1c3eaffa/attachment.bin>

From l.r.sudbery at bham.ac.uk  Fri Mar 24 09:29:31 2023
From: l.r.sudbery at bham.ac.uk (Luke Sudbery)
Date: Fri, 24 Mar 2023 09:29:31 +0000
Subject: [gpfsug-discuss] Mailman interface HTTP 500
In-Reply-To: <CWLP265MB4689854829881A259EF269F690819@CWLP265MB4689.GBRP265.PROD.OUTLOOK.COM>
References: <CD9AB56C-B632-4CF0-994A-04AE5FBA5815@ee.torontomu.ca>
 <000b01d95b4c$4ac42170$e04c6450$@gpfsug.org>
 <CWLP265MB4689854829881A259EF269F690819@CWLP265MB4689.GBRP265.PROD.OUTLOOK.COM>
Message-ID: <CWLP265MB46897BF7707C1690246129C890849@CWLP265MB4689.GBRP265.PROD.OUTLOOK.COM>

The archive links are working again now. But there seem to be no record of my thread "mmvdisk version/communication issues?" started Fri 17/03/2023 15:11.

This is unfortunate as IBM are asking for links to the messages! I can send them a copy of the email thread for now.

Many thanks,

Luke

-- 
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don?t work on Monday.

-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Luke Sudbery
Sent: 21 March 2023 09:40
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Mailman interface HTTP 500

Seem like one issue has been swapped for the other - I can access the mailman pages but the archive links:
        https://www.gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/
        https://www.spectrumscaleug.org/pipermail/gpfsug-discuss_gpfsug.org/

Are redirecting to a workpress 404 page.

Cheers,

Luke

-- 
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don?t work on Monday.

-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of systems at gpfsug.org
Sent: 20 March 2023 16:52
To: 'gpfsug main discussion list' <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Mailman interface HTTP 500

CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.


Hi David, GPFSUG Members,

We believe this issue has now been resolved, please let us know if you experience any further issues.

-- Systems

-----Original Message-----
From: systems at gpfsug.org <systems at gpfsug.org>
Sent: 07 March 2023 10:22
To: 'gpfsug main discussion list' <gpfsug-discuss at gpfsug.org>
Subject: RE: [gpfsug-discuss] Mailman interface HTTP 500

Hi David,

Thanks for letting us know, this has been reported on slack too.

There seems to be a bug in one of the scripts that is causing EGID and RGID errors; it's been reported to the software vendor and we're awaiting feedback.

-- systems

-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of David Magda
Sent: 06 March 2023 21:36
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] Mailman interface HTTP 500

To whomever runs the web server:

Apache (?) seems to be throwing errors when I go to Mailman web interface:

        https://gpfsug.org/mailman/listinfo/
        https://www.gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
        https://www.spectrumscaleug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

The archives are fine (since they?re probably just flat HTML files and not CGI):

        https://www.gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/
        https://www.spectrumscaleug.org/pipermail/gpfsug-discuss_gpfsug.org/

--
David Magda
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

From martin at uni-mainz.de  Thu Mar 30 17:44:18 2023
From: martin at uni-mainz.de (Christoph Martin)
Date: Thu, 30 Mar 2023 18:44:18 +0200
Subject: [gpfsug-discuss] AFM convertToPrimary
Message-ID: <436de125-ffed-3d10-bdea-1eb005923964@uni-mainz.de>

Hi all,

we want to convert two GPFS independent fileset to primaries for AFM-DR.
Most of the data from fileset1 we have already mirrored via rsync to the 
secondary fileset. fileset2 on the secondary server is still empty.

I now want to issue for both filesets something like:

mmafmctl storage1 convertToPrimary -j afmdrtest2 --afmtarget 
gpfs:///gpfs/storage2/afmdrtest2/ --inband --check-metadata

The help pages say, that you have to stop all IO on the primary fileset 
for the conversion. I think this might mean read and write IO.

The question is: When can we restart IO? After the mmafmctl command 
finishes or after the synchronisation of the filesets finishes.

The second can take days, especially for the second fileset, which is 
empty on the secondary site.

What happens, if read (or write) IO happens during the execution time of 
mmafmctl or the synchronisation?

Any ideas?

Regards
Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230330/11417e0a/attachment.sig>

From committee at io500.org  Fri Mar 31 00:48:11 2023
From: committee at io500.org (IO500 Committee)
Date: Thu, 30 Mar 2023 17:48:11 -0600
Subject: [gpfsug-discuss] Call for Submissions IO500 ISC23
Message-ID: <1ceb433e1ae046c5dcd515a69a807e26@io500.org>

Stabilization Period: Monday, April 3rd - Friday, April 14th, 2023
Submission Deadline: Tuesday, May 16st, 2023 AoE

The IO500 is now accepting and encouraging submissions for the upcoming 
12th semi-annual IO500 list, in conjunction with ISC23. Once again, we 
are also accepting submissions to the 10 Client Node Challenge to 
encourage the submission of small scale results. The new ranked lists 
will be announced at the ISC23  BoF [1]. We hope to see many new 
results.

What's New
1. Creation of Production and Research Lists - Starting with ISC'22, we 
proposed a separation of the list into separate Production and Research 
lists.  This better reflects the important distinction between storage 
systems that run in production environments and those that may use more 
experimental hardware and software configurations.  At ISC23, we will 
formally create these two lists and users will be able to submit to 
either of the two lists (and their 10 client-node counterparts).  Please 
see the requirements for each list on the IO500 rules page [3].
2. New Submission Tool - There is now a new IO500 submission tool that 
improves the overall submission experience.  Users can create accounts 
and then update and manage all of their submissions through that 
account.  As part of this new tool, we have improved the submission 
fields that describe the hardware and software of the system under test. 
For reproducibility and analysis reasons, we now made the easily 
obtainable fields mandatory - data from storage servers are for users 
often difficult to obtain, therefore, most remain optional. As a new 
system, there may be quirks, please reach out on Slack or the mailing 
list if you see any issues.  Further details will be released on the 
submission page [2].
3. Reproducibility - Every submission will now receive a reproducibility 
score based upon the provided system details and the reproducibility 
questionnaire. This score will inform the community on the amount of 
details provided in the submission and the obtainability of the storage 
system. Further, this score will be used to evaluate if a submission is 
eligible for the Production list.
4. New Phases - We are continuing to evaluate the inclusion of optional 
test phases for additional key workloads - split easy/hard find phases, 
4KB and 1MB random read/write phases, and concurrent metadata 
operations. This is called an extended run. At the moment, we collect 
the information to verify that additional phases do not significantly 
impact the results of a standard run and an extended run to facilitate 
comparisons between the existing and new benchmark phases. In a future 
release, we may include some or all of these results as part of the 
standard benchmark. The extended results are not currently included in 
the scoring of any ranked list.

Background
The benchmark suite is designed to be easy to run and the community has 
multiple active support channels to help with any questions. Please note 
that submissions of all sizes are welcome; the site has customizable 
sorting, so it is possible to submit on a small system and still get a 
very good per-client score, for example. Additionally, the list is about 
much more than just the raw rank; all submissions help the community by 
collecting and publishing a wider corpus of data. More details below.

Following the success of the Top500 in collecting and analyzing 
historical trends in supercomputer technology and evolution, the IO500 
was created in 2017, published its first list at SC17, and has grown 
continually since then. The need for such an initiative has long been 
known within High-Performance Computing; however, defining appropriate 
benchmarks has long been challenging. Despite this challenge, the 
community, after long and spirited discussion, finally reached consensus 
on a suite of benchmarks and a metric for resolving the scores into a 
single ranking.

The multi-fold goals of the benchmark suite are as follows:
1. Maximizing simplicity in running the benchmark suite
2. Encouraging optimization and documentation of tuning parameters for 
performance
3. Allowing submitters to highlight their "hero run" performance numbers
4. Forcing submitters to simultaneously report performance for 
challenging IO patterns.

Specifically, the benchmark suite includes a hero-run of both IOR and 
mdtest configured however possible to maximize performance and establish 
an upper-bound for performance. It also includes an IOR and mdtest run 
with highly prescribed parameters in an attempt to determine a lower 
performance bound. Finally, it includes a namespace search as this has 
been determined to be a highly sought-after feature in HPC storage 
systems that has historically not been well-measured. Submitters are 
encouraged to share their tuning insights for publication.

The goals of the community are also multi-fold:
1. Gather historical data for the sake of analysis and to aid 
predictions of storage futures
2. Collect tuning information to share valuable performance 
optimizations across the community
3. Encourage vendors and designers to optimize for workloads beyond 
"hero runs"
4. Establish bounded expectations for users, procurers, and 
administrators

The IO500 follows a two-staged approach. First, there will be a two-week 
stabilization period during which we encourage the community to verify 
that the benchmark runs properly on a variety of storage systems. During 
this period the benchmark may be updated based upon feedback from the 
community. The final benchmark will then be released. We expect that 
runs compliant with the rules made during the stabilization period will 
be valid as a final submission unless a significant defect is found.

10 Client Node I/O Challenge
The 10 Client Node Challenge is conducted using the regular IO500 
benchmark, however, with the rule that exactly 10 client nodes must be 
used to run the benchmark. You may use any shared storage with any 
number of servers. We will announce the results in the Production and 
Research lists as well as in separate derived lists.

Birds-of-a-Feather
Once again, we encourage you to submit [2] to join our community, and to 
attend the ISC23 BoF [1], where we will announce the new IO500 
Production and Research lists and their 10 client node counterparts. The 
current list includes results from twenty different storage system types 
and 70 institutions. We hope that the upcoming list grows even more.

[1] https://io500.org/pages/bof-isc23
[2] https://io500.org/submission
[3] https://io500.org/rules-submission

-- 
The IO500 Committee



From aspandem at in.ibm.com  Fri Mar 31 14:54:23 2023
From: aspandem at in.ibm.com (Ashish Pandey5)
Date: Fri, 31 Mar 2023 13:54:23 +0000
Subject: [gpfsug-discuss] AFM convertToPrimary
In-Reply-To: <mailman.1.1680260402.3940049.gpfsug-discuss_gpfsug.org@gpfsug.org>
References: <mailman.1.1680260402.3940049.gpfsug-discuss_gpfsug.org@gpfsug.org>
Message-ID: <DM6PR15MB35785C66A8E402B01F625E47FB8F9@DM6PR15MB3578.namprd15.prod.outlook.com>

Hi Christoph,

We can start IO once first AFM snapshot (psnap0) is created at the primary after starting changeSecondary command. We might get psnap0 creation failure with heavy workload.

This might not be needed if we have fileset level quiesce feature.




Thanks & Regards,
Ashish
________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of gpfsug-discuss-request at gpfsug.org <gpfsug-discuss-request at gpfsug.org>
Sent: Friday, March 31, 2023 4:30 PM
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 132, Issue 25

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
        gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. AFM convertToPrimary (Christoph Martin)
   2. Call for Submissions IO500 ISC23 (IO500 Committee)


----------------------------------------------------------------------

Message: 1
Date: Thu, 30 Mar 2023 18:44:18 +0200
From: Christoph Martin <martin at uni-mainz.de>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] AFM convertToPrimary
Message-ID: <436de125-ffed-3d10-bdea-1eb005923964 at uni-mainz.de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

Hi all,

we want to convert two GPFS independent fileset to primaries for AFM-DR.
Most of the data from fileset1 we have already mirrored via rsync to the
secondary fileset. fileset2 on the secondary server is still empty.

I now want to issue for both filesets something like:

mmafmctl storage1 convertToPrimary -j afmdrtest2 --afmtarget
gpfs:///gpfs/storage2/afmdrtest2/ --inband --check-metadata

The help pages say, that you have to stop all IO on the primary fileset
for the conversion. I think this might mean read and write IO.

The question is: When can we restart IO? After the mmafmctl command
finishes or after the synchronisation of the filesets finishes.

The second can take days, especially for the second fileset, which is
empty on the secondary site.

What happens, if read (or write) IO happens during the execution time of
mmafmctl or the synchronisation?

Any ideas?

Regards
Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230330/11417e0a/attachment-0001.sig >

------------------------------

Message: 2
Date: Thu, 30 Mar 2023 17:48:11 -0600
From: IO500 Committee <committee at io500.org>
To: BeeGFS <fhgfs-user at googlegroups.com>, Ceph <ceph-users at ceph.io>,
        GPFS <gpfsug-discuss at spectrumscale.org>, HPC Announce
        <hpc-announce at mcs.anl.gov>, IO500 <io-500 at vi4io.org>, Lustre
        <lustre-discuss at lists.lustre.org>, OrangeFS
        <users at lists.orangefs.org>, Storage Research List
        <storage-research-list at ece.cmu.edu>
Subject: [gpfsug-discuss] Call for Submissions IO500 ISC23
Message-ID: <1ceb433e1ae046c5dcd515a69a807e26 at io500.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed

Stabilization Period: Monday, April 3rd - Friday, April 14th, 2023
Submission Deadline: Tuesday, May 16st, 2023 AoE

The IO500 is now accepting and encouraging submissions for the upcoming
12th semi-annual IO500 list, in conjunction with ISC23. Once again, we
are also accepting submissions to the 10 Client Node Challenge to
encourage the submission of small scale results. The new ranked lists
will be announced at the ISC23  BoF [1]. We hope to see many new
results.

What's New
1. Creation of Production and Research Lists - Starting with ISC'22, we
proposed a separation of the list into separate Production and Research
lists.  This better reflects the important distinction between storage
systems that run in production environments and those that may use more
experimental hardware and software configurations.  At ISC23, we will
formally create these two lists and users will be able to submit to
either of the two lists (and their 10 client-node counterparts).  Please
see the requirements for each list on the IO500 rules page [3].
2. New Submission Tool - There is now a new IO500 submission tool that
improves the overall submission experience.  Users can create accounts
and then update and manage all of their submissions through that
account.  As part of this new tool, we have improved the submission
fields that describe the hardware and software of the system under test.
For reproducibility and analysis reasons, we now made the easily
obtainable fields mandatory - data from storage servers are for users
often difficult to obtain, therefore, most remain optional. As a new
system, there may be quirks, please reach out on Slack or the mailing
list if you see any issues.  Further details will be released on the
submission page [2].
3. Reproducibility - Every submission will now receive a reproducibility
score based upon the provided system details and the reproducibility
questionnaire. This score will inform the community on the amount of
details provided in the submission and the obtainability of the storage
system. Further, this score will be used to evaluate if a submission is
eligible for the Production list.
4. New Phases - We are continuing to evaluate the inclusion of optional
test phases for additional key workloads - split easy/hard find phases,
4KB and 1MB random read/write phases, and concurrent metadata
operations. This is called an extended run. At the moment, we collect
the information to verify that additional phases do not significantly
impact the results of a standard run and an extended run to facilitate
comparisons between the existing and new benchmark phases. In a future
release, we may include some or all of these results as part of the
standard benchmark. The extended results are not currently included in
the scoring of any ranked list.

Background
The benchmark suite is designed to be easy to run and the community has
multiple active support channels to help with any questions. Please note
that submissions of all sizes are welcome; the site has customizable
sorting, so it is possible to submit on a small system and still get a
very good per-client score, for example. Additionally, the list is about
much more than just the raw rank; all submissions help the community by
collecting and publishing a wider corpus of data. More details below.

Following the success of the Top500 in collecting and analyzing
historical trends in supercomputer technology and evolution, the IO500
was created in 2017, published its first list at SC17, and has grown
continually since then. The need for such an initiative has long been
known within High-Performance Computing; however, defining appropriate
benchmarks has long been challenging. Despite this challenge, the
community, after long and spirited discussion, finally reached consensus
on a suite of benchmarks and a metric for resolving the scores into a
single ranking.

The multi-fold goals of the benchmark suite are as follows:
1. Maximizing simplicity in running the benchmark suite
2. Encouraging optimization and documentation of tuning parameters for
performance
3. Allowing submitters to highlight their "hero run" performance numbers
4. Forcing submitters to simultaneously report performance for
challenging IO patterns.

Specifically, the benchmark suite includes a hero-run of both IOR and
mdtest configured however possible to maximize performance and establish
an upper-bound for performance. It also includes an IOR and mdtest run
with highly prescribed parameters in an attempt to determine a lower
performance bound. Finally, it includes a namespace search as this has
been determined to be a highly sought-after feature in HPC storage
systems that has historically not been well-measured. Submitters are
encouraged to share their tuning insights for publication.

The goals of the community are also multi-fold:
1. Gather historical data for the sake of analysis and to aid
predictions of storage futures
2. Collect tuning information to share valuable performance
optimizations across the community
3. Encourage vendors and designers to optimize for workloads beyond
"hero runs"
4. Establish bounded expectations for users, procurers, and
administrators

The IO500 follows a two-staged approach. First, there will be a two-week
stabilization period during which we encourage the community to verify
that the benchmark runs properly on a variety of storage systems. During
this period the benchmark may be updated based upon feedback from the
community. The final benchmark will then be released. We expect that
runs compliant with the rules made during the stabilization period will
be valid as a final submission unless a significant defect is found.

10 Client Node I/O Challenge
The 10 Client Node Challenge is conducted using the regular IO500
benchmark, however, with the rule that exactly 10 client nodes must be
used to run the benchmark. You may use any shared storage with any
number of servers. We will announce the results in the Production and
Research lists as well as in separate derived lists.

Birds-of-a-Feather
Once again, we encourage you to submit [2] to join our community, and to
attend the ISC23 BoF [1], where we will announce the new IO500
Production and Research lists and their 10 client node counterparts. The
current list includes results from twenty different storage system types
and 70 institutions. We hope that the upcoming list grows even more.

[1] https://io500.org/pages/bof-isc23
[2] https://io500.org/submission
[3] https://io500.org/rules-submission

--
The IO500 Committee



------------------------------

Subject: Digest Footer

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org


------------------------------

End of gpfsug-discuss Digest, Vol 132, Issue 25
***********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230331/a50117b4/attachment.htm>