From rk3199 at columbia.edu Thu Jun 6 21:45:30 2024 From: rk3199 at columbia.edu (Rob Kudyba) Date: Thu, 6 Jun 2024 16:45:30 -0400 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Message-ID: Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space left on device message when trying to write to a file. In /var/adm/ras/mmfs.log the only recent errors are this: 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode). 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode). according to https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 Check the preceding messages, and consult the earlier chapters of this > document. A frequent cause for such errors is lack of space in /var. We have plenty of space left. /usr/lpp/mmfs/bin/mmlsdisk cluster disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------ S01_MDT200_1 nsd 4096 200 Yes No ready up system S01_MDT201_1 nsd 4096 201 Yes No ready up system S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 /usr/lpp/mmfs/bin/mmdf headnode disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 14 TB) S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%) S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%) ------------- -------------------- ------------------- (pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%) Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%) S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%) S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%) S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%) S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%) S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%) S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%) ------------- -------------------- ------------------- (pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) ============= ==================== =================== (data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) (metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%) ============= ==================== =================== (total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 154807668 Total number of free inodes in all Inode spaces: 12964492 Total number of allocated inodes in all Inode spaces: 167772160 Total of Maximum number of inodes in all Inode spaces: 276971520 On the head node: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 430G 216G 215G 51% / devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 4.1G 43G 9% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/sda1 504M 114M 365M 24% /boot /dev/sda2 100M 9.9M 90M 10% /boot/efi x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share cluster 506T 405T 101T 81% /cluster tmpfs 9.3G 0 9.3G 0% /run/user/443748 tmpfs 9.3G 0 9.3G 0% /run/user/547288 tmpfs 9.3G 0 9.3G 0% /run/user/551336 tmpfs 9.3G 0 9.3G 0% /run/user/547289 The login nodes have plenty of space in /var: /dev/sda3 50G 8.7G 42G 18% /var What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart? -------------- next part -------------- An HTML attachment was scrubbed... URL: From malone12 at illinois.edu Thu Jun 6 22:01:20 2024 From: malone12 at illinois.edu (Maloney, John Daniel) Date: Thu, 6 Jun 2024 21:01:20 +0000 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: Are you seeing the issues across the whole file system or in certain areas? That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated). What does a ?df -i /cluster? show you? Or if this is only in a certain area you can ?cd? into that directory and run a ?df -i .? You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won?t have auto-inode expansion for the filesets. Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) From: gpfsug-discuss on behalf of Rob Kudyba Date: Thursday, June 6, 2024 at 3:50?PM To: gpfsug-discuss at gpfsug.org Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space left on device message when trying to write to a file. In /var/adm/ras/mmfs.log the only recent errors are this: 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode). 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode). according to https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in /var. We have plenty of space left. /usr/lpp/mmfs/bin/mmlsdisk cluster disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------ S01_MDT200_1 nsd 4096 200 Yes No ready up system S01_MDT201_1 nsd 4096 201 Yes No ready up system S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 /usr/lpp/mmfs/bin/mmdf headnode disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 14 TB) S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%) S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%) ------------- -------------------- ------------------- (pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%) Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%) S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%) S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%) S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%) S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%) S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%) S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%) ------------- -------------------- ------------------- (pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) ============= ==================== =================== (data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) (metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%) ============= ==================== =================== (total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 154807668 Total number of free inodes in all Inode spaces: 12964492 Total number of allocated inodes in all Inode spaces: 167772160 Total of Maximum number of inodes in all Inode spaces: 276971520 On the head node: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 430G 216G 215G 51% / devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 4.1G 43G 9% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/sda1 504M 114M 365M 24% /boot /dev/sda2 100M 9.9M 90M 10% /boot/efi x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share cluster 506T 405T 101T 81% /cluster tmpfs 9.3G 0 9.3G 0% /run/user/443748 tmpfs 9.3G 0 9.3G 0% /run/user/547288 tmpfs 9.3G 0 9.3G 0% /run/user/551336 tmpfs 9.3G 0 9.3G 0% /run/user/547289 The login nodes have plenty of space in /var: /dev/sda3 50G 8.7G 42G 18% /var What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rk3199 at columbia.edu Thu Jun 6 22:34:53 2024 From: rk3199 at columbia.edu (Rob Kudyba) Date: Thu, 6 Jun 2024 17:34:53 -0400 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: > > Are you seeing the issues across the whole file system or in certain > areas? > Only with accounts in GPFS, local accounts and root do not gt this. > That sounds like inode exhaustion to me (and based on it not being block > exhaustion as you?ve demonstrated). > > > > What does a ?df -i /cluster? show you? > We bumped it up a few weeks ago: df -i /cluster Filesystem Inodes IUsed IFree IUse% Mounted on cluster 276971520 154807697 122163823 56% /cluster > Or if this is only in a certain area you can ?cd? into that directory and > run a ?df -i .? > As root on a login node; df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 20971520 169536 20801984 1% / devtmpfs 12169978 528 12169450 1% /dev tmpfs 12174353 1832 12172521 1% /run tmpfs 12174353 77 12174276 1% /dev/shm tmpfs 12174353 15 12174338 1% /sys/fs/cgroup /dev/sda1 0 0 0 - /boot/efi /dev/sda3 52428800 2887 52425913 1% /var /dev/sda7 277368832 35913 277332919 1% /local /dev/sda5 104857600 398 104857202 1% /tmp tmpfs 12174353 1 12174352 1% /run/user/551336 tmpfs 12174353 1 12174352 1% /run/user/0 moto 276971520 154807697 122163823 56% /cluster tmpfs 12174353 3 12174350 1% /run/user/441245 tmpfs 12174353 12 12174341 1% /run/user/553562 tmpfs 12174353 1 12174352 1% /run/user/525583 tmpfs 12174353 1 12174352 1% /run/user/476374 tmpfs 12174353 1 12174352 1% /run/user/468934 tmpfs 12174353 5 12174348 1% /run/user/551200 tmpfs 12174353 1 12174352 1% /run/user/539143 tmpfs 12174353 1 12174352 1% /run/user/488676 tmpfs 12174353 1 12174352 1% /run/user/493713 tmpfs 12174353 1 12174352 1% /run/user/507831 tmpfs 12174353 1 12174352 1% /run/user/549822 tmpfs 12174353 1 12174352 1% /run/user/500569 tmpfs 12174353 1 12174352 1% /run/user/443748 tmpfs 12174353 1 12174352 1% /run/user/543676 tmpfs 12174353 1 12174352 1% /run/user/451446 tmpfs 12174353 1 12174352 1% /run/user/497945 tmpfs 12174353 6 12174347 1% /run/user/554672 tmpfs 12174353 32 12174321 1% /run/user/554653 tmpfs 12174353 1 12174352 1% /run/user/30094 tmpfs 12174353 1 12174352 1% /run/user/470790 tmpfs 12174353 59 12174294 1% /run/user/553037 tmpfs 12174353 1 12174352 1% /run/user/554670 tmpfs 12174353 1 12174352 1% /run/user/548236 tmpfs 12174353 1 12174352 1% /run/user/547288 tmpfs 12174353 1 12174352 1% /run/user/547289 > > > You may need to allocate more inodes to an independent inode fileset > somewhere. Especially with something as old as 4.2.3 you won?t have > auto-inode expansion for the filesets. > Do we have to restart any service after upping the inode count? > > Best, > > > > J.D. Maloney > > Lead HPC Storage Engineer | Storage Enabling Technologies Group > > National Center for Supercomputing Applications (NCSA) > Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati! > > > *From: *gpfsug-discuss on behalf of > Rob Kudyba > *Date: *Thursday, June 6, 2024 at 3:50?PM > *To: *gpfsug-discuss at gpfsug.org > *Subject: *[gpfsug-discuss] No space left on device, but plenty of quota > space for inodes and blocks > > Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space > left on device message when trying to write to a file. In /var/adm/ras/mmfs.log > the only recent errors are this: > > > > 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. > Return code -1. > 2024-06-06_15:51:22.311-0400: The previous error was detected on node > x.x.x.x (headnode). > 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. > Return code -1. > 2024-06-06_15:53:25.088-0400: The previous error was detected on node > x.x.x.x (headnode). > > > > according to > https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 > > > > > > Check the preceding messages, and consult the earlier chapters of this > document. A frequent cause for such errors is lack of space in /var. > > > > We have plenty of space left. > > > > /usr/lpp/mmfs/bin/mmlsdisk cluster > disk driver sector failure holds holds > storage > name type size group metadata data status > availability pool > ------------ -------- ------ ----------- -------- ----- ------------- > ------------ ------------ > S01_MDT200_1 nsd 4096 200 Yes No ready up > system > S01_MDT201_1 nsd 4096 201 Yes No ready up > system > S01_DAT0001_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0002_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0003_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0004_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0005_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0006_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0007_1 nsd 4096 100 No Yes ready up > data1 > > > > /usr/lpp/mmfs/bin/mmdf headnode > disk disk size failure holds holds free KB > free KB > name in KB group metadata data in full blocks > in fragments > --------------- ------------- -------- -------- ----- -------------------- > ------------------- > Disks in storage pool: system (Maximum disk size allowed is 14 TB) > S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) > 2948720 ( 0%) > S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) > 2957424 ( 0%) > ------------- -------------------- > ------------------- > (pool total) 3724541952 1938260992 ( 52%) > 5906144 ( 0%) > > Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) > S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) > 897723392 ( 1%) > S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) > 949412160 ( 1%) > S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) > 951327680 ( 1%) > S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) > 949340544 ( 1%) > S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) > 948377536 ( 1%) > S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) > 952892480 ( 1%) > S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) > 949072512 ( 1%) > ------------- -------------------- > ------------------- > (pool total) 542575165440 108113510400 ( 20%) > 6598146304 ( 1%) > > ============= ==================== > =================== > (data) 542575165440 108113510400 ( 20%) > 6598146304 ( 1%) > (metadata) 3724541952 1938260992 ( 52%) > 5906144 ( 0%) > ============= ==================== > =================== > (total) 546299707392 110051771392 ( 22%) > 6604052448 ( 1%) > > Inode Information > ----------------- > Total number of used inodes in all Inode spaces: 154807668 > Total number of free inodes in all Inode spaces: 12964492 > Total number of allocated inodes in all Inode spaces: 167772160 > Total of Maximum number of inodes in all Inode spaces: 276971520 > > > > On the head node: > > > > df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda4 430G 216G 215G 51% / > devtmpfs 47G 0 47G 0% /dev > tmpfs 47G 0 47G 0% /dev/shm > tmpfs 47G 4.1G 43G 9% /run > tmpfs 47G 0 47G 0% /sys/fs/cgroup > /dev/sda1 504M 114M 365M 24% /boot > /dev/sda2 100M 9.9M 90M 10% /boot/efi > x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share > cluster 506T 405T 101T 81% /cluster > tmpfs 9.3G 0 9.3G 0% /run/user/443748 > tmpfs 9.3G 0 9.3G 0% /run/user/547288 > tmpfs 9.3G 0 9.3G 0% /run/user/551336 > tmpfs 9.3G 0 9.3G 0% /run/user/547289 > > > > The login nodes have plenty of space in /var: > > /dev/sda3 50G 8.7G 42G 18% /var > > > > What else should we check? We are just at 81% on the GPFS mounted file > system but that should be enough for more space without these errors. Any > recommended service(s) that we can restart? > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Thu Jun 6 23:10:56 2024 From: stockf at us.ibm.com (Fred Stock) Date: Thu, 6 Jun 2024 22:10:56 +0000 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: You should check the inode counts for each of the filesets using the mmlsfileset command. You should check the local disk space on all the nodes. I presume you are aware that Scale 4.2.3 has been out of support for 4 years. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Rob Kudyba Date: Thursday, June 6, 2024 at 5:39?PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated).? ZjQcmQRYFpfptBannerStart This Message Is From an Untrusted Sender You have not previously corresponded with this sender. Report Suspicious ZjQcmQRYFpfptBannerEnd Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated). What does a ?df -i /cluster? show you? We bumped it up a few weeks ago: df -i /cluster Filesystem Inodes IUsed IFree IUse% Mounted on cluster 276971520 154807697 122163823 56% /cluster Or if this is only in a certain area you can ?cd? into that directory and run a ?df -i .? As root on a login node; df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 20971520 169536 20801984 1% / devtmpfs 12169978 528 12169450 1% /dev tmpfs 12174353 1832 12172521 1% /run tmpfs 12174353 77 12174276 1% /dev/shm tmpfs 12174353 15 12174338 1% /sys/fs/cgroup /dev/sda1 0 0 0 - /boot/efi /dev/sda3 52428800 2887 52425913 1% /var /dev/sda7 277368832 35913 277332919 1% /local /dev/sda5 104857600 398 104857202 1% /tmp tmpfs 12174353 1 12174352 1% /run/user/551336 tmpfs 12174353 1 12174352 1% /run/user/0 moto 276971520 154807697 122163823 56% /cluster tmpfs 12174353 3 12174350 1% /run/user/441245 tmpfs 12174353 12 12174341 1% /run/user/553562 tmpfs 12174353 1 12174352 1% /run/user/525583 tmpfs 12174353 1 12174352 1% /run/user/476374 tmpfs 12174353 1 12174352 1% /run/user/468934 tmpfs 12174353 5 12174348 1% /run/user/551200 tmpfs 12174353 1 12174352 1% /run/user/539143 tmpfs 12174353 1 12174352 1% /run/user/488676 tmpfs 12174353 1 12174352 1% /run/user/493713 tmpfs 12174353 1 12174352 1% /run/user/507831 tmpfs 12174353 1 12174352 1% /run/user/549822 tmpfs 12174353 1 12174352 1% /run/user/500569 tmpfs 12174353 1 12174352 1% /run/user/443748 tmpfs 12174353 1 12174352 1% /run/user/543676 tmpfs 12174353 1 12174352 1% /run/user/451446 tmpfs 12174353 1 12174352 1% /run/user/497945 tmpfs 12174353 6 12174347 1% /run/user/554672 tmpfs 12174353 32 12174321 1% /run/user/554653 tmpfs 12174353 1 12174352 1% /run/user/30094 tmpfs 12174353 1 12174352 1% /run/user/470790 tmpfs 12174353 59 12174294 1% /run/user/553037 tmpfs 12174353 1 12174352 1% /run/user/554670 tmpfs 12174353 1 12174352 1% /run/user/548236 tmpfs 12174353 1 12174352 1% /run/user/547288 tmpfs 12174353 1 12174352 1% /run/user/547289 You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won?t have auto-inode expansion for the filesets. Do we have to restart any service after upping the inode count? Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati! From: gpfsug-discuss > on behalf of Rob Kudyba > Date: Thursday, June 6, 2024 at 3:50?PM To: gpfsug-discuss at gpfsug.org > Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space left on device message when trying to write to a file. In /var/adm/ras/mmfs.log the only recent errors are this: 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode). 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode). according to https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in /var. We have plenty of space left. /usr/lpp/mmfs/bin/mmlsdisk cluster disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------ S01_MDT200_1 nsd 4096 200 Yes No ready up system S01_MDT201_1 nsd 4096 201 Yes No ready up system S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 /usr/lpp/mmfs/bin/mmdf headnode disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 14 TB) S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%) S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%) ------------- -------------------- ------------------- (pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%) Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%) S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%) S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%) S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%) S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%) S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%) S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%) ------------- -------------------- ------------------- (pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) ============= ==================== =================== (data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) (metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%) ============= ==================== =================== (total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 154807668 Total number of free inodes in all Inode spaces: 12964492 Total number of allocated inodes in all Inode spaces: 167772160 Total of Maximum number of inodes in all Inode spaces: 276971520 On the head node: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 430G 216G 215G 51% / devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 4.1G 43G 9% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/sda1 504M 114M 365M 24% /boot /dev/sda2 100M 9.9M 90M 10% /boot/efi x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share cluster 506T 405T 101T 81% /cluster tmpfs 9.3G 0 9.3G 0% /run/user/443748 tmpfs 9.3G 0 9.3G 0% /run/user/547288 tmpfs 9.3G 0 9.3G 0% /run/user/551336 tmpfs 9.3G 0 9.3G 0% /run/user/547289 The login nodes have plenty of space in /var: /dev/sda3 50G 8.7G 42G 18% /var What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rk3199 at columbia.edu Thu Jun 6 23:47:23 2024 From: rk3199 at columbia.edu (Rob Kudyba) Date: Thu, 6 Jun 2024 18:47:23 -0400 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: I guess I have my answer: /usr/lpp/mmfs/bin/mmlsfileset cluster home -L Filesets in file system 'cluster': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment home 1 1048579 0 Thu Nov 29 15:21:52 2018 1 20971520 20971520 However on some of the other filesets the AllocInodes is 0? /usr/lpp/mmfs/bin/mmlsfileset cluster groupa -L -i Collecting fileset usage information ... Filesets in file system 'moto': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes UsedInodes Comment stats 8 181207 0 Fri Nov 30 12:27:25 2018 0 0 0 7628733 Yes we realize it's old and it'll be retired at the end of 2024. On Thu, Jun 6, 2024 at 6:15?PM Fred Stock wrote: > You should check the inode counts for each of the filesets using the > mmlsfileset command. You should check the local disk space on all the > nodes. > > > > I presume you are aware that Scale 4.2.3 has been out of support for 4 > years. > > > > Fred > > > > Fred Stock, Spectrum Scale Development Advocacy > > stockf at us.ibm.com | 720-430-8821 > > > > > > > > *From: *gpfsug-discuss on behalf of > Rob Kudyba > *Date: *Thursday, June 6, 2024 at 5:39?PM > *To: *gpfsug main discussion list > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] No space left on device, but > plenty of quota space for inodes and blocks > > Are you seeing the issues across the whole file system or in certain > areas? Only with accounts in GPFS, local accounts and root do not gt this. > That sounds like inode exhaustion to me (and based on it not being block > exhaustion as you?ve demonstrated). > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an Untrusted Sender * > > You have not previously corresponded with this sender. > > > > > Report Suspicious > > > > > > > ZjQcmQRYFpfptBannerEnd > > Are you seeing the issues across the whole file system or in certain > areas? > > > > Only with accounts in GPFS, local accounts and root do not gt this. > > > > That sounds like inode exhaustion to me (and based on it not being block > exhaustion as you?ve demonstrated). > > > > What does a ?df -i /cluster? show you? > > > > We bumped it up a few weeks ago: > > df -i /cluster > Filesystem Inodes IUsed IFree IUse% Mounted on > cluster 276971520 154807697 122163823 56% /cluster > > > > > > Or if this is only in a certain area you can ?cd? into that directory and > run a ?df -i .? > > > > As root on a login node; > > df -i > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sda2 20971520 169536 20801984 1% / > devtmpfs 12169978 528 12169450 1% /dev > tmpfs 12174353 1832 12172521 1% /run > tmpfs 12174353 77 12174276 1% /dev/shm > tmpfs 12174353 15 12174338 1% /sys/fs/cgroup > /dev/sda1 0 0 0 - /boot/efi > /dev/sda3 52428800 2887 52425913 1% /var > /dev/sda7 277368832 35913 277332919 1% /local > /dev/sda5 104857600 398 104857202 1% /tmp > tmpfs 12174353 1 12174352 1% /run/user/551336 > tmpfs 12174353 1 12174352 1% /run/user/0 > moto 276971520 154807697 122163823 56% /cluster > tmpfs 12174353 3 12174350 1% /run/user/441245 > tmpfs 12174353 12 12174341 1% /run/user/553562 > tmpfs 12174353 1 12174352 1% /run/user/525583 > tmpfs 12174353 1 12174352 1% /run/user/476374 > tmpfs 12174353 1 12174352 1% /run/user/468934 > tmpfs 12174353 5 12174348 1% /run/user/551200 > tmpfs 12174353 1 12174352 1% /run/user/539143 > tmpfs 12174353 1 12174352 1% /run/user/488676 > tmpfs 12174353 1 12174352 1% /run/user/493713 > tmpfs 12174353 1 12174352 1% /run/user/507831 > tmpfs 12174353 1 12174352 1% /run/user/549822 > tmpfs 12174353 1 12174352 1% /run/user/500569 > tmpfs 12174353 1 12174352 1% /run/user/443748 > tmpfs 12174353 1 12174352 1% /run/user/543676 > tmpfs 12174353 1 12174352 1% /run/user/451446 > tmpfs 12174353 1 12174352 1% /run/user/497945 > tmpfs 12174353 6 12174347 1% /run/user/554672 > tmpfs 12174353 32 12174321 1% /run/user/554653 > tmpfs 12174353 1 12174352 1% /run/user/30094 > tmpfs 12174353 1 12174352 1% /run/user/470790 > tmpfs 12174353 59 12174294 1% /run/user/553037 > tmpfs 12174353 1 12174352 1% /run/user/554670 > tmpfs 12174353 1 12174352 1% /run/user/548236 > tmpfs 12174353 1 12174352 1% /run/user/547288 > > tmpfs 12174353 1 12174352 1% /run/user/547289 > > > > You may need to allocate more inodes to an independent inode fileset > somewhere. Especially with something as old as 4.2.3 you won?t have > auto-inode expansion for the filesets. > > > > Do we have to restart any service after upping the inode count? > > > > > > Best, > > > > J.D. Maloney > > Lead HPC Storage Engineer | Storage Enabling Technologies Group > > National Center for Supercomputing Applications (NCSA) > > > > Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati! > > > > > > *From: *gpfsug-discuss on behalf of > Rob Kudyba > *Date: *Thursday, June 6, 2024 at 3:50?PM > *To: *gpfsug-discuss at gpfsug.org > *Subject: *[gpfsug-discuss] No space left on device, but plenty of quota > space for inodes and blocks > > Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space > left on device message when trying to write to a file. In /var/adm/ras/mmfs.log > the only recent errors are this: > > > > 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. > Return code -1. > 2024-06-06_15:51:22.311-0400: The previous error was detected on node > x.x.x.x (headnode). > 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. > Return code -1. > 2024-06-06_15:53:25.088-0400: The previous error was detected on node > x.x.x.x (headnode). > > > > according to > https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 > > > > Check the preceding messages, and consult the earlier chapters of this > document. A frequent cause for such errors is lack of space in /var. > > > > We have plenty of space left. > > > > /usr/lpp/mmfs/bin/mmlsdisk cluster > disk driver sector failure holds holds > storage > name type size group metadata data status > availability pool > ------------ -------- ------ ----------- -------- ----- ------------- > ------------ ------------ > S01_MDT200_1 nsd 4096 200 Yes No ready up > system > S01_MDT201_1 nsd 4096 201 Yes No ready up > system > S01_DAT0001_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0002_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0003_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0004_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0005_1 nsd 4096 100 No Yes ready up > data1 > S01_DAT0006_1 nsd 4096 101 No Yes ready up > data1 > S01_DAT0007_1 nsd 4096 100 No Yes ready up > data1 > > > > /usr/lpp/mmfs/bin/mmdf headnode > disk disk size failure holds holds free KB > free KB > name in KB group metadata data in full blocks > in fragments > --------------- ------------- -------- -------- ----- -------------------- > ------------------- > Disks in storage pool: system (Maximum disk size allowed is 14 TB) > S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) > 2948720 ( 0%) > S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) > 2957424 ( 0%) > ------------- -------------------- > ------------------- > (pool total) 3724541952 1938260992 ( 52%) > 5906144 ( 0%) > > Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) > S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) > 897723392 ( 1%) > S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) > 949412160 ( 1%) > S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) > 951327680 ( 1%) > S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) > 949340544 ( 1%) > S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) > 948377536 ( 1%) > S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) > 952892480 ( 1%) > S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) > 949072512 ( 1%) > ------------- -------------------- > ------------------- > (pool total) 542575165440 108113510400 ( 20%) > 6598146304 ( 1%) > > ============= ==================== > =================== > (data) 542575165440 108113510400 ( 20%) > 6598146304 ( 1%) > (metadata) 3724541952 1938260992 ( 52%) > 5906144 ( 0%) > ============= ==================== > =================== > (total) 546299707392 110051771392 ( 22%) > 6604052448 ( 1%) > > Inode Information > ----------------- > Total number of used inodes in all Inode spaces: 154807668 > Total number of free inodes in all Inode spaces: 12964492 > Total number of allocated inodes in all Inode spaces: 167772160 > Total of Maximum number of inodes in all Inode spaces: 276971520 > > > > On the head node: > > > > df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda4 430G 216G 215G 51% / > devtmpfs 47G 0 47G 0% /dev > tmpfs 47G 0 47G 0% /dev/shm > tmpfs 47G 4.1G 43G 9% /run > tmpfs 47G 0 47G 0% /sys/fs/cgroup > /dev/sda1 504M 114M 365M 24% /boot > /dev/sda2 100M 9.9M 90M 10% /boot/efi > x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share > cluster 506T 405T 101T 81% /cluster > tmpfs 9.3G 0 9.3G 0% /run/user/443748 > tmpfs 9.3G 0 9.3G 0% /run/user/547288 > tmpfs 9.3G 0 9.3G 0% /run/user/551336 > tmpfs 9.3G 0 9.3G 0% /run/user/547289 > > > > The login nodes have plenty of space in /var: > > /dev/sda3 50G 8.7G 42G 18% /var > > > > What else should we check? We are just at 81% on the GPFS mounted file > system but that should be enough for more space without these errors. Any > recommended service(s) that we can restart? > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From malone12 at illinois.edu Thu Jun 6 23:58:22 2024 From: malone12 at illinois.edu (Maloney, John Daniel) Date: Thu, 6 Jun 2024 22:58:22 +0000 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: Yeah; you?ll want to bump that up on the home fileset, something like: mmchfileset cluster home --inode-limit 25000000 (that?d give you a buffer of ~4.9 million inodes) For the ones that show 0, those are dependent filesets (not independent) the inode allocations are tracked in the parent independent inode fileset. Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) From: gpfsug-discuss on behalf of Rob Kudyba Date: Thursday, June 6, 2024 at 5:51?PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks I guess I have my answer: /usr/lpp/mmfs/bin/mmlsfileset cluster home -L Filesets in file system 'cluster': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment home 1 1048579 0 Thu Nov 29 15:21:52 2018 1 20971520 20971520 However on some of the other filesets the AllocInodes is 0? /usr/lpp/mmfs/bin/mmlsfileset cluster groupa -L -i Collecting fileset usage information ... Filesets in file system 'moto': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes UsedInodes Comment stats 8 181207 0 Fri Nov 30 12:27:25 2018 0 0 0 7628733 Yes we realize it's old and it'll be retired at the end of 2024. On Thu, Jun 6, 2024 at 6:15?PM Fred Stock > wrote: You should check the inode counts for each of the filesets using the mmlsfileset command. You should check the local disk space on all the nodes. I presume you are aware that Scale 4.2.3 has been out of support for 4 years. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss > on behalf of Rob Kudyba > Date: Thursday, June 6, 2024 at 5:39?PM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated).? ZjQcmQRYFpfptBannerStart This Message Is From an Untrusted Sender You have not previously corresponded with this sender. Report Suspicious ZjQcmQRYFpfptBannerEnd Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated). What does a ?df -i /cluster? show you? We bumped it up a few weeks ago: df -i /cluster Filesystem Inodes IUsed IFree IUse% Mounted on cluster 276971520 154807697 122163823 56% /cluster Or if this is only in a certain area you can ?cd? into that directory and run a ?df -i .? As root on a login node; df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 20971520 169536 20801984 1% / devtmpfs 12169978 528 12169450 1% /dev tmpfs 12174353 1832 12172521 1% /run tmpfs 12174353 77 12174276 1% /dev/shm tmpfs 12174353 15 12174338 1% /sys/fs/cgroup /dev/sda1 0 0 0 - /boot/efi /dev/sda3 52428800 2887 52425913 1% /var /dev/sda7 277368832 35913 277332919 1% /local /dev/sda5 104857600 398 104857202 1% /tmp tmpfs 12174353 1 12174352 1% /run/user/551336 tmpfs 12174353 1 12174352 1% /run/user/0 moto 276971520 154807697 122163823 56% /cluster tmpfs 12174353 3 12174350 1% /run/user/441245 tmpfs 12174353 12 12174341 1% /run/user/553562 tmpfs 12174353 1 12174352 1% /run/user/525583 tmpfs 12174353 1 12174352 1% /run/user/476374 tmpfs 12174353 1 12174352 1% /run/user/468934 tmpfs 12174353 5 12174348 1% /run/user/551200 tmpfs 12174353 1 12174352 1% /run/user/539143 tmpfs 12174353 1 12174352 1% /run/user/488676 tmpfs 12174353 1 12174352 1% /run/user/493713 tmpfs 12174353 1 12174352 1% /run/user/507831 tmpfs 12174353 1 12174352 1% /run/user/549822 tmpfs 12174353 1 12174352 1% /run/user/500569 tmpfs 12174353 1 12174352 1% /run/user/443748 tmpfs 12174353 1 12174352 1% /run/user/543676 tmpfs 12174353 1 12174352 1% /run/user/451446 tmpfs 12174353 1 12174352 1% /run/user/497945 tmpfs 12174353 6 12174347 1% /run/user/554672 tmpfs 12174353 32 12174321 1% /run/user/554653 tmpfs 12174353 1 12174352 1% /run/user/30094 tmpfs 12174353 1 12174352 1% /run/user/470790 tmpfs 12174353 59 12174294 1% /run/user/553037 tmpfs 12174353 1 12174352 1% /run/user/554670 tmpfs 12174353 1 12174352 1% /run/user/548236 tmpfs 12174353 1 12174352 1% /run/user/547288 tmpfs 12174353 1 12174352 1% /run/user/547289 You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won?t have auto-inode expansion for the filesets. Do we have to restart any service after upping the inode count? Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati! From: gpfsug-discuss > on behalf of Rob Kudyba > Date: Thursday, June 6, 2024 at 3:50?PM To: gpfsug-discuss at gpfsug.org > Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space left on device message when trying to write to a file. In /var/adm/ras/mmfs.log the only recent errors are this: 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode). 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode). according to https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in /var. We have plenty of space left. /usr/lpp/mmfs/bin/mmlsdisk cluster disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------ S01_MDT200_1 nsd 4096 200 Yes No ready up system S01_MDT201_1 nsd 4096 201 Yes No ready up system S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 /usr/lpp/mmfs/bin/mmdf headnode disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 14 TB) S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%) S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%) ------------- -------------------- ------------------- (pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%) Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%) S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%) S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%) S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%) S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%) S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%) S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%) ------------- -------------------- ------------------- (pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) ============= ==================== =================== (data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) (metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%) ============= ==================== =================== (total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 154807668 Total number of free inodes in all Inode spaces: 12964492 Total number of allocated inodes in all Inode spaces: 167772160 Total of Maximum number of inodes in all Inode spaces: 276971520 On the head node: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 430G 216G 215G 51% / devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 4.1G 43G 9% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/sda1 504M 114M 365M 24% /boot /dev/sda2 100M 9.9M 90M 10% /boot/efi x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share cluster 506T 405T 101T 81% /cluster tmpfs 9.3G 0 9.3G 0% /run/user/443748 tmpfs 9.3G 0 9.3G 0% /run/user/547288 tmpfs 9.3G 0 9.3G 0% /run/user/551336 tmpfs 9.3G 0 9.3G 0% /run/user/547289 The login nodes have plenty of space in /var: /dev/sda3 50G 8.7G 42G 18% /var What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From danny.lang at crick.ac.uk Fri Jun 7 00:11:27 2024 From: danny.lang at crick.ac.uk (Danny Lang) Date: Thu, 6 Jun 2024 23:11:27 +0000 Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks In-Reply-To: References: Message-ID: Just to provide extra information, you can enable the gpfsgui. This will give you the webpage of the whole system and it will contain alerts. It will allow you to see everything and do stuff from the webpage rather than from the command line. It's quite useful, and it also provides system health as well as some nice graphs. ? Thanks Danny ________________________________ From: gpfsug-discuss on behalf of Maloney, John Daniel Sent: 06 June 2024 11:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks External Sender: Use caution. Yeah; you?ll want to bump that up on the home fileset, something like: mmchfileset cluster home --inode-limit 25000000 (that?d give you a buffer of ~4.9 million inodes) For the ones that show 0, those are dependent filesets (not independent) the inode allocations are tracked in the parent independent inode fileset. Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) From: gpfsug-discuss on behalf of Rob Kudyba Date: Thursday, June 6, 2024 at 5:51?PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks I guess I have my answer: /usr/lpp/mmfs/bin/mmlsfileset cluster home -L Filesets in file system 'cluster': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment home 1 1048579 0 Thu Nov 29 15:21:52 2018 1 20971520 20971520 However on some of the other filesets the AllocInodes is 0? /usr/lpp/mmfs/bin/mmlsfileset cluster groupa -L -i Collecting fileset usage information ... Filesets in file system 'moto': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes UsedInodes Comment stats 8 181207 0 Fri Nov 30 12:27:25 2018 0 0 0 7628733 Yes we realize it's old and it'll be retired at the end of 2024. On Thu, Jun 6, 2024 at 6:15?PM Fred Stock > wrote: You should check the inode counts for each of the filesets using the mmlsfileset command. You should check the local disk space on all the nodes. I presume you are aware that Scale 4.2.3 has been out of support for 4 years. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss > on behalf of Rob Kudyba > Date: Thursday, June 6, 2024 at 5:39?PM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated).? ZjQcmQRYFpfptBannerStart This Message Is From an Untrusted Sender You have not previously corresponded with this sender. Report Suspicious ZjQcmQRYFpfptBannerEnd Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you?ve demonstrated). What does a ?df -i /cluster? show you? We bumped it up a few weeks ago: df -i /cluster Filesystem Inodes IUsed IFree IUse% Mounted on cluster 276971520 154807697 122163823 56% /cluster Or if this is only in a certain area you can ?cd? into that directory and run a ?df -i .? As root on a login node; df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 20971520 169536 20801984 1% / devtmpfs 12169978 528 12169450 1% /dev tmpfs 12174353 1832 12172521 1% /run tmpfs 12174353 77 12174276 1% /dev/shm tmpfs 12174353 15 12174338 1% /sys/fs/cgroup /dev/sda1 0 0 0 - /boot/efi /dev/sda3 52428800 2887 52425913 1% /var /dev/sda7 277368832 35913 277332919 1% /local /dev/sda5 104857600 398 104857202 1% /tmp tmpfs 12174353 1 12174352 1% /run/user/551336 tmpfs 12174353 1 12174352 1% /run/user/0 moto 276971520 154807697 122163823 56% /cluster tmpfs 12174353 3 12174350 1% /run/user/441245 tmpfs 12174353 12 12174341 1% /run/user/553562 tmpfs 12174353 1 12174352 1% /run/user/525583 tmpfs 12174353 1 12174352 1% /run/user/476374 tmpfs 12174353 1 12174352 1% /run/user/468934 tmpfs 12174353 5 12174348 1% /run/user/551200 tmpfs 12174353 1 12174352 1% /run/user/539143 tmpfs 12174353 1 12174352 1% /run/user/488676 tmpfs 12174353 1 12174352 1% /run/user/493713 tmpfs 12174353 1 12174352 1% /run/user/507831 tmpfs 12174353 1 12174352 1% /run/user/549822 tmpfs 12174353 1 12174352 1% /run/user/500569 tmpfs 12174353 1 12174352 1% /run/user/443748 tmpfs 12174353 1 12174352 1% /run/user/543676 tmpfs 12174353 1 12174352 1% /run/user/451446 tmpfs 12174353 1 12174352 1% /run/user/497945 tmpfs 12174353 6 12174347 1% /run/user/554672 tmpfs 12174353 32 12174321 1% /run/user/554653 tmpfs 12174353 1 12174352 1% /run/user/30094 tmpfs 12174353 1 12174352 1% /run/user/470790 tmpfs 12174353 59 12174294 1% /run/user/553037 tmpfs 12174353 1 12174352 1% /run/user/554670 tmpfs 12174353 1 12174352 1% /run/user/548236 tmpfs 12174353 1 12174352 1% /run/user/547288 tmpfs 12174353 1 12174352 1% /run/user/547289 You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won?t have auto-inode expansion for the filesets. Do we have to restart any service after upping the inode count? Best, J.D. Maloney Lead HPC Storage Engineer | Storage Enabling Technologies Group National Center for Supercomputing Applications (NCSA) Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati! From: gpfsug-discuss > on behalf of Rob Kudyba > Date: Thursday, June 6, 2024 at 3:50?PM To: gpfsug-discuss at gpfsug.org > Subject: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks Running GPFS 4.2.3 on a DDN GridScaler and users are getting the No space left on device message when trying to write to a file. In /var/adm/ras/mmfs.log the only recent errors are this: 2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode). 2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1. 2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode). according to https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615 Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in /var. We have plenty of space left. /usr/lpp/mmfs/bin/mmlsdisk cluster disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------ S01_MDT200_1 nsd 4096 200 Yes No ready up system S01_MDT201_1 nsd 4096 201 Yes No ready up system S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 /usr/lpp/mmfs/bin/mmdf headnode disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 14 TB) S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%) S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%) ------------- -------------------- ------------------- (pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%) Disks in storage pool: data1 (Maximum disk size allowed is 578 TB) S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%) S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%) S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%) S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%) S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%) S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%) S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%) ------------- -------------------- ------------------- (pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) ============= ==================== =================== (data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%) (metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%) ============= ==================== =================== (total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 154807668 Total number of free inodes in all Inode spaces: 12964492 Total number of allocated inodes in all Inode spaces: 167772160 Total of Maximum number of inodes in all Inode spaces: 276971520 On the head node: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 430G 216G 215G 51% / devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 4.1G 43G 9% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/sda1 504M 114M 365M 24% /boot /dev/sda2 100M 9.9M 90M 10% /boot/efi x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share cluster 506T 405T 101T 81% /cluster tmpfs 9.3G 0 9.3G 0% /run/user/443748 tmpfs 9.3G 0 9.3G 0% /run/user/547288 tmpfs 9.3G 0 9.3G 0% /run/user/551336 tmpfs 9.3G 0 9.3G 0% /run/user/547289 The login nodes have plenty of space in /var: /dev/sda3 50G 8.7G 42G 18% /var What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Thu Jun 13 08:37:03 2024 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Thu, 13 Jun 2024 07:37:03 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: References: Message-ID: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Jun 13 10:32:23 2024 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 13 Jun 2024 09:32:23 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> Message-ID: Hi Walter - Create a new Rg on 3500 , and file system independently from the old environment Then set up an remote mount relationship and mount or use NFS ( if your release distance is too large for multi cluster ) Use AFM to fetch the data into the new file system You can do it by fileset pr for the whole file system at once ________________________________ Von: gpfsug-discuss im Auftrag von Walter Sklenka Gesendet: Thursday, June 13, 2024 8:37:03 AM An: gpfsug-discuss at gpfsug.org Betreff: [EXTERNAL] Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Thu Jun 13 10:50:26 2024 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Thu, 13 Jun 2024 09:50:26 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> Message-ID: <062bcd166d34466cbcfc6553bd44e6be@Mail.EDVDesign.cloudia> Hi Olaf! Thank you very much, so there is no ?dirty? way to do it within a cluster? Best regards walter Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at From: gpfsug-discuss On Behalf Of Olaf Weiser Sent: Donnerstag, 13. Juni 2024 11:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi Walter - Create a new Rg on 3500 , and file system independently from the old environment Then set up an remote mount relationship and mount or use NFS ( if your release distance is too large for multi cluster ) Use AFM to fetch the data into the new file system You can do it by fileset pr for the whole file system at once ________________________________ Von: gpfsug-discuss > im Auftrag von Walter Sklenka > Gesendet: Thursday, June 13, 2024 8:37:03 AM An: gpfsug-discuss at gpfsug.org > Betreff: [EXTERNAL] Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Thu Jun 13 13:57:39 2024 From: ewahl at osc.edu (Wahl, Edward) Date: Thu, 13 Jun 2024 12:57:39 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> Message-ID: If you are wanting seamless: Create a new pool in your existing file system, a new placement policy, and a use a movement policy to move the old data between the pools. The new placement policy will ensure new data is only written to the new location. Then when the old GL2 pool is empty you can remove it/offline it/whatever. Ed Wahl Ohio Supercomputer Center From: gpfsug-discuss On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 3:37 AM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Thu Jun 13 14:46:32 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 13 Jun 2024 14:46:32 +0100 Subject: [gpfsug-discuss] Ubuntu 24.04 Message-ID: Any indication when it might be supported? The reason I ask is that we have an issue with Microsoft DFS shares. Basically the University provides shared space which under the hood is a Hitachi VSP. We have code that lets the user mount this shared space to move data on/off the HPC system. The issue is two levels of DFS redirects(*) which works fine in RHEL7, but is broken in RHEL8, RHEL9, Ubuntu 20.04LTS and 22.04LTS. However today I took a punt and tried with a Ubuntu 24.04 LTS install and woohoo it works. So somewhere between 3.10 and 4.18 it was broken and not fixed till somewhere between 5.15 and 6.8. Obviously I only have a couple weeks left on RHEL7 but was going to flip the server we put in place to support this to CentOS 7 ELS from TuxCare while looking for a solution. So having now identified Ubuntu 24.04 as a potential solution I just need GPFS support. I presume that 24.04 will be supported in due course but just trying to gauge roughly how long, a month, six months etc. to make plans. JAB. * You mount the shared space which is a DFS share on a Windows 2019 server. You see a list of faculties. You change directory into the faculty and historically these where a share for each faculty on the VSP. However some faculties now store more data than can be accommodated on a single share from the VSP, so now when you change directory into the faculty the list of departments is another set of DFS redirects and trying to change into these does not work. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From Walter.Sklenka at EDV-Design.at Thu Jun 13 15:38:47 2024 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Thu, 13 Jun 2024 14:38:47 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> Message-ID: <60847e453f2f41fb9b92b6996131acad@Mail.EDVDesign.cloudia> Hi Ed! Thank you very much for the suggestion. So do you think I am able to create a new rg on ess3500 , create vdisks and nsds for new pool in ?GL4-provided filesystem? ? I still do not understand what the limitations are between 5.0.5 gpfs code and erase code and the new codes of 5.1.x , which prohibits me to create rgs or nsds in the same pool Best regards Walter Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at From: gpfsug-discuss On Behalf Of Wahl, Edward Sent: Donnerstag, 13. Juni 2024 14:58 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 If you are wanting seamless: Create a new pool in your existing file system, a new placement policy, and a use a movement policy to move the old data between the pools. The new placement policy will ensure new data is only written to the new location. Then when the old GL2 pool is empty you can remove it/offline it/whatever. Ed Wahl Ohio Supercomputer Center From: gpfsug-discuss > On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 3:37 AM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.rupp at us.ibm.com Thu Jun 13 16:06:49 2024 From: richard.rupp at us.ibm.com (Richard Rupp) Date: Thu, 13 Jun 2024 15:06:49 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: <60847e453f2f41fb9b92b6996131acad@Mail.EDVDesign.cloudia> References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> <60847e453f2f41fb9b92b6996131acad@Mail.EDVDesign.cloudia> Message-ID: If you bring the GL4 up to the current version 5.1.9.x and it is recommended to have the ESS3500 at that version, you can create vdisks, add them into the FS with the same pool names. An mmdelvdisk on the GL4 will then push the data over to the ESS3500 in a block style transfer which should be quicker than AFM. The GL4 needs to be completely working, with no bad drives, for this to work. Regards, Richard Rupp Senior Storage Technical Specialist, IBM Technology, US Public Market Cell: 347-510-6746 Webex: https://ibm.webex.com/join/richard.rupp From: gpfsug-discuss On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 10:39 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi Ed! Thank you very much for the suggestion. So do you think I am able to create a new rg on ess3500 , create vdisks and nsds for new pool in ?GL4-provided filesystem? ? I still do not understand what the limitations are between 5.?0.?5 gpfs Hi Ed! Thank you very much for the suggestion. So do you think I am able to create a new rg on ess3500 , create vdisks and nsds for new pool in ?GL4-provided filesystem? ? I still do not understand what the limitations are between 5.0.5 gpfs code and erase code and the new codes of 5.1.x , which prohibits me to create rgs or nsds in the same pool Best regards Walter Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at From: gpfsug-discuss > On Behalf Of Wahl, Edward Sent: Donnerstag, 13. Juni 2024 14:58 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 If you are wanting seamless: Create a new pool in your existing file system, a new placement policy, and a use a movement policy to move the old data between the pools. The new placement policy will ensure new data is only written to the new location. Then when the old GL2 pool is empty you can remove it/offline it/whatever. Ed Wahl Ohio Supercomputer Center From: gpfsug-discuss > On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 3:37 AM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Thu Jun 13 16:24:34 2024 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Thu, 13 Jun 2024 15:24:34 +0000 Subject: [gpfsug-discuss] ESS GL2 data to new ESS3500 In-Reply-To: References: <152d3b87e98e417fb6b8dc0c79dc2edc@Mail.EDVDesign.cloudia> <60847e453f2f41fb9b92b6996131acad@Mail.EDVDesign.cloudia> Message-ID: <0f66ded734f74afdbbc9de4130bc2370@Mail.EDVDesign.cloudia> Hi! My big mistake is, that I did not mention that the old GL2 is based on BE power systems, which I cannot upgrade any further ( it is at 5.0.5 ) So finally I really have to work with AFM or rsync , as a collegue already stated Thanks to all! Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at From: gpfsug-discuss On Behalf Of Richard Rupp Sent: Donnerstag, 13. Juni 2024 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 If you bring the GL4 up to the current version 5.1.9.x and it is recommended to have the ESS3500 at that version, you can create vdisks, add them into the FS with the same pool names. An mmdelvdisk on the GL4 will then push the data over to the ESS3500 in a block style transfer which should be quicker than AFM. The GL4 needs to be completely working, with no bad drives, for this to work. Regards, Richard Rupp Senior Storage Technical Specialist, IBM Technology, US Public Market Cell: 347-510-6746 Webex: https://ibm.webex.com/join/richard.rupp From: gpfsug-discuss > On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 10:39 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi Ed! Thank you very much for the suggestion. So do you think I am able to create a new rg on ess3500 , create vdisks and nsds for new pool in ?GL4-provided filesystem? ? I still do not understand what the limitations are between 5.?0.?5 gpfs Hi Ed! Thank you very much for the suggestion. So do you think I am able to create a new rg on ess3500 , create vdisks and nsds for new pool in ?GL4-provided filesystem? ? I still do not understand what the limitations are between 5.0.5 gpfs code and erase code and the new codes of 5.1.x , which prohibits me to create rgs or nsds in the same pool Best regards Walter Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at From: gpfsug-discuss > On Behalf Of Wahl, Edward Sent: Donnerstag, 13. Juni 2024 14:58 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 If you are wanting seamless: Create a new pool in your existing file system, a new placement policy, and a use a movement policy to move the old data between the pools. The new placement policy will ensure new data is only written to the new location. Then when the old GL2 pool is empty you can remove it/offline it/whatever. Ed Wahl Ohio Supercomputer Center From: gpfsug-discuss > On Behalf Of Walter Sklenka Sent: Thursday, June 13, 2024 3:37 AM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] ESS GL2 data to new ESS3500 Hi ! We want to migrate data from an old ESS GL2 ( 5.?0.?5 ) to ESS3500 solution. (5.?1.?x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member Hi ! We want to migrate data from an old ESS GL2 ( 5.0.5 ) to ESS3500 solution. (5.1.x) I wanted to ask is there any way to migrate the data with gpfs methods? We heard that you cannot create a rg on an ess3500 when it?s nodes become member of the old cluster because of GNR Version mismatch I thought I can create new rg on ess3500 , create vdisksets and nsds , then add the disks to a filesystem based on vdisks of old rgs I there really no way to transfer with gpfs methods? (like mmadddisk (new vdisks(nsds) of 3500 , and then mmdeldisk the old nsds(vdisks) So if you run into this situation you have to create a new cluster? Mit freundlichen Gr??en Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: sklenka at edv-design.at Internet: www.edv-design.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Thu Jun 20 21:14:09 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 20 Jun 2024 21:14:09 +0100 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G Message-ID: So came to light because I was checking the mmbackup logs and found that we had not been getting any successful backups for several days and seeing lots of errors like this Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Error on gpfs_iopen([/gpfs/users/xxxyyyyy/.swr],68050746): Stale file handle Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Summary of errors:: _dirscan failures:3, _serious unclassified errors:3. After some digging around wondering what was going on I came across these being logged on one of the DSS-G nodes [Wed Jun 12 22:22:05 2024] blk_update_request: I/O error, dev sdbv, sector 9144672512 op 0x1:(WRITE) flags 0x700 phys_seg 17 prio class 0 Yikes looks like I have a failed disk/ However if I do [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group all --not-ok mmvdisk: All pdisks are ok. Clearly that's a load of rubbish. After a lot more prodding [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group dssg2 --pdisk e1d2s25 -L pdisk: replacementPriority = 1000 name = "e1d2s25" device = "//gpfs1/dev/sdft(notEnabled),//gpfs1/dev/sdfu(notEnabled),//gpfs2/dev/sdfb,//gpfs2/dev/sdbv" recoveryGroup = "dssg2" declusteredArray = "DA1" state = "ok" IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 What on earth gives? Why has the disk not been failed? It's not great that a clearly bad disk is allowed to stick around in the file system and cause problems IMHO. When I try and prepare the disk for removal I get [root at gpfs2 ~]# mmvdisk pdisk replace --prepare --rg dssg2 --pdisk e1d2s25 mmvdisk: Pdisk e1d2s25 of recovery group dssg2 is not currently scheduled for replacement. mmvdisk: mmvdisk: mmvdisk: Command failed. Examine previous error messages to determine cause. Do I have to use the --force option? I would like to get this disk out the file system ASAP. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From stockf at us.ibm.com Thu Jun 20 22:02:43 2024 From: stockf at us.ibm.com (Fred Stock) Date: Thu, 20 Jun 2024 21:02:43 +0000 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: References: Message-ID: I think you are seeing two different errors. The backup is failing due to a stale file handle error which usually means the file system was unmounted while the file handle was open. The write error on the physical disk, may have contributed to the stale file handle but I doubt that is the case. As I understand a single IO error on a physical disk in an ESS (DSS) system will not cause the disk to be considered bad. This is likely why the system considers the disk to be ok. I suggest you track down the source of the stale file handle and correct that issue to see if your backups will then again be successful. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From: gpfsug-discuss on behalf of Jonathan Buzzard Date: Thursday, June 20, 2024 at 4:16?PM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] [gpfsug-discuss] Bad disk but not failed in DSS-G So came to light because I was checking the mmbackup logs and found that we had not been getting any successful backups for several days and seeing lots of errors like this Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Error on gpfs_iopen([/gpfs/users/xxxyyyyy/.swr],68050746): Stale file handle Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Summary of errors:: _dirscan failures:3, _serious unclassified errors:3. After some digging around wondering what was going on I came across these being logged on one of the DSS-G nodes [Wed Jun 12 22:22:05 2024] blk_update_request: I/O error, dev sdbv, sector 9144672512 op 0x1:(WRITE) flags 0x700 phys_seg 17 prio class 0 Yikes looks like I have a failed disk/ However if I do [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group all --not-ok mmvdisk: All pdisks are ok. Clearly that's a load of rubbish. After a lot more prodding [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group dssg2 --pdisk e1d2s25 -L pdisk: replacementPriority = 1000 name = "e1d2s25" device = "//gpfs1/dev/sdft(notEnabled),//gpfs1/dev/sdfu(notEnabled),//gpfs2/dev/sdfb,//gpfs2/dev/sdbv" recoveryGroup = "dssg2" declusteredArray = "DA1" state = "ok" IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 What on earth gives? Why has the disk not been failed? It's not great that a clearly bad disk is allowed to stick around in the file system and cause problems IMHO. When I try and prepare the disk for removal I get [root at gpfs2 ~]# mmvdisk pdisk replace --prepare --rg dssg2 --pdisk e1d2s25 mmvdisk: Pdisk e1d2s25 of recovery group dssg2 is not currently scheduled for replacement. mmvdisk: mmvdisk: mmvdisk: Command failed. Examine previous error messages to determine cause. Do I have to use the --force option? I would like to get this disk out the file system ASAP. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Thu Jun 20 23:32:26 2024 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Thu, 20 Jun 2024 22:32:26 +0000 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: References: Message-ID: Fred is most probably correct here. the two errors are not necessarily the same. i would guess that looking at # mmlsreoverygroupevents dssg2 or # mmvdisk recoverygroup list --recovery-group dssg2 --events you would see e1d2s25 multiple times, changing its state from ok to diagnosing, and back to ok If you feel this is recurring too often (and i tend to agree, given the number of IOErrors, you can always '--simulate-failing' this pdisk , and replace it # mmvdisk pdisk change --recovery-group dssg2 --pdisk e1d2s25 --simulate-failing -- Mit freundlichen Gr??en / Kind regards Achim Rehor Technical Support Specialist S?pectrum Scale and ESS (SME) Advisory Product Services Professional IBM Systems Storage Support - EMEA Achim.Rehor at de.ibm.com +49-170-4521194 IBM Deutschland GmbH Vorsitzender des Aufsichtsrats: Sebastian Krause Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Nicole Reimer, Gabriele Schwarenthorer, Christine Rupp, Frank Theisen Sitz der Gesellschaft: Ehningen / Registergericht: AmtsgerichtStuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -----Original Message----- From: Fred Stock Reply-To: gpfsug main discussion list To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Bad disk but not failed in DSS-G Date: Thu, 20 Jun 2024 21:02:43 +0000 I think you are seeing two different errors. The backup is failing due to a stale file handle error which usually means the file system was unmounted while the file handle was open. The write error on the physical disk, may have contributed #pfptBannerqlqwaj3 { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonqlqwaj3:hover, .pfptPrimaryButtonqlqwaj3:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonqlqwaj3:active { background-color: #90a4ae !important; } I think you are seeing two different errors. The backup is failing due to a stale file handle error which usually means the file system was unmounted while the file handle was open. The write error on the physical disk, may have contributed to the stale file handle but I doubt that is the case. As I understand a single IO error on a physical disk in an ESS (DSS) system will not cause the disk to be considered bad. This is likely why the system considers the disk to be ok. I suggest you track down the source of the stale file handle and correct that issue to see if your backups will then again be successful. Fred Fred Stock, Spectrum Scale Development Advocacy stockf at us.ibm.com | 720-430-8821 From:gpfsug-discuss on behalf of Jonathan Buzzard Date: Thursday, June 20, 2024 at 4:16?PM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] [gpfsug-discuss] Bad disk but not failed in DSS-G So came to light because I was checking the mmbackup logs and found that we had not been getting any successful backups for several days and seeing lots of errors like this Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Error on gpfs_iopen([/gpfs/users/xxxyyyyy/.swr],68050746): Stale file handle Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] Summary of errors:: _dirscan failures:3, _serious unclassified errors:3. After some digging around wondering what was going on I came across these being logged on one of the DSS-G nodes [Wed Jun 12 22:22:05 2024] blk_update_request: I/O error, dev sdbv, sector 9144672512 op 0x1:(WRITE) flags 0x700 phys_seg 17 prio class 0 Yikes looks like I have a failed disk/ However if I do [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group all --not-ok mmvdisk: All pdisks are ok. Clearly that's a load of rubbish. After a lot more prodding [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group dssg2 --pdisk e1d2s25 -L pdisk: replacementPriority = 1000 name = "e1d2s25" device = "//gpfs1/dev/sdft(notEnabled),//gpfs1/dev/sdfu(notEnabled),//gpfs2/dev/sdfb,//gpfs2/dev/sdbv" recoveryGroup = "dssg2" declusteredArray = "DA1" state = "ok" IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 What on earth gives? Why has the disk not been failed? It's not great that a clearly bad disk is allowed to stick around in the file system and cause problems IMHO. When I try and prepare the disk for removal I get [root at gpfs2 ~]# mmvdisk pdisk replace --prepare --rg dssg2 --pdisk e1d2s25 mmvdisk: Pdisk e1d2s25 of recovery group dssg2 is not currently scheduled for replacement. mmvdisk: mmvdisk: mmvdisk: Command failed. Examine previous error messages to determine cause. Do I have to use the --force option? I would like to get this disk out the file system ASAP. JAB. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Thu Jun 20 23:35:07 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 20 Jun 2024 23:35:07 +0100 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: References: Message-ID: <30ffbbcd-d7cd-4ca6-82b8-e37a30be272b@strath.ac.uk> On 20/06/2024 22:02, Fred Stock wrote: > > I think you are seeing two different errors.? The backup is failing due > to a stale file handle error which usually means the file system was > unmounted while the file handle was open.? The write error on the > physical disk, may have contributed to the stale file handle but I doubt > that is the case.? As I understand a single IO error on a physical disk > in an ESS (DSS) system will not cause the disk to be considered bad. > This is likely why the system considers the disk to be ok.? I suggest > you track down the source of the stale file handle and correct that > issue to see if your backups will then again be successful. > There is a *lot* more than a single IO error on the physical disk, the output of mmvdisk pdisk list for the disk shows IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 And the output of dmesg shows loads of errors. I have not attempted to count them but it is again a *lot* more than a single IO error. That disk should have been kicked out the file system and the fact that it has not is a bug IMHO. Anyone who thinks that is "normal" and not "failed" is as high as a kite. Also mmbackup has now failed for three days in a row with different stale file handles building the change lists, making this is an on going issue. So can I safely use the --force to get this dodgy disk out the file system? It is the *only* disk in the system showing IO errors so almost certainly the cause of the problems. Unless you are aware of some Linux kernel bug that causes otherwise healthy disks in an enclosure to start having problems. I guess there is an outside chance there could be an issue with the enclosure but really you start with the disk. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From LJHenson at mdanderson.org Thu Jun 20 23:41:45 2024 From: LJHenson at mdanderson.org (Henson Jr.,Larry J) Date: Thu, 20 Jun 2024 22:41:45 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: Bad disk but not failed in DSS-G In-Reply-To: <30ffbbcd-d7cd-4ca6-82b8-e37a30be272b@strath.ac.uk> References: <30ffbbcd-d7cd-4ca6-82b8-e37a30be272b@strath.ac.uk> Message-ID: Just curious what is output of 'mmlspdisk all --not-ok'? Regards, Larry Henson MD Anderson Cancer Center IT Engineering Storage Team Cell (713) 702-4896 -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: Thursday, June 20, 2024 5:35 PM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Bad disk but not failed in DSS-G SLOW DOWN! - EXTERNAL SENDER: gpfsug-discuss-bounces at gpfsug.org Be suspicious of tone, urgency, and formatting. Do not click/open links or attachments on a mobile device. Wait until you are at a computer to confirm you are absolutely certain it is a trusted source. If you are at all uncertain use the Report Phish button and our Cybersecurity team will investigate. On 20/06/2024 22:02, Fred Stock wrote: > > I think you are seeing two different errors. The backup is failing > due to a stale file handle error which usually means the file system > was unmounted while the file handle was open. The write error on the > physical disk, may have contributed to the stale file handle but I > doubt that is the case. As I understand a single IO error on a > physical disk in an ESS (DSS) system will not cause the disk to be considered bad. > This is likely why the system considers the disk to be ok. I suggest > you track down the source of the stale file handle and correct that > issue to see if your backups will then again be successful. > There is a *lot* more than a single IO error on the physical disk, the output of mmvdisk pdisk list for the disk shows IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 And the output of dmesg shows loads of errors. I have not attempted to count them but it is again a *lot* more than a single IO error. That disk should have been kicked out the file system and the fact that it has not is a bug IMHO. Anyone who thinks that is "normal" and not "failed" is as high as a kite. Also mmbackup has now failed for three days in a row with different stale file handles building the change lists, making this is an on going issue. So can I safely use the --force to get this dodgy disk out the file system? It is the *only* disk in the system showing IO errors so almost certainly the cause of the problems. Unless you are aware of some Linux kernel bug that causes otherwise healthy disks in an enclosure to start having problems. I guess there is an outside chance there could be an issue with the enclosure but really you start with the disk. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!PfbeBCCAmug!lwWt17RQZQce8WbKYm78Lpbbl6YfCk8XYVIULodmqIdCLz2mWjii9-USsOcS-mNYfRGBUxjx-hs-vGs31dd4_eQY7nUgZFd7WA$ The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. From timm.stamer at uni-oldenburg.de Fri Jun 21 06:50:04 2024 From: timm.stamer at uni-oldenburg.de (Timm Stamer) Date: Fri, 21 Jun 2024 05:50:04 +0000 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: References: Message-ID: Hi JAB, You have to run "... change --simulate-dead" or "... --simulate- failing" to get the disk out of the system. Afterwards you can start replacement procedure. mmvdisk pdisk change --recovery-group dssg2 --pdisk e1d2s25 --simulate- dead Kind regards Timm Stamer Am Donnerstag, dem 20.06.2024 um 21:14 +0100 schrieb Jonathan Buzzard: > > So came to light because I was checking the mmbackup logs and found > that > we had not been getting any successful backups for several days and > seeing lots of errors like this > > Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: > [E] > Error on gpfs_iopen([/gpfs/users/xxxyyyyy/.swr],68050746): Stale file > handle > Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: > [E] > Summary of errors:: _dirscan failures:3, _serious unclassified > errors:3. > > After some digging around wondering what was going on I came across > these being logged on one of the DSS-G nodes > > [Wed Jun 12 22:22:05 2024] blk_update_request: I/O error, dev sdbv, > sector 9144672512 op 0x1:(WRITE) flags 0x700 phys_seg 17 prio class 0 > > Yikes looks like I have a failed disk/ However if I do > > [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group all --not-ok > mmvdisk: All pdisks are ok. > > Clearly that's a load of rubbish. > > After a lot more prodding > > [root at gpfs2 ~]# mmvdisk pdisk list --recovery-group dssg2 --pdisk > e1d2s25 -L > pdisk: > ??? replacementPriority = 1000 > ??? name = "e1d2s25" > ??? device = > "//gpfs1/dev/sdft(notEnabled),//gpfs1/dev/sdfu(notEnabled),//gpfs2/de > v/sdfb,//gpfs2/dev/sdbv" > ??? recoveryGroup = "dssg2" > ??? declusteredArray = "DA1" > ??? state = "ok" > ??? IOErrors = 444 > ??? IOTimeouts = 8958 > ??? mediaErrors = 15 > > > What on earth gives? Why has the disk not been failed? It's not great > that a clearly bad disk is allowed to stick around in the file system > and cause problems IMHO. > > When I try and prepare the disk for removal I get > > [root at gpfs2 ~]# mmvdisk pdisk replace --prepare --rg dssg2 --pdisk > e1d2s25 > mmvdisk: Pdisk e1d2s25 of recovery group dssg2 is not currently > scheduled for replacement. > mmvdisk: > mmvdisk: > mmvdisk: Command failed. Examine previous error messages to determine > cause. > > Do I have to use the --force option? I would like to get this disk > out > the file system ASAP. > > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7667 bytes Desc: not available URL: From NSCHULD at de.ibm.com Fri Jun 21 07:57:34 2024 From: NSCHULD at de.ibm.com (Norbert Schuld) Date: Fri, 21 Jun 2024 06:57:34 +0000 Subject: [gpfsug-discuss] Ubuntu 24.04 In-Reply-To: References: Message-ID: Current plan is to support Ubuntu 24.04 with our 4Q release later this year. Mit freundlichen Gr??en / Kind regards Norbert Schuld Software Engineer, Release Architect IBM Storage Scale IBM Systems / 00E636 -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: Thursday, June 13, 2024 3:47 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Ubuntu 24.04 Any indication when it might be supported? The reason I ask is that we have an issue with Microsoft DFS shares. Basically the University provides shared space which under the hood is a Hitachi VSP. We have code that lets the user mount this shared space to move data on/off the HPC system. The issue is two levels of DFS redirects(*) which works fine in RHEL7, but is broken in RHEL8, RHEL9, Ubuntu 20.04LTS and 22.04LTS. However today I took a punt and tried with a Ubuntu 24.04 LTS install and woohoo it works. So somewhere between 3.10 and 4.18 it was broken and not fixed till somewhere between 5.15 and 6.8. Obviously I only have a couple weeks left on RHEL7 but was going to flip the server we put in place to support this to CentOS 7 ELS from TuxCare while looking for a solution. So having now identified Ubuntu 24.04 as a potential solution I just need GPFS support. I presume that 24.04 will be supported in due course but just trying to gauge roughly how long, a month, six months etc. to make plans. JAB. * You mount the shared space which is a DFS share on a Windows 2019 server. You see a list of faculties. You change directory into the faculty and historically these where a share for each faculty on the VSP. However some faculties now store more data than can be accommodated on a single share from the VSP, so now when you change directory into the faculty the list of departments is another set of DFS redirects and trying to change into these does not work. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From ncalimet at lenovo.com Fri Jun 21 08:32:09 2024 From: ncalimet at lenovo.com (Nicolas CALIMET) Date: Fri, 21 Jun 2024 07:32:09 +0000 Subject: [gpfsug-discuss] [External] Re: Bad disk but not failed in DSS-G In-Reply-To: <30ffbbcd-d7cd-4ca6-82b8-e37a30be272b@strath.ac.uk> References: <30ffbbcd-d7cd-4ca6-82b8-e37a30be272b@strath.ac.uk> Message-ID: The disk hospital sometimes might be a bit too conservative - or not enough in this case. What is the SMART status of this drive? Does mmhealth report anything different than the mmvdisk or mmlspdisk commands? In DSS-G the health monitor (dssghealthmon) and diskIOHang (dssgdiskIOHang) systems may help detect such a problem, the first by polling multiple metrics at regular time interval (for drives, mostly around mmlspdisk), the second by power cycling a hung drive (as signaled by the GNR diskIOHang callback, which in this case might not get triggered though). In recent DSS-G releases, checking drives is complemented by a SMART overall health status that does help detect faulty drives. Regarding the error from mmvdisk to prepare for drive replacement, and provided the drive is clearly at fault and may already be unreachable anymore, then I would definitely force remove it. That cannot be worse than keeping a bad drive that is still erroneously considered healthy. HTH -- Nicolas Calimet, PhD | HPC System Architect | Lenovo ISG | Meitnerstrasse 9, D-70563 Stuttgart, Germany | +49 71165690146 | https://www.lenovo.com/dssg -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: Friday, June 21, 2024 00:35 To: gpfsug-discuss at gpfsug.org Subject: [External] Re: [gpfsug-discuss] Bad disk but not failed in DSS-G On 20/06/2024 22:02, Fred Stock wrote: > > I think you are seeing two different errors. The backup is failing due > to a stale file handle error which usually means the file system was > unmounted while the file handle was open. The write error on the > physical disk, may have contributed to the stale file handle but I doubt > that is the case. As I understand a single IO error on a physical disk > in an ESS (DSS) system will not cause the disk to be considered bad. > This is likely why the system considers the disk to be ok. I suggest > you track down the source of the stale file handle and correct that > issue to see if your backups will then again be successful. > There is a *lot* more than a single IO error on the physical disk, the output of mmvdisk pdisk list for the disk shows IOErrors = 444 IOTimeouts = 8958 mediaErrors = 15 And the output of dmesg shows loads of errors. I have not attempted to count them but it is again a *lot* more than a single IO error. That disk should have been kicked out the file system and the fact that it has not is a bug IMHO. Anyone who thinks that is "normal" and not "failed" is as high as a kite. Also mmbackup has now failed for three days in a row with different stale file handles building the change lists, making this is an on going issue. So can I safely use the --force to get this dodgy disk out the file system? It is the *only* disk in the system showing IO errors so almost certainly the cause of the problems. Unless you are aware of some Linux kernel bug that causes otherwise healthy disks in an enclosure to start having problems. I guess there is an outside chance there could be an issue with the enclosure but really you start with the disk. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From jonathan.buzzard at strath.ac.uk Mon Jun 24 10:41:50 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 24 Jun 2024 10:41:50 +0100 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: References: Message-ID: <82308b7e-9872-42f6-902f-850596840a4a@strath.ac.uk> On 20/06/2024 23:32, Achim Rehor wrote: [SNIP] > Fred is most probably correct here. the two errors are not necessarily > the same. > Turns out Fred was incorrect and having pushed the bad disk out the file system the backups magically started working again. Not that, that should come as the slightest surprise to anyone. However finding I have a bad disk because the backups are failing is not good at all because it means I can't rely on GPFS's health monitoring to accurately report the state of the file system :-( It also begs the question with hundreds of I/O errors on a disk why was it not failed by GPFS? What criteria does GPFS use for deciding if a disk is bad as clearly they are not accurate. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From Achim.Rehor at de.ibm.com Mon Jun 24 13:16:52 2024 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Mon, 24 Jun 2024 12:16:52 +0000 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: <82308b7e-9872-42f6-902f-850596840a4a@strath.ac.uk> References: <82308b7e-9872-42f6-902f-850596840a4a@strath.ac.uk> Message-ID: <6103c18f3945d39c1950a9be759408ad15b0014b.camel@de.ibm.com> well ... not necessarily ? but on the disk ... just as i expected ... taking it out helps a lot. Now on taking it out automatically when raising too many errors was a discussion i had several times with the GNR development. The issue really is .. I/O errors on disks (as seen in the mmlsrecoverygroupevent logs) can be due to several issues (the disk itself, the expander, the IOM, the adapter, the cable ... ) in case of a more general part serving like 5 or more pdisks, that would risk the FT , if we took them out automatically. Thus ... we dont do that .. The idea is to improve the disk hospital more and more, so that the decision to switch a disk back to OK is more accurate, over time. Until then .. it might always be a good idea to scan the event log for pdisk errors ... -- Mit freundlichen Gr??en / Kind regards Achim Rehor -----Original Message----- From: Jonathan Buzzard > Reply-To: gpfsug main discussion list > To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Bad disk but not failed in DSS-G Date: Mon, 24 Jun 2024 10:41:50 +0100 On 20/06/2024 23:32, Achim Rehor wrote: [SNIP] Fred is most probably correct here. the two errors are not necessarily the same. Turns out Fred was incorrect and having pushed the bad disk out the file system the backups magically started working again. Not that, that should come as the slightest surprise to anyone. However finding I have a bad disk because the backups are failing is not good at all because it means I can't rely on GPFS's health monitoring to accurately report the state of the file system :-( It also begs the question with hundreds of I/O errors on a disk why was it not failed by GPFS? What criteria does GPFS use for deciding if a disk is bad as clearly they are not accurate. JAB. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jun 24 13:51:59 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 24 Jun 2024 13:51:59 +0100 Subject: [gpfsug-discuss] Bad disk but not failed in DSS-G In-Reply-To: <6103c18f3945d39c1950a9be759408ad15b0014b.camel@de.ibm.com> References: <82308b7e-9872-42f6-902f-850596840a4a@strath.ac.uk> <6103c18f3945d39c1950a9be759408ad15b0014b.camel@de.ibm.com> Message-ID: On 24/06/2024 13:16, Achim Rehor wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > well ... not necessarily ? > but on the disk ... just as i expected ... taking it out helps a lot. > > Now on taking it out automatically when raising too many errors was a > discussion i had several times with the GNR development. > The issue really is .. I/O errors on disks (as seen in the > mmlsrecoverygroupevent logs) can be due to several issues ?(the disk > itself, > the expander,?the IOM, the adapter, the cable ... ) > in case of a more general part serving like 5 or more pdisks, that would > risk the FT , if we took them out automatically. > Thus ... we dont do that .. > When smartctl for the disk says Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 33839 32 0 0 137434.705 32 write: 0 36 0 0 0 178408.893 0 Non-medium error count: 0 A disk with 32 read errors in smartctl is fubar, no ifs no buts. Whatever the balance in ejecting bad disks is, IMHO currently it's in the wrong place because it failed to eject an actual bad disk. At an absolute bare minimum mmhealth should be not be saying everything is fine and dandy because clearly it was not. That's the bigger issue. I can live with them not been taken out automatically, it is unacceptable that mmhealth was giving false and inaccurate information about the state of the filesystem. Had it even just changed something to a "degraded" state the problems could have been picked up much much sooner. Presumably the disk category was still good because the vdisk's where theoretically good. I suggest renaming that to VDISK to more accurately reflect what it is about and add a PDISK category. Then when a pdisk starts showing IO errors you can increment the number of disks in a degraded state and it can be picked up without end users having to roll their own monitoring. > The idea is to improve the disk hospital more and more, so that the > decision to switch a disk back to OK is more accurate, ? over time. > > Until then .. it might always be a good idea to scan the event log for > pdisk errors ... > That is my conclusion, that mmhealth is as useful as a chocolate teapot because you can't rely on it to provide correct information and I need to do my own health monitoring of the system. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Fri Jun 28 14:05:32 2024 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 28 Jun 2024 14:05:32 +0100 Subject: [gpfsug-discuss] DSS-G V5 and GUI Message-ID: <1603f202-a548-44d9-a9ae-01d07e3fd4eb@strath.ac.uk> So I skipped the v5.0a release because that only supported the new V3 SR650's. However I have finished the "get off CentOS 7" project (apart from the one server that is now on TuxCare ELS awaiting Ubuntu 24.04 support in GPFS) and so now have the time to look into v5.0b which does support the older V1 and V2 SR650's. However in the release notes I read this The Storage Scale GUI is not supported with DSS-G 5.0. Now while I could not care less about the GUI per se, you need it for the RestAPI which I really *do* care about. Please tell me that Lenovo have not just yanked this. I do notice that there are still RPM's for it but really what on earth is going on? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG