From pavel.pokorny at datera.cz Fri Jan 1 19:22:15 2016 From: pavel.pokorny at datera.cz (Pavel Pokorny) Date: Fri, 1 Jan 2016 20:22:15 +0100 Subject: [gpfsug-discuss] HAWC compare to regular pagepool Message-ID: Hello, I would like to ask whether I understand HAWC functionality correctly especially compare to regular pagepool behavior. Regular pagepool behavior: 1. Application on a node makes write call. 2. Data is moved from application data buffer to page pool buffer. 3. If not using direct IO (open with O_DIRECT or set the -D attribute); at this point, the application has completed the write system call and GPFS acknowledge write. My understanding is that with HAWC this behavior is very similar with the difference that at step 3 data are also stored in nonvolatile storage on client. Am I correct? This will mean that HAWC is giving us mainly data hardening and for direct writes also performance. Thanks, Pavel -- Ing. Pavel Pokorn? DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 1 19:57:16 2016 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 1 Jan 2016 11:57:16 -0800 Subject: [gpfsug-discuss] HAWC compare to regular pagepool In-Reply-To: References: Message-ID: Hi Pavel, HAWC is only used for stable buffered i/os, so for example if you have an application that opens files with O_SYNC instead of writing it directly to shared storage the data gets 'hardened' in the HAWC device which is typically some form of smaller , but very fast NV storage device like a NVRAM, SSD or shared FLASH DEVICE. the acknowledgment to the application happens as soon as its in the HAWC device, so it can't get lost even if you loose power. this is particular useful for database application, when you run VM's on top of GPFS or other workloads that primarily perform small stable writes. sven On Fri, Jan 1, 2016 at 11:22 AM, Pavel Pokorny wrote: > Hello, > I would like to ask whether I understand HAWC functionality correctly > especially compare to regular pagepool behavior. > > Regular pagepool behavior: > > 1. Application on a node makes write call. > 2. Data is moved from application data buffer to page pool buffer. > 3. If not using direct IO (open with O_DIRECT or set the -D > attribute); at this point, the application has completed the write system > call and GPFS acknowledge write. > > My understanding is that with HAWC this behavior is very similar with the > difference that at step 3 data are also stored in nonvolatile storage on > client. > Am I correct? This will mean that HAWC is giving us mainly data hardening > and for direct writes also performance. > Thanks, Pavel > > -- > Ing. Pavel Pokorn? > DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic > www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 1 19:57:16 2016 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 1 Jan 2016 11:57:16 -0800 Subject: [gpfsug-discuss] HAWC compare to regular pagepool In-Reply-To: References: Message-ID: Hi Pavel, HAWC is only used for stable buffered i/os, so for example if you have an application that opens files with O_SYNC instead of writing it directly to shared storage the data gets 'hardened' in the HAWC device which is typically some form of smaller , but very fast NV storage device like a NVRAM, SSD or shared FLASH DEVICE. the acknowledgment to the application happens as soon as its in the HAWC device, so it can't get lost even if you loose power. this is particular useful for database application, when you run VM's on top of GPFS or other workloads that primarily perform small stable writes. sven On Fri, Jan 1, 2016 at 11:22 AM, Pavel Pokorny wrote: > Hello, > I would like to ask whether I understand HAWC functionality correctly > especially compare to regular pagepool behavior. > > Regular pagepool behavior: > > 1. Application on a node makes write call. > 2. Data is moved from application data buffer to page pool buffer. > 3. If not using direct IO (open with O_DIRECT or set the -D > attribute); at this point, the application has completed the write system > call and GPFS acknowledge write. > > My understanding is that with HAWC this behavior is very similar with the > difference that at step 3 data are also stored in nonvolatile storage on > client. > Am I correct? This will mean that HAWC is giving us mainly data hardening > and for direct writes also performance. > Thanks, Pavel > > -- > Ing. Pavel Pokorn? > DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic > www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dacalder at co.ibm.com Sun Jan 3 15:55:59 2016 From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez) Date: Sun, 3 Jan 2016 15:55:59 +0000 Subject: [gpfsug-discuss] Resource exhausted by Pool Migration Message-ID: <201601031556.u03Futfw007019@d24av01.br.ibm.com> HI All Actually I have a 4.2 Spectrum Scale cluster with protocol service, we are managing small files (32K to 140K), when I try to migrate some files (120.000 files ) the system resources of all nodes is exhausted and the protocol nodes don't get services to client. I wan to ask if there is any way to limit the resources consuming at the migration time? Thanks to all Enviado desde IBM Verse -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Sun Jan 3 16:42:27 2016 From: oehmes at gmail.com (Sven Oehme) Date: Sun, 3 Jan 2016 08:42:27 -0800 Subject: [gpfsug-discuss] Resource exhausted by Pool Migration In-Reply-To: <201601031556.u03Futfw007019@d24av01.br.ibm.com> References: <201601031556.u03Futfw007019@d24av01.br.ibm.com> Message-ID: Hi Danny, can you be a bit more specific, which resources get exhausted ? are you talking about operating system or Spectrum Scale resources (filecache or pagepool) ? when you migrate the files ( i assume policy engine) did you specify which nodes do the migration ( -N hostnames) or did you just run mmapplypolicy without anything ? can you post either your entire mmlsconfig or at least output of : for i in maxFilesToCache pagepool maxStatCache nsdMinWorkerThreads nsdMaxWorkerThreads worker1Threads; do mmlsconfig $i ; done mmlsfs , mmlsnsd and mmlscluster output might be useful too.. sven On Sun, Jan 3, 2016 at 7:55 AM, Danny Alexander Calderon Rodriguez < dacalder at co.ibm.com> wrote: > HI All > > Actually I have a 4.2 Spectrum Scale cluster with protocol service, we are managing small files (32K to 140K), when I try to migrate some files (120.000 files ) the system resources of all nodes is exhausted and the protocol nodes don't get services to client. > > I wan to ask if there is any way to limit the resources consuming at the migration time? > > > Thanks to all > > > > Enviado desde IBM Verse > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Sun Jan 3 21:56:26 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Sun, 3 Jan 2016 21:56:26 +0000 Subject: [gpfsug-discuss] metadata replication question Message-ID: I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. Id like to add some ssd for metadata. Should I: Add raid1 ssd to the storewise? Add local ssd to the nsd servers? If I did the second, should I add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? Or something else entirely? What I want so survive is a split data centre situation or failure of a single nsd server at any point... Thoughts? Comments? I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? Id like a cost competitive solution that gives faster performance than the current sas drives. Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... Thanks Simon From bevans at pixitmedia.com Sun Jan 3 22:10:21 2016 From: bevans at pixitmedia.com (Barry Evans) Date: Sun, 3 Jan 2016 22:10:21 +0000 Subject: [gpfsug-discuss] metadata replication question In-Reply-To: References: Message-ID: <56899C4D.4050907@pixitmedia.com> Can all 4 NSD servers see all existing storwize arrays across both DC's? Cheers, Barry On 03/01/2016 21:56, Simon Thompson (Research Computing - IT Services) wrote: > I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. > > Id like to add some ssd for metadata. > > Should I: > > Add raid1 ssd to the storewise? > > Add local ssd to the nsd servers? > > If I did the second, should I > add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. > add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? > add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? > > Or something else entirely? > > What I want so survive is a split data centre situation or failure of a single nsd server at any point... > > Thoughts? Comments? > > I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? > > Id like a cost competitive solution that gives faster performance than the current sas drives. > > Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... > > Thanks > > Simon > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Barry Evans Technical Director & Co-Founder Pixit Media Mobile: +44 (0)7950 666 248 http://www.pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Sun Jan 3 22:18:24 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Sun, 3 Jan 2016 22:18:24 +0000 Subject: [gpfsug-discuss] metadata replication question In-Reply-To: <56899C4D.4050907@pixitmedia.com> References: , <56899C4D.4050907@pixitmedia.com> Message-ID: Yes there is extended san in place. The failure groups for the storage are different in each dc so we guarantee that the data replication has 1 copy per dc. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Barry Evans [bevans at pixitmedia.com] Sent: 03 January 2016 22:10 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] metadata replication question Can all 4 NSD servers see all existing storwize arrays across both DC's? Cheers, Barry On 03/01/2016 21:56, Simon Thompson (Research Computing - IT Services) wrote: I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. Id like to add some ssd for metadata. Should I: Add raid1 ssd to the storewise? Add local ssd to the nsd servers? If I did the second, should I add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? Or something else entirely? What I want so survive is a split data centre situation or failure of a single nsd server at any point... Thoughts? Comments? I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? Id like a cost competitive solution that gives faster performance than the current sas drives. Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Barry Evans Technical Director & Co-Founder Pixit Media Mobile: +44 (0)7950 666 248 http://www.pixitmedia.com [http://www.pixitmedia.com/sig/sig-cio.jpg] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. From dacalder at co.ibm.com Mon Jan 4 16:50:45 2016 From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez) Date: Mon, 4 Jan 2016 11:50:45 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 48, Issue 2 In-Reply-To: References: Message-ID: <201601041651.u04Gowuu012502@d24av03.br.ibm.com> Hi Danny, can you be a bit more specific, which resources get exhausted ? are you talking about operating system or Spectrum Scale resources (filecache or pagepool) ? when you migrate the files ( i assume policy engine) did you specify which nodes do the migration ( -N hostnames) or did you just run mmapplypolicy without anything ? can you post either your entire mmlsconfig or at least output of : for i in maxFilesToCache pagepool maxStatCache nsdMinWorkerThreads nsdMaxWorkerThreads worker1Threads; do mmlsconfig $i ; done mmlsfs , mmlsnsd and mmlscluster output might be useful too.. Hi Sven Sure. the resorces that are exhausted are CPU and RAM, i can note that when the system is make the Pool migration the SMB service is down ( very slow equal to down) when i migrate the files i was make probes with one node, two nodes (nsd nodes) and all nodes ( Nsd and protocol nodes) there is the output maxFilesToCache 1000000 pagepool 20G maxStatCache 1000 nsdMinWorkerThreads 8 nsdMinWorkerThreads 1 [cesNodes] nsdMaxWorkerThreads 512 nsdMaxWorkerThreads 2 [cesNodes] worker1Threads 48 worker1Threads 800 [cesNodes] mmlsfs ( there is two file system one for CES Shared Root and another for data ) le system attributes for /dev/datafs: ======================================= flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment size in bytes -i 4096 Inode size in bytes -I 16384 Indirect block size in bytes -m 1 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k nfs4 ACL semantics in effect -n 32 Estimated number of nodes that will mount file system -B 262144 Block size -Q none Quotas accounting enabled none Quotas enforced none Default quotas enabled --perfileset-quota No Per-fileset quota enforcement --filesetdf No Fileset df enabled? -V 15.01 (4.2.0.0) File system version --create-time Wed Dec 23 09:31:07 2015 File system creation time -z No Is DMAPI enabled? -L 4194304 Logfile size -E Yes Exact mtime mount option -S No Suppress atime mount option -K whenpossible Strict replica allocation option --fastea Yes Fast external attributes enabled? --encryption No Encryption enabled? --inode-limit 55325440 Maximum number of inodes in all inode spaces --log-replicas 0 Number of log replicas --is4KAligned Yes is4KAligned? --rapid-repair Yes rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) -P system;T12TB;T26TB Disk storage pools in file system -d nsd2;nsd3;nsd4;nsd5;nsd6;nsd7;nsd8;nsd9;nsd16;nsd17;nsd18;nsd19;nsd20;nsd15;nsd21;nsd10;nsd11;nsd12;nsd13;nsd14 Disks in file system -A yes Automatic mount option -o none Additional mount options -T /datafs Default mount point --mount-priority 0 Mount priority File system attributes for /dev/sharerfs: ========================================= flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment size in bytes -i 4096 Inode size in bytes -I 16384 Indirect block size in bytes -m 1 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j scatter Block allocation type -D nfs4 File locking semantics in effect -k nfs4 ACL semantics in effect -n 100 Estimated number of nodes that will mount file system -B 262144 Block size -Q none Quotas accounting enabled none Quotas enforced none Default quotas enabled --perfileset-quota No Per-fileset quota enforcement --filesetdf No Fileset df enabled? -V 15.01 (4.2.0.0) File system version --create-time Tue Dec 22 17:19:33 2015 File system creation time -z No Is DMAPI enabled? -L 4194304 Logfile size -E Yes Exact mtime mount option -S No Suppress atime mount option -K whenpossible Strict replica allocation option --fastea Yes Fast external attributes enabled? --encryption No Encryption enabled? --inode-limit 102656 Maximum number of inodes --log-replicas 0 Number of log replicas --is4KAligned Yes is4KAligned? --rapid-repair Yes rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) -P system Disk storage pools in file system -d nsd1 Disks in file system -A yes Automatic mount option -o none Additional mount options -T /sharedr Default mount point --mount-priority 0 Mount priority mmlsnsd File system Disk name NSD servers --------------------------------------------------------------------------- datafs nsd2 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd3 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd4 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd5 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd6 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd7 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd8 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd9 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd15 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd16 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd17 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd18 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd19 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd20 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd21 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd10 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd11 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd12 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd13 NSDSERV01_Daemon,NSDSERV02_Daemon datafs nsd14 NSDSERV01_Daemon,NSDSERV02_Daemon sharerfs nsd1 NSDSERV01_Daemon,NSDSERV02_Daemon mmlscluster GPFS cluster information ======================== GPFS cluster name: spectrum_syc.localdomain GPFS cluster id: 2719632319013564592 GPFS UID domain: spectrum_syc.localdomain Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp Repository type: CCR Node Daemon node name IP address Admin node name Designation ----------------------------------------------------------------------- 1 NSDSERV01_Daemon 172.19.20.61 NSDSERV01_Daemon quorum-manager-perfmon 2 NSDSERV02_Daemon 172.19.20.62 NSDSERV02_Daemon quorum-manager-perfmon 3 PROTSERV01_Daemon 172.19.20.63 PROTSERV01_Daemon quorum-manager-perfmon 4 PROTSERV02_Daemon 172.19.20.64 PROTSERV02_Daemon manager-perfmon and a mmdf disk disk size failure holds holds free GB free GB name in GB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 4.1 TB) nsd2 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd3 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd4 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd5 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd6 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd7 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd8 400 1 Yes Yes 390 ( 97%) 1 ( 0%) nsd9 400 1 Yes Yes 390 ( 97%) 1 ( 0%) ------------- -------------------- ------------------- (pool total) 3200 3120 ( 97%) 1 ( 0%) Disks in storage pool: T12TB (Maximum disk size allowed is 4.1 TB) nsd14 500 2 No Yes 496 ( 99%) 1 ( 0%) nsd13 500 2 No Yes 496 ( 99%) 1 ( 0%) nsd12 500 2 No Yes 496 ( 99%) 1 ( 0%) nsd11 500 2 No Yes 496 ( 99%) 1 ( 0%) nsd10 500 2 No Yes 496 ( 99%) 1 ( 0%) nsd15 500 2 No Yes 496 ( 99%) 1 ( 0%) ------------- -------------------- ------------------- (pool total) 3000 2974 ( 99%) 1 ( 0%) Disks in storage pool: T26TB (Maximum disk size allowed is 8.2 TB) nsd21 500 3 No Yes 500 (100%) 1 ( 0%) nsd20 500 3 No Yes 500 (100%) 1 ( 0%) nsd19 500 3 No Yes 500 (100%) 1 ( 0%) nsd18 500 3 No Yes 500 (100%) 1 ( 0%) nsd17 500 3 No Yes 500 (100%) 1 ( 0%) nsd16 500 3 No Yes 500 (100%) 1 ( 0%) ------------- -------------------- ------------------- (pool total) 3000 3000 (100%) 1 ( 0%) ============= ==================== =================== (data) 9200 9093 ( 99%) 2 ( 0%) (metadata) 3200 3120 ( 97%) 1 ( 0%) ============= ==================== =================== (total) 9200 9093 ( 99%) 2 ( 0%) Inode Information ----------------- Total number of used inodes in all Inode spaces: 284090 Total number of free inodes in all Inode spaces: 20318278 Total number of allocated inodes in all Inode spaces: 20602368 Total of Maximum number of inodes in all Inode spaces: 55325440 Thanks Danny Alexander Calderon R Client Technical Specialist - CTS Storage STG Colombia Phone: 57-1-6281956 Mobile: 57- 318 352 9258 Carrera 53 N?mero 100-25 Bogot?, Colombia From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 01/03/2016 05:18 PM Subject: gpfsug-discuss Digest, Vol 48, Issue 2 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Resource exhausted by Pool Migration (Danny Alexander Calderon Rodriguez) 2. Re: Resource exhausted by Pool Migration (Sven Oehme) 3. metadata replication question (Simon Thompson (Research Computing - IT Services)) 4. Re: metadata replication question (Barry Evans) 5. Re: metadata replication question (Simon Thompson (Research Computing - IT Services)) ---------------------------------------------------------------------- Message: 1 Date: Sun, 3 Jan 2016 15:55:59 +0000 From: "Danny Alexander Calderon Rodriguez" To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Resource exhausted by Pool Migration Message-ID: <201601031556.u03Futfw007019 at d24av01.br.ibm.com> Content-Type: text/plain; charset="utf-8" HI All Actually I have a 4.2 Spectrum Scale cluster with protocol service, we are managing small files (32K to 140K), when I try to migrate some files (120.000 files ) the system resources of all nodes is exhausted and the protocol nodes don't get services to client. I wan to ask if there is any way to limit the resources consuming at the migration time? Thanks to all Enviado desde IBM Verse -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160103/cfa1d022/attachment-0001.html > ------------------------------ Message: 2 Date: Sun, 3 Jan 2016 08:42:27 -0800 From: Sven Oehme To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Resource exhausted by Pool Migration Message-ID: Content-Type: text/plain; charset="utf-8" Hi Danny, can you be a bit more specific, which resources get exhausted ? are you talking about operating system or Spectrum Scale resources (filecache or pagepool) ? when you migrate the files ( i assume policy engine) did you specify which nodes do the migration ( -N hostnames) or did you just run mmapplypolicy without anything ? can you post either your entire mmlsconfig or at least output of : for i in maxFilesToCache pagepool maxStatCache nsdMinWorkerThreads nsdMaxWorkerThreads worker1Threads; do mmlsconfig $i ; done mmlsfs , mmlsnsd and mmlscluster output might be useful too.. sven On Sun, Jan 3, 2016 at 7:55 AM, Danny Alexander Calderon Rodriguez < dacalder at co.ibm.com> wrote: > HI All > > Actually I have a 4.2 Spectrum Scale cluster with protocol service, we are managing small files (32K to 140K), when I try to migrate some files (120.000 files ) the system resources of all nodes is exhausted and the protocol nodes don't get services to client. > > I wan to ask if there is any way to limit the resources consuming at the migration time? > > > Thanks to all > > > > Enviado desde IBM Verse > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160103/d5f4999c/attachment-0001.html > ------------------------------ Message: 3 Date: Sun, 3 Jan 2016 21:56:26 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] metadata replication question Message-ID: Content-Type: text/plain; charset="us-ascii" I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. Id like to add some ssd for metadata. Should I: Add raid1 ssd to the storewise? Add local ssd to the nsd servers? If I did the second, should I add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? Or something else entirely? What I want so survive is a split data centre situation or failure of a single nsd server at any point... Thoughts? Comments? I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? Id like a cost competitive solution that gives faster performance than the current sas drives. Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... Thanks Simon ------------------------------ Message: 4 Date: Sun, 3 Jan 2016 22:10:21 +0000 From: Barry Evans To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] metadata replication question Message-ID: <56899C4D.4050907 at pixitmedia.com> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Can all 4 NSD servers see all existing storwize arrays across both DC's? Cheers, Barry On 03/01/2016 21:56, Simon Thompson (Research Computing - IT Services) wrote: > I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. > > Id like to add some ssd for metadata. > > Should I: > > Add raid1 ssd to the storewise? > > Add local ssd to the nsd servers? > > If I did the second, should I > add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. > add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? > add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? > > Or something else entirely? > > What I want so survive is a split data centre situation or failure of a single nsd server at any point... > > Thoughts? Comments? > > I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? > > Id like a cost competitive solution that gives faster performance than the current sas drives. > > Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... > > Thanks > > Simon > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Barry Evans Technical Director & Co-Founder Pixit Media Mobile: +44 (0)7950 666 248 http://www.pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160103/5d463c6d/attachment-0001.html > ------------------------------ Message: 5 Date: Sun, 3 Jan 2016 22:18:24 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] metadata replication question Message-ID: Content-Type: text/plain; charset="us-ascii" Yes there is extended san in place. The failure groups for the storage are different in each dc so we guarantee that the data replication has 1 copy per dc. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Barry Evans [bevans at pixitmedia.com] Sent: 03 January 2016 22:10 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] metadata replication question Can all 4 NSD servers see all existing storwize arrays across both DC's? Cheers, Barry On 03/01/2016 21:56, Simon Thompson (Research Computing - IT Services) wrote: I currently have 4 NSD servers in a cluster, two pairs in two data centres. Data and metadata replication is currently set to 2 with metadata sitting on sas drivers in a storewise array. I also have a vm floating between the two data centres to guarantee quorum in one only in the event of split brain. Id like to add some ssd for metadata. Should I: Add raid1 ssd to the storewise? Add local ssd to the nsd servers? If I did the second, should I add ssd to each nsd server (not raid 1) and set each in a different failure group and make metadata replication of 4. add ssd to each nsd server as raid 1, use the same failure group for each data centre pair? add ssd to each nsd server not raid 1, use the dame failure group for each data centre pair? Or something else entirely? What I want so survive is a split data centre situation or failure of a single nsd server at any point... Thoughts? Comments? I'm thinking the first of the nsd local options uses 4 writes as does the second, but each nsd server then has a local copy of the metatdata locally and ssd fails, in which case it should be able to get it from its local partner pair anyway (with readlocalreplica)? Id like a cost competitive solution that gives faster performance than the current sas drives. Was also thinking I might add an ssd to each nsd server for system.log pool for hawc as well... Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Barry Evans Technical Director & Co-Founder Pixit Media Mobile: +44 (0)7950 666 248 http://www.pixitmedia.com [http://www.pixitmedia.com/sig/sig-cio.jpg] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 48, Issue 2 ********************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0F620571.gif Type: image/gif Size: 360 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Mon Jan 4 18:05:04 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 4 Jan 2016 13:05:04 -0500 Subject: [gpfsug-discuss] fix up broken connections to remote clusters In-Reply-To: References: Message-ID: <201601041805.u04I5D4P015620@d03av04.boulder.ibm.com> An OS reboot is almost never necessary. For some SpectrumScaleFileSystem (SSFS?) configuration changes you might need to `mmshutdown; mmstartup;` on some of your nodes. That can be disruptive, so should not be taken lightly on a production system, but definitely much quicker than a "full reboot". -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpatterson at systemfabricworks.com Mon Jan 4 18:54:24 2016 From: vpatterson at systemfabricworks.com (Vickie Patterson) Date: Mon, 4 Jan 2016 12:54:24 -0600 Subject: [gpfsug-discuss] Seeking an Austin/Houston Spectrum Scale Evangelist Message-ID: <011801d14721$54f59ad0$fee0d070$@systemfabricworks.com> I'm seeking a Spectrum Scale Evangelist type person who lives in the Austin/ Houston area to help educate a group on the greatness of Spectrum Scale. If you are available and interested to help out, please send me a note of introduction to: vpatterson at systemfabricworks.com. Also, considering I'm new to the this user group, can anyone tell me if there is a Austin or Houston local meetup or other forum for Spectrum Scale users? Vickie Patterson, PMP Director, Product Services System Fabric Works Mobile/Office: 512-431-2267 11754 Jollyville Road, Suite 103 Austin, Texas 78759 www.systemfabricworks.com vpatterson at systemfabricworks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 1089 bytes Desc: not available URL: From makaplan at us.ibm.com Tue Jan 5 00:13:00 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 4 Jan 2016 19:13:00 -0500 Subject: [gpfsug-discuss] Resource exhausted by Pool Migration In-Reply-To: References: <201601031556.u03Futfw007019@d24av01.br.ibm.com> Message-ID: <201601050013.u050DAXs009498@d03av04.boulder.ibm.com> "the system resources of all nodes is exhausted" - I'll second the question -- What did you mean by that? But perhaps an answer your question -- you may be in some luck -- since you are running 4.2 - you have the new feature "QOS" (Quality of Service) and the mmchqos command that allow you to cap or throttle IO intensive "maintenance" commands like mmapplypolicy and mmrestripefs. --marc of GPFS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dacalder at co.ibm.com Tue Jan 5 16:30:07 2016 From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez) Date: Tue, 5 Jan 2016 11:30:07 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 48, Issue 4 In-Reply-To: References: Message-ID: <201601051630.u05GUO7I001559@d24av01.br.ibm.com> Hi all I can fix this using the flags -n and -m as options on mmapplypolicy Thanks Danny Alexander Calderon R Client Technical Specialist - CTS Storage STG Colombia Phone: 57-1-6281956 Mobile: 57- 318 352 9258 Carrera 53 N?mero 100-25 Bogot?, Colombia From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 01/05/2016 07:28 AM Subject: gpfsug-discuss Digest, Vol 48, Issue 4 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: fix up broken connections to remote clusters (Marc A Kaplan) 2. Seeking an Austin/Houston Spectrum Scale Evangelist (Vickie Patterson) 3. Re: Resource exhausted by Pool Migration (Marc A Kaplan) ---------------------------------------------------------------------- Message: 1 Date: Mon, 4 Jan 2016 13:05:04 -0500 From: "Marc A Kaplan" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] fix up broken connections to remote clusters Message-ID: <201601041805.u04I5D4P015620 at d03av04.boulder.ibm.com> Content-Type: text/plain; charset="us-ascii" An OS reboot is almost never necessary. For some SpectrumScaleFileSystem (SSFS?) configuration changes you might need to `mmshutdown; mmstartup;` on some of your nodes. That can be disruptive, so should not be taken lightly on a production system, but definitely much quicker than a "full reboot". -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160104/c60142fe/attachment-0001.html > ------------------------------ Message: 2 Date: Mon, 4 Jan 2016 12:54:24 -0600 From: "Vickie Patterson" To: Subject: [gpfsug-discuss] Seeking an Austin/Houston Spectrum Scale Evangelist Message-ID: <011801d14721$54f59ad0$fee0d070$@systemfabricworks.com> Content-Type: text/plain; charset="us-ascii" I'm seeking a Spectrum Scale Evangelist type person who lives in the Austin/ Houston area to help educate a group on the greatness of Spectrum Scale. If you are available and interested to help out, please send me a note of introduction to: vpatterson at systemfabricworks.com. Also, considering I'm new to the this user group, can anyone tell me if there is a Austin or Houston local meetup or other forum for Spectrum Scale users? Vickie Patterson, PMP Director, Product Services System Fabric Works Mobile/Office: 512-431-2267 11754 Jollyville Road, Suite 103 Austin, Texas 78759 www.systemfabricworks.com vpatterson at systemfabricworks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160104/1289fe98/attachment-0001.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 1089 bytes Desc: not available URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160104/1289fe98/attachment-0001.gif > ------------------------------ Message: 3 Date: Mon, 4 Jan 2016 19:13:00 -0500 From: "Marc A Kaplan" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Resource exhausted by Pool Migration Message-ID: <201601050013.u050DAXs009498 at d03av04.boulder.ibm.com> Content-Type: text/plain; charset="us-ascii" "the system resources of all nodes is exhausted" - I'll second the question -- What did you mean by that? But perhaps an answer your question -- you may be in some luck -- since you are running 4.2 - you have the new feature "QOS" (Quality of Service) and the mmchqos command that allow you to cap or throttle IO intensive "maintenance" commands like mmapplypolicy and mmrestripefs. --marc of GPFS. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160104/13a66468/attachment-0001.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 48, Issue 4 ********************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 08076738.gif Type: image/gif Size: 360 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jan 8 16:10:45 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 8 Jan 2016 16:10:45 +0000 Subject: [gpfsug-discuss] GPFS 4.2 / Protocol nodes Message-ID: Happy New Year all, One of my first projects on the new year is to get GPFS 4.2 up and running on our test cluster and begin testing it out in anticipation of upgrading our production cluster sometime later this year. We?re currently running 4.1.0.8 efix2 and are thinking of bypassing 4.1.1 altogether and going straight to 4.2. We currently have 3 NSD servers also serving as CNFS servers and 1 NSD server that is not primary for any disks which serves as our SAMBA server. We are interested in going to CES. Yesterday I was reading in the 4.2 FAQ and came across question 8.3, ?What are some of the considerations when deploying the protocol functionality?? One of the considerations is that "several GPFS configuration aspects have not been explicitly tested with the protocol functionality? and one of those functions is, ?NSD server functionality and storage attached to protocol node. We recommend that Protocol nodes do not take on these functions?. Really? So it is IBM?s recommendation that we buy 3 additional very beefy (2 x hex-core processors, 256 GB RAM) servers and 3 additional server licenses just to use CES? I guess I?m very surprised by that because I?m running CNFS on 3 low end servers (1 x quad-core processor, 32 GB RAM) that also serve as NSD servers to a ~700 client HPC cluster! If we really have to buy all that, well, we probably won?t. That?s a not insignificant chunk of change. Some I?m interested in hearing feedback on both: 1) do the CES servers really have to not be NSD servers, and 2) do they really need to be such high-end boxes? And both the official party line and what you can really do in the real world if you really want to (while maintaining your support agreement with IBM) are welcome! Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Mon Jan 11 14:42:39 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 11 Jan 2016 14:42:39 +0000 Subject: [gpfsug-discuss] GPFS 4.2 / Protocol nodes In-Reply-To: References: Message-ID: <201601111442.u0BEgpCK019140@d06av06.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Jan 12 13:40:27 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 12 Jan 2016 13:40:27 +0000 Subject: [gpfsug-discuss] GPFS 4.2 / Protocol nodes Message-ID: My experience is that in small clusters, it?s acceptable to double up on some of these services, but as you scale up, breaking them out makes more sense. Spectrum Scale make it easy to add/remove the nodes non-disruptively, so you can move them to dedicated nodes. When I first started testing 4.2, I setup a 6 node cluster that had both NSD and CES on them, and it did just fine. The nodes were 4-core 32GB and I had NFS and Object running on the CES nodes. The new CES nodes run a ton more services, especially when you start working with Object. Both of you points are valid considerations ? especially with CNFS. I?m running multiple CNFS clusters and having them broken out has save me a number of times. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Daniel Kidger > Reply-To: gpfsug main discussion list > Date: Monday, January 11, 2016 at 8:42 AM To: "gpfsug-discuss at spectrumscale.org" > Cc: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] GPFS 4.2 / Protocol nodes It looks like no one has attempted to answer this, so I will step in to start the conversation. There are two issues when considering how many services to run on the same nodes - in this case the NSD servers. 1. Performance. Spectrum Scale's (nee GPFS) core differentiator is performance. The more you run on a node the more that node resource's have to be shared. Here memory bandwidth and memory space are the main ones. CPU may also be a limited resource although with modern chips this is less likely so. If performance is not the key delivered metric then running other things on the NSD server may be a good option so save both cost and server spawl in small datacentres. 2. NFS server stability. pre-4.1.1, IBM used cNFS to provide multiple NFS servers in a GPFS cluster. This used traditional kernel based NFS daemons. If one hung then the whole node had to be rebooted which might have led to disruption in NSD serving if the other NSD server of a pair was already under load. With 4.1.1 came Cluster Export Services (CES) deliverd from 'Protocol Nodes'. Since there use Ganesha there would be no need to reboot this node if the NFS serving hung and in Ganesha, all NFS activity is in userspace not the kernel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Tue Jan 12 14:15:37 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Tue, 12 Jan 2016 14:15:37 +0000 Subject: [gpfsug-discuss] weird issue when issuing mm commands Message-ID: I have just deployed a GPU node in our HPC environment and I am having couple of weird issues when trying to add it to our GPFS cluster. mmaddnode command from NSD server returns with: unknown os mmgetstate on the node itself returns with unknown os I think I've traced it down to locale issue but reinstalling glibc-common and making sure that /etc/sysconfig/i18n files are in place and OK does not seem to fix the issue. This is on RedHat 6.6 Any help is appreciated. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jan 12 19:24:18 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 12 Jan 2016 11:24:18 -0800 Subject: [gpfsug-discuss] weird issue when issuing mm commands In-Reply-To: References: Message-ID: <201601121924.u0CJOLwa003922@d01av02.pok.ibm.com> GPFS code determines the current OS by looking at the output of "uname -s". On a regular Linux node, this command returns "Linux", so check whether you see the same, when the command is run locally on the node and through an ssh session. Since mmaddnode has to do some remote processing, make sure ssh works as expected, in particular in the locale setting area. yuri From: Damir Krstic To: gpfsug-discuss at spectrumscale.org, Date: 01/12/2016 06:16 AM Subject: [gpfsug-discuss] weird issue when issuing mm commands Sent by: gpfsug-discuss-bounces at spectrumscale.org I have just deployed a GPU node in our HPC environment and I am having couple of weird issues when trying to add it to our GPFS cluster. mmaddnode command from NSD server returns with: unknown os mmgetstate on the node itself returns with unknown os I think I've traced it down to locale issue but reinstalling glibc-common and making sure that /etc/sysconfig/i18n files are in place and OK does not seem to fix the issue. This is on RedHat 6.6 Any help is appreciated. Thanks, Damir_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Tue Jan 12 19:29:18 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 12 Jan 2016 19:29:18 +0000 Subject: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash Message-ID: I'm experimenting with moving some of our more heavily used files (about 5% of the file system) from disk to a flash tier. The disk latency is around 10ms, while the flash is less than 1ms. What I'm seeing that when I move the files to flash, my NSD server queues (thread queues) have a much larger number of pending RPCs. (on the Small queues) Not surprising, since the storage subsystem is faster, the clients push more requests and the NSD server backs up. I don't see an evidence of the NSD server itself running out of steam (CPU below 20%) so I'm thinking I just need to adjust the number of threads and the threads per queue. Right now I have this: nsdMaxWorkerThreads=256, nsdMinWorkerThreads=256, nsdThreadsPerQueue=8, nsdSmallThreadRatio=1 What I'm trying to decide is whether I should increase just the MaxThreads or both the MaxThreads and the ThredsPerQueue. Can anyone offer advice? I did read through Yuri?s document. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.bergman at uphs.upenn.edu Tue Jan 12 20:17:06 2016 From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu) Date: Tue, 12 Jan 2016 15:17:06 -0500 Subject: [gpfsug-discuss] weird issue when issuing mm commands In-Reply-To: Your message of "Tue, 12 Jan 2016 11:24:18 -0800." <201601121924.u0CJOLwa003922@d01av02.pok.ibm.com> References: <201601121924.u0CJOLwa003922@d01av02.pok.ibm.com> Message-ID: <11908-1452629826.288572@eP0N.FsqM.EFX4> In the message dated: Tue, 12 Jan 2016 11:24:18 -0800, The pithy ruminations from Yuri L Volobuev on were: => => GPFS code determines the current OS by looking at the output of "uname -s". => On a regular Linux node, this command returns "Linux", so check whether you I saw the same issue on several Scientific Linux release 6.7 machines last week. These machines have been GPFS clients since ~ SL 6.3 & GPFS 3.5.9 (~3 years). This error never happened previously. Various 'mm*' commands were returning the "Unknown OS" error. Note that there was no OS named at the end of the error message, as if $osName was not defined in /usr/lpp/mmfs/bin/mmglobfuncs. The 'uname -s' command on these nodes does return "Linux" both locally and remotely via an ssh connection from a GPFS manager node. Updating OS distribution packages & rebooting the nodes & rebuilding the kernel-specific gpfs.gplbin packge resolved the problem, so I'm not sure of the exact cause. Mark => see the same, when the command is run locally on the node and through an => ssh session. Since mmaddnode has to do some remote processing, make sure => ssh works as expected, in particular in the locale setting area. => => yuri => => => => From: Damir Krstic => To: gpfsug-discuss at spectrumscale.org, => Date: 01/12/2016 06:16 AM => Subject: [gpfsug-discuss] weird issue when issuing mm commands => Sent by: gpfsug-discuss-bounces at spectrumscale.org => => => => I have just deployed a GPU node in our HPC environment and I am having => couple of weird issues when trying to add it to our GPFS cluster. mmaddnode => command from NSD server returns with: unknown os => => mmgetstate on the node itself returns with unknown os => => I think I've traced it down to locale issue but reinstalling glibc-common => and making sure that /etc/sysconfig/i18n files are in place and OK does not => seem to fix the issue. This is on RedHat 6.6 => => Any help is appreciated. => => Thanks, => Damir_______________________________________________ => gpfsug-discuss mailing list => gpfsug-discuss at spectrumscale.org => http://gpfsug.org/mailman/listinfo/gpfsug-discuss => => >>> image/gif attachment, name=graycol.gif => _______________________________________________ => gpfsug-discuss mailing list => gpfsug-discuss at spectrumscale.org => http://gpfsug.org/mailman/listinfo/gpfsug-discuss => -- Mark Bergman Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1 Stagehand '94 Yamaha GTS1000A^2 bergman at panix.com http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40panix.com I want a newsgroup with a infinite S/N ratio! Now taking CFV on: rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters 15+ So Far--Want to join? Check out: http://www.panix.com/~bergman From oehmes at us.ibm.com Tue Jan 12 20:20:29 2016 From: oehmes at us.ibm.com (Sven Oehme) Date: Tue, 12 Jan 2016 20:20:29 +0000 Subject: [gpfsug-discuss] weird issue when issuing mm commands In-Reply-To: <11908-1452629826.288572@eP0N.FsqM.EFX4> Message-ID: <201601122020.u0CKKYKp011396@d03av01.boulder.ibm.com> I have seen this with a read-only or full /var/ filesystem before. Could that be the case ? Sent from IBM Verse mark.bergman at uphs.upenn.edu --- Re: [gpfsug-discuss] weird issue when issuing mm commands --- From:"" To:"gpfsug main discussion list" Date:Tue, Jan 12, 2016 12:17 PMSubject:Re: [gpfsug-discuss] weird issue when issuing mm commands In the message dated: Tue, 12 Jan 2016 11:24:18 -0800, The pithy ruminations from Yuri L Volobuev on were: => => GPFS code determines the current OS by looking at the output of "uname -s". => On a regular Linux node, this command returns "Linux", so check whether you I saw the same issue on several Scientific Linux release 6.7 machines last week. These machines have been GPFS clients since ~ SL 6.3 & GPFS 3.5.9 (~3 years). This error never happened previously. Various 'mm*' commands were returning the "Unknown OS" error. Note that there was no OS named at the end of the error message, as if $osName was not defined in /usr/lpp/mmfs/bin/mmglobfuncs. The 'uname -s' command on these nodes does return "Linux" both locally and remotely via an ssh connection from a GPFS manager node. Updating OS distribution packages & rebooting the nodes & rebuilding the kernel-specific gpfs.gplbin packge resolved the problem, so I'm not sure of the exact cause. Mark => see the same, when the command is run locally on the node and through an => ssh session. Since mmaddnode has to do some remote processing, make sure => ssh works as expected, in particular in the locale setting area. => => yuri => => => => From: Damir Krstic => To: gpfsug-discuss at spectrumscale.org, => Date: 01/12/2016 06:16 AM => Subject: [gpfsug-discuss] weird issue when issuing mm commands => Sent by: gpfsug-discuss-bounces at spectrumscale.org => => => => I have just deployed a GPU node in our HPC environment and I am having => couple of weird issues when trying to add it to our GPFS cluster. mmaddnode => command from NSD server returns with: unknown os => => mmgetstate on the node itself returns with unknown os => => I think I've traced it down to locale issue but reinstalling glibc-common => and making sure that /etc/sysconfig/i18n files are in place and OK does not => seem to fix the issue. This is on RedHat 6.6 => => Any help is appreciated. => => Thanks, => Damir_______________________________________________ => gpfsug-discuss mailing list => gpfsug-discuss at spectrumscale.org => http://gpfsug.org/mailman/listinfo/gpfsug-discuss => => >>> image/gif attachment, name=graycol.gif => _______________________________________________ => gpfsug-discuss mailing list => gpfsug-discuss at spectrumscale.org => http://gpfsug.org/mailman/listinfo/gpfsug-discuss => -- Mark Bergman Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1 Stagehand '94 Yamaha GTS1000A^2 bergman at panix.com http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40panix.com I want a newsgroup with a infinite S/N ratio! Now taking CFV on: rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters 15+ So Far--Want to join? Check out: http://www.panix.com/~bergman _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Jan 12 20:39:24 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 12 Jan 2016 21:39:24 +0100 Subject: [gpfsug-discuss] weird issue when issuing mm commands In-Reply-To: <11908-1452629826.288572@eP0N.FsqM.EFX4> References: <201601121924.u0CJOLwa003922@d01av02.pok.ibm.com> <11908-1452629826.288572@eP0N.FsqM.EFX4> Message-ID: <5695647C.70600@cc.in2p3.fr> On 12/01/2016 21:17, mark.bergman at uphs.upenn.edu wrote: > In the message dated: Tue, 12 Jan 2016 11:24:18 -0800, > The pithy ruminations from Yuri L Volobuev on > were: > => > => GPFS code determines the current OS by looking at the output of "uname -s". > => On a regular Linux node, this command returns "Linux", so check whether you > > I saw the same issue on several Scientific Linux release 6.7 machines last > week. > > These machines have been GPFS clients since ~ SL 6.3 & GPFS 3.5.9 > (~3 years). This error never happened previously. > Hello, In my experience, this problem is often due to "/tmp" being 100% full. It appears GPFS 4.1.1 now displays a meaningful message in such a case: # df -h /tmp | grep % 985M 985M 0 100% /tmp # /usr/lpp/mmfs/bin/mmgetstate mmgetstate: Unable to obtain the OS name. The tmp file system may be full. # Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From damir.krstic at gmail.com Tue Jan 12 20:40:04 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Tue, 12 Jan 2016 20:40:04 +0000 Subject: [gpfsug-discuss] weird issue when issuing mm commands In-Reply-To: <201601122020.u0CKKYKp011396@d03av01.boulder.ibm.com> References: <11908-1452629826.288572@eP0N.FsqM.EFX4> <201601122020.u0CKKYKp011396@d03av01.boulder.ibm.com> Message-ID: I think this is it - this was a stateless image and /var/ was initially part of tmpfs...anyway i am 99% certain that this was the issue. when i get a chance to boot these again with stateless image i will report back. thanks all for your help. damir On Tue, Jan 12, 2016 at 2:20 PM Sven Oehme wrote: > I have seen this with a read-only or full /var/ filesystem before. Could that be the case ? > > > Sent from IBM Verse > > > > mark.bergman at uphs.upenn.edu --- Re: [gpfsug-discuss] weird issue when > issuing mm commands --- > > From:"" To:"gpfsug main discussion list" < > gpfsug-discuss at spectrumscale.org>Date:Tue, Jan 12, 2016 12:17 PMSubject:Re: > [gpfsug-discuss] weird issue when issuing mm commands > ------------------------------ > > In the message dated: Tue, 12 Jan 2016 11:24:18 -0800, > The pithy ruminations from Yuri L Volobuev on > were: > => > => GPFS code determines the current OS by looking at the output of "uname > -s". > => On a regular Linux node, this command returns "Linux", so check whether > you > > I saw the same issue on several Scientific Linux release 6.7 machines last > week. > > These machines have been GPFS clients since ~ SL 6.3 & GPFS 3.5.9 > (~3 years). This error never happened previously. > > Various 'mm*' commands were returning the "Unknown OS" error. Note that > there was no OS named at the end of the error message, as if $osName > was not defined in /usr/lpp/mmfs/bin/mmglobfuncs. > > The 'uname -s' command on these nodes does return "Linux" both locally > and remotely via an ssh connection from a GPFS manager node. > > Updating OS distribution packages & rebooting the nodes & rebuilding the > kernel-specific gpfs.gplbin packge resolved the problem, so I'm not sure > of the exact cause. > > Mark > > => see the same, when the command is run locally on the node and through an > => ssh session. Since mmaddnode has to do some remote processing, make sure > => ssh works as expected, in particular in the locale setting area. > => > => yuri > => > => > => > => From: Damir Krstic > => To: gpfsug-discuss at spectrumscale.org, > => Date: 01/12/2016 06:16 AM > => Subject: [gpfsug-discuss] weird issue when issuing mm commands > => Sent by: gpfsug-discuss-bounces at spectrumscale.org > => > => > => > => I have just deployed a GPU node in our HPC environment and I am having > => couple of weird issues when trying to add it to our GPFS cluster. > mmaddnode > => command from NSD server returns with: unknown os > => > => mmgetstate on the node itself returns with unknown os > => > => I think I've traced it down to locale issue but reinstalling > glibc-common > => and making sure that /etc/sysconfig/i18n files are in place and OK does > not > => seem to fix the issue. This is on RedHat 6.6 > => > => Any help is appreciated. > => > => Thanks, > => Damir_______________________________________________ > => gpfsug-discuss mailing list > => gpfsug-discuss at spectrumscale.org > => http://gpfsug.org/mailman/listinfo/gpfsug-discuss > => > => > >>> image/gif attachment, name=graycol.gif > => _______________________________________________ > => gpfsug-discuss mailing list > => gpfsug-discuss at spectrumscale.org > => http://gpfsug.org/mailman/listinfo/gpfsug-discuss > => > -- > Mark Bergman Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1 > Stagehand > '94 Yamaha GTS1000A^2 > bergman at panix.com > > http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40panix.com > > I want a newsgroup with a infinite S/N ratio! Now taking CFV on: > rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters > 15+ So Far--Want to join? Check out: http://www.panix.com/~bergman > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jan 12 21:58:25 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 12 Jan 2016 13:58:25 -0800 Subject: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash In-Reply-To: References: Message-ID: <201601122158.u0CLwTcu003135@d01av01.pok.ibm.com> If you want to have more NSD worker threads running, the recommended way to do this is increase nsdThreadsPerDisk, and also increase nsdMaxWorkerThreads. Make sure you have enough pagepool for all of the new threads though. You can also achieve a similar result by increasing nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant, although that's not how this parameter is normally used. The basic idea is that you want to achieve a certain degree of IO parallelism on the block device level, and that calls for some number of worker threads per disk. The number of NSD queues is not really relevant here, multiple queues is just a technique to achieve better SMP scalability, and with the default 256 queues there should be no contention for any individual queue on typical hardware (tens of CPU cores). Controlling the number of worker threads through nsdThreadsPerDisk gives you a way to adjust the thread count dynamically as disks are added or removed, without you having to micro-manage this by adjusting nsdMinWorkerThreads every time. It's not clear whether having more IO parallelism necessarily improves the overall throughput, vs shifting bottlenecks elsewhere. Given the basic nature of flash storage, the former intuitively seems plausible, but some hard numbers from a real performance experiment would be helpful. yuri From: "Oesterlin, Robert" To: gpfsug main discussion list , Date: 01/12/2016 11:31 AM Subject: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash Sent by: gpfsug-discuss-bounces at spectrumscale.org I'm experimenting with moving some of our more heavily used files (about 5% of the file system) from disk to a flash tier. The disk latency is around 10ms, while the flash is less than 1ms. What I'm seeing that when I move the files to flash, my NSD server queues (thread queues) have a much larger number of pending RPCs. (on the Small queues) Not surprising, since the storage subsystem is faster, the clients push more requests and the NSD server backs up. I don't see an evidence of the NSD server itself running out of steam (CPU below 20%) so I'm thinking I just need to adjust the number of threads and the threads per queue. Right now I have this: nsdMaxWorkerThreads=256, nsdMinWorkerThreads=256, nsdThreadsPerQueue=8, nsdSmallThreadRatio=1 What I'm trying to decide is whether I should increase just the MaxThreads or both the MaxThreads and the ThredsPerQueue. Can anyone offer advice? I did read through Yuri?s document. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Tue Jan 12 22:52:34 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 12 Jan 2016 22:52:34 +0000 Subject: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash In-Reply-To: <201601122158.u0CLwTcu003135@d01av01.pok.ibm.com> References: , <201601122158.u0CLwTcu003135@d01av01.pok.ibm.com> Message-ID: Thanks Yuri. As you can see I was already bumping the Min so I was effectively increasing the threads per disk as you suggest. I don't see nsdThreadsPerDisk in the config output, nor is it documented. I know it was before. Interestingly, increasing the threads didn't really help, and in some cases made things worse. I settled on 256 threads and 32 threads per queue, with a ratio of 1. This gives me 4 large and 4 shall queues. Regardless of the number of queues and threads I toss at it, I still see a number of queues with pending request in the 100's. Waiters shows I/O waiters in the 200-300ms range, but the flash system (not IBM btw) still reports latency below 1ms. Not sure where the bottleneck is. Perhaps it's the disk subsystem in Linux. Not sure how much can tell. Back to playing whack-a-mole. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid _____________________________ From: Yuri L Volobuev > Sent: Tuesday, January 12, 2016 3:58 PM Subject: Re: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash To: gpfsug main discussion list > If you want to have more NSD worker threads running, the recommended way to do this is increase nsdThreadsPerDisk, and also increase nsdMaxWorkerThreads. Make sure you have enough pagepool for all of the new threads though. You can also achieve a similar result by increasing nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant, although that's not how this parameter is normally used. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From Robert.Oesterlin at nuance.com Wed Jan 13 02:14:19 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 13 Jan 2016 02:14:19 +0000 Subject: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash In-Reply-To: References: <201601122158.u0CLwTcu003135@d01av01.pok.ibm.com> Message-ID: Following up on my own posting ? looks like it?s time to give myself the ?ole dope slap - I forgot to update the multipath.conf file for this particular vendor storage subsystem. Once I did that, problem solved! Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mikal.Sande at altibox.no Tue Jan 19 15:00:18 2016 From: Mikal.Sande at altibox.no (Sande, Mikal) Date: Tue, 19 Jan 2016 15:00:18 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? In-Reply-To: References: , Message-ID: <84C0B9AB4719954985704BC46DF8602A95BCCAA8@prdmbx01.lyse.no> The pmsensors and pmcollector rpm's for RHEL 6 seem to be missing from 4.2.0.1 standard package aswell. ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Alexander Wolf [A.Wolf-Reber at de.ibm.com] Sent: Monday, November 30, 2015 1:52 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? This was a mistake. The RHEL6 sensor packages should have been included but where somehow not picked up in the final image. We will fix this with the next PTF. Mit freundlichen Gr??en / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale GUI development lead Department M069 / Spectrum Scale Software Development +49-6131-84-6521 a.wolf-reber at de.ibm.com IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Oesterlin, Robert" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? Date: Mon, Nov 30, 2015 12:08 AM I noticed that IBM only shipped the zimon performance sensors for RH7 with version 4.2 . This is a HUGE disappointment ? most of my NSD servers are still and RH 6.6 (and the clients). gpfs.gss.pmcollector-4.2.0-0.el7.x86_64.rpm gpfs.gss.pmsensors-4.2.0-0.el7.x86_64.rpm pmswift-4.2.0-0.noarch.rpm Can IBM comment on support for RH6 system with the Performance sensors? I understand the collector node must be at RH7. Making the performance sensor RH7 only means many users won?t be able to take advantage of this function. Bob Oesterlin Sr Storage Engineer, Nuance Communications 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From chair at spectrumscale.org Tue Jan 19 17:13:42 2016 From: chair at spectrumscale.org (Spectrum scale UG Chair (Simon Thompson)) Date: Tue, 19 Jan 2016 17:13:42 +0000 Subject: [gpfsug-discuss] Next meet the devs - 24th Feb 2016 Message-ID: Dear All, We are planning the next 'Meet the Devs' event for Wednesday 24th February 2016, 11am-3:30pm. The event will be held in central Oxford. The agenda promises to be hands on and give you the opportunity to speak face to face with the developers of Spectrum Scale. Guideline agenda: * TBC - please provide input on what you'd like to see! Lunch and refreshments will be provided. Please can you let me know by email if you are interested in attending by Wednesday 17th February. Thanks and we hope to see you there. Thanks to Andy at OERC for offering to host. Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Tue Jan 19 17:31:28 2016 From: chair at spectrumscale.org (Spectrum scale UG Chair (Simon Thompson)) Date: Tue, 19 Jan 2016 17:31:28 +0000 Subject: [gpfsug-discuss] UK May user group meeting Message-ID: Hi All, Its that time of year when we are starting to plan putting together the agenda for the main UK group meeting. This year the event will be held at IBM Southbank on May 17th and 18th. This is the first year that we are running a two day event which will hopefully give us more chance to discuss and talk Spectrum Scale. As usual we are interested in your input into what we should be asking IBM to contribute to the discussion. So if you have ideas or areas which you'd like us to try and cover, please let me know. We've initially planned that the first day would follow the format of previous meetings with a mix of IBM and user focussed talks. The second day we have a number of breakout rooms available, so thought we'd try and run workshop/break out sessions. So what would you like to hear and talk about? Is cloud a hot topic? Do you want to do a deep dive q&a on performance? At the ciuk group there was a request for a frank discussion on licensing. Should we have a breakout focussed on a specific sector, e.g. Media, life sciences? We're also looking for some people to volunteer to do user talks at the event, so if you are using Spectrum Scale and think people would be interested in hearing about it, please let me know!. As a two day event, we'd like to get a social event organised between days, so any suggestions would be welcome! And we'd really appreciate a couple of corporates to offer to help facilitate the social event (aka a call for funds to support it!) Please do let us know what you want to see and hear about and we'll see what we can do! Thanks Simon (group chair) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavel.pokorny at datera.cz Tue Jan 19 18:15:47 2016 From: pavel.pokorny at datera.cz (Pavel Pokorny) Date: Tue, 19 Jan 2016 19:15:47 +0100 Subject: [gpfsug-discuss] Support for SMB Direct Message-ID: Hello, please, is it supported to access GPFS 4.2 using SMB Direct (part of SMB 3.0)? Thanks, Pavel -- Ing. Pavel Pokorn? DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Jan 20 01:09:43 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 20 Jan 2016 01:09:43 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? In-Reply-To: <84C0B9AB4719954985704BC46DF8602A95BCCAA8@prdmbx01.lyse.no> References: , <84C0B9AB4719954985704BC46DF8602A95BCCAA8@prdmbx01.lyse.no> Message-ID: And it looks like they are missing for Debian 8.x as well, although Debian 6 and 7 packages are there. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sande, Mikal Sent: Wednesday, 20 January 2016 1:00 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? The pmsensors and pmcollector rpm's for RHEL 6 seem to be missing from 4.2.0.1 standard package aswell. ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Alexander Wolf [A.Wolf-Reber at de.ibm.com] Sent: Monday, November 30, 2015 1:52 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? This was a mistake. The RHEL6 sensor packages should have been included but where somehow not picked up in the final image. We will fix this with the next PTF. Mit freundlichen Gr??en / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale GUI development lead Department M069 / Spectrum Scale Software Development +49-6131-84-6521 a.wolf-reber at de.ibm.com IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Oesterlin, Robert" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [gpfsug-discuss] Spectrum Scale 4.2 - no support for Zimon Perf sensors on RH6? Date: Mon, Nov 30, 2015 12:08 AM I noticed that IBM only shipped the zimon performance sensors for RH7 with version 4.2 . This is a HUGE disappointment - most of my NSD servers are still and RH 6.6 (and the clients). gpfs.gss.pmcollector-4.2.0-0.el7.x86_64.rpm gpfs.gss.pmsensors-4.2.0-0.el7.x86_64.rpm pmswift-4.2.0-0.noarch.rpm Can IBM comment on support for RH6 system with the Performance sensors? I understand the collector node must be at RH7. Making the performance sensor RH7 only means many users won't be able to take advantage of this function. Bob Oesterlin Sr Storage Engineer, Nuance Communications 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From MDIETZ at de.ibm.com Wed Jan 20 12:38:05 2016 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Wed, 20 Jan 2016 13:38:05 +0100 Subject: [gpfsug-discuss] Support for SMB Direct In-Reply-To: References: Message-ID: <201601201238.u0KCcAof014398@d06av05.portsmouth.uk.ibm.com> SMB Direct is not supported in GPFS 4.2 Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale Development System Health Architect IBM Certified Software Engineer ---------------------------------------------------------------------------------------------------------- IBM Deutschland Hechtsheimer Str. 2 55131 Mainz Phone: +49-6131-84-2027 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ---------------------------------------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Pavel Pokorny To: gpfsug-discuss at gpfsug.org Date: 01/19/2016 07:26 PM Subject: [gpfsug-discuss] Support for SMB Direct Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, please, is it supported to access GPFS 4.2 using SMB Direct (part of SMB 3.0)? Thanks, Pavel -- Ing. Pavel Pokorn? DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Wed Jan 20 12:38:05 2016 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Wed, 20 Jan 2016 13:38:05 +0100 Subject: [gpfsug-discuss] Support for SMB Direct In-Reply-To: References: Message-ID: <201601201238.u0KCcB2X029562@d06av11.portsmouth.uk.ibm.com> SMB Direct is not supported in GPFS 4.2 Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale Development System Health Architect IBM Certified Software Engineer ---------------------------------------------------------------------------------------------------------- IBM Deutschland Hechtsheimer Str. 2 55131 Mainz Phone: +49-6131-84-2027 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ---------------------------------------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Pavel Pokorny To: gpfsug-discuss at gpfsug.org Date: 01/19/2016 07:26 PM Subject: [gpfsug-discuss] Support for SMB Direct Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, please, is it supported to access GPFS 4.2 using SMB Direct (part of SMB 3.0)? Thanks, Pavel -- Ing. Pavel Pokorn? DATERA s.r.o. | Hadovit? 962/10 | Praha | Czech Republic www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tomasz.Wolski at ts.fujitsu.com Fri Jan 22 09:42:26 2016 From: Tomasz.Wolski at ts.fujitsu.com (Tomasz.Wolski at ts.fujitsu.com) Date: Fri, 22 Jan 2016 09:42:26 +0000 Subject: [gpfsug-discuss] Max NSD/filesystem size allowed in GPFS 4.1 Message-ID: <4bd8637076154c538bd8c7f1e3d7be35@R01UKEXCASM223.r01.fujitsu.local> Hi All, Could you please tell me, what is the maximum allowed size of NSD in GPFS 4.1? Also, are there any limitations in filesystem size? Thanks, Tomasz Wolski -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Fri Jan 22 11:29:16 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 22 Jan 2016 11:29:16 +0000 Subject: [gpfsug-discuss] Max NSD/filesystem size allowed in GPFS 4.1 In-Reply-To: <4bd8637076154c538bd8c7f1e3d7be35@R01UKEXCASM223.r01.fujitsu.local> References: <4bd8637076154c538bd8c7f1e3d7be35@R01UKEXCASM223.r01.fujitsu.local> Message-ID: <201601221129.u0MBTNcE017888@d06av01.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From linda at epcc.ed.ac.uk Mon Jan 25 16:22:14 2016 From: linda at epcc.ed.ac.uk (Linda Dewar) Date: Mon, 25 Jan 2016 16:22:14 +0000 Subject: [gpfsug-discuss] Job Opportunity at University Of Edinburgh Message-ID: <56A64BB6.2060407@epcc.ed.ac.uk> GPFS skills needed for a position at University of Edinburgh - see here for details https://www.vacancies.ed.ac.uk/pls/corehrrecruit/erq_jobspec_version_4.display_form -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From linda at epcc.ed.ac.uk Mon Jan 25 16:24:10 2016 From: linda at epcc.ed.ac.uk (Linda Dewar) Date: Mon, 25 Jan 2016 16:24:10 +0000 Subject: [gpfsug-discuss] Job Opportunity at University Of Edinburgh Message-ID: <56A64C2A.1030204@epcc.ed.ac.uk> With correct URL! https://www.vacancies.ed.ac.uk/pls/corehrrecruit/erq_jobspec_version_4.jobspec?p_id=035282 Regards, Linda -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From S.J.Thompson at bham.ac.uk Mon Jan 25 20:27:43 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 25 Jan 2016 20:27:43 +0000 Subject: [gpfsug-discuss] Job Opportunity for CLIMB cloud project Message-ID: We are currently looking for an interesting individual to join our team to support the CLIMB cloud project. If you are interested in joining the team to support this exciting project, check the job details at: http://bit.ly/20pmaK8 This post will be based supporting the Birmingham CLIMB cell. The CLIMB project uses a number of technologies including delivering storage based on IBM Spectrum Scale so the most might be of interest to an experienced Scale admin looking to work with OpenStack. Simon From russell at nordquist.info Thu Jan 28 02:17:28 2016 From: russell at nordquist.info (Russell Nordquist) Date: Wed, 27 Jan 2016 21:17:28 -0500 Subject: [gpfsug-discuss] NYC GPFS job opportunity Message-ID: <4C76D5D8-E149-4740-B98B-71D9D146AF76@nordquist.info> Since job posting are the theme of the week, here is one in NYC to work on GPFS + more scale out storage: https://jobs.pdtpartners.com/?gh_jid=151475 russell -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Thu Jan 28 08:36:21 2016 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Thu, 28 Jan 2016 08:36:21 +0000 Subject: [gpfsug-discuss] FW: GPFS Workshop at SPXXL Winter Meeting In-Reply-To: <29647BD98535414BA8781F4DFB429DCB33CFA3AA@MBX112.d.ethz.ch> References: <29647BD98535414BA8781F4DFB429DCB33CFA3AA@MBX112.d.ethz.ch> Message-ID: Hi All, Just a reminder about the upcoming SPXXL GPFS Workshop on 17th Feb, the agenda for the day is now available at: https://www.spxxl.org/?q=Garching-2016-GPFS-Day The registration for just the workshop part is also now available at the following page: https://www.eventbrite.com/e/spxxl-2016-winter-workshop-registration-198931 80037 Please note that attendance at the GPFS workshop day is free, but you need to register and select the correct option so SPXXL have an idea of numbers. Sorry for any confusion over the date, its Wednesday 17th Feb (not Wed 16th I originally sent!) Simon >________________________________________ >From: gpfsug-discuss-bounces at spectrumscale.org >[gpfsug-discuss-bounces at spectrumscale.org] on behalf of GPFS UG Chair >(Simon Thompson) [chair at spectrumscale.org] >Sent: Wednesday, December 23, 2015 1:00 PM >To: gpfsug-discuss at spectrumscale.org >Subject: [gpfsug-discuss] GPFS Workshop at SPXXL Winter Meeting > >Hi All, > >SPXXL (www.spxxl.org) are organising a GPFS Workshop as part of the Winter >(Feb 2016) meeting. This will take place on Wednesday 17th February 2016 >at Leibniz Supercomputing Centre, Germany. > >The GPFS workshop is free to attend and you don't need to be an SPXXL >member to attend the GPFS workshop part of their event, however >registration is necessary for room/catering reasons. > >As the workshop is being organised SPXXL, registration is is via the SPXXL >website at: https://www.spxxl.org/?q=Garching-2016 >(note the option to register for the GPFS workshop isn't live yet, but >advance notice is always helpful for people planning to travel!). > >They are also interested in anyone who would like to do a user talk at the >event, if so, please contact Stefano directly - sgorini at cscs.ch. > >The agenda for the workshop is still being defined and will be published >in January. > >We're of course always happy to advertise user group events happening in >other locations, so if you are aware of events in other territories, >please let me know and we can add then to the web-site and post here. > >Finally, most of us are probably out of the office for the Christmas >period, so we may be a little slower to respond than usual, and I'd like >to wish everyone a peaceful holiday season. > >Simon >(GPFS Chair) From service at metamodul.com Fri Jan 29 11:51:17 2016 From: service at metamodul.com (service at metamodul.com) Date: Fri, 29 Jan 2016 12:51:17 +0100 (CET) Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering Message-ID: <561812564.90518.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> Hi i saw in the GPFS Forum somebody mentioning IBM Spectrum Scale transparent cloud tiering http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html Thus the question. Does somebody knows how that - the tiering into clould services - is technical done and what limitations exist ? tia Hajo -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Jan 29 12:17:02 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 29 Jan 2016 12:17:02 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering Message-ID: Hi Hajo I was involved in a closed beta of this service. You can read a bit more about it from the developer at this IBM DevelperWorks posting: https://www.ibm.com/developerworks/community/forums/html/topic?id=e9a43f16-4ab2-4edf-a253-83931840895e&ps=25 If this doesn?t cover your questions, let me know and I can give a bit more detail. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of "service at metamodul.com" > Reply-To: gpfsug main discussion list > Date: Friday, January 29, 2016 at 5:51 AM To: gpfsug main discussion list > Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering Hi i saw in the GPFS Forum somebody mentioning IBM Spectrum Scale transparent cloud tiering http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html Thus the question. Does somebody knows how that - the tiering into clould services - is technical done and what limitations exist ? tia Hajo -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Fri Jan 29 12:17:34 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 29 Jan 2016 12:17:34 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: <561812564.90518.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> References: <561812564.90518.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> Message-ID: <201601291217.u0TCHfPa014059@d06av11.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From service at metamodul.com Fri Jan 29 14:46:50 2016 From: service at metamodul.com (service at metamodul.com) Date: Fri, 29 Jan 2016 15:46:50 +0100 (CET) Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: References: Message-ID: <749152688.97744.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> Hi Robert, i refered to your posting i assume ^_^ Note the following is from what I know, Since i did not had any change to work with GPFS in the last 2 years my knowledge will be outdated. The current GPFS tiering options depending on the DMAPI which i am not a big fan since it had in the past some limitations, In past i read some talking about "lightweight callbacks" - thats what i remember as the name - which could hook into the stream of open or write process ( to/from a gpfs ) Thus in case the new solution is still based on DMAPI no further info is required. If not i would like to know a little bit more ... if possible, Cheers Hajo From Robert.Oesterlin at nuance.com Fri Jan 29 15:00:12 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 29 Jan 2016 15:00:12 +0000 Subject: [gpfsug-discuss] BM Spectrum Scale transparent cloud tiering Message-ID: <8505A552-5410-4F70-AA77-3DE5EF54BE09@nuance.com> Without getting into a whole lot of detail - The service is not based around the existing DMAPI interface. This service uses the Cluster Export Service (CES) nodes in GPFS 4.2 to perform the work. There is a process running on these nodes that?s configured to use a cloud provider and it performs the data migration between cloud and GPFS. It can be done using an automated process or by manual policy migration. I?m not sure what level of technical detail I can share on the mailing list ? if you reply to Rob Basham?s post on DeveloperWorks I?m sure he can fill in the lede lot detail you need. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: "service at metamodul.com" > Date: Friday, January 29, 2016 at 8:46 AM To: Robert Oesterlin >, gpfsug main discussion list > Subject: Re: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering Hi Robert, i refered to your posting i assume ^_^ Note the following is from what I know, Since i did not had any change to work with GPFS in the last 2 years my knowledge will be outdated. The current GPFS tiering options depending on the DMAPI which i am not a big fan since it had in the past some limitations, In past i read some talking about "lightweight callbacks" - thats what i remember as the name - which could hook into the stream of open or write process ( to/from a gpfs ) Thus in case the new solution is still based on DMAPI no further info is required. If not i would like to know a little bit more ... if possible, Cheers Hajo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Fri Jan 29 15:08:17 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 29 Jan 2016 15:08:17 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: <749152688.97744.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> References: <749152688.97744.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> Message-ID: <1454080097.9807.47.camel@buzzard.phy.strath.ac.uk> On Fri, 2016-01-29 at 15:46 +0100, service at metamodul.com wrote: > Hi Robert, > i refered to your posting i assume ^_^ > > Note the following is from what I know, Since i did not had any change to work > with GPFS in the last 2 years my knowledge will be outdated. > > The current GPFS tiering options depending on the DMAPI which i am not a big fan > since it had in the past some limitations, > In past i read some talking about "lightweight callbacks" - thats what i > remember as the name - which could hook into the stream of open or write process > ( to/from a gpfs ) Er no the DMAPI was only ever required for HSM as far as I am aware. Now while HSM is a form of tiering, in GPFS parlance tiering generally referred to disk pools of varying "speeds", with file placement and movement being done via the policy engine. I still remain to be convinced that tiering to some third party cloud provider will ever be cost effective mind you. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From S.J.Thompson at bham.ac.uk Fri Jan 29 15:15:46 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 29 Jan 2016 15:15:46 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: <1454080097.9807.47.camel@buzzard.phy.strath.ac.uk> References: <749152688.97744.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> <1454080097.9807.47.camel@buzzard.phy.strath.ac.uk> Message-ID: >Er no the DMAPI was only ever required for HSM as far as I am aware. Now >while HSM is a form of tiering, in GPFS parlance tiering generally >referred to disk pools of varying "speeds", with file placement and >movement being done via the policy engine. DMAPI can do more than just HSM though. I think DDN WOS bridge uses it (Vic or someone can correct me here). And LWE could *potentially* do some of those other aspects that people might want to do. >I still remain to be convinced that tiering to some third party cloud >provider will ever be cost effective mind you. Amazon Glacier maybe? Possibly if you have data you really don?t want to touch... Now my understanding though is that its an S3 (probably) interface, so it needn't be third party, nor public cloud storage necessarily. Of course MCStore is hopefully on the list for the next Meet the Devs in Oxford (just a few places left), and on my list to get on the agenda for the May UG meeting as well. Simon From viccornell at gmail.com Fri Jan 29 15:21:49 2016 From: viccornell at gmail.com (Vic Cornell) Date: Fri, 29 Jan 2016 15:21:49 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: References: <749152688.97744.fecb8dfa-b15c-42ac-98ff-2fc085ed2a94.open-xchange@email.1und1.de> <1454080097.9807.47.camel@buzzard.phy.strath.ac.uk> Message-ID: <0168C793-7902-42C6-9118-22C26B64DA6B@gmail.com> Simon is right - we are using DMAPI to tier off to WOS our object store and to StrongBox NAS-fronted tape. We are very interested in the new ?lightweight? protocols as they may allow us to do stuff that DMAPI makes hard - like tiering off to more than one DMAPI service from a single filesystem. If the interface is S3 then it would open the doors to tiering off to any on-prem ?cloud? solution that supports S3 sufficiently well ( ours does ;-) ) Cheers, Vic > On 29 Jan 2016, at 15:15, Simon Thompson (Research Computing - IT Services) wrote: > > >> Er no the DMAPI was only ever required for HSM as far as I am aware. Now >> while HSM is a form of tiering, in GPFS parlance tiering generally >> referred to disk pools of varying "speeds", with file placement and >> movement being done via the policy engine. > > DMAPI can do more than just HSM though. I think DDN WOS bridge uses it > (Vic or someone can correct me here). And LWE could *potentially* do some > of those other aspects that people might want to do. > >> I still remain to be convinced that tiering to some third party cloud >> provider will ever be cost effective mind you. > > Amazon Glacier maybe? Possibly if you have data you really don?t want to > touch... > > Now my understanding though is that its an S3 (probably) interface, so it > needn't be third party, nor public cloud storage necessarily. > > Of course MCStore is hopefully on the list for the next Meet the Devs in > Oxford (just a few places left), and on my list to get on the agenda for > the May UG meeting as well. > > Simon > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Fri Jan 29 15:27:57 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 29 Jan 2016 15:27:57 +0000 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering Message-ID: Take a look at the new Transparent Cloud Tiering ? it works with S3 and on-prem cloud as well ? we have a local swift cluster and it?s works perfectly. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Vic Cornell > Reply-To: gpfsug main discussion list > Date: Friday, January 29, 2016 at 9:21 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering We are very interested in the new ?lightweight? protocols as they may allow us to do stuff that DMAPI makes hard - like tiering off to more than one DMAPI service from a single filesystem. If the interface is S3 then it would open the doors to tiering off to any on-prem ?cloud? solution that supports S3 sufficiently well ( ours does ;-) ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 29 15:44:20 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 29 Jan 2016 10:44:20 -0500 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: References: Message-ID: <201601291544.u0TFiP0x021813@d03av04.boulder.ibm.com> Here's a link for Transparent Cloud Tiering, IBM style http://www.storagereview.com/ibm_announces_open_beta_for_spectrum_scale_transparent_cloud_tiering -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 29 15:55:56 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 29 Jan 2016 10:55:56 -0500 Subject: [gpfsug-discuss] IBM Spectrum Scale transparent cloud tiering In-Reply-To: <201601291544.u0TFiP0x021813@d03av04.boulder.ibm.com> References: <201601291544.u0TFiP0x021813@d03av04.boulder.ibm.com> Message-ID: <201601291556.u0TFu3KQ019149@d03av04.boulder.ibm.com> >From the press release: "And Spectrum Scale extends its policy-driven targeting, migration, encryption and unified access to the data in the cloud-scale object storage." Take care with the "encryption" claim. Spectrum Scale can encrypt your data as it is stored onto Spectrum Scale controlled disks (NSDs)... I could be wrong, but last I looked, there was no way to export or backup the encrypted data blocks. Naturally when you cp, dd, tar, a file to another system you are getting the file data "in the clear". OTOH, of course if you FlashCopy a volume that is holding an encrypted Spectrum Scale file system, then those data blocks are encrypted. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Jan 29 17:18:19 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 29 Jan 2016 12:18:19 -0500 Subject: [gpfsug-discuss] BM Spectrum Scale transparent cloud tiering In-Reply-To: <8505A552-5410-4F70-AA77-3DE5EF54BE09@nuance.com> References: <8505A552-5410-4F70-AA77-3DE5EF54BE09@nuance.com> Message-ID: <201601291718.u0THIPLr009799@d01av03.pok.ibm.com> Since this official IBM website (pre)announces transparent cloud tiering ... http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html?ce=sm6024&cmp=IBMSocial&ct=M16402YW&cm=h&IIO=BSYS&csr=blog&cr=casyst&ccy=us&s_tact=M16402YW And since Oesterlin mentioned Cluster Export Service (CES), please allow me to (hopefully!) clarify: Transparent Cloud Tiering uses some new interfaces and functions within Spectrum Scale, it is not "just a rehash" of the long existing DMAPI HSM support. Transparent Cloud Tiering allows one to dynamically migrate Spectrum Scale files to and from foreign file and/or object stores. on the other hand ... Cluster Export Service, allows one to access Spectrum Scale files with foreign protocols, such as NFS, SMB, and Object(OpenStack) I suppose one could deploy both, using Spectrum Scale with Cluster Export Service for local, fast, immediate access to "hot" file and objects and some foreign object service, such as Amazon S3 or Cleversafe for long term "cold" storage. Oh, and just to add to the mix, in case you haven't heard yet, Cleversafe is a fairly recent IBM acquisition, http://www-03.ibm.com/press/us/en/pressrelease/47776.wss -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at nordquist.info Fri Jan 29 19:19:22 2016 From: russell at nordquist.info (Russell Nordquist) Date: Fri, 29 Jan 2016 14:19:22 -0500 Subject: [gpfsug-discuss] NYC GPFS job opportunity In-Reply-To: <4C76D5D8-E149-4740-B98B-71D9D146AF76@nordquist.info> References: <4C76D5D8-E149-4740-B98B-71D9D146AF76@nordquist.info> Message-ID: <72F2C2C4-4D36-4A28-BF4D-ADA3CC0B24DF@nordquist.info> An additional detail?..this position could be filled in London as well. thank you russell > On Jan 27, 2016, at 9:17 PM, Russell Nordquist wrote: > > Since job posting are the theme of the week, here is one in NYC to work on GPFS + more scale out storage: > > https://jobs.pdtpartners.com/?gh_jid=151475 > > russell > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Fri Jan 29 19:31:37 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 29 Jan 2016 19:31:37 +0000 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) Message-ID: We have recently purchased ESS appliance from IBM (GL6) with 1.5PT of storage. We are in planning stages of implementation. We would like to migrate date from our existing GPFS installation (around 300TB) to new solution. We were planning of adding ESS to our existing GPFS cluster and adding its disks and then deleting our old disks and having the data migrated this way. However, our existing block size on our projects filesystem is 1M and in order to extract as much performance out of ESS we would like its filesystem created with larger block size. Besides rsync do you have any suggestions of how to do this without downtime and in fastest way possible? I have looked at AFM but it does not seem to migrate quotas and filesets so that may not be an optimal solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Douglas.Hughes at DEShawResearch.com Fri Jan 29 19:35:24 2016 From: Douglas.Hughes at DEShawResearch.com (Hughes, Doug) Date: Fri, 29 Jan 2016 19:35:24 +0000 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) In-Reply-To: References: Message-ID: <43610bd02e384543a68242a9597224bd@mbxtoa1.winmail.deshaw.com> I have found that a tar pipe is much faster than rsync for this sort of thing. The fastest of these is ?star? (schily tar). On average it is about 2x-5x faster than rsync for doing this. After one pass with this, you can use rsync for a subsequent or last pass synch. e.g. $ cd /export/gpfs1/foo $ star ?c H=xtar | (cd /export/gpfs2/foo; star ?xp) This also will not preserve filesets and quotas, though. You should be able to automate that with a little bit of awk, perl, or whatnot. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Damir Krstic Sent: Friday, January 29, 2016 2:32 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) We have recently purchased ESS appliance from IBM (GL6) with 1.5PT of storage. We are in planning stages of implementation. We would like to migrate date from our existing GPFS installation (around 300TB) to new solution. We were planning of adding ESS to our existing GPFS cluster and adding its disks and then deleting our old disks and having the data migrated this way. However, our existing block size on our projects filesystem is 1M and in order to extract as much performance out of ESS we would like its filesystem created with larger block size. Besides rsync do you have any suggestions of how to do this without downtime and in fastest way possible? I have looked at AFM but it does not seem to migrate quotas and filesets so that may not be an optimal solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Jan 29 19:49:38 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 29 Jan 2016 19:49:38 +0000 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) Message-ID: Not talking about ESS specifically, but It?s not possible to change the block size of an existing file system ? you are pretty much stuck with file copy. Something I wish IBM would address, I?d love to change the block size of some of my file system too. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 From: > on behalf of Damir Krstic > Reply-To: gpfsug main discussion list > Date: Friday, January 29, 2016 at 1:31 PM To: gpfsug main discussion list > Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) We were planning of adding ESS to our existing GPFS cluster and adding its disks and then deleting our old disks and having the data migrated this way. However, our existing block size on our projects filesystem is 1M and in order to extract as much performance out of ESS we would like its filesystem created with larger block size. Besides rsync do you have any suggestions of how to do this without downtime and in fastest way possible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 29 20:38:44 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 29 Jan 2016 15:38:44 -0500 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance(GPFS4.1) In-Reply-To: References: Message-ID: <201601292039.u0TKdaLo012761@d03av04.boulder.ibm.com> mmbackupconfig may be of some help. The output is eyeball-able, so one could tweak and then feed into mmrestoreconfig on the new system. Even if you don't use mmrestoreconfig, you might like to have the info collected by mmbackupconfig. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.bergman at uphs.upenn.edu Fri Jan 29 22:04:01 2016 From: mark.bergman at uphs.upenn.edu (Mark Bergman) Date: Fri, 29 Jan 2016 17:04:01 -0500 Subject: [gpfsug-discuss] what's on a 'dataOnly' disk? Message-ID: <20160129170401.0ec9f72e@uphs.upenn.edu> What's on a 'dataOnly' GPFS 3.5.x NSD besides data and the NSD disk header, if anything? I'm trying to understand some file corruption, and one potential explanation would be if a (non-GPFS) server wrote to a LUN used as a GPFS dataOnly NSD. We are not seeing any 'I/O' or filesystem errors, mmfsck (online) doesn't detect any errors, and all NSDs are usable. However, some files seem to have changes in content, with no changes in metadata (modify timestamp, ownership), including files with the GPFS "immutable" ACL set. If an NSD was changed outside of GPFS control, would mmfsck detect filesystem errors, or would the GPFS filesystem be consistent, even though the content of some of the data blocks was altered? Is there any metadata or checksum information maintained by GPFS, or any means of doing a consistency check of the contents of files that would correlate with blocks stored on a particular NSD? Thanks, Mark From oehmes at gmail.com Fri Jan 29 22:36:31 2016 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 29 Jan 2016 14:36:31 -0800 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) In-Reply-To: <43610bd02e384543a68242a9597224bd@mbxtoa1.winmail.deshaw.com> References: <43610bd02e384543a68242a9597224bd@mbxtoa1.winmail.deshaw.com> Message-ID: Doug, This won't really work if you make use of ACL's or use special GPFS extended attributes or set quotas, filesets, etc so unfortunate the answer is you need to use a combination of things and there is work going on to make some of this simpler (e.g. for ACL's) , but its a longer road to get there. so until then you need to think about multiple aspects . 1. you need to get the data across and there are various ways to do this. a) AFM is the simplest of all as it not just takes care of ACL's and extended attributes and alike as it understands the GPFS internals it also is operating in parallel can prefetch data, etc so its a efficient way to do this but as already pointed out doesn't transfer quota or fileset informations. b) you can either use rsync or any other pipe based copy program. the downside is that they are typical single threaded and do a file by file approach, means very metadata intensive on the source as well as target side and cause a lot of ios on both side. c) you can use the policy engine to create a list of files to transfer to at least address the single threaded scan part, then partition the data and run multiple instances of cp or rsync in parallel, still doesn't fix the ACL / EA issues, but the data gets there faster. 2. you need to get ACL/EA informations over too. there are several command line options to dump the data and restore it, they kind of suffer the same problem as data transfers , which is why using AFM is the best way of doing this if you rely on ACL/EA informations. 3. transfer quota / fileset infos. there are several ways to do this, but all require some level of scripting to do this. if you have TSM/HSM you could also transfer the data using SOBAR it's described in the advanced admin book. sven On Fri, Jan 29, 2016 at 11:35 AM, Hughes, Doug < Douglas.Hughes at deshawresearch.com> wrote: > I have found that a tar pipe is much faster than rsync for this sort of > thing. The fastest of these is ?star? (schily tar). On average it is about > 2x-5x faster than rsync for doing this. After one pass with this, you can > use rsync for a subsequent or last pass synch. > > > > e.g. > > $ cd /export/gpfs1/foo > > $ star ?c H=xtar | (cd /export/gpfs2/foo; star ?xp) > > > > This also will not preserve filesets and quotas, though. You should be > able to automate that with a little bit of awk, perl, or whatnot. > > > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: > gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Damir Krstic > *Sent:* Friday, January 29, 2016 2:32 PM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance > (GPFS4.1) > > > > We have recently purchased ESS appliance from IBM (GL6) with 1.5PT of > storage. We are in planning stages of implementation. We would like to > migrate date from our existing GPFS installation (around 300TB) to new > solution. > > > > We were planning of adding ESS to our existing GPFS cluster and adding its > disks and then deleting our old disks and having the data migrated this > way. However, our existing block size on our projects filesystem is 1M and > in order to extract as much performance out of ESS we would like its > filesystem created with larger block size. Besides rsync do you have any > suggestions of how to do this without downtime and in fastest way possible? > > > > I have looked at AFM but it does not seem to migrate quotas and filesets > so that may not be an optimal solution. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Sat Jan 30 07:07:12 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Sat, 30 Jan 2016 08:07:12 +0100 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance(GPFS4.1) In-Reply-To: <201601292039.u0TKdaLo012761@d03av04.boulder.ibm.com> References: <201601292039.u0TKdaLo012761@d03av04.boulder.ibm.com> Message-ID: <56AC6120.7070904@ugent.be> we wrapped something base on zookeeper around rsync to be able to use rsync in parallel by splitting the path in subdirectories, and distribute those https://github.com/hpcugent/vsc-zk works really well if the number of files in directories is somewhat balanced. we use it to rsync some gpfs filesystems (200TB, 100M inodes ;) stijn On 01/29/2016 09:38 PM, Marc A Kaplan wrote: > mmbackupconfig may be of some help. The output is eyeball-able, so one > could tweak and then feed into mmrestoreconfig on the new system. > Even if you don't use mmrestoreconfig, you might like to have the info > collected by mmbackupconfig. > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From sfadden at us.ibm.com Fri Jan 29 20:16:00 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Fri, 29 Jan 2016 17:16:00 -0300 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance(GPFS4.1) In-Reply-To: References: Message-ID: <201601301021.u0UALRDW026361@d03av02.boulder.ibm.com> File copy OR AFM. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 01/29/2016 05:50 PM Subject: Re: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) Sent by: gpfsug-discuss-bounces at spectrumscale.org Not talking about ESS specifically, but It?s not possible to change the block size of an existing file system ? you are pretty much stuck with file copy. Something I wish IBM would address, I?d love to change the block size of some of my file system too. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 From: on behalf of Damir Krstic Reply-To: gpfsug main discussion list Date: Friday, January 29, 2016 at 1:31 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance (GPFS4.1) We were planning of adding ESS to our existing GPFS cluster and adding its disks and then deleting our old disks and having the data migrated this way. However, our existing block size on our projects filesystem is 1M and in order to extract as much performance out of ESS we would like its filesystem created with larger block size. Besides rsync do you have any suggestions of how to do this without downtime and in fastest way possible? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Sat Jan 30 17:09:46 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 30 Jan 2016 12:09:46 -0500 Subject: [gpfsug-discuss] migrating data from GPFS3.5 to ESS appliance(GPFS4.1) In-Reply-To: <56AC6120.7070904@ugent.be> References: <201601292039.u0TKdaLo012761@d03av04.boulder.ibm.com> <56AC6120.7070904@ugent.be> Message-ID: <201601301709.u0UH9qtD016530@d03av03.boulder.ibm.com> You may also want to use and/or adapt samples/ilm/tspcp which uses mmapplypolicy to drive parallel cp commands. The script was written by my colleague and manager, but I'm willing to entertain questions and suggestions... Here are some of the comments: # Run "cp" in parallel over a list of files/directories # # This is a sample script showing how to use the GPFS ILM policy # to copy a list of files or directories. Its takes advantage of GPFS 3.4 # input file list argument to mmapplypolicy to limit the directory scan # to only the files and directories in the input file list. It also uses # the GPFS 3.5 feature for DIRECTORY_HASH to sort candidate list by directories # and execute them in a top-down order. # # This script converts the list of files from the argument to a file list file # as input to mmapplypolicy. It then generates a simple ILM policy file to # match all of the files in the list and if -r is specified to match all files # and directories below any directories that are specified. The files are # then sorted by policy into directory order and dispatched in top down order. # Each work unit is assigned to a node and executed by a call to this script, # which simply reads its input file list and calls "cp" on each. The # original script scans stdout from each of the workers looking for # messages from "cp" or any possible errors. The script was written by my colleague and manager, but I'm willing to entertain questions and suggestions... -------------- next part -------------- An HTML attachment was scrubbed... URL: