From p.childs at qmul.ac.uk Fri Jul 1 10:50:27 2022 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 1 Jul 2022 09:50:27 +0000 Subject: [gpfsug-discuss] Quota Idea / RFE Message-ID: I've opened a Spectrum Scale Idea for Enhancement, https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-857 I suspect I'm not the first one trying to achieve this so it might be worth upvoting the idea. Basically I'd like to see some way to copy quotas between filesystems, either within scale and outside that can in theory be already done with mmedquota -F but there is no way to dump the quotas in this format currently. Thanks Peter Childs From p.ward at nhm.ac.uk Tue Jul 5 13:11:19 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Tue, 5 Jul 2022 12:11:19 +0000 Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 In-Reply-To: References: Message-ID: Hi, Are there any slides or recordings of this user group meeting. I was too unwell to attend. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of chair at gpfsug.org Sent: 13 June 2022 12:13 To: gpfsug-discuss at spectrumscaleug.org Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 Hi all, Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on 30th June 2022. Registration is open at https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eventbrite.co.uk%2Fe%2Fspectrum-scale-user-group-registration-321290978967&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc30cc58758204547cb5e08da4d2dcf71%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637907156400455377%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=krX4xVDX1KEufllOCl0l3H0uI76ZDdoj0jJgQ53IUv0%3D&reserved=0 The agenda is below 9:30 - 10:00 Arrivals and refreshments 10:00 - 10:15 Introductions and committee updates, Paul Tomlinson Group Chair and Caroline Bradley, Group Secretary 10:15 - 10:35 Strategy Update (IBM) 10:35 - 10:55 New S3 Access for AI and Analytics (IBM) 10:55 - 11:20 What is new in Spectrum Scale and ESS (IBM) 11:20 - 11:40 nvidia GPUDirect Storage (IBM) 11:40 - 12:00 New Deplyoment using Ansible and Terraform (IBM) 12:00 - 13:00 Buffet Lunch with viewings of :- Quantum, Immersive Room and AI Cars 13:00 - 13:20 Migrating Spectrum Scale using Atmepo Software (Atempo) 13:20 - 13:40 Monitoring and Serviceability Enhancements (IBM) 13:40 - 14:00 Spectrum Scale and Spectrum Discover for Data Management University of Oslo 14:00 - 14:30 Performance update (IBM) 14:30 - 15:00 Tea Break with viewing of Boston Dynamics, Spot the Robot Dog 15:00 - 15:30 Data orchestration across the global data platform (IBM) 15:30 - 16:00 AFM Deep Dive (IBM) 16:00 - 17:00 Group discussion, Challenges, Experiences and Questions Led by Paul Tomlinson 17:00 Drinks reception Thanks Paul _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc30cc58758204547cb5e08da4d2dcf71%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637907156400455377%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Aw8EnhKBs60rNk2UBnB%2BF8ekc1L9lvXDxpwqPJQ22lg%3D&reserved=0 From p.ward at nhm.ac.uk Tue Jul 5 13:11:19 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Tue, 5 Jul 2022 12:11:19 +0000 Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 In-Reply-To: References: Message-ID: Hi, Are there any slides or recordings of this user group meeting. I was too unwell to attend. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of chair at gpfsug.org Sent: 13 June 2022 12:13 To: gpfsug-discuss at spectrumscaleug.org Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 Hi all, Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on 30th June 2022. Registration is open at https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eventbrite.co.uk%2Fe%2Fspectrum-scale-user-group-registration-321290978967&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc30cc58758204547cb5e08da4d2dcf71%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637907156400455377%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=krX4xVDX1KEufllOCl0l3H0uI76ZDdoj0jJgQ53IUv0%3D&reserved=0 The agenda is below 9:30 - 10:00 Arrivals and refreshments 10:00 - 10:15 Introductions and committee updates, Paul Tomlinson Group Chair and Caroline Bradley, Group Secretary 10:15 - 10:35 Strategy Update (IBM) 10:35 - 10:55 New S3 Access for AI and Analytics (IBM) 10:55 - 11:20 What is new in Spectrum Scale and ESS (IBM) 11:20 - 11:40 nvidia GPUDirect Storage (IBM) 11:40 - 12:00 New Deplyoment using Ansible and Terraform (IBM) 12:00 - 13:00 Buffet Lunch with viewings of :- Quantum, Immersive Room and AI Cars 13:00 - 13:20 Migrating Spectrum Scale using Atmepo Software (Atempo) 13:20 - 13:40 Monitoring and Serviceability Enhancements (IBM) 13:40 - 14:00 Spectrum Scale and Spectrum Discover for Data Management University of Oslo 14:00 - 14:30 Performance update (IBM) 14:30 - 15:00 Tea Break with viewing of Boston Dynamics, Spot the Robot Dog 15:00 - 15:30 Data orchestration across the global data platform (IBM) 15:30 - 16:00 AFM Deep Dive (IBM) 16:00 - 17:00 Group discussion, Challenges, Experiences and Questions Led by Paul Tomlinson 17:00 Drinks reception Thanks Paul _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc30cc58758204547cb5e08da4d2dcf71%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637907156400455377%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Aw8EnhKBs60rNk2UBnB%2BF8ekc1L9lvXDxpwqPJQ22lg%3D&reserved=0 From Paul.Tomlinson at awe.co.uk Tue Jul 5 13:47:54 2022 From: Paul.Tomlinson at awe.co.uk (Tomlinson Paul J (DS&T) AWE) Date: Tue, 5 Jul 2022 12:47:54 +0000 Subject: [gpfsug-discuss] EXTERNAL: Re: UK Spectrum Scale User Group meeting 30th June 2022 In-Reply-To: References: Message-ID: Hi Paul, We are in the process of collecting the presentations for uploading to the website. Regards Paul Tomlinson -----Original Message----- From: gpfsug-discuss On Behalf Of Paul Ward Sent: 05 July 2022 13:11 To: gpfsug main discussion list ; gpfsug-discuss at spectrumscaleug.org Subject: EXTERNAL: Re: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 PHISHING ALERT - This email originated from outside of AWE DO YOU TRUST THIS EMAIL? If not, click here for more guidance or use the 'Report Message' button on your Outlook toolbar to send this message to the PhishTank. Hi, Are there any slides or recordings of this user group meeting. I was too unwell to attend. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of chair at gpfsug.org Sent: 13 June 2022 12:13 To: gpfsug-discuss at spectrumscaleug.org Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 Hi all, Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on 30th June 2022. Registration is open at https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eventbrite.co.uk%2Fe%2Fspectrum-scale-user-group-registration-321290978967&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IX%2B%2FxIzI9bAzMBTdoXSi2QnqXEQoZsOLnDRc7EjMkxE%3D&reserved=0 The agenda is below 9:30 - 10:00 Arrivals and refreshments 10:00 - 10:15 Introductions and committee updates, Paul Tomlinson Group Chair and Caroline Bradley, Group Secretary 10:15 - 10:35 Strategy Update (IBM) 10:35 - 10:55 New S3 Access for AI and Analytics (IBM) 10:55 - 11:20 What is new in Spectrum Scale and ESS (IBM) 11:20 - 11:40 nvidia GPUDirect Storage (IBM) 11:40 - 12:00 New Deplyoment using Ansible and Terraform (IBM) 12:00 - 13:00 Buffet Lunch with viewings of :- Quantum, Immersive Room and AI Cars 13:00 - 13:20 Migrating Spectrum Scale using Atmepo Software (Atempo) 13:20 - 13:40 Monitoring and Serviceability Enhancements (IBM) 13:40 - 14:00 Spectrum Scale and Spectrum Discover for Data Management University of Oslo 14:00 - 14:30 Performance update (IBM) 14:30 - 15:00 Tea Break with viewing of Boston Dynamics, Spot the Robot Dog 15:00 - 15:30 Data orchestration across the global data platform (IBM) 15:30 - 16:00 AFM Deep Dive (IBM) 16:00 - 17:00 Group discussion, Challenges, Experiences and Questions Led by Paul Tomlinson 17:00 Drinks reception Thanks Paul _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mjysGwVORVrWQ3243uHImcSD8TW1GrdqAdahGxtaiXg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mjysGwVORVrWQ3243uHImcSD8TW1GrdqAdahGxtaiXg%3D&reserved=0 The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR From Paul.Tomlinson at awe.co.uk Tue Jul 5 13:47:54 2022 From: Paul.Tomlinson at awe.co.uk (Tomlinson Paul J (DS&T) AWE) Date: Tue, 5 Jul 2022 12:47:54 +0000 Subject: [gpfsug-discuss] EXTERNAL: Re: UK Spectrum Scale User Group meeting 30th June 2022 In-Reply-To: References: Message-ID: Hi Paul, We are in the process of collecting the presentations for uploading to the website. Regards Paul Tomlinson -----Original Message----- From: gpfsug-discuss On Behalf Of Paul Ward Sent: 05 July 2022 13:11 To: gpfsug main discussion list ; gpfsug-discuss at spectrumscaleug.org Subject: EXTERNAL: Re: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 PHISHING ALERT - This email originated from outside of AWE DO YOU TRUST THIS EMAIL? If not, click here for more guidance or use the 'Report Message' button on your Outlook toolbar to send this message to the PhishTank. Hi, Are there any slides or recordings of this user group meeting. I was too unwell to attend. Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk -----Original Message----- From: gpfsug-discuss On Behalf Of chair at gpfsug.org Sent: 13 June 2022 12:13 To: gpfsug-discuss at spectrumscaleug.org Subject: [gpfsug-discuss] UK Spectrum Scale User Group meeting 30th June 2022 Hi all, Just a reminder that the next UK User Group meeting will be taking place in London (IBM York Road) on 30th June 2022. Registration is open at https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eventbrite.co.uk%2Fe%2Fspectrum-scale-user-group-registration-321290978967&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IX%2B%2FxIzI9bAzMBTdoXSi2QnqXEQoZsOLnDRc7EjMkxE%3D&reserved=0 The agenda is below 9:30 - 10:00 Arrivals and refreshments 10:00 - 10:15 Introductions and committee updates, Paul Tomlinson Group Chair and Caroline Bradley, Group Secretary 10:15 - 10:35 Strategy Update (IBM) 10:35 - 10:55 New S3 Access for AI and Analytics (IBM) 10:55 - 11:20 What is new in Spectrum Scale and ESS (IBM) 11:20 - 11:40 nvidia GPUDirect Storage (IBM) 11:40 - 12:00 New Deplyoment using Ansible and Terraform (IBM) 12:00 - 13:00 Buffet Lunch with viewings of :- Quantum, Immersive Room and AI Cars 13:00 - 13:20 Migrating Spectrum Scale using Atmepo Software (Atempo) 13:20 - 13:40 Monitoring and Serviceability Enhancements (IBM) 13:40 - 14:00 Spectrum Scale and Spectrum Discover for Data Management University of Oslo 14:00 - 14:30 Performance update (IBM) 14:30 - 15:00 Tea Break with viewing of Boston Dynamics, Spot the Robot Dog 15:00 - 15:30 Data orchestration across the global data platform (IBM) 15:30 - 16:00 AFM Deep Dive (IBM) 16:00 - 17:00 Group discussion, Challenges, Experiences and Questions Led by Paul Tomlinson 17:00 Drinks reception Thanks Paul _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mjysGwVORVrWQ3243uHImcSD8TW1GrdqAdahGxtaiXg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7CPaul.Tomlinson%40awe.co.uk%7C9d3a28490c4c4990f82808da5e7fc386%7C72654b74be024361a62716b132e3fdd0%7C0%7C0%7C637926200080096685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mjysGwVORVrWQ3243uHImcSD8TW1GrdqAdahGxtaiXg%3D&reserved=0 The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR From TROPPENS at de.ibm.com Tue Jul 5 17:03:08 2022 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 5 Jul 2022 16:03:08 +0000 Subject: [gpfsug-discuss] Save the date - German User Meeting - Oct 19+20, 2022 Message-ID: Greetings, IBM plans to host a Spectrum Scale User Meeting at October 19+20, 2022 in Cologne. Details will follow. Ulf Troppens Senior Technical Staff Member Spectrum Scale Development IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen / Gesch?ftsf?hrung: David Faller Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Jul 8 14:55:15 2022 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 8 Jul 2022 15:55:15 +0200 Subject: [gpfsug-discuss] Quota Idea / RFE In-Reply-To: References: Message-ID: Maybe use something like this (untested): mmrepquota -u -v -f gpfs0 | awk -F: 'NR>2 {print "quota: device=gpfs1 command=setQuota type=USR id=" $9 " blockQuota=" $12 " blockLimit=" $13}' > /tmp/quota.stanza mmsetquota -F /tmp/quota.stanza -jf On Fri, Jul 1, 2022 at 11:51 AM Peter Childs wrote: > I've opened a Spectrum Scale Idea for Enhancement, > https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-857 I suspect I'm not > the first one trying to achieve this so it might be worth upvoting the idea. > > Basically I'd like to see some way to copy quotas between filesystems, > either within scale and outside that can in theory be already done with > mmedquota -F but there is no way to dump the quotas in this format > currently. > > Thanks > > > > > Peter Childs > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Jul 8 14:55:15 2022 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 8 Jul 2022 15:55:15 +0200 Subject: [gpfsug-discuss] Quota Idea / RFE In-Reply-To: References: Message-ID: Maybe use something like this (untested): mmrepquota -u -v -f gpfs0 | awk -F: 'NR>2 {print "quota: device=gpfs1 command=setQuota type=USR id=" $9 " blockQuota=" $12 " blockLimit=" $13}' > /tmp/quota.stanza mmsetquota -F /tmp/quota.stanza -jf On Fri, Jul 1, 2022 at 11:51 AM Peter Childs wrote: > I've opened a Spectrum Scale Idea for Enhancement, > https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-857 I suspect I'm not > the first one trying to achieve this so it might be worth upvoting the idea. > > Basically I'd like to see some way to copy quotas between filesystems, > either within scale and outside that can in theory be already done with > mmedquota -F but there is no way to dump the quotas in this format > currently. > > Thanks > > > > > Peter Childs > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Jul 8 17:03:04 2022 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 8 Jul 2022 12:03:04 -0400 Subject: [gpfsug-discuss] Quota Idea / RFE In-Reply-To: References: Message-ID: <34878856-c4be-9519-64fc-9354d4445c69@rutgers.edu> I have a similar issue related to AFM: you can have two different quotas on the AFM home and cache (which I can't really see why you would ever want to do, but whatever), and there's not currently a way to syncrhornize them even if you want to (which you almost always would). The behavior you get if you allow a cache to fill, for example, but being over quota on the at-home fileset is very strange (you get persistent load average on involved machines, etc., which are trying to play back writes). On 7/1/22 05:50, Peter Childs wrote: > I've opened a Spectrum Scale Idea for Enhancement, https://ibm-sys-storage.ideas.ibm.com/ideas/GPFS-I-857 I suspect I'm not the first one trying to achieve this so it might be worth upvoting the idea. > > Basically I'd like to see some way to copy quotas between filesystems, either within scale and outside that can in theory be already done with mmedquota -F but there is no way to dump the quotas in this format currently. > > Thanks -- #BlackLivesMatter ____ || \\UTGERS, |----------------------*O*------------------------ ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus || \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark `' From sthompson2 at lenovo.com Mon Jul 11 08:45:52 2022 From: sthompson2 at lenovo.com (Simon Thompson2) Date: Mon, 11 Jul 2022 07:45:52 +0000 Subject: [gpfsug-discuss] [External] Re: Quota Idea / RFE In-Reply-To: <34878856-c4be-9519-64fc-9354d4445c69@rutgers.edu> References: <34878856-c4be-9519-64fc-9354d4445c69@rutgers.edu> Message-ID: Home and cache can have different amounts of disk space used, so "replicating quotas" is mostly pointless, unless you populate cache with all the data at home. But yes, this was something that hurt when I was running AFM some time ago. We saw the AFM queues "stall" when this happened (and eventually crash). And then you have the problem that you might have different replication factors for your AFM home and cache, and that makes what the quota should be a hard question. E.g. my super fast ingest system (1 copy, cache) and my slow long-term store (2 copies, home). Simon Simon Thompson He/Him/His Sr. Manager, HPC Storage and Performance Chineham Business Park, Crockford Lane, Basingstoke, Hampshire, RG24 8WQ HPC Customer Solutions Lenovo UK +44 7788 320635 sthompson2 at lenovo.com ? Lenovo.com/uk Twitter?|?Instagram?|?Facebook?|?Linkedin?|?YouTube?|?Privacy -----Original Message----- From: gpfsug-discuss On Behalf Of Ryan Novosielski Sent: 08 July 2022 17:03 To: gpfsug-discuss at gpfsug.org Subject: [External] Re: [gpfsug-discuss] Quota Idea / RFE I have a similar issue related to AFM: you can have two different quotas on the AFM home and cache (which I can't really see why you would ever want to do, but whatever), and there's not currently a way to syncrhornize them even if you want to (which you almost always would). The behavior you get if you allow a cache to fill, for example, but being over quota on the at-home fileset is very strange (you get persistent load average on involved machines, etc., which are trying to play back writes). On 7/1/22 05:50, Peter Childs wrote: > I've opened a Spectrum Scale Idea for Enhancement, https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fibm-sys-storage.ideas.ibm.com%2Fideas%2FGPFS-I-857&data=05%7C01%7Csthompson2%40lenovo.com%7C3970173257fd4b75415f08da60fb9221%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C637928930859469707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zhBpDIU2B5vkOIhJCvEqAX4DGWgfngZT%2BzBsjZzPV5A%3D&reserved=0 I suspect I'm not the first one trying to achieve this so it might be worth upvoting the idea. > > Basically I'd like to see some way to copy quotas between filesystems, either within scale and outside that can in theory be already done with mmedquota -F but there is no way to dump the quotas in this format currently. > > Thanks -- #BlackLivesMatter ____ || \\UTGERS, |----------------------*O*------------------------ ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus || \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark `' _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Csthompson2%40lenovo.com%7C3970173257fd4b75415f08da60fb9221%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C637928930859469707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=azIehWp1xO7DtTFuwhHfkoKa8aH3bEdL%2FQmr%2FixmhA4%3D&reserved=0 From p.childs at qmul.ac.uk Mon Jul 11 09:50:42 2022 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 11 Jul 2022 08:50:42 +0000 Subject: [gpfsug-discuss] [External] Re: Quota Idea / RFE In-Reply-To: References: <34878856-c4be-9519-64fc-9354d4445c69@rutgers.edu> Message-ID: Indeed, copying quota's within AFM is not really a very good use/case. However, there are loads of other use case where you need to copy quota's between file systems, or want to change quota's where text file editing with mmedquota, mmsetquota or using default quota's is not flexible enough. Peter Get Outlook for Android ________________________________ From: gpfsug-discuss on behalf of Simon Thompson2 Sent: Monday, July 11, 2022 8:45:52 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] [External] Re: Quota Idea / RFE CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe. Home and cache can have different amounts of disk space used, so "replicating quotas" is mostly pointless, unless you populate cache with all the data at home. But yes, this was something that hurt when I was running AFM some time ago. We saw the AFM queues "stall" when this happened (and eventually crash). And then you have the problem that you might have different replication factors for your AFM home and cache, and that makes what the quota should be a hard question. E.g. my super fast ingest system (1 copy, cache) and my slow long-term store (2 copies, home). Simon Simon Thompson He/Him/His Sr. Manager, HPC Storage and Performance Chineham Business Park, Crockford Lane, Basingstoke, Hampshire, RG24 8WQ HPC Customer Solutions Lenovo UK +44 7788 320635 sthompson2 at lenovo.com Lenovo.com/uk Twitter | Instagram | Facebook | Linkedin | YouTube | Privacy -----Original Message----- From: gpfsug-discuss On Behalf Of Ryan Novosielski Sent: 08 July 2022 17:03 To: gpfsug-discuss at gpfsug.org Subject: [External] Re: [gpfsug-discuss] Quota Idea / RFE I have a similar issue related to AFM: you can have two different quotas on the AFM home and cache (which I can't really see why you would ever want to do, but whatever), and there's not currently a way to syncrhornize them even if you want to (which you almost always would). The behavior you get if you allow a cache to fill, for example, but being over quota on the at-home fileset is very strange (you get persistent load average on involved machines, etc., which are trying to play back writes). On 7/1/22 05:50, Peter Childs wrote: > I've opened a Spectrum Scale Idea for Enhancement, https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fibm-sys-storage.ideas.ibm.com%2Fideas%2FGPFS-I-857&data=05%7C01%7Csthompson2%40lenovo.com%7C3970173257fd4b75415f08da60fb9221%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C637928930859469707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zhBpDIU2B5vkOIhJCvEqAX4DGWgfngZT%2BzBsjZzPV5A%3D&reserved=0 I suspect I'm not the first one trying to achieve this so it might be worth upvoting the idea. > > Basically I'd like to see some way to copy quotas between filesystems, either within scale and outside that can in theory be already done with mmedquota -F but there is no way to dump the quotas in this format currently. > > Thanks -- #BlackLivesMatter ____ || \\UTGERS, |----------------------*O*------------------------ ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus || \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark `' _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Csthompson2%40lenovo.com%7C3970173257fd4b75415f08da60fb9221%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C637928930859469707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=azIehWp1xO7DtTFuwhHfkoKa8aH3bEdL%2FQmr%2FixmhA4%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.ward at nhm.ac.uk Tue Jul 12 16:38:55 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Tue, 12 Jul 2022 15:38:55 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy Message-ID: Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can't find what 'KB_ Ill' means but nothing is migrating. Migrations in other policies work, but this is a 'recall' policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 5356 bytes Desc: image001.jpg URL: From keith.vickers at hpe.com Thu Jul 14 13:10:43 2022 From: keith.vickers at hpe.com (Vickers, Keith) Date: Thu, 14 Jul 2022 12:10:43 +0000 Subject: [gpfsug-discuss] Hello everyone Message-ID: Hi everyone, My new contact details at HPE. Everyone knows that HPE now support and sell Spectrum Scale on their E1000 (ClusterStor) storage right? Many thanks, Keith Keith Vickers Business Development Artificial Intelligence UKIMESA Mobile: +44 (0)7341 509 628 [HPE logo] Hewlett-Packard Limited | Registered Office: Ground Floor, 210 Wharfedale Road, Winnersh Triangle, Berkshire, RG41 5TP | Registered Number: 00690597 England | VAT number: GB 314 1496 79 This e-mail may contain confidential and/or legally privileged material for the sole use of the intended recipient. If you are not the intended recipient (or authorized to receive for the recipient) please contact the sender by reply e-mail and delete all copies of this message. If you are receiving this message internally within the Hewlett Packard Enterprise group of companies, you should consider the contents "CONFIDENTIAL". -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8270 bytes Desc: image001.png URL: From scale at us.ibm.com Mon Jul 18 20:35:26 2022 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 18 Jul 2022 15:35:26 -0400 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. From: "Paul Ward" To: "gpfsug-discuss at gpfsug.org" Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk A picture containing drawing Description automatically generated _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 17893393.jpg Type: image/jpeg Size: 5356 bytes Desc: not available URL: From p.ward at nhm.ac.uk Tue Jul 19 16:24:40 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Tue, 19 Jul 2022 15:24:40 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list ; Venkateswara R Puvvada Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From scale at us.ibm.com Tue Jul 19 19:12:20 2022 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 19 Jul 2022 23:42:20 +0530 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Paul Ward" To: "gpfsug main discussion list" Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list ; Venkateswara R Puvvada Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" To: "gpfsug-discuss at gpfsug.org" Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk A picture containing drawing Description automatically generated _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 17368645.jpg Type: image/jpeg Size: 5356 bytes Desc: not available URL: From scale at us.ibm.com Tue Jul 19 19:21:47 2022 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 19 Jul 2022 23:51:47 +0530 Subject: [gpfsug-discuss] How to shrink GPFS on DSSG's? In-Reply-To: References: Message-ID: Adding Steve and Wes to provide guidance on ESS commands. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Peter Childs" To: "gpfsug-discuss at spectrumscale.org" Date: 22-06-2022 03.36 PM Subject: Re: [gpfsug-discuss] [EXTERNAL] How to shrink GPFS on DSSG's? Sent by: "gpfsug-discuss" Having only just got an ESS I'm still learning how GNR works. as I read it there are currently two "breeds" of GNR, the version on the "DSS and ESS appliances" and the one in "Erasure Code Edition" As I understand it from past talks using mmdeldisk to remove a disk works fine in none GNR editions but is not the best way to do the task. My understanding is that you should mmchdisk suspend/empty # so new data is not put on the disk but the disk remains available for read. mmrestripefs -m # to move the data off the disk mmdeldisk # to actually remove the disk which should be fast as its already been emptied. We have done this with success in the past to migrate data between Raid6 arrays, with success. I believe there are some commands with mmvdisk to re-shape recovery groups in GNR but I've not as yet worked out how they work. Peter Childs ________________________________________ From: gpfsug-discuss on behalf of Jaime Pinto Sent: Monday, June 20, 2022 7:04 PM To: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] [gpfsug-discuss] How to shrink GPFS on DSSG's? CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe. I'm wondering if it's possible to shrink GPFS gracefully. I've seen some references to that effect on some presentations, however I can't find detailed instructions on any formal IBM documentation on how to do it. About 3 years ago we launched a new GPFS deployment with 3 DSS-G enclosures (9.6PB usable). Some 1.5 years later we added 2 more enclosures, for a total of 16PB, and only 7PB occupancy so far. Basically I'd like to return to the original 3 enclosures, and still maintain the (8+2p) parity level. Any suggestions? Thanks Jaime --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - www.scinet.utoronto.ca University of Toronto _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From laurence at qsplace.co.uk Wed Jul 20 07:40:55 2022 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Wed, 20 Jul 2022 06:40:55 +0000 Subject: [gpfsug-discuss] How to shrink GPFS on DSSG's? References: Message-ID: <7ff560e54d8828.f2bb3d16b78c7@qsplace.co.uk> Are you talking enclosures or subsystems here? i.e. is it 5 separate dss subsystems each with 2 x x86 controllers or did you add 2 enclosures to the existing 3 dss subsystems? If you added 2 new dss subsystems. Just suspend the disks on the 2 dss building blocks and remove the NSD?s created like any normal GPFS file system, the data will auto migrate to the remaining 3 building blocks. The 8+2 is local to the subsystem within the DA. Once done remove those dss controllers from gpfs and do as you please, either keep the vdisks and DA?s or rebuild them. If the latter, it becomes a bit more complex in fiddling with pdisks, and I don?t think it?s supported. ? Lauz IBM Spectrum Scale wrote: Adding Steve and Wes to provide guidance on ESS commands. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Peter Childs" ---22-06-2022 03.36.33 PM---Having only just got an ESS I'm still learning how GNR works. as I read it there are currently two " From: "Peter Childs" To: "gpfsug-discuss at spectrumscale.org" Date: 22-06-2022 03.36 PM Subject: Re: [gpfsug-discuss] [EXTERNAL] How to shrink GPFS on DSSG's? Sent by: "gpfsug-discuss" Having only just got an ESS I'm still learning how GNR works. as I read it there are currently two "breeds" of GNR, the version on the "DSS and ESS appliances" and the one in "Erasure Code Edition" As I understand it from past talks using mmdeldisk to remove a disk works fine in none GNR editions but is not the best way to do the task. My understanding is that you should mmchdisk suspend/empty # so new data is not put on the disk but the disk remains available for read. mmrestripefs -m # to move the data off the disk mmdeldisk # to actually remove the disk which should be fast as its already been emptied. We have done this with success in the past to migrate data between Raid6 arrays, with success. I believe there are some commands with mmvdisk to re-shape recovery groups in GNR but I've not as yet worked out how they work. Peter Childs ________________________________________ From: gpfsug-discuss on behalf of Jaime Pinto Sent: Monday, June 20, 2022 7:04 PM To: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] [gpfsug-discuss] How to shrink GPFS on DSSG's? CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe. I'm wondering if it's possible to shrink GPFS gracefully. I've seen some references to that effect on some presentations, however I can't find detailed instructions on any formal IBM documentation on how to do it. About 3 years ago we launched a new GPFS deployment with 3 DSS-G enclosures (9.6PB usable). Some 1.5 years later we added 2 more enclosures, for a total of 16PB, and only 7PB occupancy so far. Basically I'd like to return to the original 3 enclosures, and still maintain the (8+2p) parity level. Any suggestions? Thanks Jaime --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - www.scinet.utoronto.ca University of Toronto _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org Adding Steve and Wes to provide guidance on ESS commands. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Peter Childs" ---22-06-2022 03.36.33 PM---Having only just got an ESS I'm still learning how GNR works. as I read it there are currently two " From: "Peter Childs" To: "gpfsug-discuss at spectrumscale.org" Date: 22-06-2022 03.36 PM Subject: Re: [gpfsug-discuss] [EXTERNAL] How to shrink GPFS on DSSG's? Sent by: "gpfsug-discuss" Having only just got an ESS I'm still learning how GNR works. as I read it there are currently two "breeds" of GNR, the version on the "DSS and ESS appliances" and the one in "Erasure Code Edition" As I understand it from past talks using mmdeldisk to remove a disk works fine in none GNR editions but is not the best way to do the task. My understanding is that you should mmchdisk suspend/empty # so new data is not put on the disk but the disk remains available for read. mmrestripefs -m # to move the data off the disk mmdeldisk # to actually remove the disk which should be fast as its already been emptied. We have done this with success in the past to migrate data between Raid6 arrays, with success. I believe there are some commands with mmvdisk to re-shape recovery groups in GNR but I've not as yet worked out how they work. Peter Childs ________________________________________ From: gpfsug-discuss on behalf of Jaime Pinto Sent: Monday, June 20, 2022 7:04 PM To: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] [gpfsug-discuss] How to shrink GPFS on DSSG's? CAUTION: This email originated from outside of QMUL. Do not click links or open attachments unless you recognise the sender and know the content is safe. I'm wondering if it's possible to shrink GPFS gracefully. I've seen some references to that effect on some presentations, however I can't find detailed instructions on any formal IBM documentation on how to do it. About 3 years ago we launched a new GPFS deployment with 3 DSS-G enclosures (9.6PB usable). Some 1.5 years later we added 2 more enclosures, for a total of 16PB, and only 7PB occupancy so far. Basically I'd like to return to the original 3 enclosures, and still maintain the (8+2p) parity level. Any suggestions? Thanks Jaime --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - www.scinet.utoronto.ca University of Toronto _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From amey.gokhale at in.ibm.com Wed Jul 20 07:54:35 2022 From: amey.gokhale at in.ibm.com (Amey P Gokhale) Date: Wed, 20 Jul 2022 06:54:35 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list ; Amey P Gokhale Cc: gpfsug-discuss Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From p.ward at nhm.ac.uk Wed Jul 20 15:46:47 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Wed, 20 Jul 2022 14:46:47 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list Cc: gpfsug-discuss Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From scale at us.ibm.com Thu Jul 21 20:18:52 2022 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 22 Jul 2022 00:48:52 +0530 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Paul Ward" To: "gpfsug main discussion list" Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE ' /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/% ' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/% 2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list Cc: gpfsug-discuss Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list ; Amey P Gokhale Cc: gpfsug-discuss Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" To: "gpfsug main discussion list" Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list ; Venkateswara R Puvvada Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" To: "gpfsug-discuss at gpfsug.org" Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk A picture containing drawing Description automatically generated _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16769903.jpg Type: image/jpeg Size: 5356 bytes Desc: not available URL: From Tom.Chandler at ibm.com Thu Jul 21 20:32:01 2022 From: Tom.Chandler at ibm.com (Tom Chandler) Date: Thu, 21 Jul 2022 19:32:01 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: 30 years in IT, in 3 different continents, participated in 50 countries with IBM projects? yeah I feel stupid I?ve should have known better, very unprofessional on my part? my apologies? Tom Chandler Senior Data & AI Technical Specialist IBM Technology US Financial Services Market Email:tom.chandler at ibm.com Mobile:+1(386)-846-1578 From: gpfsug-discuss On Behalf Of IBM Spectrum Scale Sent: Thursday, July 21, 2022 3:19 PM To: gpfsug main discussion list Cc: gpfsug-discuss ; Amey P Gokhale Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ ? ? ? ? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From Tom.Chandler at ibm.com Thu Jul 21 20:35:37 2022 From: Tom.Chandler at ibm.com (Tom Chandler) Date: Thu, 21 Jul 2022 19:35:37 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Oh my gosh, embarrassing, wrong window?. Pls disregard, apologies for wasted bandwidth.. Tom Chandler Senior Data & AI Technical Specialist IBM Technology US Financial Services Market Email:tom.chandler at ibm.com Mobile:+1(386)-846-1578 From: Tom Chandler Sent: Thursday, July 21, 2022 3:32 PM To: gpfsug main discussion list Cc: gpfsug-discuss ; Amey P Gokhale Subject: RE: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy 30 years in IT, in 3 different continents, participated in 50 countries with IBM projects? yeah I feel stupid I?ve should have known better, very unprofessional on my part? my apologies? Tom Chandler Senior Data & AI Technical Specialist IBM Technology US Financial Services Market Email:tom.chandler at ibm.com Mobile:+1(386)-846-1578 From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: Thursday, July 21, 2022 3:19 PM To: gpfsug main discussion list > Cc: gpfsug-discuss >; Amey P Gokhale > Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ ? ? ? ? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From shaof777 at gmail.com Thu Jul 28 08:02:47 2022 From: shaof777 at gmail.com (shao feng) Date: Thu, 28 Jul 2022 15:02:47 +0800 Subject: [gpfsug-discuss] add local nsd back to cluster? Message-ID: Hi all, I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. The question is regarding node failure, for example only the operating system disk fails and the nsd disks are good. In that case I plan to replace the failing OS disk with a new one and install the OS on it and re-attach these nsd disks to that node, my question is: will this work? how can I add a nsd back to the cluster without restoring data from other replicas since the data/metadata is actually not corrupted on nsd. Best regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Jul 28 08:10:54 2022 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 28 Jul 2022 07:10:54 +0000 Subject: [gpfsug-discuss] add local nsd back to cluster? In-Reply-To: References: Message-ID: Hi - assuming, you'll run it withou ECE ?!? ... just with replication on the file system level ba aware, every time a node goes offline, you 'll have to restart the disks in your filesystem .. This causes a complete scan of the meta data to detect files with missing updates / replication apart from that to your Q : you may consider to backup mmsdr additionally, take a look to mmsdrrestore, in case you want to restore a nodes's SDR configuration quick and dirty.. save the content of /var/mmfs may also help you during the node is "gone".. of course.. the disk is down , after restore of SDR / node's config .. it should be able to start .. the rest runs as usual ________________________________ Von: gpfsug-discuss im Auftrag von shao feng Gesendet: Donnerstag, 28. Juli 2022 09:02 An: gpfsug main discussion list Betreff: [EXTERNAL] [gpfsug-discuss] add local nsd back to cluster? Hi all, I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. The question is regarding node failure, for example ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. The question is regarding node failure, for example only the operating system disk fails and the nsd disks are good. In that case I plan to replace the failing OS disk with a new one and install the OS on it and re-attach these nsd disks to that node, my question is: will this work? how can I add a nsd back to the cluster without restoring data from other replicas since the data/metadata is actually not corrupted on nsd. Best regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at hpe.com Thu Jul 28 10:58:29 2022 From: daniel.kidger at hpe.com (Kidger, Daniel) Date: Thu, 28 Jul 2022 09:58:29 +0000 Subject: [gpfsug-discuss] add local nsd back to cluster? In-Reply-To: References: Message-ID: Yes you can ? this is actually the same case as SAN attached storage (eg the usual twin tailed). You simply reinstall Linux on that node and add it back to the cluster (mmsdrrestore). The NSD drives are unaffected in the same way that they would have been if they were in a separate 2U SAN shelf Daniel Daniel Kidger HPC Storage Solutions Architect, EMEA daniel.kidger at hpe.com +44 (0)7818 522266 hpe.com [cid:image001.png at 01D8A270.738088F0] From: gpfsug-discuss On Behalf Of shao feng Sent: 28 July 2022 08:03 To: gpfsug main discussion list Subject: [gpfsug-discuss] add local nsd back to cluster? Hi all, I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. The question is regarding node failure, for example only the operating system disk fails and the nsd disks are good. In that case I plan to replace the failing OS disk with a new one and install the OS on it and re-attach these nsd disks to that node, my question is: will this work? how can I add a nsd back to the cluster without restoring data from other replicas since the data/metadata is actually not corrupted on nsd. Best regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2541 bytes Desc: image001.png URL: From p.ward at nhm.ac.uk Thu Jul 28 12:03:53 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Thu, 28 Jul 2022 11:03:53 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Hi, Any further ideas on this issue? Kindest regards, Paul From: Paul Ward Sent: 20 July 2022 15:47 To: gpfsug main discussion list Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From Anuj.Agnihotri at ibm.com Thu Jul 28 13:47:52 2022 From: Anuj.Agnihotri at ibm.com (Anuj Agnihotri) Date: Thu, 28 Jul 2022 12:47:52 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Hi Paul, We were able to recreate the following scenario on our end and it is working as designed. I have created 5 files of 20MB each and migrated it to the cloud and then recall them via policy. When you recall the files using policy it decodes the ?%2F? to ?/? character as generated in the policy-list file and when you try to recall manually using the path generated by policy it will fail as it is not the actual path. Below is the output attached: 1. [root at c350f3u1 ~]# mmapplypolicy /gpfs0 -f /gpfs0/s3default/anujnhm/policy-list -I defer -L 3 -P recallFromCloud.policy.template [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 5 20480 5 102400 102400 RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE(.) [I] Filesystem objects with no applicable rules: 349. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 102400KB: 5 of 5 candidates; 102400KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied system 28884992 31457280 91.822916667% [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-28 at 05:00:07.154 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. 2. [root at c350f3u1 anujnhm]# head policy-list.recall.mcstore 156416 97874145 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_1 156417 1726931404 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 156418 1426209538 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_3 156419 1880128499 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_4 156420 1841735731 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_5 3. [root at c350f3u1 anujnhm]# mmcloudgateway files recall %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 mmcloudgateway: Internal cloud services returned an error: file system object /gpfs0/s3default/anujnhm/%2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. When TCT recalls files directly via policy, we do %2F decoding that the policy adds, but when we do explicit recalls using the standalone command, we do not do any decoding, because we don't expect files to be passed with any decoding to the explicit command. Thanks, Anuj Agnihotri From: Amey P Gokhale Date: Friday, 22 July 2022 at 2:42 PM To: Anuj Agnihotri Subject: FW: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy FYI - AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha On Behalf Of scale at us.ibm.com Sent: 22 July 2022 00:49 To: gpfsug main discussion list Cc: gpfsug-discuss ; Amey P Gokhale Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 9F645794.jpg Type: image/jpeg Size: 5356 bytes Desc: 9F645794.jpg URL: From p.ward at nhm.ac.uk Fri Jul 29 11:51:25 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Fri, 29 Jul 2022 10:51:25 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Dear Anuj and Amey, Thank you so much for your testing? I repeated my tests yesterday, and actually ran the policy. All the files recalled ! yay! A few minutes later, I checked and half of the files were migrated again, and a little while later they were all migrated?. !? This is when I remembered I have a policy running every 30 mins with a rule for the parent folder to migrate any files not accessed for 90 days: /* Workspace Area rules ================================================================= */ /* Workspace Area DEFAULT rule ================================================================= */ RULE 'migrate_bulk_workspace_90' MIGRATE FROM POOL 'data' TO POOL 'migrate' WHERE (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) > 90) AND (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/%') This is what is migrating the files straight back after I ran the specific policy to recall them! I had the recall rule in the same policy, I am guessing this was conflicting with the default migrate rule: /* Workspace Area SPECIFIC rules ================================================================= */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%') So I have added this exclude: /* Exclude rules ================================================================= */ RULE 'exclude_bulk_workspace_axioimager_2021' EXCLUDE WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' And now the recalled files stay recalled ? ======================== I will try and test today with putting recall rules higher in the policy file, before default parent migrate rules. The rest of this email I wrote before realising the rule conflict?. OK so you are getting the same results and they are working as designed? Do the quantities on disk match the amounts the policy found? 14:56:13 [root at scale-sk-pn-1 2021]# du -hs . 1.3M . (need to half this value as we have two mirrored ESS, storing 2 copies) 14:56:22 [root at scale-sk-pn-1 2021]# du -bhs . 747G . Compared to the policy result: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 996 736 996 1566374532 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) KB_chosen: 1,566,374,532 = 1,566 GB more than double the amount du reports? ---------------------------------------------- I actually ran the policy: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 996 736 996 1566374532 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566374532KB: 996 of 996 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 173720590244 245111980032 70.873969612% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....|... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ......-. <1> mmcloudgateway: Command completed.cution. 996 files dispatched. /....... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....>... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ..#..... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ...@.... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. .....&.. <1> mmcloudgateway: Command completed. <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ../..... <1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....<... [I] 2022-07-28 at 14:32:48.892 Policy execution. 996 files dispatched. [I] A total of 996 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. After recall: 15:32:48 [root at scale-sk-pn-1 2021]# du -hs . 1003G . 15:33:36 [root at scale-sk-pn-1 2021]# du -bhs . 747G . So something has come back? When I check what files are migrated and what are resident: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 405 0 405 0 0 RULE 'migrated' LIST 'migrated_list' WHERE(.) 1 592 1050944608 592 1050944608 0 RULE 'resident' LIST 'resident_list' WHERE(.) not all of the files have been recalled? This is when I realised about the migrate rule running every 30 mins. I will update after re-ordering the rules. Paul From: Anuj Agnihotri Sent: 28 July 2022 13:48 To: Paul Ward Cc: Amey P Gokhale ; gpfsug-discuss at gpfsug.org Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Paul, We were able to recreate the following scenario on our end and it is working as designed. I have created 5 files of 20MB each and migrated it to the cloud and then recall them via policy. When you recall the files using policy it decodes the ?%2F? to ?/? character as generated in the policy-list file and when you try to recall manually using the path generated by policy it will fail as it is not the actual path. Below is the output attached: 1. [root at c350f3u1 ~]# mmapplypolicy /gpfs0 -f /gpfs0/s3default/anujnhm/policy-list -I defer -L 3 -P recallFromCloud.policy.template [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 5 20480 5 102400 102400 RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE(.) [I] Filesystem objects with no applicable rules: 349. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 102400KB: 5 of 5 candidates; 102400KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied system 28884992 31457280 91.822916667% [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-28 at 05:00:07.154 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. 2. [root at c350f3u1 anujnhm]# head policy-list.recall.mcstore 156416 97874145 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_1 156417 1726931404 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 156418 1426209538 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_3 156419 1880128499 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_4 156420 1841735731 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_5 3. [root at c350f3u1 anujnhm]# mmcloudgateway files recall %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 mmcloudgateway: Internal cloud services returned an error: file system object /gpfs0/s3default/anujnhm/%2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. When TCT recalls files directly via policy, we do %2F decoding that the policy adds, but when we do explicit recalls using the standalone command, we do not do any decoding, because we don't expect files to be passed with any decoding to the explicit command. Thanks, Anuj Agnihotri From: Amey P Gokhale > Date: Friday, 22 July 2022 at 2:42 PM To: Anuj Agnihotri > Subject: FW: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy FYI - AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 22 July 2022 00:49 To: gpfsug main discussion list > Cc: gpfsug-discuss >; Amey P Gokhale > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From p.ward at nhm.ac.uk Fri Jul 29 14:44:38 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Fri, 29 Jul 2022 13:44:38 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Putting the recall rule higher in the policy doesn?t seem to have recalled anything. It would seem I need to add an exclude in the every-30-min policy, do a specific separate policy to recall things, do what I need to do with the files, then remove the exclude. From: Anuj Agnihotri Sent: 28 July 2022 13:48 To: Paul Ward Cc: Amey P Gokhale ; gpfsug-discuss at gpfsug.org Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Paul, We were able to recreate the following scenario on our end and it is working as designed. I have created 5 files of 20MB each and migrated it to the cloud and then recall them via policy. When you recall the files using policy it decodes the ?%2F? to ?/? character as generated in the policy-list file and when you try to recall manually using the path generated by policy it will fail as it is not the actual path. Below is the output attached: 1. [root at c350f3u1 ~]# mmapplypolicy /gpfs0 -f /gpfs0/s3default/anujnhm/policy-list -I defer -L 3 -P recallFromCloud.policy.template [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 5 20480 5 102400 102400 RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE(.) [I] Filesystem objects with no applicable rules: 349. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 102400KB: 5 of 5 candidates; 102400KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied system 28884992 31457280 91.822916667% [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-28 at 05:00:07.154 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. 2. [root at c350f3u1 anujnhm]# head policy-list.recall.mcstore 156416 97874145 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_1 156417 1726931404 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 156418 1426209538 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_3 156419 1880128499 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_4 156420 1841735731 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_5 3. [root at c350f3u1 anujnhm]# mmcloudgateway files recall %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 mmcloudgateway: Internal cloud services returned an error: file system object /gpfs0/s3default/anujnhm/%2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. When TCT recalls files directly via policy, we do %2F decoding that the policy adds, but when we do explicit recalls using the standalone command, we do not do any decoding, because we don't expect files to be passed with any decoding to the explicit command. Thanks, Anuj Agnihotri From: Amey P Gokhale > Date: Friday, 22 July 2022 at 2:42 PM To: Anuj Agnihotri > Subject: FW: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy FYI - AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 22 July 2022 00:49 To: gpfsug main discussion list > Cc: gpfsug-discuss >; Amey P Gokhale > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From p.ward at nhm.ac.uk Fri Jul 29 15:34:29 2022 From: p.ward at nhm.ac.uk (Paul Ward) Date: Fri, 29 Jul 2022 14:34:29 +0000 Subject: [gpfsug-discuss] mass recall from on-prem COS using a policy In-Reply-To: References: Message-ID: Maybe I spoke to soon, I didn?t notice the small change in the log reports: 15:23:14 [root at scale-sk-pn-1 migrate]# grep "migrate_bulk_workspace_axioimager_2019" migrate_cos.2022-07-29* migrate_cos.2022-07-29_14-00-04.log: RULE 'migrate_bulk_workspace_axioimager_2019' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2019/%') migrate_cos.2022-07-29_14-00-04.log: 2 11114 0 11114 2941501208 2941501208 RULE 'migrate_bulk_workspace_axioimager_2019' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) migrate_cos.2022-07-29_15-00-04.log: RULE 'migrate_bulk_workspace_axioimager_2019' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2019/%') migrate_cos.2022-07-29_15-00-04.log: 2 11114 2450797296 11114 2941550204 1589770924 RULE 'migrate_bulk_workspace_axioimager_2019' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) When I checked the folder sizes Earlier today, 2019 was migrated: 13:39:37 [root at sc-ems-sl-01 zeiss_axioimager_m2]# du -hs * 1.9M 2019 47K 2020 1.5T 2021 13:39:44 [root at sc-ems-sl-01 zeiss_axioimager_m2]# du -bhs * 1.4T 2019 48G 2020 747G 2021 Just now, 2019 is recalled: 15:21:47 [root at scale-sk-pn-1 2019]# du -hs . 2.8T . 15:22:13 [root at scale-sk-pn-1 2019]# du -bhs . 1.4T . Looks like the data was recalled from within the 30 min policy. I will try with folder 2020 Paul From: Paul Ward Sent: 29 July 2022 14:45 To: Anuj Agnihotri Cc: Amey P Gokhale ; gpfsug-discuss at gpfsug.org Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy Putting the recall rule higher in the policy doesn?t seem to have recalled anything. It would seem I need to add an exclude in the every-30-min policy, do a specific separate policy to recall things, do what I need to do with the files, then remove the exclude. From: Anuj Agnihotri > Sent: 28 July 2022 13:48 To: Paul Ward > Cc: Amey P Gokhale >; gpfsug-discuss at gpfsug.org Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Paul, We were able to recreate the following scenario on our end and it is working as designed. I have created 5 files of 20MB each and migrated it to the cloud and then recall them via policy. When you recall the files using policy it decodes the ?%2F? to ?/? character as generated in the policy-list file and when you try to recall manually using the path generated by policy it will fail as it is not the actual path. Below is the output attached: 1. [root at c350f3u1 ~]# mmapplypolicy /gpfs0 -f /gpfs0/s3default/anujnhm/policy-list -I defer -L 3 -P recallFromCloud.policy.template [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 5 20480 5 102400 102400 RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE(.) [I] Filesystem objects with no applicable rules: 349. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 102400KB: 5 of 5 candidates; 102400KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied system 28884992 31457280 91.822916667% [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-28 at 05:00:07.154 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. 2. [root at c350f3u1 anujnhm]# head policy-list.recall.mcstore 156416 97874145 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_1 156417 1726931404 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 156418 1426209538 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_3 156419 1880128499 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_4 156420 1841735731 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_5 3. [root at c350f3u1 anujnhm]# mmcloudgateway files recall %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 mmcloudgateway: Internal cloud services returned an error: file system object /gpfs0/s3default/anujnhm/%2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. When TCT recalls files directly via policy, we do %2F decoding that the policy adds, but when we do explicit recalls using the standalone command, we do not do any decoding, because we don't expect files to be passed with any decoding to the explicit command. Thanks, Anuj Agnihotri From: Amey P Gokhale > Date: Friday, 22 July 2022 at 2:42 PM To: Anuj Agnihotri > Subject: FW: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy FYI - AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 22 July 2022 00:49 To: gpfsug main discussion list > Cc: gpfsug-discuss >; Amey P Gokhale > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Adding back Amey in the chain. Paul please explicitly add Amey in your replies. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 20-07-2022 08.20 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* ?????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /******************************************************************************* * Licensed Materials - Property of IBM * * OCO Source Materials * * (C) Copyright IBM Corp. 2016-2017 All Rights Reserved * * The source code for this program is not published or other- * wise divested of its trade secrets, irrespective of what has * been deposited with the U.S. Copyright Office. *******************************************************************************/ define( exclude_list, ( FALSE OR PATH_NAME LIKE '%/.mcstore/%' ) ) RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' /* RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE ( ( PATH_NAME LIKE '/gpfs/test_dir/%' ) AND NOT (exclude_list) ) */ RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE ( ( PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%' ) AND NOT (exclude_list) ) ----------------------------------- I changed to the folder listed in the path and ran this command, with the defer option: # pwd /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021 ]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol The result of the policy was: [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) [I] Filesystem objects with no applicable rules: 53. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 1566373796KB: 994 of 994 candidates; 1566373796KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: Pool_Name KB_Occupied KB_Total Percent_Occupied data 175459129252 245111980032 71.583253185% system 0 0 0.000000000% (no user data) [I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading! [I] 2022-07-20 at 14:22:23.943 Policy execution. 0 files dispatched. [I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. --------------------------------------------------------------------- Its selected about the right number, as there are : # find -type f | wc -l 996 -------------------------------------------------------- the head of the deferred output file list: 15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate 9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db 30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi 30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi 30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi 30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif ?. -------------------------- If I try to recall the file from these paths: # mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Internal cloud services returned an error: file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist mmcloudgateway: Command failed. Examine previous error messages to determine cause. It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in. ----------------------- If I Give it the correctly formed path, it works: # mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif" mmcloudgateway: Command completed. ------------------------------------------ I also ran a policy just to identify migrated files: define(ismigrated, ( MISC_ATTRIBUTES LIKE '%V%') ) Rule 'migrated' LIST 'migrated_list' WHERE ismigrated Rule 'resident' LIST 'resident_list' WHERE not (ismigrated) ---------------------------------------------------------------- The head of that file correctly forms the posix path? 15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list 9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db 30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif 30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi 30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif 30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif 30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi 30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif 30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi 30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif 30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif What?s going on? What I have done in the past is take the output from my ?ismigrated? policy. Put recall commands before each path. Divide the file into 4 parts, and run each part of one of our 4 protocol nodes. Manually doing what mmapplypolicy should do! From: gpfsug-discuss > On Behalf Of Amey P Gokhale Sent: 20 July 2022 07:55 To: scale at us.ibm.com; gpfsug main discussion list > Cc: gpfsug-discuss > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Paul ? Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template. 1. At the end of policy run, what does the stats indicate, such as ?X Files migrated, Y skipped/error? etc? I assume the path you have specified, has the files in ?non-resident? state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible. 2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy? 3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find. AMEY GOKHALE Senior Software Engineer ? Spectrum Scale Phone: 91-988 100 8675 E-mail: amey.gokhale at in.ibm.com From: Huzefa H Pancha > On Behalf Of scale at us.ibm.com Sent: 19 July 2022 23:42 To: gpfsug main discussion list >; Amey P Gokhale > Cc: gpfsug-discuss > Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Hi Amey, Can you provide them guidance from TCT angle. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS? From: "Paul Ward" > To: "gpfsug main discussion list" > Date: 19-07-2022 09.34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. ??????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT. From: gpfsug-discuss > On Behalf Of IBM Spectrum Scale Sent: 18 July 2022 20:35 To: gpfsug main discussion list >; Venkateswara R Puvvada > Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy "KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile. Copying to AFM team regarding recall in AFM-COS environment. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. [Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil From: "Paul Ward" > To: "gpfsug-discuss at gpfsug.org" > Date: 07/12/2022 11:40 AM Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy Sent by: "gpfsug-discuss" > ________________________________ Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: ????????????????????????????????????????????????? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it: RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%' Migrate is an external pool RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -' And data is the default placement pool RULE 'Placement' SET POOL 'data' When it runs it identifies matching files: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.) I can?t find what ?KB_ Ill? means but nothing is migrating. Migrations in other policies work, but this is a ?recall? policy. >From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external It suggests it should be invoked as a recall. Any ideas? Kindest regards, Paul Paul Ward TS Infrastructure Architect Natural History Museum T: 02079426450 E: p.ward at nhm.ac.uk [A picture containing drawing Description automatically generated] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 5356 bytes Desc: image002.jpg URL: From shaof777 at gmail.com Fri Jul 29 16:54:24 2022 From: shaof777 at gmail.com (shao feng) Date: Fri, 29 Jul 2022 23:54:24 +0800 Subject: [gpfsug-discuss] add local nsd back to cluster? In-Reply-To: References: Message-ID: Thanks Olaf I've setup the mmsdr backup as https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=exits-mmsdrbackup-user-exit, since my cluster is CCR enabled, it generate a CCR backup file, but when trying to restore from this file, it require quorum nodes to shutdown? Is it possible to restore without touching quorum nodes? [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz Restoring a CCR backup archive is a cluster-wide operation. The -a flag is required. mmsdrrestore: Command failed. Examine previous error messages to determine cause. [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz -a Restoring CCR backup Verifying that GPFS is inactive on quorum nodes mmsdrrestore: GPFS is still active on myquorum mmsdrrestore: Unexpected error from mmsdrrestore: CCR restore failed. Return code: 192 mmsdrrestore: Command failed. Examine previous error messages to determine cause. On Thu, Jul 28, 2022 at 3:14 PM Olaf Weiser wrote: > > > Hi - > assuming, you'll run it withou ECE ?!? ... just with replication on the > file system level > ba aware, every time a node goes offline, you 'll have to restart the > disks in your filesystem .. This causes a complete scan of the meta data to > detect files with missing updates / replication > > > apart from that to your Q : > you may consider to backup mmsdr > additionally, take a look to mmsdrrestore, in case you want to restore a > nodes's SDR configuration > > quick and dirty.. save the content of /var/mmfs may also help you > > during the node is "gone".. of course.. the disk is down , after restore > of SDR / node's config .. it should be able to start .. > the rest runs as usual > > > > ------------------------------ > *Von:* gpfsug-discuss im Auftrag von > shao feng > *Gesendet:* Donnerstag, 28. Juli 2022 09:02 > *An:* gpfsug main discussion list > *Betreff:* [EXTERNAL] [gpfsug-discuss] add local nsd back to cluster? > > Hi all, I am planning to implement a cluster with a bunch of old x86 > machines, the disks are not connected to nodes via the SAN network, instead > each x86 machine has some local attached disks. The question is regarding > node failure, for example > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi all, > > I am planning to implement a cluster with a bunch of old x86 machines, > the disks are not connected to nodes via the SAN network, instead each x86 > machine has some local attached disks. > The question is regarding node failure, for example only the operating > system disk fails and the nsd disks are good. In that case I plan to > replace the failing OS disk with a new one and install the OS on it and > re-attach these nsd disks to that node, my question is: will this work? how > can I add a nsd back to the cluster without restoring data from other > replicas since the data/metadata is actually not corrupted on nsd. > > Best regards, > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Fri Jul 29 17:48:44 2022 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 29 Jul 2022 12:48:44 -0400 Subject: [gpfsug-discuss] add local nsd back to cluster? In-Reply-To: References: Message-ID: <1DEB036E-AA3A-4498-A5B9-B66078EC87A9@ulmer.org> If there are cluster nodes up, restore from the running nodes instead of the file. I think it?s -p, but look at the manual page. -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Jul 29, 2022, at 11:20 AM, shao feng wrote: > > ? > Thanks Olaf > > I've setup the mmsdr backup as https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=exits-mmsdrbackup-user-exit, since my cluster is CCR enabled, it generate a CCR backup file, > but when trying to restore from this file, it require quorum nodes to shutdown? Is it possible to restore without touching quorum nodes? > > [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz > Restoring a CCR backup archive is a cluster-wide operation. > The -a flag is required. > mmsdrrestore: Command failed. Examine previous error messages to determine cause. > > [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz -a > Restoring CCR backup > Verifying that GPFS is inactive on quorum nodes > mmsdrrestore: GPFS is still active on myquorum > mmsdrrestore: Unexpected error from mmsdrrestore: CCR restore failed. Return code: 192 > mmsdrrestore: Command failed. Examine previous error messages to determine cause. > > >> On Thu, Jul 28, 2022 at 3:14 PM Olaf Weiser wrote: >> >> >> Hi - >> assuming, you'll run it withou ECE ?!? ... just with replication on the file system level >> ba aware, every time a node goes offline, you 'll have to restart the disks in your filesystem .. This causes a complete scan of the meta data to detect files with missing updates / replication >> >> >> apart from that to your Q : >> you may consider to backup mmsdr >> additionally, take a look to mmsdrrestore, in case you want to restore a nodes's SDR configuration >> >> quick and dirty.. save the content of /var/mmfs may also help you >> >> during the node is "gone".. of course.. the disk is down , after restore of SDR / node's config .. it should be able to start .. >> the rest runs as usual >> >> >> >> Von: gpfsug-discuss im Auftrag von shao feng >> Gesendet: Donnerstag, 28. Juli 2022 09:02 >> An: gpfsug main discussion list >> Betreff: [EXTERNAL] [gpfsug-discuss] add local nsd back to cluster? >> >> This Message Is From an External Sender >> This message came from outside your organization. >> >> Hi all, >> >> I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. >> The question is regarding node failure, for example only the operating system disk fails and the nsd disks are good. In that case I plan to replace the failing OS disk with a new one and install the OS on it and re-attach these nsd disks to that node, my question is: will this work? how can I add a nsd back to the cluster without restoring data from other replicas since the data/metadata is actually not corrupted on nsd. >> >> Best regards, >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From truongv at us.ibm.com Sat Jul 30 01:30:57 2022 From: truongv at us.ibm.com (Truong Vu) Date: Sat, 30 Jul 2022 00:30:57 +0000 Subject: [gpfsug-discuss] add local nsd back to cluster? Message-ID: <9004117D-3C4D-4A76-931B-1DCB2B631B2F@us.ibm.com> Starting GPFS 5.1.4, you can use the CCR archive to restore the local node (the node that is issuing the mmsdrrestore command) beside restoring the entire cluster. Prior to GPFS5.1.4, as the error message reviewed, you can only use the CCR archive to restore the entire cluster. GPFS must be down any node that is being restored. If is a good node in the cluster, use the -p option -p NodeName Specifies the node from which to obtain a valid GPFS configuration file. The node must be either the primary configuration server or a node that has a valid backup copy of the mmsdrfs file. If this parameter is not specified, the command uses the configuration file on the node from which the command is issued. Thanks, Tru. ?On 7/29/22, 12:51 PM, "gpfsug-discuss on behalf of gpfsug-discuss-request at gpfsug.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at gpfsug.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at gpfsug.org You can reach the person managing the list at gpfsug-discuss-owner at gpfsug.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: add local nsd back to cluster? (shao feng) 2. Re: add local nsd back to cluster? (Stephen Ulmer) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 Jul 2022 23:54:24 +0800 From: shao feng To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] add local nsd back to cluster? Message-ID: Content-Type: text/plain; charset="utf-8" Thanks Olaf I've setup the mmsdr backup as https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=exits-mmsdrbackup-user-exit, since my cluster is CCR enabled, it generate a CCR backup file, but when trying to restore from this file, it require quorum nodes to shutdown? Is it possible to restore without touching quorum nodes? [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz Restoring a CCR backup archive is a cluster-wide operation. The -a flag is required. mmsdrrestore: Command failed. Examine previous error messages to determine cause. [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz -a Restoring CCR backup Verifying that GPFS is inactive on quorum nodes mmsdrrestore: GPFS is still active on myquorum mmsdrrestore: Unexpected error from mmsdrrestore: CCR restore failed. Return code: 192 mmsdrrestore: Command failed. Examine previous error messages to determine cause. On Thu, Jul 28, 2022 at 3:14 PM Olaf Weiser wrote: > > > Hi - > assuming, you'll run it withou ECE ?!? ... just with replication on the > file system level > ba aware, every time a node goes offline, you 'll have to restart the > disks in your filesystem .. This causes a complete scan of the meta data to > detect files with missing updates / replication > > > apart from that to your Q : > you may consider to backup mmsdr > additionally, take a look to mmsdrrestore, in case you want to restore a > nodes's SDR configuration > > quick and dirty.. save the content of /var/mmfs may also help you > > during the node is "gone".. of course.. the disk is down , after restore > of SDR / node's config .. it should be able to start .. > the rest runs as usual > > > > ------------------------------ > *Von:* gpfsug-discuss im Auftrag von > shao feng > *Gesendet:* Donnerstag, 28. Juli 2022 09:02 > *An:* gpfsug main discussion list > *Betreff:* [EXTERNAL] [gpfsug-discuss] add local nsd back to cluster? > > Hi all, I am planning to implement a cluster with a bunch of old x86 > machines, the disks are not connected to nodes via the SAN network, instead > each x86 machine has some local attached disks. The question is regarding > node failure, for example > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi all, > > I am planning to implement a cluster with a bunch of old x86 machines, > the disks are not connected to nodes via the SAN network, instead each x86 > machine has some local attached disks. > The question is regarding node failure, for example only the operating > system disk fails and the nsd disks are good. In that case I plan to > replace the failing OS disk with a new one and install the OS on it and > re-attach these nsd disks to that node, my question is: will this work? how > can I add a nsd back to the cluster without restoring data from other > replicas since the data/metadata is actually not corrupted on nsd. > > Best regards, > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Fri, 29 Jul 2022 12:48:44 -0400 From: Stephen Ulmer To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] add local nsd back to cluster? Message-ID: <1DEB036E-AA3A-4498-A5B9-B66078EC87A9 at ulmer.org> Content-Type: text/plain; charset="utf-8" If there are cluster nodes up, restore from the running nodes instead of the file. I think it?s -p, but look at the manual page. -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Jul 29, 2022, at 11:20 AM, shao feng wrote: > > ? > Thanks Olaf > > I've setup the mmsdr backup as https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=exits-mmsdrbackup-user-exit, since my cluster is CCR enabled, it generate a CCR backup file, > but when trying to restore from this file, it require quorum nodes to shutdown? Is it possible to restore without touching quorum nodes? > > [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz > Restoring a CCR backup archive is a cluster-wide operation. > The -a flag is required. > mmsdrrestore: Command failed. Examine previous error messages to determine cause. > > [root at tofail ~]# mmsdrrestore -F CCRBackup.986.2022.07.29.23.06.19.myquorum.tar.gz -a > Restoring CCR backup > Verifying that GPFS is inactive on quorum nodes > mmsdrrestore: GPFS is still active on myquorum > mmsdrrestore: Unexpected error from mmsdrrestore: CCR restore failed. Return code: 192 > mmsdrrestore: Command failed. Examine previous error messages to determine cause. > > >> On Thu, Jul 28, 2022 at 3:14 PM Olaf Weiser wrote: >> >> >> Hi - >> assuming, you'll run it withou ECE ?!? ... just with replication on the file system level >> ba aware, every time a node goes offline, you 'll have to restart the disks in your filesystem .. This causes a complete scan of the meta data to detect files with missing updates / replication >> >> >> apart from that to your Q : >> you may consider to backup mmsdr >> additionally, take a look to mmsdrrestore, in case you want to restore a nodes's SDR configuration >> >> quick and dirty.. save the content of /var/mmfs may also help you >> >> during the node is "gone".. of course.. the disk is down , after restore of SDR / node's config .. it should be able to start .. >> the rest runs as usual >> >> >> >> Von: gpfsug-discuss im Auftrag von shao feng >> Gesendet: Donnerstag, 28. Juli 2022 09:02 >> An: gpfsug main discussion list >> Betreff: [EXTERNAL] [gpfsug-discuss] add local nsd back to cluster? >> >> This Message Is From an External Sender >> This message came from outside your organization. >> >> Hi all, >> >> I am planning to implement a cluster with a bunch of old x86 machines, the disks are not connected to nodes via the SAN network, instead each x86 machine has some local attached disks. >> The question is regarding node failure, for example only the operating system disk fails and the nsd disks are good. In that case I plan to replace the failing OS disk with a new one and install the OS on it and re-attach these nsd disks to that node, my question is: will this work? how can I add a nsd back to the cluster without restoring data from other replicas since the data/metadata is actually not corrupted on nsd. >> >> Best regards, >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org ------------------------------ End of gpfsug-discuss Digest, Vol 126, Issue 21 ***********************************************