From jonathan.buzzard at strath.ac.uk Wed Jul 5 23:20:44 2023 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 5 Jul 2023 23:20:44 +0100 Subject: [gpfsug-discuss] Special characters in filenames Message-ID: After another support incident that eventually transpired to be down to the user using what I will call stupid characters in their filenames (we include a section on not doing this in our mandatory training so no excuse) I have been musing on using the policy engine to periodically produce lists of files that have stupid characters in their filenames so we can proactively educate the users and get them to rename their files to something sensible :-) The issue is of course the stupid characters include all the regular expression wildcard characters in addition to \n, \r and backticks. I am coming up short on escaping them correctly in REGEX() for the policy engine. The documentation appears to be devoid of help on the subject, because of course only an fool would be including these characters in their filenames... Anyone any idea on how to do this? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Wed Jul 5 23:31:17 2023 From: anacreo at gmail.com (Alec) Date: Wed, 5 Jul 2023 15:31:17 -0700 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: Would it not just be easier to eliminate all characters you deem not to be "stupid" and then match against the remainder? ie. /[^a-zA-Z.]+/ and then go from there? On Wed, Jul 5, 2023 at 3:23?PM Jonathan Buzzard < jonathan.buzzard at strath.ac.uk> wrote: > > After another support incident that eventually transpired to be down to > the user using what I will call stupid characters in their filenames (we > include a section on not doing this in our mandatory training so no > excuse) I have been musing on using the policy engine to periodically > produce lists of files that have stupid characters in their filenames so > we can proactively educate the users and get them to rename their files > to something sensible :-) > > The issue is of course the stupid characters include all the regular > expression wildcard characters in addition to \n, \r and backticks. I am > coming up short on escaping them correctly in REGEX() for the policy > engine. > > The documentation appears to be devoid of help on the subject, because > of course only an fool would be including these characters in their > filenames... > > Anyone any idea on how to do this? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Jul 6 00:02:19 2023 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 5 Jul 2023 23:02:19 +0000 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: Hallo Jonathan, I haven't used it for a while, but I can remember a customer, where we masked "all" special characters with ESCAPE In fact, as far as I remember.. this was an iterative progress ... ? ? You're right, the doc's are not really self-explaining here.. from my personal notes I found a litte better example: In GPFS 3.5 we introduce an (optional) ESCAPE clause to the EXTERNAL LIST and EXTERNAL POOL rules, which allow the user-administrator to specify that path names and SHOW(strings) within the associated file lists are encoded using an encoding based on the RFC3986 URI-percent-encoding scheme. For example: RULE 'xp' EXTERNAL POOL 'pool-name' EXEC 'script-name' ESCAPE '%' RULE 'xl' EXTERNAL LIST 'list-name' EXEC 'script-name' ESCAPE '%/+@#' ESCAPE '%' specifies that all characters except the "unreserved" characters in the set a-zA-Z0-9-_.~ are encoded as %XX where XX comprises 2 hexadecimal digits. The GPFS ESCAPE clause allows you to add to the set of "unreserved" characters. For example, ESCAPE '%/+@#', specifies that none of the characters in "/+@#" are escaped, so that a path name like "/root/directory/@abc+def#ghi.jkl" will appear in a file list with no escape sequences, whereas under ESCAPE '%', specifying a rigorous RFC3986 encoding yields "%2Froot%2Fdirectory%2F%40abc%2Bdef%23ghi.jkl". at least for us, it was doing the trick (back then) by using ESCAPE.. Maybe it is useful for your case here as well cheers laff ________________________________ Von: gpfsug-discuss im Auftrag von Jonathan Buzzard Gesendet: Donnerstag, 6. Juli 2023 00:20 An: gpfsug main discussion list Betreff: [EXTERNAL] [gpfsug-discuss] Special characters in filenames After another support incident that eventually transpired to be down to the user using what I will call stupid characters in their filenames (we include a section on not doing this in our mandatory training so no excuse) I have been musing on using the policy engine to periodically produce lists of files that have stupid characters in their filenames so we can proactively educate the users and get them to rename their files to something sensible :-) The issue is of course the stupid characters include all the regular expression wildcard characters in addition to \n, \r and backticks. I am coming up short on escaping them correctly in REGEX() for the policy engine. The documentation appears to be devoid of help on the subject, because of course only an fool would be including these characters in their filenames... Anyone any idea on how to do this? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri Jul 7 13:37:26 2023 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 7 Jul 2023 12:37:26 +0000 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: <812e5278512d385c155ff702fb099d78e7973088.camel@strath.ac.uk> [Invalid UTF-8] From juergen.hannappel at desy.de Fri Jul 7 17:22:27 2023 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Fri, 7 Jul 2023 18:22:27 +0200 (CEST) Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: <812e5278512d385c155ff702fb099d78e7973088.camel@strath.ac.uk> References: <812e5278512d385c155ff702fb099d78e7973088.camel@strath.ac.uk> Message-ID: <1006799600.17047565.1688746947628.JavaMail.zimbra@desy.de> Hi, not exactly to the subject, but here is a C++ function to check if a file name is "bad". It returns a string with the reason, or nullptr if it's fine: const char* hasGoodName(const std::string& name) const { if (isspace(name().front())) { static const char* reason = "spaceInFront"; return reason; } if (isspace(name().back())) { static const char* reason = "spaceInBack"; return reason; } // only relevant if files go to dCache... if (name().size() > (255 - sizeof(".(get)()(checksums)"))) { // to long for dot command magic static const char* reason = "tooLong"; return reason; } size_t n=0; // number of extension-bytes uint32_t code=0; for (auto c : name()) { if (n==0) { // not an extension byte if ((c & 0b1000'0000) == 0) { // ASCII if (! isprint(c)) { static const char* reason = "nonPrintableAScii"; return reason; } if (c == '/') { // this must not be! static const char* reason = "slashInName"; return reason; } } else if ((c & 0b1110'0000) == 0b1100'0000) { // one ext byte code = c & 0b0001'1111; n=1; } else if ((c & 0b1111'0000) == 0b1110'0000) { // two ext bytes code = c & 0b0000'1111; n=2; } else if ((c & 0b1111'1000) == 0b1111'0000) { // three ext bytes code = c & 0b0000'0111; n=3; } else { // no UTF8 coding static const char* reason = "nonUTFStarter"; return reason; } } else { // this is an extension byte if ((c & 0b1100'0000) == 0b1000'0000) { // and well-formed code = (code << 6) | (c & 0b0011'1111); // ' n--; if (n==0) {// last extension byte seen: code finished if (0x0080 <= code && code <= 0x009F) { static const char* reason = "C1ControlCode"; return reason; } else if ((0x200E <= code && code <= 0x200F) || (0x202A <= code && code <= 0x202E)){ static const char* reason = "directionMarks"; return reason; } else if ((0xFFFE <= (code&0xFFFF) && (code&0xFFFF) <= 0xFFFF) || (0xFDD0 <= code && code <= 0xFDEF)) { static const char* reason = "NonCharacter"; return reason; } else if (0xE000 <= code && code <= 0xF8FF) { static const char* reason = "privateUseUTF"; return reason; } else if (0xf0000 <= code && code <= 0xfffff) { static const char* reason = "privateUseAreaA"; return reason; } else if (0x100000 <= code && code <= 0x10fffd) { static const char* reason = "privateUseAreaB"; return reason; } else if (0x110000 <= code) { static const char* reason = "beyondLastPlane"; return reason; } } } else { // ill-formed static const char* reason = "missingExtByte"; return reason; } } } return nullptr; } ----- Original Message ----- > From: "Jonathan Buzzard" > To: "gpfsug-discuss" > Sent: Friday, 7 July, 2023 14:37:26 > Subject: Re: [gpfsug-discuss] Special characters in filenames > [Invalid UTF-8] > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From wsawdon at us.ibm.com Fri Jul 7 18:55:07 2023 From: wsawdon at us.ibm.com (Wayne Sawdon) Date: Fri, 7 Jul 2023 17:55:07 +0000 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: The policy code uses a more or less standard linux regexp library, so your regular expressions used for grep should work. The catch is the policy file is preprocessed with M4 which makes writing regexs a bit tricky. I grabbed a comment from the code: Note: The policy SQL parser normally does M4 macro preprocessing with [ ] set as the quote characters. SOOOO.... we highly recommend you add an extra set of [ ] around your REGEX pattern string. Like this: ... WHERE REGEX(name,['^[a-z]*$']) only accept lowercase alphabetic file names Once you?ve gotten past M4, you can either match for not good characters or directly for bad characters REGEX(. FILENAME, [?[^a-zA-Z0-9\_\-\.]?] ) ### match when you find a character not in the good set REGEX(. FILENAME, [?[\n\*\\]?] ). ### match when you find a bad character I am not sure which is more difficult to enumerate. The ESCAPE clause described by Olaf is the trick we use to pass file names with bad characters through the surrounding scripts (like mmbackup, mmxcp, etc). There is code in samples/ilm that show how to use it. -Wayne From: gpfsug-discuss on behalf of Olaf Weiser Date: Wednesday, July 5, 2023 at 7:06 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Special characters in filenames Hallo Jonathan, I haven't used it for a while, but I can remember a customer, where we masked "all" special characters with ESCAPE In fact, as far as I remember.?. this was an iterative progress ..?. ? ? You're right, the doc's are ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious ? ZjQcmQRYFpfptBannerEnd Hallo Jonathan, I haven't used it for a while, but I can remember a customer, where we masked "all" special characters with ESCAPE In fact, as far as I remember.. this was an iterative progress ... ? ? You're right, the doc's are not really self-explaining here.. from my personal notes I found a litte better example: In GPFS 3.5 we introduce an (optional) ESCAPE clause to the EXTERNAL LIST and EXTERNAL POOL rules, which allow the user-administrator to specify that path names and SHOW(strings) within the associated file lists are encoded using an encoding based on the RFC3986 URI-percent-encoding scheme. For example: RULE 'xp' EXTERNAL POOL 'pool-name' EXEC 'script-name' ESCAPE '%' RULE 'xl' EXTERNAL LIST 'list-name' EXEC 'script-name' ESCAPE '%/+@#' ESCAPE '%' specifies that all characters except the "unreserved" characters in the set a-zA-Z0-9-_.~ are encoded as %XX where XX comprises 2 hexadecimal digits. The GPFS ESCAPE clause allows you to add to the set of "unreserved" characters. For example, ESCAPE '%/+@#', specifies that none of the characters in "/+@#" are escaped, so that a path name like "/root/directory/@abc+def#ghi.jkl" will appear in a file list with no escape sequences, whereas under ESCAPE '%', specifying a rigorous RFC3986 encoding yields "%2Froot%2Fdirectory%2F%40abc%2Bdef%23ghi.jkl". at least for us, it was doing the trick (back then) by using ESCAPE.. Maybe it is useful for your case here as well cheers laff ________________________________ Von: gpfsug-discuss im Auftrag von Jonathan Buzzard Gesendet: Donnerstag, 6. Juli 2023 00:20 An: gpfsug main discussion list Betreff: [EXTERNAL] [gpfsug-discuss] Special characters in filenames After another support incident that eventually transpired to be down to the user using what I will call stupid characters in their filenames (we include a section on not doing this in our mandatory training so no excuse) I have been musing on using the policy engine to periodically produce lists of files that have stupid characters in their filenames so we can proactively educate the users and get them to rename their files to something sensible :-) The issue is of course the stupid characters include all the regular expression wildcard characters in addition to \n, \r and backticks. I am coming up short on escaping them correctly in REGEX() for the policy engine. The documentation appears to be devoid of help on the subject, because of course only an fool would be including these characters in their filenames... Anyone any idea on how to do this? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From wsawdon at us.ibm.com Fri Jul 7 19:07:06 2023 From: wsawdon at us.ibm.com (Wayne Sawdon) Date: Fri, 7 Jul 2023 18:07:06 +0000 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: The policy code uses a more or less standard linux regexp library, so your regular expressions used for grep should work. The catch is the policy file is preprocessed with M4 which makes writing regexs a bit tricky. I grabbed a comment from the code: The policy SQL parser normally does M4 macros processing with [ ] set as the quote characters. SOOOO?. We highly recommend you add an extra set of [ ] around your REGEX pattern string like this: . . . WHERE REGEX(name, [?^[a-z]*$?]) To only match lowercase alphabetic names. Once you?ve gotten past M4, you can either match for not good characters or directly for bad characters REGEX(. FILENAME, [?[^a-zA-Z0-9\_\-\.]?] ) ### match when you find a character not in the good set REGEX(. FILENAME, [?[\n\*\\]?] ). ### match when you find a bad character I am not sure which is more difficult to enumerate. The ESCAPE clause described by Olaf is the trick we use to pass file names with bad characters through the surrounding scripts (like mmbackup, mmxcp, etc). There is code in samples/ilm that show how to use it. -Wayne From: gpfsug-discuss on behalf of Olaf Weiser Date: Wednesday, July 5, 2023 at 7:06 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Special characters in filenames Hallo Jonathan, I haven't used it for a while, but I can remember a customer, where we masked "all" special characters with ESCAPE In fact, as far as I remember.?. this was an iterative progress ..?. ? ? You're right, the doc's are ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious ? ZjQcmQRYFpfptBannerEnd Hallo Jonathan, I haven't used it for a while, but I can remember a customer, where we masked "all" special characters with ESCAPE In fact, as far as I remember.. this was an iterative progress ... ? ? You're right, the doc's are not really self-explaining here.. from my personal notes I found a litte better example: In GPFS 3.5 we introduce an (optional) ESCAPE clause to the EXTERNAL LIST and EXTERNAL POOL rules, which allow the user-administrator to specify that path names and SHOW(strings) within the associated file lists are encoded using an encoding based on the RFC3986 URI-percent-encoding scheme. For example: RULE 'xp' EXTERNAL POOL 'pool-name' EXEC 'script-name' ESCAPE '%' RULE 'xl' EXTERNAL LIST 'list-name' EXEC 'script-name' ESCAPE '%/+@#' ESCAPE '%' specifies that all characters except the "unreserved" characters in the set a-zA-Z0-9-_.~ are encoded as %XX where XX comprises 2 hexadecimal digits. The GPFS ESCAPE clause allows you to add to the set of "unreserved" characters. For example, ESCAPE '%/+@#', specifies that none of the characters in "/+@#" are escaped, so that a path name like "/root/directory/@abc+def#ghi.jkl" will appear in a file list with no escape sequences, whereas under ESCAPE '%', specifying a rigorous RFC3986 encoding yields "%2Froot%2Fdirectory%2F%40abc%2Bdef%23ghi.jkl". at least for us, it was doing the trick (back then) by using ESCAPE.. Maybe it is useful for your case here as well cheers laff ________________________________ Von: gpfsug-discuss im Auftrag von Jonathan Buzzard Gesendet: Donnerstag, 6. Juli 2023 00:20 An: gpfsug main discussion list Betreff: [EXTERNAL] [gpfsug-discuss] Special characters in filenames After another support incident that eventually transpired to be down to the user using what I will call stupid characters in their filenames (we include a section on not doing this in our mandatory training so no excuse) I have been musing on using the policy engine to periodically produce lists of files that have stupid characters in their filenames so we can proactively educate the users and get them to rename their files to something sensible :-) The issue is of course the stupid characters include all the regular expression wildcard characters in addition to \n, \r and backticks. I am coming up short on escaping them correctly in REGEX() for the policy engine. The documentation appears to be devoid of help on the subject, because of course only an fool would be including these characters in their filenames... Anyone any idea on how to do this? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacreo at gmail.com Fri Jul 7 21:26:09 2023 From: anacreo at gmail.com (Alec) Date: Fri, 7 Jul 2023 13:26:09 -0700 Subject: [gpfsug-discuss] Special characters in filenames In-Reply-To: References: Message-ID: Building off of this you could use the file placement engine to give a bad pool name for files that have bad names... Then new files couldn't be created with bad names. However files can still be renamed so you need a policy to deal with those cases. Alec On Fri, Jul 7, 2023, 11:11 AM Wayne Sawdon wrote: > > > The policy code uses a more or less standard linux regexp library, so your > regular expressions used for grep should work. The catch is the policy file > is preprocessed with M4 which makes writing regexs a bit tricky. I grabbed > a comment from the code: > > > > > > The policy SQL parser normally does M4 macros processing with [ ] > set as the quote characters. > > SOOOO?. We highly recommend you add an extra set of [ ] around your > REGEX pattern string like this: > > > > . . . WHERE REGEX(name, [?^[a-z]*$?]) > > > > To only match lowercase alphabetic names. > > > > > > Once you?ve gotten past M4, you can either match for not good characters > or directly for bad characters > > > > REGEX(. FILENAME, [?[^a-zA-Z0-9\_\-\.]?] ) ### match when you find a > character not in the good set > > REGEX(. FILENAME, [?[\n\*\\]?] ). ### match when you > find a bad character > > > > I am not sure which is more difficult to enumerate. > > > > The ESCAPE clause described by Olaf is the trick we use to pass file names > with bad characters through the surrounding scripts (like mmbackup, mmxcp, > etc). There is code in samples/ilm that show how to use it. > > > > > > -Wayne > > > > > > *From: *gpfsug-discuss on behalf of > Olaf Weiser > *Date: *Wednesday, July 5, 2023 at 7:06 PM > *To: *gpfsug main discussion list > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] Special characters in filenames > > Hallo Jonathan, I haven't used it for a while, but I can remember a > customer, where we masked "all" special characters with ESCAPE In fact, as > far as I remember. . this was an iterative progress .. . ? ? You're > right, the doc's are > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > * Report Suspicious * > ? > > > ZjQcmQRYFpfptBannerEnd > > Hallo Jonathan, > > > > I haven't used it for a while, but I can remember a customer, where we > masked "all" special characters with *ESCAPE* > > In fact, as far as I remember.. this was an iterative progress ... ? ? > > > > You're right, the doc's are not really self-explaining here.. > > > > from my personal notes I found a litte better example: > > > > In GPFS 3.5 we introduce an (optional) ESCAPE clause to the EXTERNAL LIST > and EXTERNAL POOL rules, which allow the user-administrator to specify that > path names and SHOW(strings) within the associated file lists are encoded > using an encoding based on the RFC3986 URI-percent-encoding scheme. For > example: > > RULE 'xp' EXTERNAL POOL 'pool-name' EXEC 'script-name' ESCAPE '%' > > RULE 'xl' EXTERNAL LIST 'list-name' EXEC 'script-name' ESCAPE '%/+@#' > > > > ESCAPE '%' specifies that all characters except the "unreserved" > characters in the set a-zA-Z0-9-_.~ are encoded as %XX where XX comprises 2 > hexadecimal digits. The GPFS ESCAPE clause allows you to add to the set of > "unreserved" characters. > > For example, ESCAPE '%/+@#', specifies that none of the characters in > "/+@#" are escaped, so that a path name like > "/root/directory/@abc+def#ghi.jkl" will appear in a file list with no > escape sequences, whereas under ESCAPE '%', specifying a rigorous RFC3986 > encoding yields "%2Froot%2Fdirectory%2F%40abc%2Bdef%23ghi.jkl". > > > > at least for us, it was doing the trick (back then) by using ESCAPE.. > > Maybe it is useful for your case here as well > > > > cheers > > laff > > > ------------------------------ > > *Von:* gpfsug-discuss im Auftrag von > Jonathan Buzzard > *Gesendet:* Donnerstag, 6. Juli 2023 00:20 > *An:* gpfsug main discussion list > *Betreff:* [EXTERNAL] [gpfsug-discuss] Special characters in filenames > > > > > After another support incident that eventually transpired to be down to > the user using what I will call stupid characters in their filenames (we > include a section on not doing this in our mandatory training so no > excuse) I have been musing on using the policy engine to periodically > produce lists of files that have stupid characters in their filenames so > we can proactively educate the users and get them to rename their files > to something sensible :-) > > The issue is of course the stupid characters include all the regular > expression wildcard characters in addition to \n, \r and backticks. I am > coming up short on escaping them correctly in REGEX() for the policy > engine. > > The documentation appears to be devoid of help on the subject, because > of course only an fool would be including these characters in their > filenames... > > Anyone any idea on how to do this? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.hanlon at kcl.ac.uk Wed Jul 12 08:36:41 2023 From: daniel.hanlon at kcl.ac.uk (Daniel Hanlon) Date: Wed, 12 Jul 2023 07:36:41 +0000 Subject: [gpfsug-discuss] External storage pool Message-ID: Hello! (sorry to those who?ve seen a similar message on Slack, but I understand that more GPFS devs have visibility of this mailing list) I run the GPFS-based (currently 5.1.2) general purpose research data storage facility at KCL and I?ve been experimenting with external storage pools. Using a VM running the Developer Edition I?m trying to implement a proof of concept. I?ve defined the external storage pool: RULE EXTERNAL POOL ?dt? EXEC ?/root/rule/dt_esp? pointing to my go code that understands the ?verbs? from the policy engine ? TEST, MIGRATE, PREMIGRATE etc. and I have RULE ?migrate_to_dt? MIGRATE FROM POOL ?system? to POOL ?dt? WHERE PATH_NAME LIKE ?/test1/fileset1/%? I?ve implemented MIGRATE, to start with, and I get the list of files and copy them over the the external storage. All that seems to be working fine and the files are getting copied over as expected. However, I?m struggling to find any documentation or examples of stubbing the file in GPFS or how the RECALL action should work. Is it up to the implementation of any particular external storage pool type to decide what to leave in place and set whichever extended attribute it wants? I wondered whether there are any examples of this and what?s needed (DMAPI?) to get the RECALL. Many thanks! Daniel. /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ Daniel Hanlon Head of e-Research Data, e-Research, RMID, King?s College London. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpr9c at virginia.edu Mon Jul 17 19:13:21 2023 From: jpr9c at virginia.edu (Ruffner, Scott (jpr9c)) Date: Mon, 17 Jul 2023 18:13:21 +0000 Subject: [gpfsug-discuss] replacing a "missing" disk Message-ID: I've got a couple of disks which were failing/draining and never completed this stage, and now simply report as missing after a revive and then resume. I've also reseated the drive, but no blinking lights, so I'm thinking it's dead. Can one force a drain? What's the procedure for a drive lost when it's altogether dead? TIA Scott Ruffner From ewahl at osc.edu Mon Jul 17 20:59:10 2023 From: ewahl at osc.edu (Wahl, Edward) Date: Mon, 17 Jul 2023 19:59:10 +0000 Subject: [gpfsug-discuss] replacing a "missing" disk In-Reply-To: References: Message-ID: I've run into this a few times, especially drives like SSDs just vanishing and going "missing". No amount of physical work will help those. Normally just '--simulate-fail' is enough to kick the disk hospital into getting the erase code rebuild rolling. But with a few disks I've had to use --simulate-fail and --simulate-dead BOTH, and that was what it took to get an SSD to prep. The worst part here is that since the drive is 'missing' you cannot just rely on 'replace' to kick things off for you, as '--prepare' will NOT allow you to swap the drive at LEAST until the simulate-fail gets rolling. Ed Wahl Ohio Supercomputer Center -----Original Message----- From: gpfsug-discuss On Behalf Of Ruffner, Scott (jpr9c) Sent: Monday, July 17, 2023 2:13 PM To: gpfsug-discuss at gpfsug.org Subject: [gpfsug-discuss] replacing a "missing" disk I've got a couple of disks which were failing/draining and never completed this stage, and now simply report as missing after a revive and then resume. I've also reseated the drive, but no blinking lights, so I'm thinking it's dead. Can one force a drain? What's the procedure for a drive lost when it's altogether dead? TIA Scott Ruffner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!zma2dhPZ346NVBqTcn_thK-fGa5-6knNNp-3Rjpof3mqIkvqy5QoUQgY-M2u-mnm7djYJpWXokqFGQI$ From chair at gpfsug.org Wed Jul 19 07:21:59 2023 From: chair at gpfsug.org (chair at gpfsug.org) Date: Wed, 19 Jul 2023 07:21:59 +0100 Subject: [gpfsug-discuss] GPFS Uk Chair - Vacancy Message-ID: <73b9b3d7ff380e6fdc4b4aba03c411e6@gpfsug.org> Dear All, At the recent UK GPFS UG meeting, I stepped down as the chair, and due to no volunteers at that meeting, OCF are now custodians of the User Group. If you are interested in taking the role of chair, please let us know so we can start the voting process in plenty of time before the next planned meeting at CIUK Regards Paul