[gpfsug-discuss] Client Latency and High NSD Server Load Average

[Saula, Oluwasijibomi]

Vladis/Kums/Fred/Kevin/Stephen,

Thanks so much for your insights, thoughts, and pointers! - Certainly increased my knowledge and understanding of potential culprits to watch for...

So we finally discovered the root issue to this problem: An unattended TSM restore exercise profusely writing to a single file, over and over again into the GBs!!..I'm opening up a ticket with TSM support to learn how to mitigate this in the future.

But with the RAID 6 writing costs Vladis explained, it now makes sense why the write IO was badly affected...

Excerpt from output file:


--- User Action is Required ---

File '/gpfs1/X/Y/Z/fileABC' is write protected


Select an appropriate action

  1. Force an overwrite for this object

  2. Force an overwrite on all objects that are write protected

  3. Skip this object

  4. Skip all objects that are write protected

  A. Abort this operation

Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]

Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]
Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]
Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]
...


Thanks,


Oluwasijibomi (Siji) Saula

HPC Systems Administrator  /  Information Technology



Research 2 Building 220B / Fargo ND 58108-6050

p: 701.231.7749 / www.ndsu.edu<http://www.ndsu.edu/>



[cid:image001.gif at 01D57DE0.91C300C0]



________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of gpfsug-discuss-request at spectrumscale.org <gpfsug-discuss-request at spectrumscale.org>
Sent: Friday, June 5, 2020 6:00 AM
To: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
Subject: gpfsug-discuss Digest, Vol 101, Issue 12

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: Client Latency and High NSD Server Load Average
      (Valdis Kl=?utf-8?Q?=c4=93?=tnieks)


----------------------------------------------------------------------

Message: 1
Date: Thu, 04 Jun 2020 21:17:08 -0400
From: "Valdis Kl=?utf-8?Q?=c4=93?=tnieks" <valdis.kletnieks at vt.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Client Latency and High NSD Server Load
        Average
Message-ID: <309214.1591319828 at turing-police>
Content-Type: text/plain; charset="us-ascii"

On Thu, 04 Jun 2020 15:33:18 -0000, "Saula, Oluwasijibomi" said:

> However, I still can't understand why write IO operations are 5x more latent
> than ready operations to the same class of disks.

Two things that may be biting you:

First, on a RAID 5 or 6 LUN, most of the time you only need to do 2 physical
reads (data and parity block). To do a write, you have to read the old parity
block, compute the new value, and write the data block and new parity block.
This is often called the "RAID write penalty".

Second, if a read size is smaller than the physical block size, the storage array can read
a block, and return only the fragment needed.  But on a write, it has to read
the whole block, splice in the new data, and write back the block - a RMW (read
modify write) cycle.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200604/da016913/attachment-0001.sig>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 101, Issue 12
***********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200605/6989ff30/attachment.htm>