Bug#946305: linux-image-5.3.0-2-amd64, linux-image-5.3.0-3-amd64 problems copying a large group of files
Under linux kernel-image-4.19.0-6-amd64 Linux 4.19 for 64-bit PCs (signed)
(4.19.67-2 + deb10u1) no problems occur when copying folders with files
total volume > 100 Gb.
Problems were found when working under the linux-image kernel-5.3.0-3-amd64
Linux 5.3 for 64-bit PCs (signed) (5.3.15-1)
After copying an array of files in 1.4 Tb from one disk to another, by testing
copied "*.rar-files" some files are found with partially distorted content.
How this is possible is not yet clear. "rar t" on such files gives messages
"checksum error". There may not be any such bad files or be from 1 to 6-7
on the entire array of files. And from copy to copy such failures occur
in arbitrary files.
A total of 4511 rar files of 1.4 Tb were copied.
This happens when you copy through:
- copy with " Double Commander"
- "cp-Rp"../Download" " / media/u1/u-z/U-Z/"
- time ionice -c Best-effort-n 7 cp-Rp "../Download" "/media/user/disk2/"
In all 3 cases, copying is about 3 or more hours.
After that testing the copied files using
time ionice -c Best-effort-n 7 rar t-r "*.rar" &> "log.txt"
allows you to detect failures that occur when copying rar archives, the message
"checksum error" on such files. I. when you test such copied archives, then
you find the replacement of entire blocks on the unknown where the dirt has taken,
but the size of the files does not change. So at least you can find it. What's
going on with files not protected by CRC codes, one can only guess.
This shouldn't be happening!
I re-copy the failed files manually through "Double Commander" and then test them
this archive. And it passes without mistakes.
Testing a copied 1.4 Tb archive array takes between 2.5 hours and 3 hours.
Then defragment copied through "e4defrag -v 'path to files massive'".
Again I check rar-files through
time ionice -c Best-effort-n 7 rar t-r "*.rar" &> "log.txt"
And then it turns out that again arbitrarily appear archives, which are reported
"checksum error". But at the subsequent individual testing of such archives
no "checksum error" messages appear. It turned out that after
defragment the copied files, you just need to reset the cache:
sync; echo 3 > /proc/sys/vm/drop_caches (with administrator rights)
And after resetting the cache, start testing the copied archives. Then
"checksum error" messages do not appear.
When copying individual folders from this array with "*.rar files"
there are no such problems.
The problem is related to caching information in Debian. The method
of "scientific poke" picked up parameters for Seagate ST2000 disks
(disks with" Hot-Plug " Barracuda ST2000DMD001, Constellation ES.3 ST2000NM0033
and CS ST2000NC001, not " Hot-Plug» Barracuda ST2000DM008).
About the "vm.dirty_background_ratio " look at the installed package
"linux-source" in the folder " /usr/src/linux-source-5.3.tar.xz". Open the archive
tar-x-f "/usr/src / linux-source-5.3.tar.xz"
And look in the folder " linux-source-5.3/Documentation/admin-guide/sysctl/vm.rst".
In General, in the folder "linux-source-5.3/Documentation" a lot of very important
documentation on configuring Debian.
It is necessary to install for Linux kernel-image-5.3.0-3-amd64 Linux 5.3
for 64-bit PCs (signed) (5.3.15-1):
In " /etc/sysctl.d/local.conf"
vm.dirty_background_ratio = 5
vm.dirty_background_ratio = 10 (default), =9, = 8, =7, and even at =6
when copying through all three copy methods (see above), there are
distortion of information when copying from disk to disk.
In "Double Commander" are visible temporary slowdown copying and even
temporarily repeatedly pausing copying. Using "Double Commander" you can clearly
see the change in the speed of copying and the process itself. Interesting,
that in the process copying repeatedly were attempts the most
Debian to increase the speed of copying, but with the simultaneous further
speed drop to 120 Mb/s, although copying started at 170 Mb/s.
When "vm.dirty_background_ratio = 5" stopped occurring after copying
1.4 Tb of rar files of the message "checksum error" at "rar t".
That is, we can assume that the information is now copied without distortion.
On other computers, the acceptable value is " vm.dirty_background_ratio " can
be another.
When set to " vm.dirty_background_ratio=5 " copying an array of 1.4 Gb files
in "Double Commander" takes the same time as in
time ionice -c Best-effort-n 7 cp-Rp "../Download" " /media/u1/disk2/"
, i.e. about 3 hours and, in the future, when testing the copied information
no errors occur.
Reply to: