[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Uninterruptible sleep apache process while aceessing nfs on debian 12 bookworm



Hi

We recently migrated to new nfs server running on debian 12 bookworm

On the client Apache processes started randomly switching to D state,

In apache fluststatus Process 93661 a mis 10786 sec
=============

4-1 93661 1598/ W 15.92 10786 0 2367404 0.0 71.45 142.44 172.20.1.47 http/1.1 sisca.groupe-mfc.fr:80 POST /deverrouille-fiche-ajax.php?sTable=prospects&iCode=243239
====


ps aux ==> Process 93661 un interruptible sleep
============
root@hexaom-v2-vm-prod-front2:~# while true; do date; ps auxf | awk '{if($8=="D") print $0;}'; sleep 1; done
Fri 26 Apr 2024 12:37:59 PM CEST
www-data   93661  0.1  1.4 315100 120468 ?       D    08:45   0:14  \_ /usr/sbin/apache2 -k start
www-data  119374  0.2  0.0      0     0 ?        D    11:33   0:10  \_ [apache2]
www-data  127425  0.1  0.8 214520 68308 ?        D    12:27   0:00  \_ /usr/sbin/apache2 -k start
============


process stack :  (can't attach using gdp gcore etc ....)
===========
root@hexaom-v2-vm-prod-front2:~# cat /proc/93661/stack 
[<0>] wait_on_commit+0x71/0xb0 [nfs]
[<0>] __nfs_commit_inode+0x131/0x180 [nfs]
[<0>] nfs_wb_all+0xb4/0x100 [nfs]
[<0>] nfs4_file_flush+0x6f/0xa0 [nfsv4]
[<0>] filp_close+0x2f/0x70
[<0>] __x64_sys_close+0x1e/0x60
[<0>] do_syscall_64+0x30/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x62/0xc7
=========



In the client  debian 11 
=====
 rpcdebug -m nfs -s all
Apr 26 11:30:15 hexaom-v2-vm-prod-front2 kernel: [51318.693854] decode_attr_fs_locations: fs_locations done, error = 0
Apr 26 11:30:15 hexaom-v2-vm-prod-front2 kernel: [51318.693871] nfs41_sequence_process: Error 0 free the slot 
Apr 26 11:30:15 hexaom-v2-vm-prod-front2 kernel: [51318.694161] nfs41_sequence_process: Error 0 free the slot 
Apr 26 11:30:15 hexaom-v2-vm-prod-front2 kernel: [51318.694301] nfs41_sequence_process: Error 0 free the slot 
=====

No error in nfds server even with debug all : rpcdebug -m nfsd -s all

Information :  on client and server
*******************



client : 

========
root@hexaom-v2-vm-prod-front2:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"


root@hexaom-v2-vm-prod-front2:~# uname -a
Linux hexaom-v2-vm-prod-front2 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux


root@hexaom-v2-vm-prod-front2:~# dpkg -l | grep -i nfs
ii  liblockfile1:amd64                1.17-1+b1                                                                  amd64        NFS-safe locking library
ii  libnfsidmap2:amd64                0.25-6                                                                     amd64        NFS idmapping library
ii  nfs-common                        1:1.3.4-6                                                                  amd64        NFS support files common to client and server


fstab:
192.20.2.30:/NFS/sessions_v2 /srv/sessions nfs defaults,rw,relatime,vers=4.1,hard,timeo=100,retrans=4,_netdev     0 0


=============




Server:
=============
root@SERVSESSION01:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"


Linux SERVSESSION01 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

*********
root@SERVSESSION01:~# dpkg -l | grep nfs
ii  libnfsidmap1:amd64                1:2.6.2-4                           amd64        NFS idmapping library
ii  nfs-common                        1:2.6.2-4                           amd64        NFS support files common to client and server
ii  nfs-kernel-server                 1:2.6.2-4                           amd64        support for NFS kernel server
root@SERVSESSION01:~# dpkg -l | grep rpc
ii  libtirpc-common                   1.3.3+ds-1                          all          transport-independent RPC library - common files
ii  libtirpc3:amd64                   1.3.3+ds-1                          amd64        transport-independent RPC library
ii  rpcbind                           1.2.6-6+b1                          amd64        converts RPC program numbers into universal addresses
root@SERVSESSION01:~# 
**********


*********
root@SERVSESSION01:~# cat /etc/default/nfs-common 
# If you do not set values for the NEED_ options, they will be attempted
# autodetected; this should be sufficient for most people. Valid alternatives
# for the NEED_ options are "yes" and "no".

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=

# Options for rpc.statd.
#   Should rpc.statd listen on a specific port? This is especially useful
#   when you have a port-based firewall. To use a fixed port, set this
#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS
STATDOPTS=

# Do you want to start the idmapd daemon? It is only needed for NFSv4.
#NEED_IDMAPD=
NEED_IDMAPD="yes"

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=
************

************
root@SERVSESSION01:~# cat /etc/default/nfs-kernel-server 
# Number of servers to start up
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids"

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=""

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""
*********


********
root@SERVSESSION01:~# cat /etc/nfs.conf | grep -vP '^#'
[general]
pipefs-directory=/run/rpc_pipefs
[nfsrahead]
[exports]
[exportfs]
[gssd]
[lockd]
[exportd]
[mountd]
manage-gids=y
[nfsdcld]
[nfsdcltrack]
[nfsd]

[statd]
[sm-notify]
[svcgssd]
******


****
root@SERVSESSION01:~# cat /etc/idmapd.conf 
[General]

Verbosity = 0
# set your own domain here, if it differs from FQDN minus hostname
# Domain = localdomain

[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup
****
=============

Could any one help, did any one encounter this bug ? 

Best regards 

























Reply to: