容器停机跨服务器迁移后无法stop问题

操作过程

• 在debian 12上启动一个Docker容器，执行一些增删改查后停止容器。
• 找到容器的Upper dir， 并将其打包，scp到Ubuntu上。
• 在Ubuntu上创建一个容器， 将迁移过来的Upper dir覆盖到新创建的容器对应目录下，然后启动新创建的容器。

按照预期的流程：

• 将旧容器的读写层（diff目录），打包成tar文件;
• scp导出的tar文件到新机器;
• 创建新容器（默认参数）， 覆盖新容器的diff目录;
• 启动新容器，容器正常启动且改动的数据是与旧容器同步的;

问题出在docker stop 新容器的时候，会提示： Error response from daemon: cannot stop container: nginx: permission denied

问题分析

本质是因为apparmor模块拒绝了stop的操作，docker stop是会发送 sigterm信号给容器，超时后发送sigkill信号给容器，从syslog中的 apparmor="DENIED"的相关日志可以看到：这个singal处理不知道为什么被拒绝了？

2025-05-06T09:32:21.904139+00:00 ubuntu kernel: audit: type=1400 audit(1746523941.901:157): apparmor="DENIED" operation="ptrace" class="ptrace" profile="snap.docker.dockerd" pid=3405 comm="runc" requested_mask="read" denied_mask="read" peer="docker-container-custom"  
2025-05-06T09:32:21.904157+00:00 ubuntu kernel: audit: type=1400 audit(1746523941.902:158): apparmor="DENIED" operation="ptrace" class="ptrace" profile="snap.docker.dockerd" pid=3405 comm="runc" requested_mask="read" denied_mask="read" peer="docker-container-custom"  
2025-05-06T09:32:21.904159+00:00 ubuntu kernel: audit: type=1400 audit(1746523941.902:159): apparmor="DENIED" operation="signal" class="signal" profile="snap.docker.dockerd" pid=3405 comm="runc" requested_mask="send" denied_mask="send" signal=quit peer="docker-container-custom"  
2025-05-06T09:32:23.915630+00:00 ubuntu docker.dockerd[1184]: time="2025-05-06T09:32:23.914903416Z" level=error msg="Error sending stop (signal 3) to container" container=656acd42fa37cd15e563ca44481a5680c3e3316efc6dbc9989a5f7de7db3af22 error="Cannot kill container 656acd42fa37cd15e563ca44481a5680c3e3316efc6dbc9989a5f7de7db3af22: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"  
2025-05-06T09:32:23.941093+00:00 ubuntu kernel: audit: type=1400 audit(1746523943.939:160): apparmor="DENIED" operation="ptrace" class="ptrace" profile="snap.docker.dockerd" pid=3411 comm="runc" requested_mask="read" denied_mask="read" peer="docker-container-custom"  
2025-05-06T09:32:23.942141+00:00 ubuntu kernel: audit: type=1400 audit(1746523943.940:161): apparmor="DENIED" operation="ptrace" class="ptrace" profile="snap.docker.dockerd" pid=3411 comm="runc" requested_mask="read" denied_mask="read" peer="docker-container-custom"  
2025-05-06T09:32:23.942149+00:00 ubuntu kernel: audit: type=1400 audit(1746523943.940:162): apparmor="DENIED" operation="signal" class="signal" profile="snap.docker.dockerd" pid=3411 comm="runc" requested_mask="send" denied_mask="send" signal=kill peer="docker-container-custom"  
2025-05-06T09:32:35.341179+00:00 ubuntu kernel: audit: type=1400 audit(1746523955.339:163): apparmor="DENIED" operation="signal" class="signal" profile="snap.docker.dockerd" pid=1184 comm="dockerd" requested_mask="send" denied_mask="send" signal=kill peer="docker-container-custom"  
2025-05-06T09:32:37.355195+00:00 ubuntu docker.dockerd[1184]: time="2025-05-06T09:32:37.354219068Z" level=error msg="error killing container: context deadline exceeded" container=656acd42fa37cd15e563ca44481a5680c3e3316efc6dbc9989a5f7de7db3af22 error="permission denied"  
2025-05-06T09:32:37.355335+00:00 ubuntu docker.dockerd[1184]: time="2025-05-06T09:32:37.354335353Z" level=error msg="Handler for POST /v1.45/containers/nginx/stop returned error: cannot stop container: nginx: permission denied"

从上述来看是snap的原因，进一步查询资料后发现Ubuntu系统下，通过snap安装的程序的apparmor策略配置文件与apt安装的目录不一致。

解决办法

一、禁用apparmor模块

systemctl stop apparmor
# 或者
aa-remove-unknown

重新走创建容器之后import的流程。 容器可以正常stop.

二、容器配置为unconfined

创建容器时增加 --security-opt "apparmor=unconfined" 参数，表示不限制容器的访问，然后走import流程，容器可以正常stop 。

docker create --security-opt "apparmor=unconfined" --name=nginx nginx:test-layer

三、使用自定义的apparmor配置

不使用--security-opt apparmor=unconfined参数的实验


修改 vi /var/lib/snapd/apparmor/profiles/snap.docker.dockerd 
增加内容：
signal (send, receive) peer=unconfined,  
signal (send, receive) peer=docker-default,  
signal (send, receive) peer=docker-container-custom,  
signal (send) set=(kill, term, quit, stop, cont, hup, int),  

ptrace (trace, read) peer=docker-default,  
ptrace (trace, read) peer=unconfined,  
ptrace (trace, read, tracedby, readby) peer=docker-container-custom,

保存文件改动，执行命令加载改动：apparmor_parser -r -W /var/lib/snapd/apparmor/profiles/snap.docker.dockerd

编辑自定义的apparmor配置文件docker-container-custom：
内容如下：

#include <tunables/global>  
  
profile docker-container-custom flags=(attach_disconnected,mediate_deleted) {  
 #include <abstractions/base>  
  
 # 基本网络和容器操作权限  
 network,  
 capability,  
 file,  
 umount,  
 mount,  
  
 # 增强信号处理权限  
 signal (send, receive) peer=unconfined,  
 signal (send, receive) peer=docker-container-custom,  
 signal (send, receive) peer=docker-default,  
 signal (send, receive) peer=snap.docker.dockerd,  
  
 # 允许接收所有停止相关信号  
 signal (receive) set=(kill, term, quit, stop, cont, hup, int),  
  
 # 增强 ptrace 权限  
 ptrace (trace, read, tracedby, readby) peer=docker-container-custom,  
 ptrace (trace, read, tracedby, readby) peer=unconfined,  
 ptrace (trace, read, tracedby, readby) peer=docker-default,  
 ptrace (read, trace, tracedby, readby) peer=snap.docker.dockerd,  
  
 # 允许访问容器文件系统  
 /var/snap/docker/common/var-lib-docker/overlay2/** rwk,  
 /var/lib/docker/overlay2/** rwk,  
  
 # 允许访问 proc 文件系统  
 @{PROC}/** r,  
 @{PROC}/[0-9]*/attr/current w,  
  
 # 允许访问必要的系统文件  
 /sys/fs/cgroup/** rwk,

# 允许容器内部操作  
 /** rwk,  
  
 # 允许与 Docker 守护进程通信  
 unix (send, receive) type=stream peer=(label=snap.docker.dockerd),  
}

保存文件内容，加载自定义配置文件： apparmor_parser -r -W /home/arlen/docker-container-custom

创建容器时增加参数，指定容器使用自定义的apparmor配置文件： --security-opt "apparmor=docker-container-custom" 

docker create --security-opt "apparmor=docker-container-custom" --name=nginx nginx:test-layer