2026-04-01T15:36:15.299 INFO:root:teuthology version: 1.2.4.dev37+ga59626679 2026-04-01T15:36:15.305 DEBUG:teuthology.report:Pushing job info to http://localhost:8080 2026-04-01T15:36:15.326 INFO:teuthology.run:Config: archive_path: /archive/supriti-2026-04-01_13:45:16-rgw-wip-sse-s3-on-v20.2.0-none-default-vps/4821 branch: wip-sse-s3-on-v20.2.0 description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides rocky_latest s3tests-branch} email: null first_in_suite: false flavor: default job_id: '4821' last_in_suite: false machine_type: vps name: supriti-2026-04-01_13:45:16-rgw-wip-sse-s3-on-v20.2.0-none-default-vps no_nested_subset: false os_type: rocky os_version: '9.7' overrides: admin_socket: branch: wip-sse-s3-on-v20.2.0 ansible.cephlab: branch: main repo: https://github.com/kshtsk/ceph-cm-ansible.git skip_tags: nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs vars: logical_volumes: lv_1: scratch_dev: true size: 25%VG vg: vg_nvme lv_2: scratch_dev: true size: 25%VG vg: vg_nvme lv_3: scratch_dev: true size: 25%VG vg: vg_nvme lv_4: scratch_dev: true size: 25%VG vg: vg_nvme timezone: UTC volume_groups: vg_nvme: pvs: /dev/vdb,/dev/vdc,/dev/vdd,/dev/vde ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption_keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw s3 auth use sts: true rgw sts key: abcdefghijklmnop setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd mclock iops capacity threshold hdd: 49000 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: e2ce8426bc5387b913d1b06bd31280ced108bbb6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} cephadm: cephadm_binary_url: https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm containers: image: harbor.clyso.com/custom-ceph/ceph/ceph:sse-s3-kmip-preview-not-for-production-2 install: ceph: flavor: default sha1: e2ce8426bc5387b913d1b06bd31280ced108bbb6 extra_system_packages: deb: - python3-jmespath - python3-xmltodict - s3cmd rpm: - bzip2 - perl-Test-Harness - python3-jmespath - python3-xmltodict - s3cmd repos: - name: ceph-source priority: 1 url: https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/SRPMS - name: ceph-noarch priority: 1 url: https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/noarch - name: ceph priority: 1 url: https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/x86_64 rgw: compression type: random storage classes: FROZEN: null LUKEWARM: - --compression - zlib s3tests: accounts: iam alt root: RGW99999999999999999 iam root: RGW88888888888888888 force-branch: ceph-tentacle sha1: e0c4ff71baef6d5126a0201df5fe54196d89b296 storage classes: LUKEWARM, FROZEN selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: tt-20.2.0-sse-s3-kmip-preview-not-for-production-2 sha1: 0d46c98b681455ef9cd34c2d3c6981da4e9f1aea owner: supriti priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 343 sha1: e2ce8426bc5387b913d1b06bd31280ced108bbb6 sleep_before_teardown: 0 suite: rgw suite_branch: tt-20.2.0-sse-s3-kmip-preview-not-for-production-2 suite_path: /home/teuthos/src/git.local_ceph_0d46c98b681455ef9cd34c2d3c6981da4e9f1aea/qa suite_relpath: qa suite_repo: http://git.local/ceph.git suite_sha1: 0d46c98b681455ef9cd34c2d3c6981da4e9f1aea targets: vm07.local: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSgXVFmsTpLwLidMh2xFtU8MeDlh5SEJGOmlCW9pFQZQruNvZaGwlGg9uW0ohiG7JP344vtmZFK9GyLWw83iCw= tasks: - install: null - ceph: null - rgw: client.0: null - tox: - client.0 - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: uv2 teuthology_repo: https://github.com/kshtsk/teuthology teuthology_sha1: a59626679648f962bca99d20d35578f2998c8f37 timestamp: 2026-04-01_13:45:16 tube: vps user: supriti verbose: false worker_log: /home/teuthos/.teuthology/dispatcher/dispatcher.vps.1399776 2026-04-01T15:36:15.326 INFO:teuthology.run:suite_path is set to /home/teuthos/src/git.local_ceph_0d46c98b681455ef9cd34c2d3c6981da4e9f1aea/qa; will attempt to use it 2026-04-01T15:36:15.327 INFO:teuthology.run:Found tasks at /home/teuthos/src/git.local_ceph_0d46c98b681455ef9cd34c2d3c6981da4e9f1aea/qa/tasks 2026-04-01T15:36:15.327 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-04-01T15:36:15.327 INFO:teuthology.task.internal:Saving configuration 2026-04-01T15:36:15.333 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-04-01T15:36:15.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-04-01T15:36:15.341 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'vm07.local', 'description': '/archive/supriti-2026-04-01_13:45:16-rgw-wip-sse-s3-on-v20.2.0-none-default-vps/4821', 'up': True, 'machine_type': 'vps', 'is_vm': True, 'vm_host': {'name': 'localhost', 'description': None, 'up': True, 'machine_type': 'libvirt', 'is_vm': False, 'vm_host': None, 'os_type': None, 'os_version': None, 'arch': None, 'locked': True, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': None}, 'os_type': 'rocky', 'os_version': '9.7', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-04-01 15:35:43.990040', 'locked_by': 'supriti', 'mac_address': '52:55:00:00:00:07', 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSgXVFmsTpLwLidMh2xFtU8MeDlh5SEJGOmlCW9pFQZQruNvZaGwlGg9uW0ohiG7JP344vtmZFK9GyLWw83iCw='} 2026-04-01T15:36:15.341 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-04-01T15:36:15.341 INFO:teuthology.task.internal:roles: ubuntu@vm07.local - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-04-01T15:36:15.342 INFO:teuthology.run_tasks:Running task console_log... 2026-04-01T15:36:15.347 DEBUG:teuthology.task.console_log:vm07 does not support IPMI; excluding 2026-04-01T15:36:15.347 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe798b92440>, signals=[15]) 2026-04-01T15:36:15.347 INFO:teuthology.run_tasks:Running task internal.connect... 2026-04-01T15:36:15.348 INFO:teuthology.task.internal:Opening connections... 2026-04-01T15:36:15.348 DEBUG:teuthology.task.internal:connecting to ubuntu@vm07.local 2026-04-01T15:36:15.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm07.local', 'username': 'ubuntu', 'timeout': 60} 2026-04-01T15:36:15.421 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-04-01T15:36:15.422 DEBUG:teuthology.orchestra.run.vm07:> uname -m 2026-04-01T15:36:15.576 INFO:teuthology.orchestra.run.vm07.stdout:x86_64 2026-04-01T15:36:15.576 DEBUG:teuthology.orchestra.run.vm07:> cat /etc/os-release 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:NAME="Rocky Linux" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:VERSION="9.7 (Blue Onyx)" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:ID="rocky" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:ID_LIKE="rhel centos fedora" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:VERSION_ID="9.7" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:PLATFORM_ID="platform:el9" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:PRETTY_NAME="Rocky Linux 9.7 (Blue Onyx)" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:ANSI_COLOR="0;32" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:LOGO="fedora-logo-icon" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:CPE_NAME="cpe:/o:rocky:rocky:9::baseos" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:HOME_URL="https://rockylinux.org/" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:VENDOR_NAME="RESF" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:VENDOR_URL="https://resf.org/" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:SUPPORT_END="2032-05-31" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="9.7" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux" 2026-04-01T15:36:15.633 INFO:teuthology.orchestra.run.vm07.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="9.7" 2026-04-01T15:36:15.633 INFO:teuthology.lock.ops:Updating vm07.local on lock server 2026-04-01T15:36:15.638 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-04-01T15:36:15.640 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-04-01T15:36:15.640 INFO:teuthology.task.internal:Checking for old test directory... 2026-04-01T15:36:15.641 DEBUG:teuthology.orchestra.run.vm07:> test '!' -e /home/ubuntu/cephtest 2026-04-01T15:36:15.690 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-04-01T15:36:15.691 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-04-01T15:36:15.691 DEBUG:teuthology.orchestra.run.vm07:> test -z $(ls -A /var/lib/ceph) 2026-04-01T15:36:15.747 INFO:teuthology.orchestra.run.vm07.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-04-01T15:36:15.747 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-04-01T15:36:15.756 DEBUG:teuthology.orchestra.run.vm07:> test -e /ceph-qa-ready 2026-04-01T15:36:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-04-01T15:36:16.006 INFO:teuthology.run_tasks:Running task internal.base... 2026-04-01T15:36:16.007 INFO:teuthology.task.internal:Creating test directory... 2026-04-01T15:36:16.008 DEBUG:teuthology.orchestra.run.vm07:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-04-01T15:36:16.024 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-04-01T15:36:16.025 INFO:teuthology.run_tasks:Running task internal.archive... 2026-04-01T15:36:16.026 INFO:teuthology.task.internal:Creating archive directory... 2026-04-01T15:36:16.026 DEBUG:teuthology.orchestra.run.vm07:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-04-01T15:36:16.084 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-04-01T15:36:16.085 INFO:teuthology.task.internal:Enabling coredump saving... 2026-04-01T15:36:16.085 DEBUG:teuthology.orchestra.run.vm07:> test -f /run/.containerenv -o -f /.dockerenv 2026-04-01T15:36:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-04-01T15:36:16.140 DEBUG:teuthology.orchestra.run.vm07:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-04-01T15:36:16.206 INFO:teuthology.orchestra.run.vm07.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-04-01T15:36:16.215 INFO:teuthology.orchestra.run.vm07.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-04-01T15:36:16.216 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-04-01T15:36:16.217 INFO:teuthology.task.internal:Configuring sudo... 2026-04-01T15:36:16.218 DEBUG:teuthology.orchestra.run.vm07:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-04-01T15:36:16.280 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-04-01T15:36:16.282 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-04-01T15:36:16.282 DEBUG:teuthology.orchestra.run.vm07:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-04-01T15:36:16.334 DEBUG:teuthology.orchestra.run.vm07:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-04-01T15:36:16.396 DEBUG:teuthology.orchestra.run.vm07:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-04-01T15:36:16.455 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:36:16.455 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-04-01T15:36:16.515 DEBUG:teuthology.orchestra.run.vm07:> sudo service rsyslog restart 2026-04-01T15:36:16.586 INFO:teuthology.orchestra.run.vm07.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-04-01T15:36:17.021 INFO:teuthology.run_tasks:Running task internal.timer... 2026-04-01T15:36:17.022 INFO:teuthology.task.internal:Starting timer... 2026-04-01T15:36:17.022 INFO:teuthology.run_tasks:Running task pcp... 2026-04-01T15:36:17.025 INFO:teuthology.run_tasks:Running task selinux... 2026-04-01T15:36:17.027 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2026-04-01T15:36:17.027 INFO:teuthology.task.selinux:Excluding vm07: VMs are not yet supported 2026-04-01T15:36:17.027 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-04-01T15:36:17.027 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-04-01T15:36:17.027 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-04-01T15:36:17.027 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-04-01T15:36:17.029 DEBUG:teuthology.task:Applying overrides for task ansible.cephlab: {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}} 2026-04-01T15:36:17.030 DEBUG:teuthology.repo_utils:Setting repo remote to https://github.com/kshtsk/ceph-cm-ansible.git 2026-04-01T15:36:17.035 INFO:teuthology.repo_utils:Fetching github.com_kshtsk_ceph-cm-ansible_main from origin 2026-04-01T15:36:17.731 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main to origin/main 2026-04-01T15:36:17.750 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-04-01T15:36:17.751 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu", "logical_volumes": {"lv_1": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_2": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_3": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_4": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}}, "timezone": "UTC", "volume_groups": {"vg_nvme": {"pvs": "/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde"}}}' -i /tmp/teuth_ansible_inventory78s7yt6o --limit vm07.local /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main/cephlab.yml --skip-tags nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs 2026-04-01T15:37:47.057 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@vm07.local')] 2026-04-01T15:37:47.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@vm07.local' 2026-04-01T15:37:47.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm07.local', 'username': 'ubuntu', 'timeout': 60} 2026-04-01T15:37:47.121 DEBUG:teuthology.orchestra.run.vm07:> true 2026-04-01T15:37:47.201 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@vm07.local' 2026-04-01T15:37:47.201 INFO:teuthology.run_tasks:Running task clock... 2026-04-01T15:37:47.204 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-04-01T15:37:47.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-04-01T15:37:47.204 DEBUG:teuthology.orchestra.run.vm07:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-04-01T15:37:47.274 INFO:teuthology.orchestra.run.vm07.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-04-01T15:37:47.288 INFO:teuthology.orchestra.run.vm07.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-04-01T15:37:47.313 INFO:teuthology.orchestra.run.vm07.stderr:sudo: ntpd: command not found 2026-04-01T15:37:47.324 INFO:teuthology.orchestra.run.vm07.stdout:506 Cannot talk to daemon 2026-04-01T15:37:47.339 INFO:teuthology.orchestra.run.vm07.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-04-01T15:37:47.354 INFO:teuthology.orchestra.run.vm07.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-04-01T15:37:47.408 INFO:teuthology.orchestra.run.vm07.stderr:bash: line 1: ntpq: command not found 2026-04-01T15:37:47.410 INFO:teuthology.orchestra.run.vm07.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-04-01T15:37:47.410 INFO:teuthology.orchestra.run.vm07.stdout:=============================================================================== 2026-04-01T15:37:47.410 INFO:teuthology.run_tasks:Running task install... 2026-04-01T15:37:47.412 DEBUG:teuthology.task.install:project ceph 2026-04-01T15:37:47.412 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'repos': [{'name': 'ceph-source', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/SRPMS'}, {'name': 'ceph-noarch', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/noarch'}, {'name': 'ceph', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/x86_64'}]} 2026-04-01T15:37:47.412 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6', 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}} 2026-04-01T15:37:47.412 INFO:teuthology.task.install:Using flavor: default 2026-04-01T15:37:47.415 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-04-01T15:37:47.415 INFO:teuthology.task.install:extra packages: [] 2026-04-01T15:37:47.415 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6', 'tag': None, 'wait_for_package': False, 'repos': [{'name': 'ceph-source', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/SRPMS'}, {'name': 'ceph-noarch', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/noarch'}, {'name': 'ceph', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/x86_64'}]} 2026-04-01T15:37:47.415 DEBUG:teuthology.task.install.rpm:Adding repos: [{'name': 'ceph-source', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/SRPMS'}, {'name': 'ceph-noarch', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/noarch'}, {'name': 'ceph', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/x86_64'}] 2026-04-01T15:37:47.415 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:37:47.415 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/etc/yum.repos.d/ceph-source.repo 2026-04-01T15:37:47.483 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:37:47.483 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/etc/yum.repos.d/ceph-noarch.repo 2026-04-01T15:37:47.553 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:37:47.553 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-04-01T15:37:47.619 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, bzip2, perl-Test-Harness, python3-jmespath, python3-xmltodict, s3cmd on remote rpm x86_64 2026-04-01T15:37:47.619 DEBUG:teuthology.orchestra.run.vm07:> sudo yum clean all 2026-04-01T15:37:47.796 INFO:teuthology.orchestra.run.vm07.stdout:47 files removed 2026-04-01T15:37:47.818 DEBUG:teuthology.orchestra.run.vm07:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd bzip2 perl-Test-Harness python3-jmespath python3-xmltodict s3cmd 2026-04-01T15:37:48.402 INFO:teuthology.orchestra.run.vm07.stdout:ceph 213 kB/s | 90 kB 00:00 2026-04-01T15:37:48.760 INFO:teuthology.orchestra.run.vm07.stdout:ceph-noarch 75 kB/s | 25 kB 00:00 2026-04-01T15:37:48.952 INFO:teuthology.orchestra.run.vm07.stdout:ceph-source 13 kB/s | 2.3 kB 00:00 2026-04-01T15:37:49.332 INFO:teuthology.orchestra.run.vm07.stdout:Extra Packages for Enterprise Linux 56 MB/s | 20 MB 00:00 2026-04-01T15:37:54.008 INFO:teuthology.orchestra.run.vm07.stdout:lab-extras 57 kB/s | 50 kB 00:00 2026-04-01T15:37:54.824 INFO:teuthology.orchestra.run.vm07.stdout:Rocky Linux 9 - BaseOS 23 MB/s | 17 MB 00:00 2026-04-01T15:37:56.679 INFO:teuthology.orchestra.run.vm07.stdout:Rocky Linux 9 - AppStream 24 MB/s | 17 MB 00:00 2026-04-01T15:37:58.914 INFO:teuthology.orchestra.run.vm07.stdout:Rocky Linux 9 - CRB 7.7 MB/s | 4.3 MB 00:00 2026-04-01T15:37:59.805 INFO:teuthology.orchestra.run.vm07.stdout:Rocky Linux 9 - Extras 49 kB/s | 17 kB 00:00 2026-04-01T15:38:00.978 INFO:teuthology.orchestra.run.vm07.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-04-01T15:38:00.978 INFO:teuthology.orchestra.run.vm07.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-04-01T15:38:01.001 INFO:teuthology.orchestra.run.vm07.stdout:Dependencies resolved. 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout:============================================================================================= 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: Package Arch Version Repository Size 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout:============================================================================================= 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout:Installing: 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: bzip2 x86_64 1.0.8-10.el9_5 baseos 51 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 6.5 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-base x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 5.9 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-fuse x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 941 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 154 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 961 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-cephadm noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 173 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-dashboard noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 15 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 7.4 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-rook noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 50 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-radosgw x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 24 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-test x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 85 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: ceph-volume noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 297 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: cephadm noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 1.0 M 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-devel x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 34 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs2 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 868 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: librados-devel x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 126 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: perl-Test-Harness noarch 1:3.42-461.el9 appstream 267 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: python3-cephfs x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 163 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: python3-jmespath noarch 1.0.1-1.el9_7 appstream 43 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: python3-rados x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 316 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: python3-rbd x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 304 k 2026-04-01T15:38:01.006 INFO:teuthology.orchestra.run.vm07.stdout: python3-rgw x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 99 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: rbd-fuse x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 91 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: rbd-mirror x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 2.9 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: rbd-nbd x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 180 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: s3cmd noarch 2.4.0-1.el9 epel 206 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout:Upgrading: 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: librados2 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 3.5 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: librbd1 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 2.8 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout:Installing dependencies: 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: boost-program-options x86_64 1.75.0-13.el9_7 appstream 104 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: c-ares x86_64 1.19.1-2.el9_4 baseos 110 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-common x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 24 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-grafana-dashboards noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 43 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mds x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 2.3 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-modules-core noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 289 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mon x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 5.0 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-osd x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 17 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-prometheus-alerts noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 17 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ceph-selinux x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 25 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: cryptsetup x86_64 2.7.2-4.el9 baseos 310 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas x86_64 3.0.4-8.el9.0.1 appstream 30 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas-netlib x86_64 3.0.4-8.el9.0.1 appstream 3.0 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9.0.1 appstream 15 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: fuse x86_64 2.9.9-17.el9 baseos 78 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 41 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-proxy2 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 24 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libcephsqlite x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 164 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libconfig x86_64 1.7.2-9.el9 baseos 71 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libgfortran x86_64 11.5.0-11.el9 baseos 794 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libnbd x86_64 1.20.3-4.el9 appstream 171 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 159 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libquadmath x86_64 11.5.0-11.el9 baseos 184 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 44 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libradosstriper1 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 250 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: librgw2 x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 6.4 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 243 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: libxslt x86_64 1.1.34-13.el9_6 appstream 239 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: lmdb-libs x86_64 0.9.29-3.el9 baseos 60 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 282 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: lua x86_64 5.4.4-4.el9 appstream 187 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: lua-devel x86_64 5.4.4-4.el9 crb 21 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: mailcap noarch 2.1.49-5.el9.0.2 baseos 32 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: openblas x86_64 0.3.29-1.el9 appstream 41 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: pciutils x86_64 3.7.0-7.el9 baseos 92 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: perl-Benchmark noarch 1.23-481.1.el9_6 appstream 25 k 2026-04-01T15:38:01.007 INFO:teuthology.orchestra.run.vm07.stdout: protobuf x86_64 3.14.0-17.el9_7 appstream 1.0 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: protobuf-compiler x86_64 3.14.0-17.el9_7 crb 862 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-babel noarch 2.9.1-2.el9 appstream 5.8 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-ceph-argparse x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 45 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-ceph-common x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 163 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 241 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-cheroot noarch 10.0.1-5.el9 epel 173 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-cherrypy noarch 18.10.0-5.el9 epel 290 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-cryptography x86_64 36.0.1-5.el9_6 baseos 1.2 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-devel x86_64 3.9.23-2.el9 appstream 205 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-jinja2 noarch 2.11.3-8.el9_5 appstream 228 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 166 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 32 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-numpy x86_64 1:1.23.5-2.el9_7 appstream 5.8 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9_7 appstream 368 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-packaging noarch 20.9-5.el9 appstream 69 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-ply noarch 3.11-14.el9.0.1 baseos 103 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-protobuf noarch 3.14.0-17.el9_7 appstream 237 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyasn1 noarch 0.4.8-7.el9_7 appstream 132 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9_7 appstream 210 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-pycparser noarch 2.20-6.el9 baseos 124 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyparsing noarch 2.4.7-9.el9.0.1 baseos 150 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-requests noarch 2.25.1-10.el9_6 baseos 115 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 43 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-toml noarch 0.10.2-6.el9.0.1 appstream 44 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-urllib3 noarch 1.26.5-6.el9_7.1 baseos 191 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-04-01T15:38:01.008 INFO:teuthology.orchestra.run.vm07.stdout: qatlib x86_64 24.09.0-1.el9 appstream 221 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 65 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: socat x86_64 1.7.4.1-8.el9 appstream 299 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: unzip x86_64 6.0-59.el9 baseos 180 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 188 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 89 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 63 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: zip x86_64 3.0-35.el9 baseos 263 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Installing weak dependencies: 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph-noarch 22 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-daemon x86_64 2:20.2.0-9.ge2ce8426bc5.el9.clyso ceph 35 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.0 M 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: qatlib-service x86_64 24.09.0-1.el9 appstream 36 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: smartmontools x86_64 1:7.2-9.el9 baseos 551 k 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Transaction Summary 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:============================================================================================= 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Install 150 Packages 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Upgrade 2 Packages 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Total download size: 274 M 2026-04-01T15:38:01.009 INFO:teuthology.orchestra.run.vm07.stdout:Downloading Packages: 2026-04-01T15:38:02.120 INFO:teuthology.orchestra.run.vm07.stdout:(1/152): ceph-20.2.0-9.ge2ce8426bc5.el9.clyso.x 77 kB/s | 6.5 kB 00:00 2026-04-01T15:38:02.369 INFO:teuthology.orchestra.run.vm07.stdout:(2/152): ceph-fuse-20.2.0-9.ge2ce8426bc5.el9.cl 3.7 MB/s | 941 kB 00:00 2026-04-01T15:38:02.426 INFO:teuthology.orchestra.run.vm07.stdout:(3/152): ceph-immutable-object-cache-20.2.0-9.g 2.6 MB/s | 154 kB 00:00 2026-04-01T15:38:02.623 INFO:teuthology.orchestra.run.vm07.stdout:(4/152): ceph-mds-20.2.0-9.ge2ce8426bc5.el9.cly 12 MB/s | 2.3 MB 00:00 2026-04-01T15:38:02.737 INFO:teuthology.orchestra.run.vm07.stdout:(5/152): ceph-base-20.2.0-9.ge2ce8426bc5.el9.cl 8.4 MB/s | 5.9 MB 00:00 2026-04-01T15:38:02.740 INFO:teuthology.orchestra.run.vm07.stdout:(6/152): ceph-mgr-20.2.0-9.ge2ce8426bc5.el9.cly 8.0 MB/s | 961 kB 00:00 2026-04-01T15:38:02.993 INFO:teuthology.orchestra.run.vm07.stdout:(7/152): ceph-mon-20.2.0-9.ge2ce8426bc5.el9.cly 20 MB/s | 5.0 MB 00:00 2026-04-01T15:38:03.054 INFO:teuthology.orchestra.run.vm07.stdout:(8/152): ceph-common-20.2.0-9.ge2ce8426bc5.el9. 23 MB/s | 24 MB 00:01 2026-04-01T15:38:03.104 INFO:teuthology.orchestra.run.vm07.stdout:(9/152): ceph-selinux-20.2.0-9.ge2ce8426bc5.el9 506 kB/s | 25 kB 00:00 2026-04-01T15:38:03.333 INFO:teuthology.orchestra.run.vm07.stdout:(10/152): ceph-osd-20.2.0-9.ge2ce8426bc5.el9.cl 29 MB/s | 17 MB 00:00 2026-04-01T15:38:03.489 INFO:teuthology.orchestra.run.vm07.stdout:(11/152): libcephfs-daemon-20.2.0-9.ge2ce8426bc 228 kB/s | 35 kB 00:00 2026-04-01T15:38:03.623 INFO:teuthology.orchestra.run.vm07.stdout:(12/152): libcephfs-devel-20.2.0-9.ge2ce8426bc5 256 kB/s | 34 kB 00:00 2026-04-01T15:38:03.661 INFO:teuthology.orchestra.run.vm07.stdout:(13/152): libcephfs-proxy2-20.2.0-9.ge2ce8426bc 646 kB/s | 24 kB 00:00 2026-04-01T15:38:03.768 INFO:teuthology.orchestra.run.vm07.stdout:(14/152): libcephfs2-20.2.0-9.ge2ce8426bc5.el9. 8.0 MB/s | 868 kB 00:00 2026-04-01T15:38:03.817 INFO:teuthology.orchestra.run.vm07.stdout:(15/152): libcephsqlite-20.2.0-9.ge2ce8426bc5.e 3.3 MB/s | 164 kB 00:00 2026-04-01T15:38:03.858 INFO:teuthology.orchestra.run.vm07.stdout:(16/152): librados-devel-20.2.0-9.ge2ce8426bc5. 3.1 MB/s | 126 kB 00:00 2026-04-01T15:38:03.934 INFO:teuthology.orchestra.run.vm07.stdout:(17/152): ceph-radosgw-20.2.0-9.ge2ce8426bc5.el 25 MB/s | 24 MB 00:00 2026-04-01T15:38:03.937 INFO:teuthology.orchestra.run.vm07.stdout:(18/152): libradosstriper1-20.2.0-9.ge2ce8426bc 3.1 MB/s | 250 kB 00:00 2026-04-01T15:38:03.999 INFO:teuthology.orchestra.run.vm07.stdout:(19/152): python3-ceph-argparse-20.2.0-9.ge2ce8 725 kB/s | 45 kB 00:00 2026-04-01T15:38:04.080 INFO:teuthology.orchestra.run.vm07.stdout:(20/152): python3-ceph-common-20.2.0-9.ge2ce842 2.0 MB/s | 163 kB 00:00 2026-04-01T15:38:04.186 INFO:teuthology.orchestra.run.vm07.stdout:(21/152): python3-cephfs-20.2.0-9.ge2ce8426bc5. 1.5 MB/s | 163 kB 00:00 2026-04-01T15:38:04.240 INFO:teuthology.orchestra.run.vm07.stdout:(22/152): librgw2-20.2.0-9.ge2ce8426bc5.el9.cly 21 MB/s | 6.4 MB 00:00 2026-04-01T15:38:04.310 INFO:teuthology.orchestra.run.vm07.stdout:(23/152): python3-rbd-20.2.0-9.ge2ce8426bc5.el9 4.3 MB/s | 304 kB 00:00 2026-04-01T15:38:04.358 INFO:teuthology.orchestra.run.vm07.stdout:(24/152): python3-rgw-20.2.0-9.ge2ce8426bc5.el9 2.0 MB/s | 99 kB 00:00 2026-04-01T15:38:04.519 INFO:teuthology.orchestra.run.vm07.stdout:(25/152): python3-rados-20.2.0-9.ge2ce8426bc5.e 950 kB/s | 316 kB 00:00 2026-04-01T15:38:04.602 INFO:teuthology.orchestra.run.vm07.stdout:(26/152): rbd-fuse-20.2.0-9.ge2ce8426bc5.el9.cl 375 kB/s | 91 kB 00:00 2026-04-01T15:38:04.643 INFO:teuthology.orchestra.run.vm07.stdout:(27/152): rbd-nbd-20.2.0-9.ge2ce8426bc5.el9.cly 4.4 MB/s | 180 kB 00:00 2026-04-01T15:38:04.690 INFO:teuthology.orchestra.run.vm07.stdout:(28/152): ceph-grafana-dashboards-20.2.0-9.ge2c 918 kB/s | 43 kB 00:00 2026-04-01T15:38:04.742 INFO:teuthology.orchestra.run.vm07.stdout:(29/152): ceph-mgr-cephadm-20.2.0-9.ge2ce8426bc 3.3 MB/s | 173 kB 00:00 2026-04-01T15:38:04.772 INFO:teuthology.orchestra.run.vm07.stdout:(30/152): rbd-mirror-20.2.0-9.ge2ce8426bc5.el9. 12 MB/s | 2.9 MB 00:00 2026-04-01T15:38:05.060 INFO:teuthology.orchestra.run.vm07.stdout:(31/152): ceph-test-20.2.0-9.ge2ce8426bc5.el9.c 43 MB/s | 85 MB 00:01 2026-04-01T15:38:05.095 INFO:teuthology.orchestra.run.vm07.stdout:(32/152): ceph-mgr-k8sevents-20.2.0-9.ge2ce8426 638 kB/s | 22 kB 00:00 2026-04-01T15:38:05.152 INFO:teuthology.orchestra.run.vm07.stdout:(33/152): ceph-mgr-modules-core-20.2.0-9.ge2ce8 5.0 MB/s | 289 kB 00:00 2026-04-01T15:38:05.185 INFO:teuthology.orchestra.run.vm07.stdout:(34/152): ceph-mgr-rook-20.2.0-9.ge2ce8426bc5.e 1.5 MB/s | 50 kB 00:00 2026-04-01T15:38:05.216 INFO:teuthology.orchestra.run.vm07.stdout:(35/152): ceph-prometheus-alerts-20.2.0-9.ge2ce 554 kB/s | 17 kB 00:00 2026-04-01T15:38:05.269 INFO:teuthology.orchestra.run.vm07.stdout:(36/152): ceph-volume-20.2.0-9.ge2ce8426bc5.el9 5.5 MB/s | 297 kB 00:00 2026-04-01T15:38:05.320 INFO:teuthology.orchestra.run.vm07.stdout:(37/152): ceph-mgr-dashboard-20.2.0-9.ge2ce8426 26 MB/s | 15 MB 00:00 2026-04-01T15:38:05.350 INFO:teuthology.orchestra.run.vm07.stdout:(38/152): ceph-mgr-diskprediction-local-20.2.0- 13 MB/s | 7.4 MB 00:00 2026-04-01T15:38:05.353 INFO:teuthology.orchestra.run.vm07.stdout:(39/152): abseil-cpp-20211102.0-4.el9.x86_64.rp 16 MB/s | 551 kB 00:00 2026-04-01T15:38:05.355 INFO:teuthology.orchestra.run.vm07.stdout:(40/152): grpc-data-1.46.7-10.el9.noarch.rpm 8.0 MB/s | 19 kB 00:00 2026-04-01T15:38:05.365 INFO:teuthology.orchestra.run.vm07.stdout:(41/152): gperftools-libs-2.9.1-3.el9.x86_64.rp 19 MB/s | 308 kB 00:00 2026-04-01T15:38:05.369 INFO:teuthology.orchestra.run.vm07.stdout:(42/152): libarrow-doc-9.0.0-15.el9.noarch.rpm 7.2 MB/s | 25 kB 00:00 2026-04-01T15:38:05.373 INFO:teuthology.orchestra.run.vm07.stdout:(43/152): liboath-2.6.12-1.el9.x86_64.rpm 11 MB/s | 49 kB 00:00 2026-04-01T15:38:05.379 INFO:teuthology.orchestra.run.vm07.stdout:(44/152): libunwind-1.6.2-1.el9.x86_64.rpm 12 MB/s | 67 kB 00:00 2026-04-01T15:38:05.388 INFO:teuthology.orchestra.run.vm07.stdout:(45/152): luarocks-3.9.2-5.el9.noarch.rpm 17 MB/s | 151 kB 00:00 2026-04-01T15:38:05.423 INFO:teuthology.orchestra.run.vm07.stdout:(46/152): libarrow-9.0.0-15.el9.x86_64.rpm 66 MB/s | 4.4 MB 00:00 2026-04-01T15:38:05.433 INFO:teuthology.orchestra.run.vm07.stdout:(47/152): python3-asyncssh-2.13.2-5.el9.noarch. 54 MB/s | 548 kB 00:00 2026-04-01T15:38:05.437 INFO:teuthology.orchestra.run.vm07.stdout:(48/152): parquet-libs-9.0.0-15.el9.x86_64.rpm 17 MB/s | 838 kB 00:00 2026-04-01T15:38:05.437 INFO:teuthology.orchestra.run.vm07.stdout:(49/152): python3-autocommand-2.2.2-8.el9.noarc 7.5 MB/s | 29 kB 00:00 2026-04-01T15:38:05.440 INFO:teuthology.orchestra.run.vm07.stdout:(50/152): python3-backports-tarfile-1.2.0-1.el9 21 MB/s | 60 kB 00:00 2026-04-01T15:38:05.440 INFO:teuthology.orchestra.run.vm07.stdout:(51/152): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 14 MB/s | 43 kB 00:00 2026-04-01T15:38:05.443 INFO:teuthology.orchestra.run.vm07.stdout:(52/152): python3-cachetools-4.2.4-1.el9.noarch 12 MB/s | 32 kB 00:00 2026-04-01T15:38:05.443 INFO:teuthology.orchestra.run.vm07.stdout:(53/152): python3-certifi-2023.05.07-4.el9.noar 5.4 MB/s | 14 kB 00:00 2026-04-01T15:38:05.447 INFO:teuthology.orchestra.run.vm07.stdout:(54/152): python3-cheroot-10.0.1-5.el9.noarch.r 38 MB/s | 173 kB 00:00 2026-04-01T15:38:05.453 INFO:teuthology.orchestra.run.vm07.stdout:(55/152): python3-google-auth-2.45.0-1.el9.noar 42 MB/s | 254 kB 00:00 2026-04-01T15:38:05.455 INFO:teuthology.orchestra.run.vm07.stdout:(56/152): python3-cherrypy-18.10.0-5.el9.noarch 24 MB/s | 290 kB 00:00 2026-04-01T15:38:05.464 INFO:teuthology.orchestra.run.vm07.stdout:(57/152): python3-grpcio-tools-1.46.7-10.el9.x8 16 MB/s | 144 kB 00:00 2026-04-01T15:38:05.474 INFO:teuthology.orchestra.run.vm07.stdout:(58/152): python3-influxdb-5.3.1-1.el9.noarch.r 14 MB/s | 139 kB 00:00 2026-04-01T15:38:05.482 INFO:teuthology.orchestra.run.vm07.stdout:(59/152): python3-grpcio-1.46.7-10.el9.x86_64.r 71 MB/s | 2.0 MB 00:00 2026-04-01T15:38:05.483 INFO:teuthology.orchestra.run.vm07.stdout:(60/152): python3-isodate-0.6.1-3.el9.noarch.rp 6.4 MB/s | 56 kB 00:00 2026-04-01T15:38:05.485 INFO:teuthology.orchestra.run.vm07.stdout:(61/152): python3-jaraco-8.2.1-3.el9.noarch.rpm 3.9 MB/s | 11 kB 00:00 2026-04-01T15:38:05.486 INFO:teuthology.orchestra.run.vm07.stdout:(62/152): python3-jaraco-classes-3.2.1-5.el9.no 6.4 MB/s | 18 kB 00:00 2026-04-01T15:38:05.488 INFO:teuthology.orchestra.run.vm07.stdout:(63/152): python3-jaraco-collections-3.0.0-8.el 8.8 MB/s | 23 kB 00:00 2026-04-01T15:38:05.489 INFO:teuthology.orchestra.run.vm07.stdout:(64/152): python3-jaraco-context-6.0.1-3.el9.no 7.8 MB/s | 20 kB 00:00 2026-04-01T15:38:05.490 INFO:teuthology.orchestra.run.vm07.stdout:(65/152): python3-jaraco-functools-3.5.0-2.el9. 8.0 MB/s | 19 kB 00:00 2026-04-01T15:38:05.491 INFO:teuthology.orchestra.run.vm07.stdout:(66/152): python3-jaraco-text-4.0.0-2.el9.noarc 10 MB/s | 26 kB 00:00 2026-04-01T15:38:05.499 INFO:teuthology.orchestra.run.vm07.stdout:(67/152): python3-more-itertools-8.12.0-2.el9.n 11 MB/s | 79 kB 00:00 2026-04-01T15:38:05.506 INFO:teuthology.orchestra.run.vm07.stdout:(68/152): python3-kubernetes-26.1.0-3.el9.noarc 68 MB/s | 1.0 MB 00:00 2026-04-01T15:38:05.507 INFO:teuthology.orchestra.run.vm07.stdout:(69/152): python3-msgpack-1.0.3-2.el9.x86_64.rp 10 MB/s | 86 kB 00:00 2026-04-01T15:38:05.509 INFO:teuthology.orchestra.run.vm07.stdout:(70/152): python3-natsort-7.1.1-5.el9.noarch.rp 18 MB/s | 58 kB 00:00 2026-04-01T15:38:05.510 INFO:teuthology.orchestra.run.vm07.stdout:(71/152): python3-portend-3.1.0-2.el9.noarch.rp 6.0 MB/s | 16 kB 00:00 2026-04-01T15:38:05.513 INFO:teuthology.orchestra.run.vm07.stdout:(72/152): python3-pyOpenSSL-21.0.0-1.el9.noarch 23 MB/s | 90 kB 00:00 2026-04-01T15:38:05.514 INFO:teuthology.orchestra.run.vm07.stdout:(73/152): python3-repoze-lru-0.7-16.el9.noarch. 7.5 MB/s | 31 kB 00:00 2026-04-01T15:38:05.518 INFO:teuthology.orchestra.run.vm07.stdout:(74/152): python3-routes-2.5.1-5.el9.noarch.rpm 39 MB/s | 188 kB 00:00 2026-04-01T15:38:05.519 INFO:teuthology.orchestra.run.vm07.stdout:(75/152): python3-rsa-4.9-2.el9.noarch.rpm 11 MB/s | 59 kB 00:00 2026-04-01T15:38:05.522 INFO:teuthology.orchestra.run.vm07.stdout:(76/152): python3-saml-1.16.0-1.el9.noarch.rpm 32 MB/s | 125 kB 00:00 2026-04-01T15:38:05.523 INFO:teuthology.orchestra.run.vm07.stdout:(77/152): python3-tempora-5.0.0-2.el9.noarch.rp 9.4 MB/s | 36 kB 00:00 2026-04-01T15:38:05.525 INFO:teuthology.orchestra.run.vm07.stdout:(78/152): python3-typing-extensions-4.15.0-1.el 25 MB/s | 86 kB 00:00 2026-04-01T15:38:05.527 INFO:teuthology.orchestra.run.vm07.stdout:(79/152): python3-websocket-client-1.2.3-2.el9. 24 MB/s | 90 kB 00:00 2026-04-01T15:38:05.529 INFO:teuthology.orchestra.run.vm07.stdout:(80/152): python3-xmlsec-1.3.13-1.el9.x86_64.rp 15 MB/s | 48 kB 00:00 2026-04-01T15:38:05.530 INFO:teuthology.orchestra.run.vm07.stdout:(81/152): python3-xmltodict-0.12.0-15.el9.noarc 8.2 MB/s | 22 kB 00:00 2026-04-01T15:38:05.531 INFO:teuthology.orchestra.run.vm07.stdout:(82/152): python3-zc-lockfile-2.0-10.el9.noarch 8.4 MB/s | 20 kB 00:00 2026-04-01T15:38:05.535 INFO:teuthology.orchestra.run.vm07.stdout:(83/152): re2-20211101-20.el9.x86_64.rpm 37 MB/s | 191 kB 00:00 2026-04-01T15:38:05.537 INFO:teuthology.orchestra.run.vm07.stdout:(84/152): s3cmd-2.4.0-1.el9.noarch.rpm 36 MB/s | 206 kB 00:00 2026-04-01T15:38:05.557 INFO:teuthology.orchestra.run.vm07.stdout:(85/152): thrift-0.15.0-4.el9.x86_64.rpm 73 MB/s | 1.6 MB 00:00 2026-04-01T15:38:05.679 INFO:teuthology.orchestra.run.vm07.stdout:(86/152): bzip2-1.0.8-10.el9_5.x86_64.rpm 363 kB/s | 51 kB 00:00 2026-04-01T15:38:05.715 INFO:teuthology.orchestra.run.vm07.stdout:(87/152): c-ares-1.19.1-2.el9_4.x86_64.rpm 694 kB/s | 110 kB 00:00 2026-04-01T15:38:05.742 INFO:teuthology.orchestra.run.vm07.stdout:(88/152): fuse-2.9.9-17.el9.x86_64.rpm 2.8 MB/s | 78 kB 00:00 2026-04-01T15:38:05.757 INFO:teuthology.orchestra.run.vm07.stdout:(89/152): cryptsetup-2.7.2-4.el9.x86_64.rpm 3.9 MB/s | 310 kB 00:00 2026-04-01T15:38:05.769 INFO:teuthology.orchestra.run.vm07.stdout:(90/152): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.5 MB/s | 41 kB 00:00 2026-04-01T15:38:05.784 INFO:teuthology.orchestra.run.vm07.stdout:(91/152): libconfig-1.7.2-9.el9.x86_64.rpm 2.6 MB/s | 71 kB 00:00 2026-04-01T15:38:05.813 INFO:teuthology.orchestra.run.vm07.stdout:(92/152): libquadmath-11.5.0-11.el9.x86_64.rpm 6.4 MB/s | 184 kB 00:00 2026-04-01T15:38:05.840 INFO:teuthology.orchestra.run.vm07.stdout:(93/152): lmdb-libs-0.9.29-3.el9.x86_64.rpm 2.2 MB/s | 60 kB 00:00 2026-04-01T15:38:05.851 INFO:teuthology.orchestra.run.vm07.stdout:(94/152): libgfortran-11.5.0-11.el9.x86_64.rpm 9.5 MB/s | 794 kB 00:00 2026-04-01T15:38:05.866 INFO:teuthology.orchestra.run.vm07.stdout:(95/152): mailcap-2.1.49-5.el9.0.2.noarch.rpm 1.2 MB/s | 32 kB 00:00 2026-04-01T15:38:05.894 INFO:teuthology.orchestra.run.vm07.stdout:(96/152): pciutils-3.7.0-7.el9.x86_64.rpm 3.3 MB/s | 92 kB 00:00 2026-04-01T15:38:05.905 INFO:teuthology.orchestra.run.vm07.stdout:(97/152): nvme-cli-2.13-1.el9.x86_64.rpm 18 MB/s | 1.0 MB 00:00 2026-04-01T15:38:05.923 INFO:teuthology.orchestra.run.vm07.stdout:(98/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 8.1 MB/s | 241 kB 00:00 2026-04-01T15:38:05.947 INFO:teuthology.orchestra.run.vm07.stdout:(99/152): python3-cryptography-36.0.1-5.el9_6.x 28 MB/s | 1.2 MB 00:00 2026-04-01T15:38:05.951 INFO:teuthology.orchestra.run.vm07.stdout:(100/152): python3-ply-3.11-14.el9.0.1.noarch.r 3.6 MB/s | 103 kB 00:00 2026-04-01T15:38:05.976 INFO:teuthology.orchestra.run.vm07.stdout:(101/152): python3-pycparser-2.20-6.el9.noarch. 4.3 MB/s | 124 kB 00:00 2026-04-01T15:38:05.981 INFO:teuthology.orchestra.run.vm07.stdout:(102/152): python3-pyparsing-2.4.7-9.el9.0.1.no 4.9 MB/s | 150 kB 00:00 2026-04-01T15:38:06.041 INFO:teuthology.orchestra.run.vm07.stdout:(103/152): python3-requests-2.25.1-10.el9_6.noa 1.7 MB/s | 115 kB 00:00 2026-04-01T15:38:06.042 INFO:teuthology.orchestra.run.vm07.stdout:(104/152): python3-urllib3-1.26.5-6.el9_7.1.noa 3.0 MB/s | 191 kB 00:00 2026-04-01T15:38:06.075 INFO:teuthology.orchestra.run.vm07.stdout:(105/152): unzip-6.0-59.el9.x86_64.rpm 5.4 MB/s | 180 kB 00:00 2026-04-01T15:38:06.077 INFO:teuthology.orchestra.run.vm07.stdout:(106/152): smartmontools-7.2-9.el9.x86_64.rpm 15 MB/s | 551 kB 00:00 2026-04-01T15:38:06.106 INFO:teuthology.orchestra.run.vm07.stdout:(107/152): boost-program-options-1.75.0-13.el9_ 3.5 MB/s | 104 kB 00:00 2026-04-01T15:38:06.108 INFO:teuthology.orchestra.run.vm07.stdout:(108/152): zip-3.0-35.el9.x86_64.rpm 7.8 MB/s | 263 kB 00:00 2026-04-01T15:38:06.135 INFO:teuthology.orchestra.run.vm07.stdout:(109/152): flexiblas-3.0.4-8.el9.0.1.x86_64.rpm 1.0 MB/s | 30 kB 00:00 2026-04-01T15:38:06.162 INFO:teuthology.orchestra.run.vm07.stdout:(110/152): flexiblas-openblas-openmp-3.0.4-8.el 561 kB/s | 15 kB 00:00 2026-04-01T15:38:06.196 INFO:teuthology.orchestra.run.vm07.stdout:(111/152): libnbd-1.20.3-4.el9.x86_64.rpm 4.9 MB/s | 171 kB 00:00 2026-04-01T15:38:06.234 INFO:teuthology.orchestra.run.vm07.stdout:(112/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.2 MB/s | 159 kB 00:00 2026-04-01T15:38:06.253 INFO:teuthology.orchestra.run.vm07.stdout:(113/152): flexiblas-netlib-3.0.4-8.el9.0.1.x86 21 MB/s | 3.0 MB 00:00 2026-04-01T15:38:06.261 INFO:teuthology.orchestra.run.vm07.stdout:(114/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.6 MB/s | 44 kB 00:00 2026-04-01T15:38:06.286 INFO:teuthology.orchestra.run.vm07.stdout:(115/152): librdkafka-1.6.1-102.el9.x86_64.rpm 20 MB/s | 662 kB 00:00 2026-04-01T15:38:06.290 INFO:teuthology.orchestra.run.vm07.stdout:(116/152): libstoragemgmt-1.10.1-1.el9.x86_64.r 8.3 MB/s | 243 kB 00:00 2026-04-01T15:38:06.315 INFO:teuthology.orchestra.run.vm07.stdout:(117/152): libxslt-1.1.34-13.el9_6.x86_64.rpm 8.1 MB/s | 239 kB 00:00 2026-04-01T15:38:06.321 INFO:teuthology.orchestra.run.vm07.stdout:(118/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.8 MB/s | 282 kB 00:00 2026-04-01T15:38:06.345 INFO:teuthology.orchestra.run.vm07.stdout:(119/152): lua-5.4.4-4.el9.x86_64.rpm 6.2 MB/s | 187 kB 00:00 2026-04-01T15:38:06.348 INFO:teuthology.orchestra.run.vm07.stdout:(120/152): openblas-0.3.29-1.el9.x86_64.rpm 1.5 MB/s | 41 kB 00:00 2026-04-01T15:38:06.399 INFO:teuthology.orchestra.run.vm07.stdout:(121/152): perl-Benchmark-1.23-481.1.el9_6.noar 499 kB/s | 25 kB 00:00 2026-04-01T15:38:06.432 INFO:teuthology.orchestra.run.vm07.stdout:(122/152): perl-Test-Harness-3.42-461.el9.noarc 8.0 MB/s | 267 kB 00:00 2026-04-01T15:38:06.475 INFO:teuthology.orchestra.run.vm07.stdout:(123/152): openblas-openmp-0.3.29-1.el9.x86_64. 41 MB/s | 5.3 MB 00:00 2026-04-01T15:38:06.479 INFO:teuthology.orchestra.run.vm07.stdout:(124/152): protobuf-3.14.0-17.el9_7.x86_64.rpm 22 MB/s | 1.0 MB 00:00 2026-04-01T15:38:06.509 INFO:teuthology.orchestra.run.vm07.stdout:(125/152): python3-devel-3.9.23-2.el9.x86_64.rp 6.6 MB/s | 205 kB 00:00 2026-04-01T15:38:06.547 INFO:teuthology.orchestra.run.vm07.stdout:(126/152): python3-jinja2-2.11.3-8.el9_5.noarch 5.9 MB/s | 228 kB 00:00 2026-04-01T15:38:06.575 INFO:teuthology.orchestra.run.vm07.stdout:(127/152): python3-babel-2.9.1-2.el9.noarch.rpm 58 MB/s | 5.8 MB 00:00 2026-04-01T15:38:06.576 INFO:teuthology.orchestra.run.vm07.stdout:(128/152): python3-jmespath-1.0.1-1.el9_7.noarc 1.5 MB/s | 43 kB 00:00 2026-04-01T15:38:06.603 INFO:teuthology.orchestra.run.vm07.stdout:(129/152): python3-libstoragemgmt-1.10.1-1.el9. 5.9 MB/s | 166 kB 00:00 2026-04-01T15:38:06.616 INFO:teuthology.orchestra.run.vm07.stdout:(130/152): python3-lxml-4.6.5-3.el9.x86_64.rpm 30 MB/s | 1.2 MB 00:00 2026-04-01T15:38:06.630 INFO:teuthology.orchestra.run.vm07.stdout:(131/152): python3-markupsafe-1.1.1-12.el9.x86_ 1.2 MB/s | 32 kB 00:00 2026-04-01T15:38:06.663 INFO:teuthology.orchestra.run.vm07.stdout:(132/152): python3-numpy-f2py-1.23.5-2.el9_7.x8 11 MB/s | 368 kB 00:00 2026-04-01T15:38:06.691 INFO:teuthology.orchestra.run.vm07.stdout:(133/152): python3-packaging-20.9-5.el9.noarch. 2.5 MB/s | 69 kB 00:00 2026-04-01T15:38:06.727 INFO:teuthology.orchestra.run.vm07.stdout:(134/152): python3-numpy-1.23.5-2.el9_7.x86_64. 53 MB/s | 5.8 MB 00:00 2026-04-01T15:38:06.728 INFO:teuthology.orchestra.run.vm07.stdout:(135/152): python3-protobuf-3.14.0-17.el9_7.noa 6.1 MB/s | 237 kB 00:00 2026-04-01T15:38:06.754 INFO:teuthology.orchestra.run.vm07.stdout:(136/152): python3-pyasn1-0.4.8-7.el9_7.noarch. 4.7 MB/s | 132 kB 00:00 2026-04-01T15:38:06.756 INFO:teuthology.orchestra.run.vm07.stdout:(137/152): python3-pyasn1-modules-0.4.8-7.el9_7 7.3 MB/s | 210 kB 00:00 2026-04-01T15:38:06.781 INFO:teuthology.orchestra.run.vm07.stdout:(138/152): python3-requests-oauthlib-1.3.0-12.e 1.6 MB/s | 43 kB 00:00 2026-04-01T15:38:06.808 INFO:teuthology.orchestra.run.vm07.stdout:(139/152): python3-toml-0.10.2-6.el9.0.1.noarch 1.6 MB/s | 44 kB 00:00 2026-04-01T15:38:06.837 INFO:teuthology.orchestra.run.vm07.stdout:(140/152): qatlib-24.09.0-1.el9.x86_64.rpm 7.5 MB/s | 221 kB 00:00 2026-04-01T15:38:06.864 INFO:teuthology.orchestra.run.vm07.stdout:(141/152): qatlib-service-24.09.0-1.el9.x86_64. 1.3 MB/s | 36 kB 00:00 2026-04-01T15:38:06.891 INFO:teuthology.orchestra.run.vm07.stdout:(142/152): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.4 MB/s | 65 kB 00:00 2026-04-01T15:38:06.922 INFO:teuthology.orchestra.run.vm07.stdout:(143/152): socat-1.7.4.1-8.el9.x86_64.rpm 9.6 MB/s | 299 kB 00:00 2026-04-01T15:38:06.950 INFO:teuthology.orchestra.run.vm07.stdout:(144/152): xmlsec1-1.2.29-13.el9.x86_64.rpm 6.6 MB/s | 188 kB 00:00 2026-04-01T15:38:07.005 INFO:teuthology.orchestra.run.vm07.stdout:(145/152): python3-scipy-1.9.3-2.el9.x86_64.rpm 76 MB/s | 19 MB 00:00 2026-04-01T15:38:07.007 INFO:teuthology.orchestra.run.vm07.stdout:(146/152): xmlsec1-openssl-1.2.29-13.el9.x86_64 1.5 MB/s | 89 kB 00:00 2026-04-01T15:38:07.032 INFO:teuthology.orchestra.run.vm07.stdout:(147/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.3 MB/s | 63 kB 00:00 2026-04-01T15:38:07.033 INFO:teuthology.orchestra.run.vm07.stdout:(148/152): lua-devel-5.4.4-4.el9.x86_64.rpm 818 kB/s | 21 kB 00:00 2026-04-01T15:38:07.068 INFO:teuthology.orchestra.run.vm07.stdout:(149/152): protobuf-compiler-3.14.0-17.el9_7.x8 24 MB/s | 862 kB 00:00 2026-04-01T15:38:07.408 INFO:teuthology.orchestra.run.vm07.stdout:(150/152): librbd1-20.2.0-9.ge2ce8426bc5.el9.cl 8.4 MB/s | 2.8 MB 00:00 2026-04-01T15:38:09.675 INFO:teuthology.orchestra.run.vm07.stdout:(151/152): cephadm-20.2.0-9.ge2ce8426bc5.el9.cl 228 kB/s | 1.0 MB 00:04 2026-04-01T15:38:09.836 INFO:teuthology.orchestra.run.vm07.stdout:(152/152): librados2-20.2.0-9.ge2ce8426bc5.el9. 1.3 MB/s | 3.5 MB 00:02 2026-04-01T15:38:09.840 INFO:teuthology.orchestra.run.vm07.stdout:-------------------------------------------------------------------------------- 2026-04-01T15:38:09.840 INFO:teuthology.orchestra.run.vm07.stdout:Total 31 MB/s | 274 MB 00:08 2026-04-01T15:38:10.489 INFO:teuthology.orchestra.run.vm07.stdout:Running transaction check 2026-04-01T15:38:10.542 INFO:teuthology.orchestra.run.vm07.stdout:Transaction check succeeded. 2026-04-01T15:38:10.542 INFO:teuthology.orchestra.run.vm07.stdout:Running transaction test 2026-04-01T15:38:11.553 INFO:teuthology.orchestra.run.vm07.stdout:Transaction test succeeded. 2026-04-01T15:38:11.553 INFO:teuthology.orchestra.run.vm07.stdout:Running transaction 2026-04-01T15:38:12.628 INFO:teuthology.orchestra.run.vm07.stdout: Preparing : 1/1 2026-04-01T15:38:12.636 INFO:teuthology.orchestra.run.vm07.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/154 2026-04-01T15:38:12.649 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/154 2026-04-01T15:38:12.828 INFO:teuthology.orchestra.run.vm07.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 3/154 2026-04-01T15:38:12.830 INFO:teuthology.orchestra.run.vm07.stdout: Upgrading : librados2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 4/154 2026-04-01T15:38:12.885 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: librados2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 4/154 2026-04-01T15:38:12.886 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libcephfs2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 5/154 2026-04-01T15:38:12.900 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libcephfs2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 5/154 2026-04-01T15:38:12.903 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libcephfs-daemon-2:20.2.0-9.ge2ce8426bc5.el9.cly 6/154 2026-04-01T15:38:12.905 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libcephfs-proxy2-2:20.2.0-9.ge2ce8426bc5.el9.cly 7/154 2026-04-01T15:38:12.935 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-9.ge2ce8426bc5.el9.cly 7/154 2026-04-01T15:38:12.942 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-rados-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 8/154 2026-04-01T15:38:12.952 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libxslt-1.1.34-13.el9_6.x86_64 9/154 2026-04-01T15:38:12.955 INFO:teuthology.orchestra.run.vm07.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 10/154 2026-04-01T15:38:12.960 INFO:teuthology.orchestra.run.vm07.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 11/154 2026-04-01T15:38:12.964 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 12/154 2026-04-01T15:38:12.967 INFO:teuthology.orchestra.run.vm07.stdout: Installing : lmdb-libs-0.9.29-3.el9.x86_64 13/154 2026-04-01T15:38:12.972 INFO:teuthology.orchestra.run.vm07.stdout: Installing : liboath-2.6.12-1.el9.x86_64 14/154 2026-04-01T15:38:13.116 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 15/154 2026-04-01T15:38:13.118 INFO:teuthology.orchestra.run.vm07.stdout: Upgrading : librbd1-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 16/154 2026-04-01T15:38:13.135 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: librbd1-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 16/154 2026-04-01T15:38:13.172 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 17/154 2026-04-01T15:38:13.181 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 18/154 2026-04-01T15:38:13.191 INFO:teuthology.orchestra.run.vm07.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 19/154 2026-04-01T15:38:13.193 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libcephsqlite-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 20/154 2026-04-01T15:38:13.219 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libcephsqlite-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 20/154 2026-04-01T15:38:13.220 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libradosstriper1-2:20.2.0-9.ge2ce8426bc5.el9.cly 21/154 2026-04-01T15:38:13.258 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libradosstriper1-2:20.2.0-9.ge2ce8426bc5.el9.cly 21/154 2026-04-01T15:38:13.269 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-pyasn1-0.4.8-7.el9_7.noarch 22/154 2026-04-01T15:38:13.276 INFO:teuthology.orchestra.run.vm07.stdout: Installing : protobuf-3.14.0-17.el9_7.x86_64 23/154 2026-04-01T15:38:13.280 INFO:teuthology.orchestra.run.vm07.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/154 2026-04-01T15:38:13.285 INFO:teuthology.orchestra.run.vm07.stdout: Installing : flexiblas-3.0.4-8.el9.0.1.x86_64 25/154 2026-04-01T15:38:13.316 INFO:teuthology.orchestra.run.vm07.stdout: Installing : unzip-6.0-59.el9.x86_64 26/154 2026-04-01T15:38:13.341 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-urllib3-1.26.5-6.el9_7.1.noarch 27/154 2026-04-01T15:38:13.346 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-requests-2.25.1-10.el9_6.noarch 28/154 2026-04-01T15:38:13.354 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libquadmath-11.5.0-11.el9.x86_64 29/154 2026-04-01T15:38:13.357 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libgfortran-11.5.0-11.el9.x86_64 30/154 2026-04-01T15:38:13.361 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 31/154 2026-04-01T15:38:13.403 INFO:teuthology.orchestra.run.vm07.stdout: Installing : re2-1:20211101-20.el9.x86_64 32/154 2026-04-01T15:38:13.436 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 33/154 2026-04-01T15:38:13.444 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-ceph-common-2:20.2.0-9.ge2ce8426bc5.el9. 34/154 2026-04-01T15:38:13.454 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-ceph-argparse-2:20.2.0-9.ge2ce8426bc5.el 35/154 2026-04-01T15:38:13.467 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cephfs-2:20.2.0-9.ge2ce8426bc5.el9.clyso 36/154 2026-04-01T15:38:13.475 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 37/154 2026-04-01T15:38:13.504 INFO:teuthology.orchestra.run.vm07.stdout: Installing : zip-3.0-35.el9.x86_64 38/154 2026-04-01T15:38:13.510 INFO:teuthology.orchestra.run.vm07.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/154 2026-04-01T15:38:13.518 INFO:teuthology.orchestra.run.vm07.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 40/154 2026-04-01T15:38:13.534 INFO:teuthology.orchestra.run.vm07.stdout: Installing : protobuf-compiler-3.14.0-17.el9_7.x86_64 41/154 2026-04-01T15:38:13.594 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/154 2026-04-01T15:38:13.598 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9_7.noarch 43/154 2026-04-01T15:38:13.605 INFO:teuthology.orchestra.run.vm07.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 44/154 2026-04-01T15:38:13.611 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 45/154 2026-04-01T15:38:13.632 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/154 2026-04-01T15:38:13.636 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-rbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 47/154 2026-04-01T15:38:13.646 INFO:teuthology.orchestra.run.vm07.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 48/154 2026-04-01T15:38:13.656 INFO:teuthology.orchestra.run.vm07.stdout: Installing : librados-devel-2:20.2.0-9.ge2ce8426bc5.el9.clyso 49/154 2026-04-01T15:38:13.666 INFO:teuthology.orchestra.run.vm07.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 50/154 2026-04-01T15:38:13.671 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-toml-0.10.2-6.el9.0.1.noarch 51/154 2026-04-01T15:38:13.680 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 52/154 2026-04-01T15:38:13.685 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 53/154 2026-04-01T15:38:13.720 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 54/154 2026-04-01T15:38:13.729 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-protobuf-3.14.0-17.el9_7.noarch 55/154 2026-04-01T15:38:13.738 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 56/154 2026-04-01T15:38:13.782 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jmespath-1.0.1-1.el9_7.noarch 57/154 2026-04-01T15:38:14.061 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-devel-3.9.23-2.el9.x86_64 58/154 2026-04-01T15:38:14.092 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 59/154 2026-04-01T15:38:14.096 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jinja2-2.11.3-8.el9_5.noarch 60/154 2026-04-01T15:38:14.101 INFO:teuthology.orchestra.run.vm07.stdout: Installing : perl-Benchmark-1.23-481.1.el9_6.noarch 61/154 2026-04-01T15:38:14.164 INFO:teuthology.orchestra.run.vm07.stdout: Installing : openblas-0.3.29-1.el9.x86_64 62/154 2026-04-01T15:38:14.167 INFO:teuthology.orchestra.run.vm07.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 63/154 2026-04-01T15:38:14.191 INFO:teuthology.orchestra.run.vm07.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.0.1.x86_64 64/154 2026-04-01T15:38:14.576 INFO:teuthology.orchestra.run.vm07.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.0.1.x86_64 65/154 2026-04-01T15:38:14.667 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-numpy-1:1.23.5-2.el9_7.x86_64 66/154 2026-04-01T15:38:15.423 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9_7.x86_64 67/154 2026-04-01T15:38:15.443 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 68/154 2026-04-01T15:38:15.456 INFO:teuthology.orchestra.run.vm07.stdout: Installing : boost-program-options-1.75.0-13.el9_7.x86_64 69/154 2026-04-01T15:38:15.458 INFO:teuthology.orchestra.run.vm07.stdout: Installing : smartmontools-1:7.2-9.el9.x86_64 70/154 2026-04-01T15:38:15.473 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: smartmontools-1:7.2-9.el9.x86_64 70/154 2026-04-01T15:38:15.473 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/smartd.service → /usr/lib/systemd/system/smartd.service. 2026-04-01T15:38:15.473 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:15.491 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-pyparsing-2.4.7-9.el9.0.1.noarch 71/154 2026-04-01T15:38:15.500 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-packaging-20.9-5.el9.noarch 72/154 2026-04-01T15:38:15.517 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-ply-3.11-14.el9.0.1.noarch 73/154 2026-04-01T15:38:15.538 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 74/154 2026-04-01T15:38:15.630 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 75/154 2026-04-01T15:38:15.643 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cryptography-36.0.1-5.el9_6.x86_64 76/154 2026-04-01T15:38:15.670 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 77/154 2026-04-01T15:38:15.677 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cheroot-10.0.1-5.el9.noarch 78/154 2026-04-01T15:38:15.683 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 79/154 2026-04-01T15:38:15.685 INFO:teuthology.orchestra.run.vm07.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 80/154 2026-04-01T15:38:15.713 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: qatlib-24.09.0-1.el9.x86_64 81/154 2026-04-01T15:38:15.717 INFO:teuthology.orchestra.run.vm07.stdout: Installing : qatlib-24.09.0-1.el9.x86_64 81/154 2026-04-01T15:38:15.718 INFO:teuthology.orchestra.run.vm07.stdout: Installing : qatlib-service-24.09.0-1.el9.x86_64 82/154 2026-04-01T15:38:15.736 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: qatlib-service-24.09.0-1.el9.x86_64 82/154 2026-04-01T15:38:15.866 INFO:teuthology.orchestra.run.vm07.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 83/154 2026-04-01T15:38:15.870 INFO:teuthology.orchestra.run.vm07.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 84/154 2026-04-01T15:38:16.189 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 84/154 2026-04-01T15:38:16.190 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-04-01T15:38:16.190 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:16.199 INFO:teuthology.orchestra.run.vm07.stdout: Installing : mailcap-2.1.49-5.el9.0.2.noarch 85/154 2026-04-01T15:38:16.204 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 86/154 2026-04-01T15:38:16.231 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 87/154 2026-04-01T15:38:16.231 INFO:teuthology.orchestra.run.vm07.stdout:Creating group 'libstoragemgmt' with GID 992. 2026-04-01T15:38:16.231 INFO:teuthology.orchestra.run.vm07.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 992 and GID 992. 2026-04-01T15:38:16.231 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:16.241 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 87/154 2026-04-01T15:38:16.275 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 87/154 2026-04-01T15:38:16.275 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-04-01T15:38:16.275 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:16.295 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 88/154 2026-04-01T15:38:16.323 INFO:teuthology.orchestra.run.vm07.stdout: Installing : fuse-2.9.9-17.el9.x86_64 89/154 2026-04-01T15:38:16.329 INFO:teuthology.orchestra.run.vm07.stdout: Installing : cryptsetup-2.7.2-4.el9.x86_64 90/154 2026-04-01T15:38:16.337 INFO:teuthology.orchestra.run.vm07.stdout: Installing : c-ares-1.19.1-2.el9_4.x86_64 91/154 2026-04-01T15:38:16.341 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 92/154 2026-04-01T15:38:16.359 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 93/154 2026-04-01T15:38:16.368 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 94/154 2026-04-01T15:38:16.430 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 95/154 2026-04-01T15:38:16.440 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 96/154 2026-04-01T15:38:16.455 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 97/154 2026-04-01T15:38:16.468 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 98/154 2026-04-01T15:38:16.477 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 99/154 2026-04-01T15:38:16.505 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 100/154 2026-04-01T15:38:16.521 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 101/154 2026-04-01T15:38:16.542 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 102/154 2026-04-01T15:38:16.549 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 103/154 2026-04-01T15:38:16.560 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 104/154 2026-04-01T15:38:16.608 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 105/154 2026-04-01T15:38:16.991 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 106/154 2026-04-01T15:38:17.008 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 107/154 2026-04-01T15:38:17.015 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 108/154 2026-04-01T15:38:17.023 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 109/154 2026-04-01T15:38:17.030 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 110/154 2026-04-01T15:38:17.036 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 111/154 2026-04-01T15:38:17.071 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 112/154 2026-04-01T15:38:17.078 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-cherrypy-18.10.0-5.el9.noarch 113/154 2026-04-01T15:38:17.086 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 114/154 2026-04-01T15:38:17.090 INFO:teuthology.orchestra.run.vm07.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 115/154 2026-04-01T15:38:17.096 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 116/154 2026-04-01T15:38:17.415 INFO:teuthology.orchestra.run.vm07.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 117/154 2026-04-01T15:38:17.421 INFO:teuthology.orchestra.run.vm07.stdout: Installing : librgw2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 118/154 2026-04-01T15:38:17.441 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: librgw2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 118/154 2026-04-01T15:38:17.443 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-rgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 119/154 2026-04-01T15:38:18.719 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 120/154 2026-04-01T15:38:18.725 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 120/154 2026-04-01T15:38:19.066 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 120/154 2026-04-01T15:38:19.073 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-base-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 121/154 2026-04-01T15:38:19.116 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-base-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 121/154 2026-04-01T15:38:19.116 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-04-01T15:38:19.116 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-04-01T15:38:19.116 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:19.121 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-selinux-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 122/154 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-selinux-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 122/154 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /sys 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /proc 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /mnt 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /var/tmp 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /home 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /root 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout:skipping the directory /tmp 2026-04-01T15:38:25.542 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:25.630 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-volume-2:20.2.0-9.ge2ce8426bc5.el9.clyso.no 123/154 2026-04-01T15:38:25.646 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-volume-2:20.2.0-9.ge2ce8426bc5.el9.clyso.no 123/154 2026-04-01T15:38:25.646 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:25.646 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-04-01T15:38:25.646 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:26.453 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-osd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 124/154 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-osd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 124/154 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-04-01T15:38:26.480 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:26.591 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mds-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 125/154 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mds-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 125/154 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-04-01T15:38:26.612 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:26.951 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mon-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 126/154 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mon-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 126/154 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-04-01T15:38:26.971 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:27.045 INFO:teuthology.orchestra.run.vm07.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 127/154 2026-04-01T15:38:27.150 INFO:teuthology.orchestra.run.vm07.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 128/154 2026-04-01T15:38:27.167 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 129/154 2026-04-01T15:38:27.171 INFO:teuthology.orchestra.run.vm07.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 130/154 2026-04-01T15:38:27.237 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: cephadm-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 131/154 2026-04-01T15:38:27.241 INFO:teuthology.orchestra.run.vm07.stdout: Installing : cephadm-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 131/154 2026-04-01T15:38:27.250 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-9.ge2ce8426bc5.e 132/154 2026-04-01T15:38:27.281 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-9.ge2ce8426bc5. 133/154 2026-04-01T15:38:27.284 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-9.ge2ce8426bc5.el9.cly 134/154 2026-04-01T15:38:29.149 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-9.ge2ce8426bc5.el9.cly 134/154 2026-04-01T15:38:29.163 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-9.ge2ce8426bc5.el9.c 135/154 2026-04-01T15:38:29.694 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-9.ge2ce8426bc5.el9.c 135/154 2026-04-01T15:38:29.697 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-9.ge2ce84 136/154 2026-04-01T15:38:29.712 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-9.ge2ce84 136/154 2026-04-01T15:38:29.713 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-k8sevents-2:20.2.0-9.ge2ce8426bc5.el9.c 137/154 2026-04-01T15:38:29.778 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-9.ge2ce8426bc5.el9.c 137/154 2026-04-01T15:38:29.826 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-9.ge2ce8426bc5.el 138/154 2026-04-01T15:38:29.829 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 139/154 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 139/154 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-04-01T15:38:29.851 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:29.866 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-mgr-rook-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 140/154 2026-04-01T15:38:29.876 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 140/154 2026-04-01T15:38:31.093 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 141/154 2026-04-01T15:38:31.097 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-radosgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 142/154 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 142/154 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-04-01T15:38:31.120 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:31.133 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-9.ge2ce8426 143/154 2026-04-01T15:38:31.156 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-9.ge2ce8426 143/154 2026-04-01T15:38:31.156 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:31.156 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-04-01T15:38:31.156 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:31.305 INFO:teuthology.orchestra.run.vm07.stdout: Installing : rbd-mirror-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 144/154 2026-04-01T15:38:31.325 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: rbd-mirror-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 144/154 2026-04-01T15:38:31.326 INFO:teuthology.orchestra.run.vm07.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-04-01T15:38:31.326 INFO:teuthology.orchestra.run.vm07.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-04-01T15:38:31.326 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-04-01T15:38:31.326 INFO:teuthology.orchestra.run.vm07.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-04-01T15:38:31.326 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:35.829 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-test-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 145/154 2026-04-01T15:38:35.864 INFO:teuthology.orchestra.run.vm07.stdout: Installing : ceph-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 146/154 2026-04-01T15:38:35.872 INFO:teuthology.orchestra.run.vm07.stdout: Installing : perl-Test-Harness-1:3.42-461.el9.noarch 147/154 2026-04-01T15:38:35.878 INFO:teuthology.orchestra.run.vm07.stdout: Installing : libcephfs-devel-2:20.2.0-9.ge2ce8426bc5.el9.clys 148/154 2026-04-01T15:38:35.890 INFO:teuthology.orchestra.run.vm07.stdout: Installing : rbd-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 149/154 2026-04-01T15:38:35.898 INFO:teuthology.orchestra.run.vm07.stdout: Installing : rbd-nbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 150/154 2026-04-01T15:38:35.917 INFO:teuthology.orchestra.run.vm07.stdout: Installing : bzip2-1.0.8-10.el9_5.x86_64 151/154 2026-04-01T15:38:35.921 INFO:teuthology.orchestra.run.vm07.stdout: Installing : s3cmd-2.4.0-1.el9.noarch 152/154 2026-04-01T15:38:35.921 INFO:teuthology.orchestra.run.vm07.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/154 2026-04-01T15:38:35.942 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/154 2026-04-01T15:38:35.942 INFO:teuthology.orchestra.run.vm07.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 154/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 154/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 1/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-base-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 2/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 3/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 4/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-9.ge2ce8426 5/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mds-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 6/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 7/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mon-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 8/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-osd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 9/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-radosgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 10/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-selinux-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x 11/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-test-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 12/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libcephfs-daemon-2:20.2.0-9.ge2ce8426bc5.el9.cly 13/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libcephfs-devel-2:20.2.0-9.ge2ce8426bc5.el9.clys 14/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libcephfs-proxy2-2:20.2.0-9.ge2ce8426bc5.el9.cly 15/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libcephfs2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 16/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libcephsqlite-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 17/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librados-devel-2:20.2.0-9.ge2ce8426bc5.el9.clyso 18/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libradosstriper1-2:20.2.0-9.ge2ce8426bc5.el9.cly 19/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librgw2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 20/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-ceph-argparse-2:20.2.0-9.ge2ce8426bc5.el 21/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-ceph-common-2:20.2.0-9.ge2ce8426bc5.el9. 22/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cephfs-2:20.2.0-9.ge2ce8426bc5.el9.clyso 23/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-rados-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 24/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-rbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 25/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-rgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x8 26/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : rbd-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_6 27/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : rbd-mirror-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86 28/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : rbd-nbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 29/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-9.ge2ce8426bc5. 30/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-9.ge2ce8426bc5.el9.cly 31/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-9.ge2ce8426bc5.el9.c 32/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-9.ge2ce84 33/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-9.ge2ce8426bc5.el9.c 34/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-9.ge2ce8426bc5.el 35/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-mgr-rook-2:20.2.0-9.ge2ce8426bc5.el9.clyso. 36/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-9.ge2ce8426bc5.e 37/154 2026-04-01T15:38:37.645 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ceph-volume-2:20.2.0-9.ge2ce8426bc5.el9.clyso.no 38/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : cephadm-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 39/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 40/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 41/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 42/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 43/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 44/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 45/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 46/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 47/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 48/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 49/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 50/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 51/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 52/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 53/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 54/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cheroot-10.0.1-5.el9.noarch 55/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cherrypy-18.10.0-5.el9.noarch 56/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 57/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 58/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 60/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 61/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 62/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 63/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 64/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 65/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 66/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 67/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 68/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 69/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 70/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 71/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 72/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 73/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 74/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 75/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 76/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 77/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 78/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 79/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 80/154 2026-04-01T15:38:37.648 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 81/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 82/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 83/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : re2-1:20211101-20.el9.x86_64 84/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : s3cmd-2.4.0-1.el9.noarch 85/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 86/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : bzip2-1.0.8-10.el9_5.x86_64 87/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : c-ares-1.19.1-2.el9_4.x86_64 88/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : cryptsetup-2.7.2-4.el9.x86_64 89/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : fuse-2.9.9-17.el9.x86_64 90/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 91/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 92/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libgfortran-11.5.0-11.el9.x86_64 93/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libquadmath-11.5.0-11.el9.x86_64 94/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : lmdb-libs-0.9.29-3.el9.x86_64 95/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : mailcap-2.1.49-5.el9.0.2.noarch 96/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 97/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 98/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 99/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-cryptography-36.0.1-5.el9_6.x86_64 100/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-ply-3.11-14.el9.0.1.noarch 101/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 102/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.0.1.noarch 103/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-requests-2.25.1-10.el9_6.noarch 104/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-urllib3-1.26.5-6.el9_7.1.noarch 105/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : smartmontools-1:7.2-9.el9.x86_64 106/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : unzip-6.0-59.el9.x86_64 107/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : zip-3.0-35.el9.x86_64 108/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : boost-program-options-1.75.0-13.el9_7.x86_64 109/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : flexiblas-3.0.4-8.el9.0.1.x86_64 110/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.0.1.x86_64 111/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.0.1.x86_64 112/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 113/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 114/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 115/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 116/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 117/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : libxslt-1.1.34-13.el9_6.x86_64 118/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 119/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : lua-5.4.4-4.el9.x86_64 120/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 121/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 122/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : perl-Benchmark-1.23-481.1.el9_6.noarch 123/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : perl-Test-Harness-1:3.42-461.el9.noarch 124/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : protobuf-3.14.0-17.el9_7.x86_64 125/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 126/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-devel-3.9.23-2.el9.x86_64 127/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jinja2-2.11.3-8.el9_5.noarch 128/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-jmespath-1.0.1-1.el9_7.noarch 129/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 130/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 131/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 132/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-numpy-1:1.23.5-2.el9_7.x86_64 133/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9_7.x86_64 134/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 135/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-protobuf-3.14.0-17.el9_7.noarch 136/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-pyasn1-0.4.8-7.el9_7.noarch 137/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9_7.noarch 138/154 2026-04-01T15:38:37.649 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 139/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 140/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : python3-toml-0.10.2-6.el9.0.1.noarch 141/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : qatlib-24.09.0-1.el9.x86_64 142/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : qatlib-service-24.09.0-1.el9.x86_64 143/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 144/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 145/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 146/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 147/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 148/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 149/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : protobuf-compiler-3.14.0-17.el9_7.x86_64 150/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librados2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_ 151/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/154 2026-04-01T15:38:37.650 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librbd1-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 153/154 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/154 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout:Upgraded: 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: librados2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: librbd1-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout:Installed: 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: boost-program-options-1.75.0-13.el9_7.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: bzip2-1.0.8-10.el9_5.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: c-ares-1.19.1-2.el9_4.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-base-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-grafana-dashboards-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-immutable-object-cache-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mds-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-cephadm-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-dashboard-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-diskprediction-local-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-k8sevents-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-modules-core-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mgr-rook-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-mon-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-osd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-prometheus-alerts-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-radosgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-selinux-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-test-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ceph-volume-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: cephadm-2:20.2.0-9.ge2ce8426bc5.el9.clyso.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: cryptsetup-2.7.2-4.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas-3.0.4-8.el9.0.1.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas-netlib-3.0.4-8.el9.0.1.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.0.1.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: fuse-2.9.9-17.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: grpc-data-1.46.7-10.el9.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: libarrow-9.0.0-15.el9.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-daemon-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.740 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-devel-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs-proxy2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libcephfs2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libcephsqlite-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libconfig-1.7.2-9.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libgfortran-11.5.0-11.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libnbd-1.20.3-4.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: liboath-2.6.12-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libquadmath-11.5.0-11.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: librados-devel-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libradosstriper1-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: librgw2-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libunwind-1.6.2-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: libxslt-1.1.34-13.el9_6.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: lmdb-libs-0.9.29-3.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: lua-5.4.4-4.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: luarocks-3.9.2-5.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: mailcap-2.1.49-5.el9.0.2.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: nvme-cli-2.13-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: openblas-0.3.29-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: pciutils-3.7.0-7.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: perl-Benchmark-1.23-481.1.el9_6.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: perl-Test-Harness-1:3.42-461.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: protobuf-3.14.0-17.el9_7.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: protobuf-compiler-3.14.0-17.el9_7.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-babel-2.9.1-2.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-ceph-argparse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-ceph-common-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cephfs-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cheroot-10.0.1-5.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cherrypy-18.10.0-5.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-cryptography-36.0.1-5.el9_6.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-devel-3.9.23-2.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jinja2-2.11.3-8.el9_5.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-jmespath-1.0.1-1.el9_7.noarch 2026-04-01T15:38:37.741 INFO:teuthology.orchestra.run.vm07.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-numpy-1:1.23.5-2.el9_7.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-numpy-f2py-1:1.23.5-2.el9_7.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-packaging-20.9-5.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-ply-3.11-14.el9.0.1.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-portend-3.1.0-2.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-protobuf-3.14.0-17.el9_7.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyasn1-0.4.8-7.el9_7.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyasn1-modules-0.4.8-7.el9_7.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-pycparser-2.20-6.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-pyparsing-2.4.7-9.el9.0.1.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-rados-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-rbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-requests-2.25.1-10.el9_6.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-rgw-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-routes-2.5.1-5.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-rsa-4.9-2.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-saml-1.16.0-1.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-toml-0.10.2-6.el9.0.1.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-urllib3-1.26.5-6.el9_7.1.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: qatlib-24.09.0-1.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: qatlib-service-24.09.0-1.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: rbd-fuse-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: rbd-mirror-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: rbd-nbd-2:20.2.0-9.ge2ce8426bc5.el9.clyso.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: re2-1:20211101-20.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: s3cmd-2.4.0-1.el9.noarch 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: smartmontools-1:7.2-9.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: socat-1.7.4.1-8.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: thrift-0.15.0-4.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: unzip-6.0-59.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: zip-3.0-35.el9.x86_64 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:37.742 INFO:teuthology.orchestra.run.vm07.stdout:Complete! 2026-04-01T15:38:37.856 DEBUG:teuthology.parallel:result is None 2026-04-01T15:38:37.856 INFO:teuthology.task.install:Skipping version verification because we have custom repos... 2026-04-01T15:38:37.856 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-04-01T15:38:37.856 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:37.856 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-04-01T15:38:37.883 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-04-01T15:38:37.883 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:37.883 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/usr/bin/daemon-helper 2026-04-01T15:38:37.947 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-04-01T15:38:38.013 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-04-01T15:38:38.013 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:38.013 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/usr/bin/adjust-ulimits 2026-04-01T15:38:38.083 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-04-01T15:38:38.147 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-04-01T15:38:38.147 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:38.147 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/usr/bin/stdin-killer 2026-04-01T15:38:38.212 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-04-01T15:38:38.277 INFO:teuthology.run_tasks:Running task ceph... 2026-04-01T15:38:38.322 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-04-01T15:38:38.322 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod 777 /var/log/ceph 2026-04-01T15:38:38.348 INFO:tasks.ceph:Disabling ceph logrotate... 2026-04-01T15:38:38.348 DEBUG:teuthology.orchestra.run.vm07:> sudo rm -f -- /etc/logrotate.d/ceph 2026-04-01T15:38:38.412 INFO:tasks.ceph:Creating extra log directories... 2026-04-01T15:38:38.412 DEBUG:teuthology.orchestra.run.vm07:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-04-01T15:38:38.479 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-04-01T15:38:38.479 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd mclock iops capacity threshold hdd': 49000, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-04-01T15:38:38.479 INFO:tasks.ceph:ctx.config {'archive_path': '/archive/supriti-2026-04-01_13:45:16-rgw-wip-sse-s3-on-v20.2.0-none-default-vps/4821', 'branch': 'wip-sse-s3-on-v20.2.0', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides rocky_latest s3tests-branch}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '4821', 'last_in_suite': False, 'machine_type': 'vps', 'name': 'supriti-2026-04-01_13:45:16-rgw-wip-sse-s3-on-v20.2.0-none-default-vps', 'no_nested_subset': False, 'os_type': 'rocky', 'os_version': '9.7', 'overrides': {'admin_socket': {'branch': 'wip-sse-s3-on-v20.2.0'}, 'ansible.cephlab': {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd mclock iops capacity threshold hdd': 49000, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'cephadm': {'cephadm_binary_url': 'https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm', 'containers': {'image': 'harbor.clyso.com/custom-ceph/ceph/ceph:sse-s3-kmip-preview-not-for-production-2'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'repos': [{'name': 'ceph-source', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/SRPMS'}, {'name': 'ceph-noarch', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/noarch'}, {'name': 'ceph', 'priority': 1, 'url': 'https://s3.clyso.com/ces-packages/components/ceph-debug/rpm-20.2.0-9-ge2ce8426bc5/el9.clyso/x86_64'}]}, 'rgw': {'compression type': 'random', 'storage classes': {'FROZEN': None, 'LUKEWARM': ['--compression', 'zlib']}}, 's3tests': {'accounts': {'iam alt root': 'RGW99999999999999999', 'iam root': 'RGW88888888888888888'}, 'force-branch': 'ceph-tentacle', 'sha1': 'e0c4ff71baef6d5126a0201df5fe54196d89b296', 'storage classes': 'LUKEWARM, FROZEN'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'tt-20.2.0-sse-s3-kmip-preview-not-for-production-2', 'sha1': '0d46c98b681455ef9cd34c2d3c6981da4e9f1aea'}}, 'owner': 'supriti', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 343, 'sha1': 'e2ce8426bc5387b913d1b06bd31280ced108bbb6', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'tt-20.2.0-sse-s3-kmip-preview-not-for-production-2', 'suite_path': '/home/teuthos/src/git.local_ceph_0d46c98b681455ef9cd34c2d3c6981da4e9f1aea/qa', 'suite_relpath': 'qa', 'suite_repo': 'http://git.local/ceph.git', 'suite_sha1': '0d46c98b681455ef9cd34c2d3c6981da4e9f1aea', 'targets': {'vm07.local': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSgXVFmsTpLwLidMh2xFtU8MeDlh5SEJGOmlCW9pFQZQruNvZaGwlGg9uW0ohiG7JP344vtmZFK9GyLWw83iCw='}, 'tasks': [{'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': {'client.0': None}}, {'tox': ['client.0']}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'uv2', 'teuthology_repo': 'https://github.com/kshtsk/teuthology', 'teuthology_sha1': 'a59626679648f962bca99d20d35578f2998c8f37', 'timestamp': '2026-04-01_13:45:16', 'tube': 'vps', 'user': 'supriti', 'verbose': False, 'worker_log': '/home/teuthos/.teuthology/dispatcher/dispatcher.vps.1399776'} 2026-04-01T15:38:38.479 DEBUG:teuthology.orchestra.run.vm07:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-04-01T15:38:38.532 DEBUG:teuthology.orchestra.run.vm07:> sudo install -d -m0777 -- /var/run/ceph 2026-04-01T15:38:38.597 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:38.597 DEBUG:teuthology.orchestra.run.vm07:> dd if=/scratch_devs of=/dev/stdout 2026-04-01T15:38:38.653 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-04-01T15:38:38.653 DEBUG:teuthology.orchestra.run.vm07:> stat /dev/vg_nvme/lv_1 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Device: 5h/5d Inode: 1047 Links: 1 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Context: system_u:object_r:device_t:s0 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Access: 2026-04-01 15:38:36.234697679 +0000 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Modify: 2026-04-01 15:38:16.267737831 +0000 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout:Change: 2026-04-01 15:38:16.267737831 +0000 2026-04-01T15:38:38.708 INFO:teuthology.orchestra.run.vm07.stdout: Birth: 2026-04-01 15:38:16.267737831 +0000 2026-04-01T15:38:38.709 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-04-01T15:38:38.772 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records in 2026-04-01T15:38:38.772 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records out 2026-04-01T15:38:38.772 INFO:teuthology.orchestra.run.vm07.stderr:512 bytes copied, 0.000162935 s, 3.1 MB/s 2026-04-01T15:38:38.773 DEBUG:teuthology.orchestra.run.vm07:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-04-01T15:38:38.830 DEBUG:teuthology.orchestra.run.vm07:> stat /dev/vg_nvme/lv_2 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Device: 5h/5d Inode: 1053 Links: 1 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Context: system_u:object_r:device_t:s0 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Access: 2026-04-01 15:38:36.235697681 +0000 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Modify: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout:Change: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:38.886 INFO:teuthology.orchestra.run.vm07.stdout: Birth: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:38.886 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-04-01T15:38:38.949 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records in 2026-04-01T15:38:38.949 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records out 2026-04-01T15:38:38.949 INFO:teuthology.orchestra.run.vm07.stderr:512 bytes copied, 0.000192269 s, 2.7 MB/s 2026-04-01T15:38:38.950 DEBUG:teuthology.orchestra.run.vm07:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-04-01T15:38:39.007 DEBUG:teuthology.orchestra.run.vm07:> stat /dev/vg_nvme/lv_3 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Device: 5h/5d Inode: 1058 Links: 1 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Context: system_u:object_r:device_t:s0 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Access: 2026-04-01 15:38:36.235697681 +0000 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Modify: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout:Change: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:39.062 INFO:teuthology.orchestra.run.vm07.stdout: Birth: 2026-04-01 15:38:16.271737822 +0000 2026-04-01T15:38:39.062 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-04-01T15:38:39.126 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records in 2026-04-01T15:38:39.126 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records out 2026-04-01T15:38:39.126 INFO:teuthology.orchestra.run.vm07.stderr:512 bytes copied, 0.000207728 s, 2.5 MB/s 2026-04-01T15:38:39.127 DEBUG:teuthology.orchestra.run.vm07:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-04-01T15:38:39.185 DEBUG:teuthology.orchestra.run.vm07:> stat /dev/vg_nvme/lv_4 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Device: 5h/5d Inode: 1061 Links: 1 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Context: system_u:object_r:device_t:s0 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Access: 2026-04-01 15:38:36.236697682 +0000 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Modify: 2026-04-01 15:38:16.276737811 +0000 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout:Change: 2026-04-01 15:38:16.276737811 +0000 2026-04-01T15:38:39.241 INFO:teuthology.orchestra.run.vm07.stdout: Birth: 2026-04-01 15:38:16.276737811 +0000 2026-04-01T15:38:39.241 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-04-01T15:38:39.306 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records in 2026-04-01T15:38:39.306 INFO:teuthology.orchestra.run.vm07.stderr:1+0 records out 2026-04-01T15:38:39.307 INFO:teuthology.orchestra.run.vm07.stderr:512 bytes copied, 0.000136437 s, 3.8 MB/s 2026-04-01T15:38:39.307 DEBUG:teuthology.orchestra.run.vm07:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-04-01T15:38:39.364 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-04-01T15:38:39.364 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@vm07.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-04-01T15:38:39.364 INFO:tasks.ceph:Generating config... 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] debug rgw = 20 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] rgw s3 auth use sts = True 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] rgw sts key = abcdefghijklmnop 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] setgroup = ceph 2026-04-01T15:38:39.364 INFO:tasks.ceph:[client] setuser = ceph 2026-04-01T15:38:39.364 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-04-01T15:38:39.364 INFO:tasks.ceph:[mgr] debug ms = 1 2026-04-01T15:38:39.364 INFO:tasks.ceph:[mon] debug mon = 20 2026-04-01T15:38:39.364 INFO:tasks.ceph:[mon] debug ms = 1 2026-04-01T15:38:39.364 INFO:tasks.ceph:[mon] debug paxos = 20 2026-04-01T15:38:39.364 INFO:tasks.ceph:[osd] debug ms = 1 2026-04-01T15:38:39.364 INFO:tasks.ceph:[osd] debug osd = 20 2026-04-01T15:38:39.364 INFO:tasks.ceph:[osd] osd mclock iops capacity threshold hdd = 49000 2026-04-01T15:38:39.364 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-04-01T15:38:39.365 INFO:tasks.ceph:Setting up mon.a... 2026-04-01T15:38:39.365 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-04-01T15:38:39.443 INFO:teuthology.orchestra.run.vm07.stdout:creating /etc/ceph/ceph.keyring 2026-04-01T15:38:39.446 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-04-01T15:38:39.524 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-04-01T15:38:39.587 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '192.168.123.107')] 2026-04-01T15:38:39.587 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '192.168.123.107'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd mclock iops capacity threshold hdd': 49000, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2026-04-01T15:38:39.587 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:39.587 DEBUG:teuthology.orchestra.run.vm07:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-04-01T15:38:39.642 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 192.168.123.107 --print /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:39.717 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:38:39.717 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:38:39.717 INFO:teuthology.orchestra.run.vm07.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:monmaptool: generated fsid 47de3fe0-7c25-476e-ac72-6a289e94af77 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:setting min_mon_release = tentacle 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:epoch 0 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:fsid 47de3fe0-7c25-476e-ac72-6a289e94af77 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:last_changed 2026-04-01T15:38:39.717388+0000 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:created 2026-04-01T15:38:39.717388+0000 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:min_mon_release 20 (tentacle) 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:election_strategy: 1 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:0: [v2:192.168.123.107:3300/0,v1:192.168.123.107:6789/0] mon.a 2026-04-01T15:38:39.718 INFO:teuthology.orchestra.run.vm07.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-04-01T15:38:39.719 DEBUG:teuthology.orchestra.run.vm07:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-04-01T15:38:39.775 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 47de3fe0-7c25-476e-ac72-6a289e94af77... 2026-04-01T15:38:39.775 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout:[global] 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: chdir = "" 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: auth supported = cephx 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: filestore xattr use omap = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon clock drift allowed = 1.000 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: osd crush chooseleaf type = 0 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: auth debug = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: ms die on old message = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: ms die on bug = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon max pg per osd = 10000 # >= luminous 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon pg warn max object skew = 0 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: # disable pg_autoscaler by default for new pools 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: osd_pool_default_pg_autoscale_mode = off 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: osd pool default size = 2 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon osd allow primary affinity = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon osd allow pg remap = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon warn on legacy crush tunables = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon warn on crush straw calc version zero = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon warn on no sortbitwise = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon warn on osd down out interval zero = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon warn on too few osds = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon_warn_on_pool_no_redundancy = false 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon_allow_pool_size_one = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: osd default data pool replay window = 5 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon allow pool delete = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon cluster log file level = debug 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: debug asserts on shutdown = true 2026-04-01T15:38:39.855 INFO:teuthology.orchestra.run.vm07.stdout: mon health detail to clog = false 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon host = 192.168.123.107 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: fsid = 47de3fe0-7c25-476e-ac72-6a289e94af77 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout:[osd] 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd journal size = 100 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd scrub load threshold = 5.0 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd scrub max interval = 600 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd mclock profile = high_recovery_ops 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd recover clone overlap = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd recovery max chunk = 1048576 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug shutdown = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug op order = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug verify stray on activate = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug trim objects = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd open classes on start = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug pg log writeout = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd deep scrub update digest min age = 30 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd map max advance = 10 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: journal zero on create = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: filestore ondisk finisher threads = 3 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: filestore apply finisher threads = 3 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: bdev debug aio = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd debug misdirected ops = true 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug ms = 1 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug osd = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd mclock iops capacity threshold hdd = 49000 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: osd shutdown pgref assert = True 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout:[mgr] 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug ms = 1 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug mgr = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug mon = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug auth = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon reweight min pgs per osd = 4 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon reweight min bytes per osd = 10 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mgr/telemetry/nag = false 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout:[mon] 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug ms = 1 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug mon = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug paxos = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: debug auth = 20 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon data avail warn = 5 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon mgr mkfs grace = 240 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon reweight min pgs per osd = 4 2026-04-01T15:38:39.856 INFO:teuthology.orchestra.run.vm07.stdout: mon osd reporter subtree level = osd 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon osd prime pg temp = true 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon reweight min bytes per osd = 10 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: auth mon ticket ttl = 660 # 11m 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: auth service ticket ttl = 240 # 4m 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: # don't complain about insecure global_id in the test suite 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: # 1m isn't quite enough 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon_down_mkfs_grace = 2m 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: mon_warn_on_filestore_osds = false 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout:[client] 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw cache enabled = true 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw enable ops log = true 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw enable usage log = true 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: debug rgw = 20 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw crypt require ssl = False 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw crypt s3 kms backend = testing 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw s3 auth use sts = True 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: rgw sts key = abcdefghijklmnop 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: setgroup = ceph 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout: setuser = ceph 2026-04-01T15:38:39.857 INFO:teuthology.orchestra.run.vm07.stdout:[mon.a] 2026-04-01T15:38:39.867 INFO:tasks.ceph:Creating admin key on mon.a... 2026-04-01T15:38:39.867 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-04-01T15:38:39.950 INFO:tasks.ceph:Copying monmap to all nodes... 2026-04-01T15:38:39.950 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:39.950 DEBUG:teuthology.orchestra.run.vm07:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-04-01T15:38:40.007 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:40.007 DEBUG:teuthology.orchestra.run.vm07:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-04-01T15:38:40.064 INFO:tasks.ceph:Sending monmap to node ubuntu@vm07.local 2026-04-01T15:38:40.064 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:40.064 DEBUG:teuthology.orchestra.run.vm07:> sudo dd of=/etc/ceph/ceph.keyring 2026-04-01T15:38:40.064 DEBUG:teuthology.orchestra.run.vm07:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-04-01T15:38:40.137 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:40.137 DEBUG:teuthology.orchestra.run.vm07:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:40.195 INFO:tasks.ceph:Setting up mon nodes... 2026-04-01T15:38:40.195 INFO:tasks.ceph:Setting up mgr nodes... 2026-04-01T15:38:40.195 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-04-01T15:38:40.277 INFO:teuthology.orchestra.run.vm07.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-04-01T15:38:40.280 INFO:tasks.ceph:Setting up mds nodes... 2026-04-01T15:38:40.280 INFO:tasks.ceph_client:Setting up client nodes... 2026-04-01T15:38:40.280 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-04-01T15:38:40.359 INFO:teuthology.orchestra.run.vm07.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-04-01T15:38:40.370 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-04-01T15:38:40.370 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@vm07.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-04-01T15:38:40.370 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-04-01T15:38:40.433 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-04-01T15:38:40.433 INFO:tasks.ceph:role: osd.0 2026-04-01T15:38:40.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@vm07.local 2026-04-01T15:38:40.433 DEBUG:teuthology.orchestra.run.vm07:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=1310464 blks 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 attr=2, projid32bit=1 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout: = sunit=0 swidth=0 blks 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-04-01T15:38:40.499 INFO:teuthology.orchestra.run.vm07.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-04-01T15:38:40.506 INFO:teuthology.orchestra.run.vm07.stdout:Discarding blocks...Done. 2026-04-01T15:38:40.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@vm07.local -o noatime 2026-04-01T15:38:40.511 DEBUG:teuthology.orchestra.run.vm07:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-04-01T15:38:40.583 DEBUG:teuthology.orchestra.run.vm07:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-04-01T15:38:40.652 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-04-01T15:38:40.716 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-04-01T15:38:40.716 INFO:tasks.ceph:role: osd.1 2026-04-01T15:38:40.716 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@vm07.local 2026-04-01T15:38:40.716 DEBUG:teuthology.orchestra.run.vm07:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=1310464 blks 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 attr=2, projid32bit=1 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout: = sunit=0 swidth=0 blks 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-04-01T15:38:40.782 INFO:teuthology.orchestra.run.vm07.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-04-01T15:38:40.787 INFO:teuthology.orchestra.run.vm07.stdout:Discarding blocks...Done. 2026-04-01T15:38:40.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@vm07.local -o noatime 2026-04-01T15:38:40.789 DEBUG:teuthology.orchestra.run.vm07:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-04-01T15:38:40.858 DEBUG:teuthology.orchestra.run.vm07:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-04-01T15:38:40.925 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-04-01T15:38:40.990 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-04-01T15:38:40.990 INFO:tasks.ceph:role: osd.2 2026-04-01T15:38:40.990 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@vm07.local 2026-04-01T15:38:40.990 DEBUG:teuthology.orchestra.run.vm07:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=1310464 blks 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 attr=2, projid32bit=1 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout: = sunit=0 swidth=0 blks 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-04-01T15:38:41.056 INFO:teuthology.orchestra.run.vm07.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-04-01T15:38:41.060 INFO:teuthology.orchestra.run.vm07.stdout:Discarding blocks...Done. 2026-04-01T15:38:41.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@vm07.local -o noatime 2026-04-01T15:38:41.063 DEBUG:teuthology.orchestra.run.vm07:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-04-01T15:38:41.130 DEBUG:teuthology.orchestra.run.vm07:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-04-01T15:38:41.197 DEBUG:teuthology.orchestra.run.vm07:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:41.279 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.277+0000 7f20fdf9e900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-04-01T15:38:41.279 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.278+0000 7f20fdf9e900 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-04-01T15:38:41.279 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.278+0000 7f20fdf9e900 -1 bdev(0x559bd46a1800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-04-01T15:38:41.280 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.278+0000 7f20fdf9e900 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-04-01T15:38:41.823 DEBUG:teuthology.orchestra.run.vm07:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-04-01T15:38:41.891 DEBUG:teuthology.orchestra.run.vm07:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:41.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.971+0000 7f347912a900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-04-01T15:38:41.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.971+0000 7f347912a900 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-04-01T15:38:41.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.971+0000 7f347912a900 -1 bdev(0x5635ff207800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-04-01T15:38:41.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:41.971+0000 7f347912a900 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-04-01T15:38:42.455 DEBUG:teuthology.orchestra.run.vm07:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-04-01T15:38:42.518 DEBUG:teuthology.orchestra.run.vm07:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:42.601 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:42.600+0000 7f0cb2e12900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-04-01T15:38:42.601 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:42.600+0000 7f0cb2e12900 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-04-01T15:38:42.601 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:42.600+0000 7f0cb2e12900 -1 bdev(0x55770d77d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-04-01T15:38:42.602 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:38:42.600+0000 7f0cb2e12900 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-04-01T15:38:43.113 DEBUG:teuthology.orchestra.run.vm07:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-04-01T15:38:43.136 INFO:tasks.ceph:Reading keys from all nodes... 2026-04-01T15:38:43.136 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:43.136 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-04-01T15:38:43.199 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:43.200 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-04-01T15:38:43.267 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:43.268 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-04-01T15:38:43.332 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:43.332 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-04-01T15:38:43.395 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:43.395 DEBUG:teuthology.orchestra.run.vm07:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-04-01T15:38:43.453 INFO:tasks.ceph:Adding keys to all mons... 2026-04-01T15:38:43.453 DEBUG:teuthology.orchestra.run.vm07:> sudo tee -a /etc/ceph/ceph.keyring 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout:[mgr.x] 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout: key = AQAAPM1pVAp+EBAAGQXBUjtRmC+BdkauBBWSuA== 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout:[osd.0] 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout: key = AQABPM1pQV+dEBAANgXm2jCQWOvoqB64/Qi1zA== 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout:[osd.1] 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout: key = AQABPM1pf07uORAAqhqRUxZ0xRE+c0a3NrHokg== 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout:[osd.2] 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout: key = AQACPM1pwtXUIxAA5OpH4A3AfiOSHfHohN8XAg== 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout:[client.0] 2026-04-01T15:38:43.518 INFO:teuthology.orchestra.run.vm07.stdout: key = AQAAPM1p6o1fFRAAw38VclPwggvgt31XkPlTyA== 2026-04-01T15:38:43.519 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-04-01T15:38:43.605 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-04-01T15:38:43.650 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-04-01T15:38:43.695 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-04-01T15:38:43.737 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-04-01T15:38:43.822 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-04-01T15:38:43.822 DEBUG:teuthology.orchestra.run.vm07:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-04-01T15:38:43.850 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-04-01T15:38:43.950 DEBUG:teuthology.orchestra.run.vm07:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-04-01T15:38:43.975 DEBUG:teuthology.orchestra.run.vm07:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-04-01T15:38:44.031 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-04-01T15:38:44.031 INFO:tasks.ceph.mon.a:Restarting daemon 2026-04-01T15:38:44.031 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-04-01T15:38:44.072 INFO:tasks.ceph.mon.a:Started 2026-04-01T15:38:44.072 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-04-01T15:38:44.072 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-04-01T15:38:44.072 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-04-01T15:38:44.073 INFO:tasks.ceph.mgr.x:Started 2026-04-01T15:38:44.073 DEBUG:tasks.ceph:set 0 configs 2026-04-01T15:38:44.073 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph config dump 2026-04-01T15:38:44.205 INFO:teuthology.orchestra.run.vm07.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-04-01T15:38:44.214 INFO:tasks.ceph:Setting crush tunables to default 2026-04-01T15:38:44.215 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph osd crush tunables default 2026-04-01T15:38:44.341 INFO:teuthology.orchestra.run.vm07.stderr:adjusted tunables profile to default 2026-04-01T15:38:44.357 INFO:tasks.ceph:check_enable_crimson: False 2026-04-01T15:38:44.357 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-04-01T15:38:44.357 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:44.357 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-04-01T15:38:44.383 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:44.383 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-04-01T15:38:44.450 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:38:44.450 DEBUG:teuthology.orchestra.run.vm07:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-04-01T15:38:44.514 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph osd new 805a4f17-23de-40be-83bf-0468502d49d6 0 2026-04-01T15:38:44.674 INFO:teuthology.orchestra.run.vm07.stdout:0 2026-04-01T15:38:44.684 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph osd new dcc263c4-19ff-4447-91dd-4bf6a65d6d50 1 2026-04-01T15:38:44.804 INFO:teuthology.orchestra.run.vm07.stdout:1 2026-04-01T15:38:44.816 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph osd new 33be657c-b1e6-44fe-9c33-721222533a71 2 2026-04-01T15:38:44.935 INFO:teuthology.orchestra.run.vm07.stdout:2 2026-04-01T15:38:44.945 INFO:tasks.ceph.osd.0:Restarting daemon 2026-04-01T15:38:44.946 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-04-01T15:38:44.947 INFO:tasks.ceph.osd.0:Started 2026-04-01T15:38:44.947 INFO:tasks.ceph.osd.1:Restarting daemon 2026-04-01T15:38:44.947 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-04-01T15:38:44.948 INFO:tasks.ceph.osd.1:Started 2026-04-01T15:38:44.948 INFO:tasks.ceph.osd.2:Restarting daemon 2026-04-01T15:38:44.948 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-04-01T15:38:44.950 INFO:tasks.ceph.osd.2:Started 2026-04-01T15:38:44.950 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-04-01T15:38:45.057 INFO:tasks.ceph.mgr.x.vm07.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-04-01T15:38:45.057 INFO:tasks.ceph.mgr.x.vm07.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-04-01T15:38:45.057 INFO:tasks.ceph.mgr.x.vm07.stderr: from numpy import show_config as show_numpy_config 2026-04-01T15:38:45.097 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:45.097 INFO:teuthology.orchestra.run.vm07.stdout:{"epoch":5,"fsid":"47de3fe0-7c25-476e-ac72-6a289e94af77","created":"2026-04-01T15:38:44.147500+0000","modified":"2026-04-01T15:38:44.933921+0000","last_up_change":"0.000000","last_in_change":"2026-04-01T15:38:44.933921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"805a4f17-23de-40be-83bf-0468502d49d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcc263c4-19ff-4447-91dd-4bf6a65d6d50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"33be657c-b1e6-44fe-9c33-721222533a71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-04-01T15:38:45.108 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-04-01T15:38:45.108 INFO:tasks.ceph:Waiting for OSDs to come up 2026-04-01T15:38:45.114 INFO:tasks.ceph.osd.1.vm07.stderr:2026-04-01T15:38:45.112+0000 7f3c01452900 -1 Falling back to public interface 2026-04-01T15:38:45.133 INFO:tasks.ceph.osd.0.vm07.stderr:2026-04-01T15:38:45.129+0000 7fc749acd900 -1 Falling back to public interface 2026-04-01T15:38:45.135 INFO:tasks.ceph.osd.2.vm07.stderr:2026-04-01T15:38:45.134+0000 7f784ba1f900 -1 Falling back to public interface 2026-04-01T15:38:45.255 INFO:tasks.ceph.osd.1.vm07.stderr:2026-04-01T15:38:45.253+0000 7f3c01452900 -1 osd.1 0 log_to_monitors true 2026-04-01T15:38:45.255 INFO:tasks.ceph.osd.0.vm07.stderr:2026-04-01T15:38:45.254+0000 7fc749acd900 -1 osd.0 0 log_to_monitors true 2026-04-01T15:38:45.273 INFO:tasks.ceph.osd.2.vm07.stderr:2026-04-01T15:38:45.272+0000 7f784ba1f900 -1 osd.2 0 log_to_monitors true 2026-04-01T15:38:45.410 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-04-01T15:38:45.510 INFO:teuthology.misc.health.vm07.stdout: 2026-04-01T15:38:45.510 INFO:teuthology.misc.health.vm07.stdout:{"epoch":5,"fsid":"47de3fe0-7c25-476e-ac72-6a289e94af77","created":"2026-04-01T15:38:44.147500+0000","modified":"2026-04-01T15:38:44.933921+0000","last_up_change":"0.000000","last_in_change":"2026-04-01T15:38:44.933921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"805a4f17-23de-40be-83bf-0468502d49d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcc263c4-19ff-4447-91dd-4bf6a65d6d50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"33be657c-b1e6-44fe-9c33-721222533a71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-04-01T15:38:45.518 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-04-01T15:38:47.883 INFO:tasks.ceph.mgr.x.vm07.stderr:2026-04-01T15:38:47.881+0000 7fb3574c2640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-04-01T15:38:48.997 INFO:tasks.ceph.osd.2.vm07.stderr:2026-04-01T15:38:48.995+0000 7f78471c9640 -1 osd.2 0 waiting for initial osdmap 2026-04-01T15:38:49.007 INFO:tasks.ceph.osd.0.vm07.stderr:2026-04-01T15:38:49.005+0000 7fc745a5b640 -1 osd.0 0 waiting for initial osdmap 2026-04-01T15:38:49.008 INFO:tasks.ceph.osd.2.vm07.stderr:2026-04-01T15:38:49.007+0000 7f7841fcd640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-04-01T15:38:49.016 INFO:tasks.ceph.osd.1.vm07.stderr:2026-04-01T15:38:49.014+0000 7f3bfd3e2640 -1 osd.1 0 waiting for initial osdmap 2026-04-01T15:38:49.023 INFO:tasks.ceph.osd.0.vm07.stderr:2026-04-01T15:38:49.022+0000 7fc74085f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-04-01T15:38:49.027 INFO:tasks.ceph.osd.1.vm07.stderr:2026-04-01T15:38:49.026+0000 7f3bf81e6640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-04-01T15:38:51.819 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-04-01T15:38:52.011 INFO:teuthology.misc.health.vm07.stdout: 2026-04-01T15:38:52.011 INFO:teuthology.misc.health.vm07.stdout:{"epoch":9,"fsid":"47de3fe0-7c25-476e-ac72-6a289e94af77","created":"2026-04-01T15:38:44.147500+0000","modified":"2026-04-01T15:38:51.185496+0000","last_up_change":"2026-04-01T15:38:50.007870+0000","last_in_change":"2026-04-01T15:38:44.933921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-04-01T15:38:50.864644+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"805a4f17-23de-40be-83bf-0468502d49d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6808","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6809","nonce":2746847543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6810","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6811","nonce":2746847543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6814","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6815","nonce":2746847543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6812","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6813","nonce":2746847543}]},"public_addr":"192.168.123.107:6809/2746847543","cluster_addr":"192.168.123.107:6811/2746847543","heartbeat_back_addr":"192.168.123.107:6815/2746847543","heartbeat_front_addr":"192.168.123.107:6813/2746847543","state":["exists","up"]},{"osd":1,"uuid":"dcc263c4-19ff-4447-91dd-4bf6a65d6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6800","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6801","nonce":3869334286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6802","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6803","nonce":3869334286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6806","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6807","nonce":3869334286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6804","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6805","nonce":3869334286}]},"public_addr":"192.168.123.107:6801/3869334286","cluster_addr":"192.168.123.107:6803/3869334286","heartbeat_back_addr":"192.168.123.107:6807/3869334286","heartbeat_front_addr":"192.168.123.107:6805/3869334286","state":["exists","up"]},{"osd":2,"uuid":"33be657c-b1e6-44fe-9c33-721222533a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6816","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6817","nonce":4235848468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6818","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6819","nonce":4235848468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6822","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6823","nonce":4235848468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6820","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6821","nonce":4235848468}]},"public_addr":"192.168.123.107:6817/4235848468","cluster_addr":"192.168.123.107:6819/4235848468","heartbeat_back_addr":"192.168.123.107:6823/4235848468","heartbeat_front_addr":"192.168.123.107:6821/4235848468","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-04-01T15:38:52.020 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-04-01T15:38:52.021 INFO:tasks.ceph:Creating RBD pool 2026-04-01T15:38:52.021 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph --cluster ceph osd pool create rbd 8 2026-04-01T15:38:53.200 INFO:teuthology.orchestra.run.vm07.stderr:pool 'rbd' created 2026-04-01T15:38:53.214 DEBUG:teuthology.orchestra.run.vm07:> rbd --cluster ceph pool init rbd 2026-04-01T15:38:53.248 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:38:53.248 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:38:56.218 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-04-01T15:38:56.219 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-04-01T15:38:56.219 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-04-01T15:38:56.460 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:56.470 INFO:teuthology.orchestra.run.vm07.stdout:[{"version":4,"timestamp":"2026-04-01T15:38:49.015547+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_hdd","new_value":"21574.628117"}]}] 2026-04-01T15:38:56.470 INFO:tasks.ceph_manager:config epoch is 4 2026-04-01T15:38:56.470 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-04-01T15:38:56.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-04-01T15:38:56.470 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-04-01T15:38:56.738 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:56.751 INFO:teuthology.orchestra.run.vm07.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6824","nonce":2091104389},{"type":"v1","addr":"192.168.123.107:6825","nonce":2091104389}]},"active_addr":"192.168.123.107:6825/2091104389","active_change":"2026-04-01T15:38:46.839014+0000","active_mgr_features":4541880224203014143,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.2.0","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"192.168.123.107:0","nonce":2126578091}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"192.168.123.107:0","nonce":757947178}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"192.168.123.107:0","nonce":1535980443}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"192.168.123.107:0","nonce":1239573498}]}]} 2026-04-01T15:38:56.752 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-04-01T15:38:56.752 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-04-01T15:38:56.752 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-04-01T15:38:56.951 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:56.951 INFO:teuthology.orchestra.run.vm07.stdout:{"epoch":14,"fsid":"47de3fe0-7c25-476e-ac72-6a289e94af77","created":"2026-04-01T15:38:44.147500+0000","modified":"2026-04-01T15:38:56.207252+0000","last_up_change":"2026-04-01T15:38:50.007870+0000","last_in_change":"2026-04-01T15:38:44.933921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-04-01T15:38:50.864644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-04-01T15:38:52.233737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"805a4f17-23de-40be-83bf-0468502d49d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6808","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6809","nonce":2746847543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6810","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6811","nonce":2746847543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6814","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6815","nonce":2746847543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6812","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6813","nonce":2746847543}]},"public_addr":"192.168.123.107:6809/2746847543","cluster_addr":"192.168.123.107:6811/2746847543","heartbeat_back_addr":"192.168.123.107:6815/2746847543","heartbeat_front_addr":"192.168.123.107:6813/2746847543","state":["exists","up"]},{"osd":1,"uuid":"dcc263c4-19ff-4447-91dd-4bf6a65d6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6800","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6801","nonce":3869334286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6802","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6803","nonce":3869334286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6806","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6807","nonce":3869334286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6804","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6805","nonce":3869334286}]},"public_addr":"192.168.123.107:6801/3869334286","cluster_addr":"192.168.123.107:6803/3869334286","heartbeat_back_addr":"192.168.123.107:6807/3869334286","heartbeat_front_addr":"192.168.123.107:6805/3869334286","state":["exists","up"]},{"osd":2,"uuid":"33be657c-b1e6-44fe-9c33-721222533a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6816","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6817","nonce":4235848468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6818","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6819","nonce":4235848468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6822","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6823","nonce":4235848468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6820","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6821","nonce":4235848468}]},"public_addr":"192.168.123.107:6817/4235848468","cluster_addr":"192.168.123.107:6819/4235848468","heartbeat_back_addr":"192.168.123.107:6823/4235848468","heartbeat_front_addr":"192.168.123.107:6821/4235848468","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-04-01T15:38:56.963 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-04-01T15:38:56.963 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-04-01T15:38:57.169 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:38:57.169 INFO:teuthology.orchestra.run.vm07.stdout:{"epoch":14,"fsid":"47de3fe0-7c25-476e-ac72-6a289e94af77","created":"2026-04-01T15:38:44.147500+0000","modified":"2026-04-01T15:38:56.207252+0000","last_up_change":"2026-04-01T15:38:50.007870+0000","last_in_change":"2026-04-01T15:38:44.933921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-04-01T15:38:50.864644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-04-01T15:38:52.233737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"805a4f17-23de-40be-83bf-0468502d49d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6808","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6809","nonce":2746847543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6810","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6811","nonce":2746847543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6814","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6815","nonce":2746847543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6812","nonce":2746847543},{"type":"v1","addr":"192.168.123.107:6813","nonce":2746847543}]},"public_addr":"192.168.123.107:6809/2746847543","cluster_addr":"192.168.123.107:6811/2746847543","heartbeat_back_addr":"192.168.123.107:6815/2746847543","heartbeat_front_addr":"192.168.123.107:6813/2746847543","state":["exists","up"]},{"osd":1,"uuid":"dcc263c4-19ff-4447-91dd-4bf6a65d6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6800","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6801","nonce":3869334286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6802","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6803","nonce":3869334286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6806","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6807","nonce":3869334286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6804","nonce":3869334286},{"type":"v1","addr":"192.168.123.107:6805","nonce":3869334286}]},"public_addr":"192.168.123.107:6801/3869334286","cluster_addr":"192.168.123.107:6803/3869334286","heartbeat_back_addr":"192.168.123.107:6807/3869334286","heartbeat_front_addr":"192.168.123.107:6805/3869334286","state":["exists","up"]},{"osd":2,"uuid":"33be657c-b1e6-44fe-9c33-721222533a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6816","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6817","nonce":4235848468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6818","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6819","nonce":4235848468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6822","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6823","nonce":4235848468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.107:6820","nonce":4235848468},{"type":"v1","addr":"192.168.123.107:6821","nonce":4235848468}]},"public_addr":"192.168.123.107:6817/4235848468","cluster_addr":"192.168.123.107:6819/4235848468","heartbeat_back_addr":"192.168.123.107:6823/4235848468","heartbeat_front_addr":"192.168.123.107:6821/4235848468","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-04-01T15:38:57.182 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-04-01T15:38:57.182 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-04-01T15:38:57.183 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-04-01T15:38:57.286 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:38:57.286 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-04-01T15:38:57.287 INFO:teuthology.orchestra.run.vm07.stdout:34359738372 2026-04-01T15:38:57.287 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-04-01T15:38:57.292 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:38:57.293 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-04-01T15:38:57.494 INFO:teuthology.orchestra.run.vm07.stdout:34359738370 2026-04-01T15:38:57.499 INFO:teuthology.orchestra.run.vm07.stdout:34359738370 2026-04-01T15:38:57.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-04-01T15:38:57.508 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:38:57.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2026-04-01T15:38:57.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2026-04-01T15:38:58.507 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-04-01T15:38:58.513 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-04-01T15:38:58.522 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-04-01T15:38:58.710 INFO:teuthology.orchestra.run.vm07.stdout:34359738370 2026-04-01T15:38:58.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-04-01T15:38:58.764 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:38:58.774 INFO:teuthology.orchestra.run.vm07.stdout:34359738370 2026-04-01T15:38:58.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2026-04-01T15:38:58.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2026-04-01T15:38:59.722 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-04-01T15:38:59.777 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-04-01T15:38:59.787 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-04-01T15:38:59.930 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:38:59.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2026-04-01T15:38:59.947 DEBUG:teuthology.parallel:result is None 2026-04-01T15:38:59.988 INFO:teuthology.orchestra.run.vm07.stdout:34359738372 2026-04-01T15:39:00.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2026-04-01T15:39:00.001 DEBUG:teuthology.parallel:result is None 2026-04-01T15:39:00.040 INFO:teuthology.orchestra.run.vm07.stdout:34359738371 2026-04-01T15:39:00.051 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2026-04-01T15:39:00.051 DEBUG:teuthology.parallel:result is None 2026-04-01T15:39:00.051 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-04-01T15:39:00.051 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-04-01T15:39:00.284 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:00.284 INFO:teuthology.orchestra.run.vm07.stderr:dumped all 2026-04-01T15:39:00.296 INFO:teuthology.orchestra.run.vm07.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-04-01T15:38:58.853804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81792,"kb_used_data":1272,"kb_used_omap":20,"kb_used_meta":80427,"kb_avail":314491008,"statfs":{"total":322122547200,"available":322038792192,"internally_reserved":0,"allocated":1302528,"data_stored":1024630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21124,"internal_metadata":82357628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.644769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210393+0000","last_change":"2026-04-01T15:38:56.210559+0000","last_active":"2026-04-01T15:38:56.210393+0000","last_peered":"2026-04-01T15:38:56.210393+0000","last_clean":"2026-04-01T15:38:56.210393+0000","last_became_active":"2026-04-01T15:38:54.207473+0000","last_became_peered":"2026-04-01T15:38:54.207473+0000","last_unstale":"2026-04-01T15:38:56.210393+0000","last_undegraded":"2026-04-01T15:38:56.210393+0000","last_fullsized":"2026-04-01T15:38:56.210393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T00:09:41.598204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027203900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210804+0000","last_change":"2026-04-01T15:38:56.210878+0000","last_active":"2026-04-01T15:38:56.210804+0000","last_peered":"2026-04-01T15:38:56.210804+0000","last_clean":"2026-04-01T15:38:56.210804+0000","last_became_active":"2026-04-01T15:38:54.207580+0000","last_became_peered":"2026-04-01T15:38:54.207580+0000","last_unstale":"2026-04-01T15:38:56.210804+0000","last_undegraded":"2026-04-01T15:38:56.210804+0000","last_fullsized":"2026-04-01T15:38:56.210804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T19:26:37.160387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050929399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210453+0000","last_change":"2026-04-01T15:38:56.210518+0000","last_active":"2026-04-01T15:38:56.210453+0000","last_peered":"2026-04-01T15:38:56.210453+0000","last_clean":"2026-04-01T15:38:56.210453+0000","last_became_active":"2026-04-01T15:38:54.207656+0000","last_became_peered":"2026-04-01T15:38:54.207656+0000","last_unstale":"2026-04-01T15:38:56.210453+0000","last_undegraded":"2026-04-01T15:38:56.210453+0000","last_fullsized":"2026-04-01T15:38:56.210453+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T00:40:28.684948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017688099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210534+0000","last_change":"2026-04-01T15:38:56.210746+0000","last_active":"2026-04-01T15:38:56.210534+0000","last_peered":"2026-04-01T15:38:56.210534+0000","last_clean":"2026-04-01T15:38:56.210534+0000","last_became_active":"2026-04-01T15:38:54.207630+0000","last_became_peered":"2026-04-01T15:38:54.207630+0000","last_unstale":"2026-04-01T15:38:56.210534+0000","last_undegraded":"2026-04-01T15:38:56.210534+0000","last_fullsized":"2026-04-01T15:38:56.210534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T23:43:37.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000394459,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.213608+0000","last_change":"2026-04-01T15:38:56.213608+0000","last_active":"2026-04-01T15:38:56.213608+0000","last_peered":"2026-04-01T15:38:56.213608+0000","last_clean":"2026-04-01T15:38:56.213608+0000","last_became_active":"2026-04-01T15:38:54.208281+0000","last_became_peered":"2026-04-01T15:38:54.208281+0000","last_unstale":"2026-04-01T15:38:56.213608+0000","last_undegraded":"2026-04-01T15:38:56.213608+0000","last_fullsized":"2026-04-01T15:38:56.213608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T22:58:27.770348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038739500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.357613+0000","last_change":"2026-04-01T15:38:56.357687+0000","last_active":"2026-04-01T15:38:56.357613+0000","last_peered":"2026-04-01T15:38:56.357613+0000","last_clean":"2026-04-01T15:38:56.357613+0000","last_became_active":"2026-04-01T15:38:54.207257+0000","last_became_peered":"2026-04-01T15:38:54.207257+0000","last_unstale":"2026-04-01T15:38:56.357613+0000","last_undegraded":"2026-04-01T15:38:56.357613+0000","last_fullsized":"2026-04-01T15:38:56.357613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T22:58:27.770348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026029699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.357637+0000","last_change":"2026-04-01T15:38:56.357755+0000","last_active":"2026-04-01T15:38:56.357637+0000","last_peered":"2026-04-01T15:38:56.357637+0000","last_clean":"2026-04-01T15:38:56.357637+0000","last_became_active":"2026-04-01T15:38:54.206815+0000","last_became_peered":"2026-04-01T15:38:54.206815+0000","last_unstale":"2026-04-01T15:38:56.357637+0000","last_undegraded":"2026-04-01T15:38:56.357637+0000","last_fullsized":"2026-04-01T15:38:56.357637+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T19:23:16.111950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030983999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210605+0000","last_change":"2026-04-01T15:38:56.210899+0000","last_active":"2026-04-01T15:38:56.210605+0000","last_peered":"2026-04-01T15:38:56.210605+0000","last_clean":"2026-04-01T15:38:56.210605+0000","last_became_active":"2026-04-01T15:38:54.206787+0000","last_became_peered":"2026-04-01T15:38:54.206787+0000","last_unstale":"2026-04-01T15:38:56.210605+0000","last_undegraded":"2026-04-01T15:38:56.210605+0000","last_fullsized":"2026-04-01T15:38:56.210605+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T03:36:46.525305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049784299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'32","reported_seq":63,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210428+0000","last_change":"2026-04-01T15:38:52.196734+0000","last_active":"2026-04-01T15:38:56.210428+0000","last_peered":"2026-04-01T15:38:56.210428+0000","last_clean":"2026-04-01T15:38:56.210428+0000","last_became_active":"2026-04-01T15:38:52.196580+0000","last_became_peered":"2026-04-01T15:38:52.196580+0000","last_unstale":"2026-04-01T15:38:56.210428+0000","last_undegraded":"2026-04-01T15:38:56.210428+0000","last_fullsized":"2026-04-01T15:38:56.210428+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:51.185496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:51.185496+0000","last_clean_scrub_stamp":"2026-04-01T15:38:51.185496+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T21:20:56.136781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":26960,"kb_used_data":120,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":104830640,"statfs":{"total":107374182400,"available":107346575360,"internally_reserved":0,"allocated":122880,"data_stored":35344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6170,"internal_metadata":27453414},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27416,"kb_used_data":576,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":104830184,"statfs":{"total":107374182400,"available":107346108416,"internally_reserved":0,"allocated":589824,"data_stored":494643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6827,"internal_metadata":27452757},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27416,"kb_used_data":576,"kb_used_omap":7,"kb_used_meta":26808,"kb_avail":104830184,"statfs":{"total":107374182400,"available":107346108416,"internally_reserved":0,"allocated":589824,"data_stored":494643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8127,"internal_metadata":27451457},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-04-01T15:39:00.296 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-04-01T15:39:00.488 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:00.488 INFO:teuthology.orchestra.run.vm07.stderr:dumped all 2026-04-01T15:39:00.499 INFO:teuthology.orchestra.run.vm07.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-04-01T15:38:58.853804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81792,"kb_used_data":1272,"kb_used_omap":20,"kb_used_meta":80427,"kb_avail":314491008,"statfs":{"total":322122547200,"available":322038792192,"internally_reserved":0,"allocated":1302528,"data_stored":1024630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21124,"internal_metadata":82357628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.644769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210393+0000","last_change":"2026-04-01T15:38:56.210559+0000","last_active":"2026-04-01T15:38:56.210393+0000","last_peered":"2026-04-01T15:38:56.210393+0000","last_clean":"2026-04-01T15:38:56.210393+0000","last_became_active":"2026-04-01T15:38:54.207473+0000","last_became_peered":"2026-04-01T15:38:54.207473+0000","last_unstale":"2026-04-01T15:38:56.210393+0000","last_undegraded":"2026-04-01T15:38:56.210393+0000","last_fullsized":"2026-04-01T15:38:56.210393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T00:09:41.598204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027203900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210804+0000","last_change":"2026-04-01T15:38:56.210878+0000","last_active":"2026-04-01T15:38:56.210804+0000","last_peered":"2026-04-01T15:38:56.210804+0000","last_clean":"2026-04-01T15:38:56.210804+0000","last_became_active":"2026-04-01T15:38:54.207580+0000","last_became_peered":"2026-04-01T15:38:54.207580+0000","last_unstale":"2026-04-01T15:38:56.210804+0000","last_undegraded":"2026-04-01T15:38:56.210804+0000","last_fullsized":"2026-04-01T15:38:56.210804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T19:26:37.160387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050929399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210453+0000","last_change":"2026-04-01T15:38:56.210518+0000","last_active":"2026-04-01T15:38:56.210453+0000","last_peered":"2026-04-01T15:38:56.210453+0000","last_clean":"2026-04-01T15:38:56.210453+0000","last_became_active":"2026-04-01T15:38:54.207656+0000","last_became_peered":"2026-04-01T15:38:54.207656+0000","last_unstale":"2026-04-01T15:38:56.210453+0000","last_undegraded":"2026-04-01T15:38:56.210453+0000","last_fullsized":"2026-04-01T15:38:56.210453+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T00:40:28.684948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017688099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210534+0000","last_change":"2026-04-01T15:38:56.210746+0000","last_active":"2026-04-01T15:38:56.210534+0000","last_peered":"2026-04-01T15:38:56.210534+0000","last_clean":"2026-04-01T15:38:56.210534+0000","last_became_active":"2026-04-01T15:38:54.207630+0000","last_became_peered":"2026-04-01T15:38:54.207630+0000","last_unstale":"2026-04-01T15:38:56.210534+0000","last_undegraded":"2026-04-01T15:38:56.210534+0000","last_fullsized":"2026-04-01T15:38:56.210534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T23:43:37.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000394459,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.213608+0000","last_change":"2026-04-01T15:38:56.213608+0000","last_active":"2026-04-01T15:38:56.213608+0000","last_peered":"2026-04-01T15:38:56.213608+0000","last_clean":"2026-04-01T15:38:56.213608+0000","last_became_active":"2026-04-01T15:38:54.208281+0000","last_became_peered":"2026-04-01T15:38:54.208281+0000","last_unstale":"2026-04-01T15:38:56.213608+0000","last_undegraded":"2026-04-01T15:38:56.213608+0000","last_fullsized":"2026-04-01T15:38:56.213608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T22:58:27.770348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038739500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.357613+0000","last_change":"2026-04-01T15:38:56.357687+0000","last_active":"2026-04-01T15:38:56.357613+0000","last_peered":"2026-04-01T15:38:56.357613+0000","last_clean":"2026-04-01T15:38:56.357613+0000","last_became_active":"2026-04-01T15:38:54.207257+0000","last_became_peered":"2026-04-01T15:38:54.207257+0000","last_unstale":"2026-04-01T15:38:56.357613+0000","last_undegraded":"2026-04-01T15:38:56.357613+0000","last_fullsized":"2026-04-01T15:38:56.357613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T22:58:27.770348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026029699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.357637+0000","last_change":"2026-04-01T15:38:56.357755+0000","last_active":"2026-04-01T15:38:56.357637+0000","last_peered":"2026-04-01T15:38:56.357637+0000","last_clean":"2026-04-01T15:38:56.357637+0000","last_became_active":"2026-04-01T15:38:54.206815+0000","last_became_peered":"2026-04-01T15:38:54.206815+0000","last_unstale":"2026-04-01T15:38:56.357637+0000","last_undegraded":"2026-04-01T15:38:56.357637+0000","last_fullsized":"2026-04-01T15:38:56.357637+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T19:23:16.111950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030983999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210605+0000","last_change":"2026-04-01T15:38:56.210899+0000","last_active":"2026-04-01T15:38:56.210605+0000","last_peered":"2026-04-01T15:38:56.210605+0000","last_clean":"2026-04-01T15:38:56.210605+0000","last_became_active":"2026-04-01T15:38:54.206787+0000","last_became_peered":"2026-04-01T15:38:54.206787+0000","last_unstale":"2026-04-01T15:38:56.210605+0000","last_undegraded":"2026-04-01T15:38:56.210605+0000","last_fullsized":"2026-04-01T15:38:56.210605+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:53.195790+0000","last_clean_scrub_stamp":"2026-04-01T15:38:53.195790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-03T03:36:46.525305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049784299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'32","reported_seq":63,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-04-01T15:38:56.210428+0000","last_change":"2026-04-01T15:38:52.196734+0000","last_active":"2026-04-01T15:38:56.210428+0000","last_peered":"2026-04-01T15:38:56.210428+0000","last_clean":"2026-04-01T15:38:56.210428+0000","last_became_active":"2026-04-01T15:38:52.196580+0000","last_became_peered":"2026-04-01T15:38:52.196580+0000","last_unstale":"2026-04-01T15:38:56.210428+0000","last_undegraded":"2026-04-01T15:38:56.210428+0000","last_fullsized":"2026-04-01T15:38:56.210428+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-04-01T15:38:51.185496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-04-01T15:38:51.185496+0000","last_clean_scrub_stamp":"2026-04-01T15:38:51.185496+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T21:20:56.136781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":26960,"kb_used_data":120,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":104830640,"statfs":{"total":107374182400,"available":107346575360,"internally_reserved":0,"allocated":122880,"data_stored":35344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6170,"internal_metadata":27453414},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27416,"kb_used_data":576,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":104830184,"statfs":{"total":107374182400,"available":107346108416,"internally_reserved":0,"allocated":589824,"data_stored":494643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6827,"internal_metadata":27452757},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27416,"kb_used_data":576,"kb_used_omap":7,"kb_used_meta":26808,"kb_avail":104830184,"statfs":{"total":107374182400,"available":107346108416,"internally_reserved":0,"allocated":589824,"data_stored":494643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8127,"internal_metadata":27451457},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-04-01T15:39:00.500 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-04-01T15:39:00.500 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-04-01T15:39:00.500 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-04-01T15:39:00.500 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-04-01T15:39:00.720 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:00.720 INFO:teuthology.orchestra.run.vm07.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-04-01T15:39:00.732 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-04-01T15:39:00.732 INFO:teuthology.run_tasks:Running task rgw... 2026-04-01T15:39:00.737 DEBUG:tasks.rgw:config is {'client.0': None} 2026-04-01T15:39:00.737 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2026-04-01T15:39:00.737 INFO:tasks.rgw:Creating data pools 2026-04-01T15:39:00.737 DEBUG:tasks.rgw:Obtaining remote for client client.0 2026-04-01T15:39:00.737 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2026-04-01T15:39:01.226 INFO:teuthology.orchestra.run.vm07.stderr:pool 'default.rgw.buckets.data' created 2026-04-01T15:39:01.251 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2026-04-01T15:39:02.228 INFO:teuthology.orchestra.run.vm07.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2026-04-01T15:39:02.267 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2026-04-01T15:39:03.262 INFO:teuthology.orchestra.run.vm07.stderr:pool 'default.rgw.buckets.index' created 2026-04-01T15:39:03.295 DEBUG:teuthology.orchestra.run.vm07:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2026-04-01T15:39:04.142 INFO:teuthology.orchestra.run.vm07.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2026-04-01T15:39:04.179 DEBUG:tasks.rgw:Pools created 2026-04-01T15:39:04.179 INFO:tasks.rgw:Configuring compression type = random 2026-04-01T15:39:04.179 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2026-04-01T15:39:04.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2026-04-01T15:39:04.180 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2026-04-01T15:39:04.217 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:04.217 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:06.185 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.184+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 realm 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.185+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.186 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.186+0000 7f9ef898b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:06.187 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.186+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.187 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.186+0000 7f9ef898b900 20 rados_obj.operate() r=0 bl.length=1060 2026-04-01T15:39:06.187 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.186+0000 7f9ef898b900 20 searching for the correct realm 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados_obj.operate() r=0 bl.length=436 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 zone default found 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 4 Realm: () 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 zonegroup default 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.192+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:06.193 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.193+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:06.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:06.193+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:08.174 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:08.172+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:08.174 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:08.173+0000 7f9ef898b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:08.174 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:08.173+0000 7f9ef898b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:08.174 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:08.173+0000 7f9ef898b900 20 started sync module instance, tier type = 2026-04-01T15:39:08.174 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:08.173+0000 7f9ef898b900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:10.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.204+0000 7f9ef898b900 20 add_watcher() i=5 2026-04-01T15:39:10.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.211+0000 7f9ef898b900 20 add_watcher() i=7 2026-04-01T15:39:10.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.211+0000 7f9ef898b900 20 add_watcher() i=4 2026-04-01T15:39:10.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.212+0000 7f9ef898b900 20 add_watcher() i=1 2026-04-01T15:39:10.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.212+0000 7f9ef898b900 20 add_watcher() i=6 2026-04-01T15:39:10.213 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.212+0000 7f9ef898b900 20 add_watcher() i=0 2026-04-01T15:39:10.214 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.213+0000 7f9ef898b900 20 add_watcher() i=3 2026-04-01T15:39:10.214 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.213+0000 7f9ef898b900 20 add_watcher() i=2 2026-04-01T15:39:10.214 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.213+0000 7f9ef898b900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:10.215 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.214+0000 7f9eed7fa640 5 boost::asio::awaitable, obj_version> > logback_generations::read(const DoutPrefixProvider*):446: oid=data_loggenerations_metadata not found 2026-04-01T15:39:10.215 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.214+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.0 2026-04-01T15:39:10.215 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.214+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.0 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.1 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.1 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.2 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.2 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.3 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.3 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.4 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.215+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.4 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.5 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.5 does not exist 2026-04-01T15:39:10.216 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.6 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.6 does not exist 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.7 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.7 does not exist 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.8 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.8 does not exist 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.9 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.9 does not exist 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.10 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.216+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.217 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.10 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.11 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.11 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.12 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.12 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.13 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.13 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.14 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.14 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.15 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.217+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.15 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.16 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.16 does not exist 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.17 2026-04-01T15:39:10.218 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.17 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.18 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.18 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.19 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.19 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.20 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.20 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.21 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.21 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.22 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.218+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.22 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.23 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.23 does not exist 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.24 2026-04-01T15:39:10.219 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.24 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.25 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.25 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.26 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.26 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.27 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.27 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.28 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.28 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.29 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.219+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.29 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.30 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.30 does not exist 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.31 2026-04-01T15:39:10.220 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.31 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.32 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.32 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.33 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.33 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.34 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.34 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.35 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.35 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.36 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.220+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.36 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.37 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.37 does not exist 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.38 2026-04-01T15:39:10.221 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.38 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.39 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.39 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.40 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.40 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.41 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.41 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.42 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.42 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.43 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.43 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.44 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.221+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.44 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.45 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.45 does not exist 2026-04-01T15:39:10.222 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.46 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.46 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.47 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.47 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.48 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.48 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.49 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.49 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.50 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.50 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.51 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.51 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.52 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.222+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.52 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.53 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.53 does not exist 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.54 2026-04-01T15:39:10.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.54 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.55 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.55 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.56 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.56 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.57 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.57 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.58 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.58 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.59 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.223+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.59 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.60 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.60 does not exist 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.61 2026-04-01T15:39:10.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.61 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.62 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.62 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.63 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.63 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.64 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.64 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.65 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.65 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.66 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.224+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.66 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.67 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.67 does not exist 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.68 2026-04-01T15:39:10.225 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.68 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.69 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.69 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.70 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.70 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.71 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.71 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.72 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.72 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.73 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.225+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.73 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.74 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.74 does not exist 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.75 2026-04-01T15:39:10.226 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.75 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.76 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.76 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.77 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.77 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.78 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.78 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.79 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.79 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.80 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.226+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.80 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.81 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.81 does not exist 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.82 2026-04-01T15:39:10.227 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.82 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.83 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.83 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.84 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.84 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.85 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.85 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.86 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.86 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.87 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.87 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.88 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.227+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.88 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.89 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.89 does not exist 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.90 2026-04-01T15:39:10.228 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.90 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.91 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.91 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.92 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.92 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.93 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.93 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.94 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.94 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.95 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.228+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.95 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.96 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.96 does not exist 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.97 2026-04-01T15:39:10.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.97 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.98 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.98 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.99 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.99 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.100 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.100 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.101 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.101 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.102 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.229+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.102 does not exist 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.103 2026-04-01T15:39:10.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.103 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.104 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.104 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.105 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.105 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.106 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.106 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.107 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.107 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.108 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.108 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.109 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.230+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.109 does not exist 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.110 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.110 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.111 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.111 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.112 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.112 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.113 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.113 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.114 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.114 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.115 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.115 does not exist 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.116 2026-04-01T15:39:10.232 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.231+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.116 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.117 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.117 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.118 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.118 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.119 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.119 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.120 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eed7fa640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.120 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eecff9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.121 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.121 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ee7fff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.122 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.232+0000 7f9ee7fff640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.122 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef56e9640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.123 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef56e9640 20 do_open: entering 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.123 does not exist 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eeeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.124 2026-04-01T15:39:10.233 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eeeffd640 20 do_open: entering 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.124 does not exist 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eee7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.125 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eee7fc640 20 do_open: entering 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.125 does not exist 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eedffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.126 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eedffb640 20 do_open: entering 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.126 does not exist 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef6972640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.127 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9ef6972640 20 do_open: entering 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eed7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.127 does not exist 2026-04-01T15:39:10.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.233+0000 7f9eed7fa640 20 do_create: entering 2026-04-01T15:39:10.236 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.235+0000 7f9eecff9640 20 do_open: entering 2026-04-01T15:39:10.239 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.238+0000 7f9ef898b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:10.239 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:10.238+0000 7f9ef898b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.204 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.203+0000 7f9ef898b900 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2026-04-01T15:39:13.204 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.203+0000 7f9ef898b900 5 note: GC not initialized 2026-04-01T15:39:13.204 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.203+0000 7f9e9cff1640 20 reqs_thread_entry: start 2026-04-01T15:39:13.253 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.252+0000 7f9ef898b900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:13.253 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.252+0000 7f9ef898b900 20 Filter name: none 2026-04-01T15:39:13.253 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.252+0000 7f9e967fc640 20 reqs_thread_entry: start 2026-04-01T15:39:13.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.261+0000 7f9ef898b900 20 remove_watcher() i=3 2026-04-01T15:39:13.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.261+0000 7f9ef898b900 2 removed watcher, disabling cache 2026-04-01T15:39:13.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.261+0000 7f9ef898b900 20 remove_watcher() i=5 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=2 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=1 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=6 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=4 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=0 2026-04-01T15:39:13.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.262+0000 7f9ef898b900 20 remove_watcher() i=7 2026-04-01T15:39:13.269 INFO:teuthology.orchestra.run.vm07.stdout:[] 2026-04-01T15:39:13.269 DEBUG:tasks.util.rgw: json result: [] 2026-04-01T15:39:13.269 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2026-04-01T15:39:13.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2026-04-01T15:39:13.269 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2026-04-01T15:39:13.348 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.348 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.359 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.358+0000 7fd0659c9900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.359 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.358+0000 7fd0659c9900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.359 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.359+0000 7fd00dff3640 20 reqs_thread_entry: start 2026-04-01T15:39:13.369 INFO:teuthology.orchestra.run.vm07.stdout:{"id":"26234e52-e0ad-4466-b829-a8bdf0048973","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","dedup_pool":"default.rgw.dedup","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","restore_pool":"default.rgw.log:restore"} 2026-04-01T15:39:13.369 DEBUG:tasks.util.rgw: json result: {'id': '26234e52-e0ad-4466-b829-a8bdf0048973', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'dedup_pool': 'default.rgw.dedup', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'restore_pool': 'default.rgw.log:restore'} 2026-04-01T15:39:13.369 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2026-04-01T15:39:13.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2026-04-01T15:39:13.369 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2026-04-01T15:39:13.445 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.445 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.459 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.458+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.459 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 realm 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 rados_obj.operate() r=0 bl.length=1076 2026-04-01T15:39:13.460 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.459+0000 7fd14660f900 20 searching for the correct realm 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.464+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados_obj.operate() r=0 bl.length=436 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 zone default found 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 4 Realm: () 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 zonegroup default 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.465+0000 7fd14660f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.466+0000 7fd14660f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.466+0000 7fd14660f900 20 started sync module instance, tier type = 2026-04-01T15:39:13.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.466+0000 7fd14660f900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:13.470 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.469+0000 7fd14660f900 20 add_watcher() i=0 2026-04-01T15:39:13.470 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.469+0000 7fd14660f900 20 add_watcher() i=1 2026-04-01T15:39:13.470 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.470+0000 7fd14660f900 20 add_watcher() i=4 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 20 add_watcher() i=5 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 20 add_watcher() i=2 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 20 add_watcher() i=7 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 20 add_watcher() i=3 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 20 add_watcher() i=6 2026-04-01T15:39:13.474 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.474+0000 7fd14660f900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:13.476 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.476+0000 7fd14660f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.476 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.476+0000 7fd14660f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.476 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.476+0000 7fd14660f900 5 note: GC not initialized 2026-04-01T15:39:13.477 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.476+0000 7fd0eefed640 20 reqs_thread_entry: start 2026-04-01T15:39:13.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.510+0000 7fd14660f900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:13.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.510+0000 7fd14660f900 20 Filter name: none 2026-04-01T15:39:13.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.510+0000 7fd0ecfe9640 20 reqs_thread_entry: start 2026-04-01T15:39:13.519 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.518+0000 7fd14660f900 20 remove_watcher() i=0 2026-04-01T15:39:13.519 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.518+0000 7fd14660f900 2 removed watcher, disabling cache 2026-04-01T15:39:13.519 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.518+0000 7fd14660f900 20 remove_watcher() i=3 2026-04-01T15:39:13.519 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=1 2026-04-01T15:39:13.520 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=5 2026-04-01T15:39:13.520 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=4 2026-04-01T15:39:13.520 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=6 2026-04-01T15:39:13.520 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=2 2026-04-01T15:39:13.520 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.519+0000 7fd14660f900 20 remove_watcher() i=7 2026-04-01T15:39:13.525 INFO:teuthology.orchestra.run.vm07.stdout:[] 2026-04-01T15:39:13.525 DEBUG:tasks.util.rgw: json result: [] 2026-04-01T15:39:13.525 INFO:tasks.rgw:Configuring storage class = FROZEN 2026-04-01T15:39:13.525 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-04-01T15:39:13.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-04-01T15:39:13.525 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2026-04-01T15:39:13.604 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.604 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.616 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.615+0000 7f439ab39900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.616 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.615+0000 7f439ab39900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.616 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.616+0000 7f433ffff640 20 reqs_thread_entry: start 2026-04-01T15:39:13.626 INFO:teuthology.orchestra.run.vm07.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","STANDARD"]}}] 2026-04-01T15:39:13.626 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'STANDARD']}}] 2026-04-01T15:39:13.626 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-04-01T15:39:13.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-04-01T15:39:13.626 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2026-04-01T15:39:13.708 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.708 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.718 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.717+0000 7f364332b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.718 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.717+0000 7f364332b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.718 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.717+0000 7f35ed7ea640 20 reqs_thread_entry: start 2026-04-01T15:39:13.726 INFO:teuthology.orchestra.run.vm07.stdout:{"id":"26234e52-e0ad-4466-b829-a8bdf0048973","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","dedup_pool":"default.rgw.dedup","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","restore_pool":"default.rgw.log:restore"} 2026-04-01T15:39:13.726 DEBUG:tasks.util.rgw: json result: {'id': '26234e52-e0ad-4466-b829-a8bdf0048973', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'dedup_pool': 'default.rgw.dedup', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'restore_pool': 'default.rgw.log:restore'} 2026-04-01T15:39:13.726 INFO:tasks.rgw:Configuring storage class = LUKEWARM 2026-04-01T15:39:13.726 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-04-01T15:39:13.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-04-01T15:39:13.726 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2026-04-01T15:39:13.802 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.803 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.814 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.813+0000 7f577ab66900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.814 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.813+0000 7f577ab66900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.814 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.813+0000 7f571ffff640 20 reqs_thread_entry: start 2026-04-01T15:39:13.821 INFO:teuthology.orchestra.run.vm07.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2026-04-01T15:39:13.821 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2026-04-01T15:39:13.821 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm', '--compression', 'zlib'] 2026-04-01T15:39:13.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm', '--compression', 'zlib'] 2026-04-01T15:39:13.821 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm --compression zlib 2026-04-01T15:39:13.906 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:13.906 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:13.919 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.917+0000 7f4cb98f2900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:13.919 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.918+0000 7f4cb98f2900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:13.919 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:13.918+0000 7f4c5ffff640 20 reqs_thread_entry: start 2026-04-01T15:39:13.928 INFO:teuthology.orchestra.run.vm07.stdout:{"id":"26234e52-e0ad-4466-b829-a8bdf0048973","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","dedup_pool":"default.rgw.dedup","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm","compression_type":"zlib"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","restore_pool":"default.rgw.log:restore"} 2026-04-01T15:39:13.928 DEBUG:tasks.util.rgw: json result: {'id': '26234e52-e0ad-4466-b829-a8bdf0048973', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'dedup_pool': 'default.rgw.dedup', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm', 'compression_type': 'zlib'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'restore_pool': 'default.rgw.log:restore'} 2026-04-01T15:39:13.928 INFO:tasks.rgw:Starting rgw... 2026-04-01T15:39:13.928 INFO:tasks.rgw:rgw client.0 config is {} 2026-04-01T15:39:13.928 INFO:tasks.rgw:Using beast as radosgw frontend 2026-04-01T15:39:13.928 DEBUG:teuthology.orchestra.run.vm07:> sudo echo -n http://vm07.local:80 | sudo tee /home/ubuntu/cephtest/url_file 2026-04-01T15:39:13.956 INFO:teuthology.orchestra.run.vm07.stdout:http://vm07.local:80 2026-04-01T15:39:13.956 DEBUG:teuthology.orchestra.run.vm07:> sudo chown ceph /home/ubuntu/cephtest/url_file 2026-04-01T15:39:14.022 INFO:tasks.rgw.client.0:Restarting daemon 2026-04-01T15:39:14.022 DEBUG:teuthology.orchestra.run.vm07:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2026-04-01T15:39:14.064 INFO:tasks.rgw.client.0:Started 2026-04-01T15:39:14.065 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://vm07.local:80/ 2026-04-01T15:39:14.065 DEBUG:teuthology.orchestra.run.vm07:> curl http://vm07.local:80/ 2026-04-01T15:39:14.083 INFO:teuthology.orchestra.run.vm07.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-04-01T15:39:14.083 INFO:teuthology.orchestra.run.vm07.stderr: Dload Upload Total Spent Left Speed 2026-04-01T15:39:14.086 INFO:teuthology.orchestra.run.vm07.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2026-04-01T15:39:14.086 INFO:teuthology.orchestra.run.vm07.stderr:curl: (7) Failed to connect to vm07.local port 80: Connection refused 2026-04-01T15:39:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 7 2026-04-01T15:39:15.087 DEBUG:teuthology.orchestra.run.vm07:> curl http://vm07.local:80/ 2026-04-01T15:39:15.108 INFO:teuthology.orchestra.run.vm07.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-04-01T15:39:15.108 INFO:teuthology.orchestra.run.vm07.stderr: Dload Upload Total Spent Left Speed 2026-04-01T15:39:15.109 INFO:teuthology.orchestra.run.vm07.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2026-04-01T15:39:15.109 INFO:teuthology.orchestra.run.vm07.stdout:anonymous 2026-04-01T15:39:15.110 INFO:teuthology.run_tasks:Running task tox... 2026-04-01T15:39:15.112 INFO:tasks.tox:Deploying tox from pip... 2026-04-01T15:39:15.112 DEBUG:teuthology.orchestra.run.vm07:> python3 -m venv /home/ubuntu/cephtest/tox-venv 2026-04-01T15:39:16.420 DEBUG:teuthology.orchestra.run.vm07:> source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox 2026-04-01T15:39:16.711 INFO:teuthology.orchestra.run.vm07.stdout:Collecting tox 2026-04-01T15:39:16.751 INFO:teuthology.orchestra.run.vm07.stdout: Downloading tox-4.30.3-py3-none-any.whl (175 kB) 2026-04-01T15:39:16.828 INFO:teuthology.orchestra.run.vm07.stdout:Collecting tomli>=2.2.1 2026-04-01T15:39:16.837 INFO:teuthology.orchestra.run.vm07.stdout: Downloading tomli-2.4.1-py3-none-any.whl (14 kB) 2026-04-01T15:39:16.869 INFO:teuthology.orchestra.run.vm07.stdout:Collecting packaging>=25 2026-04-01T15:39:16.878 INFO:teuthology.orchestra.run.vm07.stdout: Downloading packaging-26.0-py3-none-any.whl (74 kB) 2026-04-01T15:39:16.908 INFO:teuthology.orchestra.run.vm07.stdout:Collecting typing-extensions>=4.14.1 2026-04-01T15:39:16.916 INFO:teuthology.orchestra.run.vm07.stdout: Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2026-04-01T15:39:16.949 INFO:teuthology.orchestra.run.vm07.stdout:Collecting cachetools>=6.1 2026-04-01T15:39:16.957 INFO:teuthology.orchestra.run.vm07.stdout: Downloading cachetools-6.2.6-py3-none-any.whl (11 kB) 2026-04-01T15:39:17.007 INFO:teuthology.orchestra.run.vm07.stdout:Collecting chardet>=5.2 2026-04-01T15:39:17.016 INFO:teuthology.orchestra.run.vm07.stdout: Downloading chardet-5.2.0-py3-none-any.whl (199 kB) 2026-04-01T15:39:17.105 INFO:teuthology.orchestra.run.vm07.stdout:Collecting virtualenv>=20.31.2 2026-04-01T15:39:17.114 INFO:teuthology.orchestra.run.vm07.stdout: Downloading virtualenv-21.2.0-py3-none-any.whl (5.8 MB) 2026-04-01T15:39:17.209 INFO:teuthology.orchestra.run.vm07.stdout:Collecting colorama>=0.4.6 2026-04-01T15:39:17.217 INFO:teuthology.orchestra.run.vm07.stdout: Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB) 2026-04-01T15:39:17.251 INFO:teuthology.orchestra.run.vm07.stdout:Collecting platformdirs>=4.3.8 2026-04-01T15:39:17.259 INFO:teuthology.orchestra.run.vm07.stdout: Downloading platformdirs-4.4.0-py3-none-any.whl (18 kB) 2026-04-01T15:39:17.285 INFO:teuthology.orchestra.run.vm07.stdout:Collecting pluggy>=1.6 2026-04-01T15:39:17.293 INFO:teuthology.orchestra.run.vm07.stdout: Downloading pluggy-1.6.0-py3-none-any.whl (20 kB) 2026-04-01T15:39:17.330 INFO:teuthology.orchestra.run.vm07.stdout:Collecting filelock>=3.18 2026-04-01T15:39:17.338 INFO:teuthology.orchestra.run.vm07.stdout: Downloading filelock-3.19.1-py3-none-any.whl (15 kB) 2026-04-01T15:39:17.362 INFO:teuthology.orchestra.run.vm07.stdout:Collecting pyproject-api>=1.9.1 2026-04-01T15:39:17.371 INFO:teuthology.orchestra.run.vm07.stdout: Downloading pyproject_api-1.9.1-py3-none-any.whl (13 kB) 2026-04-01T15:39:17.409 INFO:teuthology.orchestra.run.vm07.stdout:Collecting distlib<1,>=0.3.7 2026-04-01T15:39:17.418 INFO:teuthology.orchestra.run.vm07.stdout: Downloading distlib-0.4.0-py2.py3-none-any.whl (469 kB) 2026-04-01T15:39:17.446 INFO:teuthology.orchestra.run.vm07.stdout:Collecting python-discovery>=1 2026-04-01T15:39:17.455 INFO:teuthology.orchestra.run.vm07.stdout: Downloading python_discovery-1.2.1-py3-none-any.whl (31 kB) 2026-04-01T15:39:17.512 INFO:teuthology.orchestra.run.vm07.stdout:Installing collected packages: platformdirs, filelock, typing-extensions, tomli, python-discovery, packaging, distlib, virtualenv, pyproject-api, pluggy, colorama, chardet, cachetools, tox 2026-04-01T15:39:17.862 INFO:teuthology.orchestra.run.vm07.stdout:Successfully installed cachetools-6.2.6 chardet-5.2.0 colorama-0.4.6 distlib-0.4.0 filelock-3.19.1 packaging-26.0 platformdirs-4.4.0 pluggy-1.6.0 pyproject-api-1.9.1 python-discovery-1.2.1 tomli-2.4.1 tox-4.30.3 typing-extensions-4.15.0 virtualenv-21.2.0 2026-04-01T15:39:17.936 INFO:teuthology.orchestra.run.vm07.stderr:WARNING: You are using pip version 21.3.1; however, version 26.0.1 is available. 2026-04-01T15:39:17.936 INFO:teuthology.orchestra.run.vm07.stderr:You should consider upgrading via the '/home/ubuntu/cephtest/tox-venv/bin/python3 -m pip install --upgrade pip' command. 2026-04-01T15:39:17.980 INFO:teuthology.run_tasks:Running task s3tests... 2026-04-01T15:39:17.985 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'accounts': {'iam alt root': 'RGW99999999999999999', 'iam root': 'RGW88888888888888888'}, 'force-branch': 'ceph-tentacle', 'sha1': 'e0c4ff71baef6d5126a0201df5fe54196d89b296', 'storage classes': 'LUKEWARM, FROZEN'}} 2026-04-01T15:39:17.985 INFO:tasks.s3tests:Downloading s3-tests... 2026-04-01T15:39:17.985 INFO:tasks.s3tests:Using branch 'ceph-tentacle' for s3tests 2026-04-01T15:39:17.986 DEBUG:teuthology.orchestra.run.vm07:> git clone -b ceph-tentacle https://github.com/ceph/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2026-04-01T15:39:18.001 INFO:teuthology.orchestra.run.vm07.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2026-04-01T15:39:19.090 DEBUG:teuthology.orchestra.run.vm07:> cd /home/ubuntu/cephtest/s3-tests-client.0 && git reset --hard e0c4ff71baef6d5126a0201df5fe54196d89b296 2026-04-01T15:39:19.153 INFO:teuthology.orchestra.run.vm07.stdout:HEAD is now at e0c4ff7 functional: fix invalid escape sequence deprication warning 2026-04-01T15:39:19.153 INFO:tasks.s3tests:Creating rgw users... 2026-04-01T15:39:19.153 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2026-04-01T15:39:19.153 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name Mr.foo.client.0 --email foo+test@test.test --access-key ZWFWRHXCMUDPESQFULHC --secret 5LGtsQ8YsvE0rnEBgxadVPoIXBc1RcQ3mMK9XhE3+DLFm+EUWFaesA== --cluster ceph 2026-04-01T15:39:19.233 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:19.233 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:19.245 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.244+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.245 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 realm 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:19.246 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.245+0000 7f0c9253e900 20 searching for the correct realm 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.254+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 zone default found 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 4 Realm: () 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 zonegroup default 2026-04-01T15:39:19.255 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 started sync module instance, tier type = 2026-04-01T15:39:19.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.255+0000 7f0c9253e900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:19.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.261+0000 7f0c9253e900 20 add_watcher() i=0 2026-04-01T15:39:19.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.261+0000 7f0c9253e900 20 add_watcher() i=5 2026-04-01T15:39:19.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.261+0000 7f0c9253e900 20 add_watcher() i=7 2026-04-01T15:39:19.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.262+0000 7f0c9253e900 20 add_watcher() i=4 2026-04-01T15:39:19.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.262+0000 7f0c9253e900 20 add_watcher() i=1 2026-04-01T15:39:19.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.263+0000 7f0c9253e900 20 add_watcher() i=6 2026-04-01T15:39:19.263 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.263+0000 7f0c9253e900 20 add_watcher() i=3 2026-04-01T15:39:19.264 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.263+0000 7f0c9253e900 20 add_watcher() i=2 2026-04-01T15:39:19.264 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.263+0000 7f0c9253e900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:19.266 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.265+0000 7f0c9253e900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:19.266 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.265+0000 7f0c9253e900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:19.266 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.265+0000 7f0c9253e900 5 note: GC not initialized 2026-04-01T15:39:19.266 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.265+0000 7f0c36ffd640 20 reqs_thread_entry: start 2026-04-01T15:39:19.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 Filter name: none 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c34ff9640 20 reqs_thread_entry: start 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache get: name=default.rgw.meta+users.email+foo+test@test.test : miss 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 adding default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache get: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC : miss 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC info.flags=0x0 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 adding default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC to cache LRU end 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache get: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC : hit (negative entry) 2026-04-01T15:39:19.295 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.294+0000 7f0c9253e900 10 cache get: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC : hit (negative entry) 2026-04-01T15:39:19.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-04-01T15:39:19.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c9253e900 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:19.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c9253e900 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2026-04-01T15:39:19.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c6dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 94337108688064 notifier 4273 bl.length()=645 2026-04-01T15:39:19.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c6dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-04-01T15:39:19.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.296+0000 7f0c6dffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.297+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x7 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.297+0000 7f0c9253e900 10 moving default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.297+0000 7f0c9253e900 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:foo+test@test.test, ofs0, ns] 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.298+0000 7f0c6d7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 94337108694032 notifier 4273 bl.length()=185 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.298+0000 7f0c6d7fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x7 2026-04-01T15:39:19.298 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.298+0000 7f0c6d7fa640 10 rgw watcher librados: moving default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2026-04-01T15:39:19.299 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c9253e900 10 cache put: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC info.flags=0x7 2026-04-01T15:39:19.299 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c9253e900 10 moving default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC to cache LRU end 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c9253e900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:ZWFWRHXCMUDPESQFULHC, ofs0, ns] 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c6dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94337108685088 notifier 4273 bl.length()=186 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c6dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC info.flags=0x7 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.299+0000 7f0c6dffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+ZWFWRHXCMUDPESQFULHC to cache LRU end 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "foo.client.0", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.foo.client.0", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "email": "foo+test@test.test", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "user": "foo.client.0", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "ZWFWRHXCMUDPESQFULHC", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "5LGtsQ8YsvE0rnEBgxadVPoIXBc1RcQ3mMK9XhE3+DLFm+EUWFaesA==", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:19.295724Z" 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:19.300 INFO:teuthology.orchestra.run.vm07.stdout: "type": "rgw", 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "", 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:19.295720Z", 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:19.301 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:19.303 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=2 2026-04-01T15:39:19.303 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 2 removed watcher, disabling cache 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=5 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=0 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=3 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=4 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=7 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=1 2026-04-01T15:39:19.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.303+0000 7f0c9253e900 20 remove_watcher() i=6 2026-04-01T15:39:19.309 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 1295646771 --totp-seed RJ7CYV5DRIPXTQYYAM2LGMFU6ZQ3AITD6GWTG3XOKUP37PYFGEV6KG2QLOQV25AW --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:19.388 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:19.388 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:19.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.401+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 realm 2026-04-01T15:39:19.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:19.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.402+0000 7f4b83d44900 20 searching for the correct realm 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.407+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 zone default found 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 4 Realm: () 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 zonegroup default 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.408+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.409+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.409+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.409+0000 7f4b83d44900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.409+0000 7f4b83d44900 20 started sync module instance, tier type = 2026-04-01T15:39:19.409 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.409+0000 7f4b83d44900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:19.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.413+0000 7f4b83d44900 20 add_watcher() i=0 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.413+0000 7f4b83d44900 20 add_watcher() i=1 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.413+0000 7f4b83d44900 20 add_watcher() i=3 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.413+0000 7f4b83d44900 20 add_watcher() i=2 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.414+0000 7f4b83d44900 20 add_watcher() i=5 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.414+0000 7f4b83d44900 20 add_watcher() i=4 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.414+0000 7f4b83d44900 20 add_watcher() i=6 2026-04-01T15:39:19.414 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.414+0000 7f4b83d44900 20 add_watcher() i=7 2026-04-01T15:39:19.415 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.414+0000 7f4b83d44900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:19.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.416+0000 7f4b83d44900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:19.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.416+0000 7f4b83d44900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:19.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.416+0000 7f4b83d44900 5 note: GC not initialized 2026-04-01T15:39:19.417 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.416+0000 7f4b2cfe9640 20 reqs_thread_entry: start 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.447+0000 7f4b83d44900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.447+0000 7f4b83d44900 20 Filter name: none 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.447+0000 7f4b267fc640 20 reqs_thread_entry: start 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.447+0000 7f4b83d44900 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.447+0000 7f4b83d44900 20 rados->read ofs=0 len=0 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.448+0000 7f4b83d44900 20 rados_obj.operate() r=0 bl.length=460 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.448+0000 7f4b83d44900 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.448+0000 7f4b83d44900 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:19.448 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:19.448+0000 7f4b83d44900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.243+0000 7f4b83d44900 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.243+0000 7f4b83d44900 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.243+0000 7f4b83d44900 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.244+0000 7f4b5affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051585 cookie 94882746407136 notifier 4280 bl.length()=659 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.244+0000 7f4b5affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-04-01T15:39:21.244 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.244+0000 7f4b5affd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 20 remove_watcher() i=2 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 2 removed watcher, disabling cache 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 20 remove_watcher() i=3 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 20 remove_watcher() i=0 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 20 remove_watcher() i=7 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.277+0000 7f4b83d44900 20 remove_watcher() i=5 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.278+0000 7f4b83d44900 20 remove_watcher() i=1 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.278+0000 7f4b83d44900 20 remove_watcher() i=6 2026-04-01T15:39:21.278 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.278+0000 7f4b83d44900 20 remove_watcher() i=4 2026-04-01T15:39:21.283 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2026-04-01T15:39:21.283 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name Mr.bar.client.0 --email bar+test@test.test --access-key VTHCUXXAKWGNFWMWKYSP --secret jNqTwvlpdkMVOve1UgXoMTa3OVCtsWupNFo+UwQ2QZ/q1EhXF5yu8w== --cluster ceph 2026-04-01T15:39:21.365 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:21.365 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:21.377 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.376+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 realm 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.377+0000 7fdafe121900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.378+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.378+0000 7fdafe121900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:21.378 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.378+0000 7fdafe121900 20 searching for the correct realm 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.383 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 zone default found 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 4 Realm: () 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 zonegroup default 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.383+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.384+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.384+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.384+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.384+0000 7fdafe121900 20 started sync module instance, tier type = 2026-04-01T15:39:21.384 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.384+0000 7fdafe121900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:21.388 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.387+0000 7fdafe121900 20 add_watcher() i=1 2026-04-01T15:39:21.388 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=5 2026-04-01T15:39:21.388 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=3 2026-04-01T15:39:21.388 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=4 2026-04-01T15:39:21.389 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=6 2026-04-01T15:39:21.389 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=0 2026-04-01T15:39:21.389 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=2 2026-04-01T15:39:21.389 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 20 add_watcher() i=7 2026-04-01T15:39:21.389 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.388+0000 7fdafe121900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:21.391 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.390+0000 7fdafe121900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:21.391 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.390+0000 7fdafe121900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:21.391 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.390+0000 7fdafe121900 5 note: GC not initialized 2026-04-01T15:39:21.391 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.390+0000 7fdaa6fed640 20 reqs_thread_entry: start 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 Filter name: none 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdaa4fe9640 20 reqs_thread_entry: start 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 cache get: name=default.rgw.meta+users.email+bar+test@test.test : miss 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 adding default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 10 cache get: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP : miss 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.418+0000 7fdafe121900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.419+0000 7fdafe121900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.419+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP info.flags=0x0 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.419+0000 7fdafe121900 10 adding default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP to cache LRU end 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.419+0000 7fdafe121900 10 cache get: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP : hit (negative entry) 2026-04-01T15:39:21.419 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.419+0000 7fdafe121900 10 cache get: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP : hit (negative entry) 2026-04-01T15:39:21.420 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2026-04-01T15:39:21.420 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdafe121900 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.420 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdafe121900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2026-04-01T15:39:21.420 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdad8ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051586 cookie 93922388971472 notifier 4286 bl.length()=645 2026-04-01T15:39:21.421 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdad8ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2026-04-01T15:39:21.421 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.420+0000 7fdad8ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x7 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdafe121900 10 moving default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdafe121900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:bar+test@test.test, ofs0, ns] 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdad8ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051586 cookie 93922388965088 notifier 4286 bl.length()=185 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdad8ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x7 2026-04-01T15:39:21.422 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.421+0000 7fdad8ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdafe121900 10 cache put: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP info.flags=0x7 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdafe121900 10 moving default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP to cache LRU end 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdafe121900 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:VTHCUXXAKWGNFWMWKYSP, ofs0, ns] 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdad8ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051587 cookie 93922388989392 notifier 4286 bl.length()=186 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdad8ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP info.flags=0x7 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.423+0000 7fdad8ff9640 10 rgw watcher librados: moving default.rgw.meta+users.keys+VTHCUXXAKWGNFWMWKYSP to cache LRU end 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:21.423 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "bar.client.0", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.bar.client.0", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "email": "bar+test@test.test", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "user": "bar.client.0", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "VTHCUXXAKWGNFWMWKYSP", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "jNqTwvlpdkMVOve1UgXoMTa3OVCtsWupNFo+UwQ2QZ/q1EhXF5yu8w==", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.419819Z" 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "type": "rgw", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.419815Z", 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:21.424 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:21.427 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=5 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 2 removed watcher, disabling cache 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=1 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=4 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=6 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=0 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=7 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=3 2026-04-01T15:39:21.428 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.427+0000 7fdafe121900 20 remove_watcher() i=2 2026-04-01T15:39:21.433 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 4104311559 --totp-seed C3BCVBQW3LJ67DXYNFDC7EDVOQHCC2NODVVVHZV6NGHBFMT2DWSPKIA3CJT6LGNO --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:21.517 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:21.517 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:21.529 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.528+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.529 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.529 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 realm 2026-04-01T15:39:21.529 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.529 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:21.530 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.529+0000 7f8fe3123900 20 searching for the correct realm 2026-04-01T15:39:21.534 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:21.534 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 zone default found 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 4 Realm: () 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 zonegroup default 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.534+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.535+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.535+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.535+0000 7f8fe3123900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.535+0000 7f8fe3123900 20 started sync module instance, tier type = 2026-04-01T15:39:21.535 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.535+0000 7f8fe3123900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:21.539 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.537+0000 7f8fe3123900 20 add_watcher() i=4 2026-04-01T15:39:21.539 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.538+0000 7f8fe3123900 20 add_watcher() i=1 2026-04-01T15:39:21.539 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.538+0000 7f8fe3123900 20 add_watcher() i=6 2026-04-01T15:39:21.539 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.538+0000 7f8fe3123900 20 add_watcher() i=0 2026-04-01T15:39:21.540 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.539+0000 7f8fe3123900 20 add_watcher() i=3 2026-04-01T15:39:21.540 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.539+0000 7f8fe3123900 20 add_watcher() i=7 2026-04-01T15:39:21.540 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.539+0000 7f8fe3123900 20 add_watcher() i=2 2026-04-01T15:39:21.540 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.539+0000 7f8fe3123900 20 add_watcher() i=5 2026-04-01T15:39:21.540 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.539+0000 7f8fe3123900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:21.541 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.541+0000 7f8fe3123900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:21.541 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.541+0000 7f8fe3123900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:21.541 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.541+0000 7f8fe3123900 5 note: GC not initialized 2026-04-01T15:39:21.541 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.541+0000 7f8f877fe640 20 reqs_thread_entry: start 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.569+0000 7f8fe3123900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.569+0000 7f8fe3123900 20 Filter name: none 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8f857fa640 20 reqs_thread_entry: start 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 20 rados_obj.operate() r=0 bl.length=460 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.570+0000 7f8fe3123900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fe3123900 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fe3123900 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fe3123900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fbdffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051588 cookie 94621277265600 notifier 4293 bl.length()=659 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fbdffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2026-04-01T15:39:21.573 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.572+0000 7f8fbdffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.576+0000 7f8fe3123900 20 remove_watcher() i=0 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.576+0000 7f8fe3123900 2 removed watcher, disabling cache 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.576+0000 7f8fe3123900 20 remove_watcher() i=5 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.576+0000 7f8fe3123900 20 remove_watcher() i=1 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.576+0000 7f8fe3123900 20 remove_watcher() i=2 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.577+0000 7f8fe3123900 20 remove_watcher() i=4 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.577+0000 7f8fe3123900 20 remove_watcher() i=3 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.577+0000 7f8fe3123900 20 remove_watcher() i=7 2026-04-01T15:39:21.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.577+0000 7f8fe3123900 20 remove_watcher() i=6 2026-04-01T15:39:21.582 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2026-04-01T15:39:21.582 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr.testx$tenanteduser.client.0' --email 'testx$tenanteduser+test@test.test' --access-key MZGZEHDDEGJMHVGQWJSM --secret mCb73/hzLiKQrE4wrlDKZSpnqq96umqm06TPa4RJgEzj4Raa+VXrIA== --cluster ceph 2026-04-01T15:39:21.621 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:21.621 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:21.634 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.633+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.634 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.634 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 realm 2026-04-01T15:39:21.634 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.634 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.634+0000 7f0c72d6e900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:21.635 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.635+0000 7f0c72d6e900 20 searching for the correct realm 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.639+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.640 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 zone default found 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 4 Realm: () 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 zonegroup default 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.640+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.641+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.641+0000 7f0c72d6e900 20 started sync module instance, tier type = 2026-04-01T15:39:21.641 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.641+0000 7f0c72d6e900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:21.645 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.644+0000 7f0c72d6e900 20 add_watcher() i=1 2026-04-01T15:39:21.645 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.644+0000 7f0c72d6e900 20 add_watcher() i=2 2026-04-01T15:39:21.645 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=6 2026-04-01T15:39:21.645 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=5 2026-04-01T15:39:21.645 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=4 2026-04-01T15:39:21.646 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=3 2026-04-01T15:39:21.646 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=0 2026-04-01T15:39:21.646 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 20 add_watcher() i=7 2026-04-01T15:39:21.646 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.645+0000 7f0c72d6e900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:21.647 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.647+0000 7f0c72d6e900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:21.647 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.647+0000 7f0c72d6e900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:21.647 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.647+0000 7f0c72d6e900 5 note: GC not initialized 2026-04-01T15:39:21.647 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.647+0000 7f0c177fe640 20 reqs_thread_entry: start 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 Filter name: none 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c157fa640 20 reqs_thread_entry: start 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test : miss 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 adding default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 10 cache get: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM : miss 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.674+0000 7f0c72d6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.675+0000 7f0c72d6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.675+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM info.flags=0x0 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.675+0000 7f0c72d6e900 10 adding default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM to cache LRU end 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.675+0000 7f0c72d6e900 10 cache get: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM : hit (negative entry) 2026-04-01T15:39:21.675 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.675+0000 7f0c72d6e900 10 cache get: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM : hit (negative entry) 2026-04-01T15:39:21.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2026-04-01T15:39:21.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c72d6e900 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c72d6e900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2026-04-01T15:39:21.677 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c4dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051587 cookie 94507530440784 notifier 4300 bl.length()=719 2026-04-01T15:39:21.677 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c4dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2026-04-01T15:39:21.677 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.676+0000 7f0c4dffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x7 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c72d6e900 10 moving default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c72d6e900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser+test@test.test, ofs0, ns] 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c4e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051588 cookie 94507530422000 notifier 4300 bl.length()=215 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c4e7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x7 2026-04-01T15:39:21.678 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.678+0000 7f0c4e7fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c72d6e900 10 cache put: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM info.flags=0x7 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c72d6e900 10 moving default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM to cache LRU end 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c72d6e900 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:MZGZEHDDEGJMHVGQWJSM, ofs0, ns] 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c4dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051584 cookie 94507530433456 notifier 4300 bl.length()=201 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c4dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM info.flags=0x7 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.679+0000 7f0c4dffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+MZGZEHDDEGJMHVGQWJSM to cache LRU end 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "testx$tenanteduser.client.0", 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.testx$tenanteduser.client.0", 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "email": "testx$tenanteduser+test@test.test", 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "user": "testx$tenanteduser.client.0", 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "MZGZEHDDEGJMHVGQWJSM", 2026-04-01T15:39:21.680 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "mCb73/hzLiKQrE4wrlDKZSpnqq96umqm06TPa4RJgEzj4Raa+VXrIA==", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.675965Z" 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "type": "rgw", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.675958Z", 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:21.681 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:21.684 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.683+0000 7f0c72d6e900 20 remove_watcher() i=2 2026-04-01T15:39:21.684 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.683+0000 7f0c72d6e900 2 removed watcher, disabling cache 2026-04-01T15:39:21.684 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=3 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=4 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=0 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=5 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=1 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=6 2026-04-01T15:39:21.685 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.684+0000 7f0c72d6e900 20 remove_watcher() i=7 2026-04-01T15:39:21.691 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 9485882452 --totp-seed HUFGMATX3V6OO5BAF4CX6VXVHI6ZI4SETLBWYKAO6YX6UNYJ5VAWUOZKBACFH5AA --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:21.770 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:21.770 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:21.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.781+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 realm 2026-04-01T15:39:21.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:21.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.782+0000 7fdbf1d2f900 20 searching for the correct realm 2026-04-01T15:39:21.787 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:21.787 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 zone default found 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 4 Realm: () 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 zonegroup default 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.787+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.788 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.789 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.789 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 started sync module instance, tier type = 2026-04-01T15:39:21.789 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.788+0000 7fdbf1d2f900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:21.792 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=4 2026-04-01T15:39:21.792 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=0 2026-04-01T15:39:21.792 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=7 2026-04-01T15:39:21.792 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=5 2026-04-01T15:39:21.792 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=1 2026-04-01T15:39:21.793 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=3 2026-04-01T15:39:21.793 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=2 2026-04-01T15:39:21.793 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 20 add_watcher() i=6 2026-04-01T15:39:21.793 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.792+0000 7fdbf1d2f900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:21.795 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.794+0000 7fdbf1d2f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:21.795 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.794+0000 7fdbf1d2f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:21.795 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.794+0000 7fdbf1d2f900 5 note: GC not initialized 2026-04-01T15:39:21.795 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.794+0000 7fdb9afed640 20 reqs_thread_entry: start 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.822+0000 7fdbf1d2f900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.822+0000 7fdbf1d2f900 20 Filter name: none 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.822+0000 7fdb98fe9640 20 reqs_thread_entry: start 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.822+0000 7fdbf1d2f900 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.822+0000 7fdbf1d2f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.823+0000 7fdbf1d2f900 20 rados_obj.operate() r=0 bl.length=519 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.823+0000 7fdbf1d2f900 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.823+0000 7fdbf1d2f900 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.823+0000 7fdbf1d2f900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:21.825 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbf1d2f900 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2026-04-01T15:39:21.825 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbf1d2f900 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.825 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbf1d2f900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2026-04-01T15:39:21.826 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbccff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051589 cookie 94084201311600 notifier 4306 bl.length()=733 2026-04-01T15:39:21.826 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbccff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2026-04-01T15:39:21.826 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.825+0000 7fdbccff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.828+0000 7fdbf1d2f900 20 remove_watcher() i=0 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.828+0000 7fdbf1d2f900 2 removed watcher, disabling cache 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.828+0000 7fdbf1d2f900 20 remove_watcher() i=7 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=4 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=3 2026-04-01T15:39:21.829 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=2 2026-04-01T15:39:21.830 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=5 2026-04-01T15:39:21.830 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=6 2026-04-01T15:39:21.830 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.829+0000 7fdbf1d2f900 20 remove_watcher() i=1 2026-04-01T15:39:21.835 DEBUG:tasks.s3tests:Creating user foobar.client.0 on client.0 2026-04-01T15:39:21.835 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foobar.client.0 --display-name Mr.foobar.client.0 --email foobar+test@test.test --access-key RZFKORFICEAKVFPVQFDI --secret eh3/LZr4dlx2VI2L5Kcojx0J88NmUhFPAXpk8hvmgAt//z1B0eTvGw== --cluster ceph 2026-04-01T15:39:21.912 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:21.913 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.924+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 realm 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:21.925 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:21.926 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.925+0000 7fe955131900 20 searching for the correct realm 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.930+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 zone default found 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 4 Realm: () 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 zonegroup default 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.931 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.932 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.932 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 started sync module instance, tier type = 2026-04-01T15:39:21.932 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.931+0000 7fe955131900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:21.935 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.934+0000 7fe955131900 20 add_watcher() i=1 2026-04-01T15:39:21.935 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.934+0000 7fe955131900 20 add_watcher() i=0 2026-04-01T15:39:21.935 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=5 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=6 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=2 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=3 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=7 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 20 add_watcher() i=4 2026-04-01T15:39:21.936 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.935+0000 7fe955131900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:21.937 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.936+0000 7fe955131900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:21.937 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.936+0000 7fe955131900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:21.937 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.936+0000 7fe955131900 5 note: GC not initialized 2026-04-01T15:39:21.937 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.937+0000 7fe8fdfeb640 20 reqs_thread_entry: start 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.966+0000 7fe955131900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.966+0000 7fe955131900 20 Filter name: none 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.966+0000 7fe8f77fe640 20 reqs_thread_entry: start 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.966+0000 7fe955131900 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.966+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.967 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache get: name=default.rgw.meta+users.email+foobar+test@test.test : miss 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 adding default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache get: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI : miss 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 20 rados->read ofs=0 len=0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI info.flags=0x0 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 adding default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI to cache LRU end 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache get: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI : hit (negative entry) 2026-04-01T15:39:21.968 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.967+0000 7fe955131900 10 cache get: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI : hit (negative entry) 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe955131900 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe955131900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe927fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051590 cookie 94576194912672 notifier 4312 bl.length()=660 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe927fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:21.969 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.968+0000 7fe927fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:21.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x7 2026-04-01T15:39:21.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe955131900 10 moving default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2026-04-01T15:39:21.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe955131900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:foobar+test@test.test, ofs0, ns] 2026-04-01T15:39:21.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe9277fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051589 cookie 94576194883728 notifier 4312 bl.length()=191 2026-04-01T15:39:21.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe9277fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x7 2026-04-01T15:39:21.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.970+0000 7fe9277fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe955131900 10 cache put: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI info.flags=0x7 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe955131900 10 moving default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI to cache LRU end 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe955131900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:RZFKORFICEAKVFPVQFDI, ofs0, ns] 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe9277fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051585 cookie 94576194903504 notifier 4312 bl.length()=189 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe9277fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI info.flags=0x7 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.971+0000 7fe9277fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+RZFKORFICEAKVFPVQFDI to cache LRU end 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "foobar.client.0", 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.foobar.client.0", 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "email": "foobar+test@test.test", 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "user": "foobar.client.0", 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "RZFKORFICEAKVFPVQFDI", 2026-04-01T15:39:21.972 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "eh3/LZr4dlx2VI2L5Kcojx0J88NmUhFPAXpk8hvmgAt//z1B0eTvGw==", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.968246Z" 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "type": "rgw", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.968241Z", 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:21.973 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.975+0000 7fe955131900 20 remove_watcher() i=1 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.975+0000 7fe955131900 2 removed watcher, disabling cache 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=4 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=6 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=0 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=2 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=3 2026-04-01T15:39:21.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=5 2026-04-01T15:39:21.977 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:21.976+0000 7fe955131900 20 remove_watcher() i=7 2026-04-01T15:39:21.982 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foobar.client.0 --totp-serial 2446925840 --totp-seed JJXZNHR6V2R2OCBM6ZYO4NGCWUB6GPJVTEGO4UZ2NIH2T56YGB6XGZG5TWF3UREV --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:22.064 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.065 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.077 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.076+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 20 realm 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.077+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.078 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.078+0000 7f5509b6e900 20 searching for the correct realm 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.084 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 zone default found 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 4 Realm: () 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 zonegroup default 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.084+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.085+0000 7f5509b6e900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.085+0000 7f5509b6e900 20 started sync module instance, tier type = 2026-04-01T15:39:22.085 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.085+0000 7f5509b6e900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=3 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=2 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=0 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=5 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=1 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=6 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=7 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 20 add_watcher() i=4 2026-04-01T15:39:22.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.089+0000 7f5509b6e900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.091+0000 7f5509b6e900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.091+0000 7f5509b6e900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.091+0000 7f5509b6e900 5 note: GC not initialized 2026-04-01T15:39:22.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.091+0000 7f54b2fed640 20 reqs_thread_entry: start 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 20 Filter name: none 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f54b0fe9640 20 reqs_thread_entry: start 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 20 rados_obj.operate() r=0 bl.length=472 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.125+0000 7f5509b6e900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:22.128 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f5509b6e900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:22.128 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f5509b6e900 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.128 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f5509b6e900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2026-04-01T15:39:22.129 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f54e4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051591 cookie 94842302122592 notifier 4318 bl.length()=674 2026-04-01T15:39:22.129 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f54e4ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:22.129 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.128+0000 7f54e4ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.132 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.131+0000 7f5509b6e900 20 remove_watcher() i=2 2026-04-01T15:39:22.132 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.131+0000 7f5509b6e900 2 removed watcher, disabling cache 2026-04-01T15:39:22.132 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.131+0000 7f5509b6e900 20 remove_watcher() i=5 2026-04-01T15:39:22.132 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=7 2026-04-01T15:39:22.132 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=0 2026-04-01T15:39:22.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=1 2026-04-01T15:39:22.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=6 2026-04-01T15:39:22.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=4 2026-04-01T15:39:22.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.132+0000 7f5509b6e900 20 remove_watcher() i=3 2026-04-01T15:39:22.137 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'oidc-provider=*;roles=*;user-policy=*' --cluster ceph 2026-04-01T15:39:22.178 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.178 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.193+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.193+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.193+0000 7f8ef1521900 20 realm 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.194 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.195 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.194+0000 7f8ef1521900 20 searching for the correct realm 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.199+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.200 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 zone default found 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 4 Realm: () 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 zonegroup default 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.200+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.201+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.201+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.201+0000 7f8ef1521900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.201+0000 7f8ef1521900 20 started sync module instance, tier type = 2026-04-01T15:39:22.201 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.201+0000 7f8ef1521900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.204+0000 7f8ef1521900 20 add_watcher() i=2 2026-04-01T15:39:22.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.205+0000 7f8ef1521900 20 add_watcher() i=1 2026-04-01T15:39:22.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.205+0000 7f8ef1521900 20 add_watcher() i=0 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.205+0000 7f8ef1521900 20 add_watcher() i=5 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.205+0000 7f8ef1521900 20 add_watcher() i=4 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.205+0000 7f8ef1521900 20 add_watcher() i=6 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.206+0000 7f8ef1521900 20 add_watcher() i=3 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.206+0000 7f8ef1521900 20 add_watcher() i=7 2026-04-01T15:39:22.206 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.206+0000 7f8ef1521900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.207+0000 7f8ef1521900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.207+0000 7f8ef1521900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.207+0000 7f8ef1521900 5 note: GC not initialized 2026-04-01T15:39:22.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.208+0000 7f8e9a7ec640 20 reqs_thread_entry: start 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.253+0000 7f8ef1521900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.253+0000 7f8ef1521900 20 Filter name: none 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.253+0000 7f8e93fff640 20 reqs_thread_entry: start 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 20 rados_obj.operate() r=0 bl.length=486 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.254 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.254+0000 7f8ef1521900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ef1521900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ef1521900 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ef1521900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ec3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018888 cookie 93977007354080 notifier 4325 bl.length()=727 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ec3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.255+0000 7f8ec3fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "foobar.client.0", 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.foobar.client.0", 2026-04-01T15:39:22.256 INFO:teuthology.orchestra.run.vm07.stdout: "email": "foobar+test@test.test", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "user": "foobar.client.0", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "RZFKORFICEAKVFPVQFDI", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "eh3/LZr4dlx2VI2L5Kcojx0J88NmUhFPAXpk8hvmgAt//z1B0eTvGw==", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.968246Z" 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [ 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "type": "oidc-provider", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "perm": "*" 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "type": "roles", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "perm": "*" 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "type": "user-policy", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "perm": "*" 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "type": "rgw", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [ 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "2446925840" 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:21.968241Z", 2026-04-01T15:39:22.257 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:22.258 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:22.258 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:22.258 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.260+0000 7f8ef1521900 20 remove_watcher() i=0 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.260+0000 7f8ef1521900 2 removed watcher, disabling cache 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.260+0000 7f8ef1521900 20 remove_watcher() i=1 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=4 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=2 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=5 2026-04-01T15:39:22.261 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=6 2026-04-01T15:39:22.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=3 2026-04-01T15:39:22.262 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.261+0000 7f8ef1521900 20 remove_watcher() i=7 2026-04-01T15:39:22.267 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph account create --account-id RGW88888888888888888 --account-name Mr.RGW88888888888888888 --email RGW88888888888888888+test@test.test 2026-04-01T15:39:22.343 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.343 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.357 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 realm 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.357+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.358 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.358+0000 7f128eee6900 20 searching for the correct realm 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.363+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 zone default found 2026-04-01T15:39:22.364 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 4 Realm: () 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 zonegroup default 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.364+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.365+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.365+0000 7f128eee6900 20 started sync module instance, tier type = 2026-04-01T15:39:22.365 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.365+0000 7f128eee6900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.369 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.368+0000 7f128eee6900 20 add_watcher() i=2 2026-04-01T15:39:22.369 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.368+0000 7f128eee6900 20 add_watcher() i=0 2026-04-01T15:39:22.369 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=4 2026-04-01T15:39:22.369 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=7 2026-04-01T15:39:22.370 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=3 2026-04-01T15:39:22.370 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=5 2026-04-01T15:39:22.370 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=1 2026-04-01T15:39:22.370 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 20 add_watcher() i=6 2026-04-01T15:39:22.370 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.369+0000 7f128eee6900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.371 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.371+0000 7f128eee6900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.371 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.371+0000 7f128eee6900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.372 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.371+0000 7f128eee6900 5 note: GC not initialized 2026-04-01T15:39:22.372 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.371+0000 7f1239feb640 20 reqs_thread_entry: start 2026-04-01T15:39:22.401 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.401+0000 7f128eee6900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.401+0000 7f128eee6900 20 Filter name: none 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.401+0000 7f12337fe640 20 reqs_thread_entry: start 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.401+0000 7f128eee6900 10 cache get: name=default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 : miss 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.401+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 10 cache put: name=default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 info.flags=0x0 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 10 adding default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 20 failed to read name.$Mr.RGW88888888888888888 with: (2) No such file or directory 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 10 cache get: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test : miss 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.402 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 10 cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x0 2026-04-01T15:39:22.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 10 adding default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2026-04-01T15:39:22.403 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.402+0000 7f128eee6900 20 failed to read rgw88888888888888888+test@test.test with: (2) No such file or directory 2026-04-01T15:39:22.404 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.403+0000 7f128eee6900 10 cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2026-04-01T15:39:22.404 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.403+0000 7f128eee6900 10 adding default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.404 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.403+0000 7f128eee6900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:accounts:account.RGW88888888888888888, ofs0, ns] 2026-04-01T15:39:22.405 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.404+0000 7f12637fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018886 cookie 94341238734928 notifier 4331 bl.length()=384 2026-04-01T15:39:22.405 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.404+0000 7f12637fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2026-04-01T15:39:22.405 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.404+0000 7f12637fe640 10 rgw watcher librados: moving default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.405+0000 7f128eee6900 10 cache put: name=default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 info.flags=0x17 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.405+0000 7f128eee6900 10 moving default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.405+0000 7f128eee6900 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:accounts:name.$Mr.RGW88888888888888888, ofs0, ns] 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.406+0000 7f1263fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018887 cookie 94341238715216 notifier 4331 bl.length()=225 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.406+0000 7f1263fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 info.flags=0x17 2026-04-01T15:39:22.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.406+0000 7f1263fff640 10 rgw watcher librados: moving default.rgw.meta+accounts+name.$Mr.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.407+0000 7f128eee6900 10 cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x17 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.407+0000 7f128eee6900 10 moving default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.407+0000 7f128eee6900 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:rgw88888888888888888+test@test.test, ofs0, ns] 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.408+0000 7f12637fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94341238718272 notifier 4331 bl.length()=234 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.408+0000 7f12637fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x17 2026-04-01T15:39:22.408 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.408+0000 7f12637fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2026-04-01T15:39:22.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=0 2026-04-01T15:39:22.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 2 removed watcher, disabling cache 2026-04-01T15:39:22.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=5 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=2 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=4 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=1 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=6 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=3 2026-04-01T15:39:22.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.412+0000 7f128eee6900 20 remove_watcher() i=7 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "id": "RGW88888888888888888", 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "tenant": "", 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "name": "Mr.RGW88888888888888888", 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "email": "RGW88888888888888888+test@test.test", 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "quota": { 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_users": 1000, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_roles": 1000, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_groups": 1000, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout: "max_access_keys": 4 2026-04-01T15:39:22.416 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:22.418 DEBUG:tasks.s3tests:Creating user RGW88888888888888888 on client.0 2026-04-01T15:39:22.418 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid root1.client.0 --display-name Mr.RGW88888888888888888 --email root1+test@test.test --access-key POMSSXZFMOTDNYCCGIME --secret lviVb9XAG7PbnzLjvgqxObxqKpbmsj1ScjgL0XV1TpijVR8KMqsNNg== --cluster ceph --account-id RGW88888888888888888 --account-root 2026-04-01T15:39:22.498 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.498 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.510 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.509+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.510 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.510 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 realm 2026-04-01T15:39:22.510 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.510+0000 7fea0340f900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.511 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.511+0000 7fea0340f900 20 searching for the correct realm 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.515 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 zone default found 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 4 Realm: () 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.515+0000 7fea0340f900 20 zonegroup default 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.516 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.517 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.517 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 started sync module instance, tier type = 2026-04-01T15:39:22.517 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.516+0000 7fea0340f900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.521 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.520+0000 7fea0340f900 20 add_watcher() i=0 2026-04-01T15:39:22.521 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.520+0000 7fea0340f900 20 add_watcher() i=1 2026-04-01T15:39:22.521 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.520+0000 7fea0340f900 20 add_watcher() i=4 2026-04-01T15:39:22.521 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.520+0000 7fea0340f900 20 add_watcher() i=5 2026-04-01T15:39:22.521 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.521+0000 7fea0340f900 20 add_watcher() i=6 2026-04-01T15:39:22.522 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.521+0000 7fea0340f900 20 add_watcher() i=3 2026-04-01T15:39:22.522 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.521+0000 7fea0340f900 20 add_watcher() i=7 2026-04-01T15:39:22.522 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.521+0000 7fea0340f900 20 add_watcher() i=2 2026-04-01T15:39:22.522 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.521+0000 7fea0340f900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.523 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.523+0000 7fea0340f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.523 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.523+0000 7fea0340f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.524 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.523+0000 7fea0340f900 5 note: GC not initialized 2026-04-01T15:39:22.524 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.523+0000 7fe9a77fe640 20 reqs_thread_entry: start 2026-04-01T15:39:22.551 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.551+0000 7fea0340f900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.551+0000 7fea0340f900 20 Filter name: none 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.551+0000 7fe9a57fa640 20 reqs_thread_entry: start 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.551+0000 7fea0340f900 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.551+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache get: name=default.rgw.meta+users.email+root1+test@test.test : miss 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x0 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 adding default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2026-04-01T15:39:22.552 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache get: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME : miss 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME info.flags=0x0 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 adding default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME to cache LRU end 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 10 cache get: name=default.rgw.meta+accounts+account.RGW88888888888888888 : miss 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.552+0000 7fea0340f900 20 rados_obj.operate() r=0 bl.length=184 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.553+0000 7fea0340f900 10 cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.553+0000 7fea0340f900 10 adding default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.553+0000 7fea0340f900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.553+0000 7fea0340f900 10 cache get: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME : hit (negative entry) 2026-04-01T15:39:22.553 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.553+0000 7fea0340f900 10 cache get: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME : hit (negative entry) 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fea0340f900 10 moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fea0340f900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root1.client.0, ofs0, ns] 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fe9de7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018888 cookie 94716833096368 notifier 4338 bl.length()=681 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fe9de7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2026-04-01T15:39:22.555 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.554+0000 7fe9de7fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.556 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x7 2026-04-01T15:39:22.556 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fea0340f900 10 moving default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2026-04-01T15:39:22.556 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fea0340f900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:root1+test@test.test, ofs0, ns] 2026-04-01T15:39:22.557 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fe9de7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018889 cookie 94716833090896 notifier 4338 bl.length()=189 2026-04-01T15:39:22.557 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fe9de7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x7 2026-04-01T15:39:22.557 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.556+0000 7fe9de7fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.557+0000 7fea0340f900 10 cache put: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME info.flags=0x7 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.557+0000 7fea0340f900 10 moving default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME to cache LRU end 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.557+0000 7fea0340f900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:POMSSXZFMOTDNYCCGIME, ofs0, ns] 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.558+0000 7fe9de7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94716833094096 notifier 4338 bl.length()=188 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.558+0000 7fe9de7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME info.flags=0x7 2026-04-01T15:39:22.558 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.558+0000 7fe9de7fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+POMSSXZFMOTDNYCCGIME to cache LRU end 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.559+0000 7fea0340f900 20 linked user root1.client.0 to account RGW88888888888888888 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "root1.client.0", 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.RGW88888888888888888", 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "email": "root1+test@test.test", 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:22.559 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "user": "root1.client.0", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "POMSSXZFMOTDNYCCGIME", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "lviVb9XAG7PbnzLjvgqxObxqKpbmsj1ScjgL0XV1TpijVR8KMqsNNg==", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:22.553981Z" 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "type": "root", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "RGW88888888888888888", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:22.553947Z", 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:22.560 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:22.563 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=2 2026-04-01T15:39:22.563 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 2 removed watcher, disabling cache 2026-04-01T15:39:22.563 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=0 2026-04-01T15:39:22.563 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=5 2026-04-01T15:39:22.563 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=3 2026-04-01T15:39:22.564 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=1 2026-04-01T15:39:22.564 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=4 2026-04-01T15:39:22.564 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=6 2026-04-01T15:39:22.564 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.563+0000 7fea0340f900 20 remove_watcher() i=7 2026-04-01T15:39:22.568 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid root1.client.0 --totp-serial 9309597297 --totp-seed RLX7VGPR4INMCQAAWO73JJQ7LUS7FM5SNSNQJC4S7G4QMLS7S3OFP5J6K4VI742G --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:22.648 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.648 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.660 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 20 realm 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.660+0000 7f8b6ad4b900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.661 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.661+0000 7f8b6ad4b900 20 searching for the correct realm 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.667+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.668 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 zone default found 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 4 Realm: () 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 zonegroup default 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.668+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.669 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.670 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.670 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 started sync module instance, tier type = 2026-04-01T15:39:22.670 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.669+0000 7f8b6ad4b900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.673 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.673+0000 7f8b6ad4b900 20 add_watcher() i=1 2026-04-01T15:39:22.673 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.673+0000 7f8b6ad4b900 20 add_watcher() i=5 2026-04-01T15:39:22.673 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.673+0000 7f8b6ad4b900 20 add_watcher() i=0 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.673+0000 7f8b6ad4b900 20 add_watcher() i=7 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.673+0000 7f8b6ad4b900 20 add_watcher() i=6 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.674+0000 7f8b6ad4b900 20 add_watcher() i=4 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.674+0000 7f8b6ad4b900 20 add_watcher() i=3 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.674+0000 7f8b6ad4b900 20 add_watcher() i=2 2026-04-01T15:39:22.674 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.674+0000 7f8b6ad4b900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.675+0000 7f8b6ad4b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.675+0000 7f8b6ad4b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.675+0000 7f8b6ad4b900 5 note: GC not initialized 2026-04-01T15:39:22.676 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.676+0000 7f8b0f7fe640 20 reqs_thread_entry: start 2026-04-01T15:39:22.703 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.703+0000 7f8b6ad4b900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.703 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.703+0000 7f8b6ad4b900 20 Filter name: none 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.703+0000 7f8b0d7fa640 20 reqs_thread_entry: start 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.703+0000 7f8b6ad4b900 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.703+0000 7f8b6ad4b900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.704+0000 7f8b6ad4b900 20 rados_obj.operate() r=0 bl.length=494 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.704+0000 7f8b6ad4b900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x13 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.704+0000 7f8b6ad4b900 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.704 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.704+0000 7f8b6ad4b900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:22.706 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b6ad4b900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2026-04-01T15:39:22.706 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b6ad4b900 10 moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.707 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b6ad4b900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root1.client.0, ofs0, ns] 2026-04-01T15:39:22.707 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b45ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018889 cookie 94710068676784 notifier 4345 bl.length()=695 2026-04-01T15:39:22.707 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b45ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2026-04-01T15:39:22.707 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.706+0000 7f8b45ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 20 remove_watcher() i=3 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 2 removed watcher, disabling cache 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 20 remove_watcher() i=2 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 20 remove_watcher() i=4 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 20 remove_watcher() i=0 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.710+0000 7f8b6ad4b900 20 remove_watcher() i=6 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.711+0000 7f8b6ad4b900 20 remove_watcher() i=1 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.711+0000 7f8b6ad4b900 20 remove_watcher() i=5 2026-04-01T15:39:22.711 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.711+0000 7f8b6ad4b900 20 remove_watcher() i=7 2026-04-01T15:39:22.716 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph account create --account-id RGW99999999999999999 --account-name Mr.RGW99999999999999999 --email RGW99999999999999999+test@test.test 2026-04-01T15:39:22.795 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.795 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.808+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 realm 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.809 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.810 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.809+0000 7eff7bf67900 20 searching for the correct realm 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.815 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 zone default found 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 4 Realm: () 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 zonegroup default 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.815+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.816+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.816+0000 7eff7bf67900 20 started sync module instance, tier type = 2026-04-01T15:39:22.816 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.816+0000 7eff7bf67900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.819 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.819+0000 7eff7bf67900 20 add_watcher() i=1 2026-04-01T15:39:22.819 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.819+0000 7eff7bf67900 20 add_watcher() i=6 2026-04-01T15:39:22.820 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.820+0000 7eff7bf67900 20 add_watcher() i=7 2026-04-01T15:39:22.820 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.820+0000 7eff7bf67900 20 add_watcher() i=4 2026-04-01T15:39:22.820 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.820+0000 7eff7bf67900 20 add_watcher() i=5 2026-04-01T15:39:22.821 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.820+0000 7eff7bf67900 20 add_watcher() i=0 2026-04-01T15:39:22.821 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.821+0000 7eff7bf67900 20 add_watcher() i=2 2026-04-01T15:39:22.821 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.821+0000 7eff7bf67900 20 add_watcher() i=3 2026-04-01T15:39:22.821 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.821+0000 7eff7bf67900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.823+0000 7eff7bf67900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.823+0000 7eff7bf67900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.823+0000 7eff7bf67900 5 note: GC not initialized 2026-04-01T15:39:22.823 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.823+0000 7eff1ffff640 20 reqs_thread_entry: start 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.851+0000 7eff7bf67900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.851+0000 7eff7bf67900 20 Filter name: none 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.851+0000 7eff1dffb640 20 reqs_thread_entry: start 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.851+0000 7eff7bf67900 10 cache get: name=default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 : miss 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.851+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 10 cache put: name=default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 info.flags=0x0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 10 adding default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 20 failed to read name.$Mr.RGW99999999999999999 with: (2) No such file or directory 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 10 cache get: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test : miss 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 10 cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x0 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 10 adding default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2026-04-01T15:39:22.852 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.852+0000 7eff7bf67900 20 failed to read rgw99999999999999999+test@test.test with: (2) No such file or directory 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.853+0000 7eff7bf67900 10 cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.853+0000 7eff7bf67900 10 adding default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.853+0000 7eff7bf67900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:accounts:account.RGW99999999999999999, ofs0, ns] 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.853+0000 7eff56ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018890 cookie 94829508478880 notifier 4352 bl.length()=384 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.854+0000 7eff56ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2026-04-01T15:39:22.854 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.854+0000 7eff56ffd640 10 rgw watcher librados: moving default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:22.855 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff7bf67900 10 cache put: name=default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 info.flags=0x17 2026-04-01T15:39:22.855 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff7bf67900 10 moving default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:22.855 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff7bf67900 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:accounts:name.$Mr.RGW99999999999999999, ofs0, ns] 2026-04-01T15:39:22.856 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff56ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018891 cookie 94829508494080 notifier 4352 bl.length()=225 2026-04-01T15:39:22.856 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff56ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 info.flags=0x17 2026-04-01T15:39:22.856 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.855+0000 7eff56ffd640 10 rgw watcher librados: moving default.rgw.meta+accounts+name.$Mr.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:22.857 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff7bf67900 10 cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x17 2026-04-01T15:39:22.857 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff7bf67900 10 moving default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2026-04-01T15:39:22.857 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff7bf67900 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:rgw99999999999999999+test@test.test, ofs0, ns] 2026-04-01T15:39:22.857 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff567fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94829508497056 notifier 4352 bl.length()=234 2026-04-01T15:39:22.858 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff567fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x17 2026-04-01T15:39:22.858 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.857+0000 7eff567fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2026-04-01T15:39:22.862 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.861+0000 7eff7bf67900 20 remove_watcher() i=0 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 2 removed watcher, disabling cache 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=1 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=3 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=5 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=6 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=2 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.862+0000 7eff7bf67900 20 remove_watcher() i=4 2026-04-01T15:39:22.863 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.863+0000 7eff7bf67900 20 remove_watcher() i=7 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "id": "RGW99999999999999999", 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "tenant": "", 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "name": "Mr.RGW99999999999999999", 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "email": "RGW99999999999999999+test@test.test", 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "quota": { 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.866 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_users": 1000, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_roles": 1000, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_groups": 1000, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout: "max_access_keys": 4 2026-04-01T15:39:22.867 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:22.869 DEBUG:tasks.s3tests:Creating user RGW99999999999999999 on client.0 2026-04-01T15:39:22.869 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid root2.client.0 --display-name Mr.RGW99999999999999999 --email root2+test@test.test --access-key DDGCSEXNYDLFPZMJDUQP --secret 0x4kfomXVK7BCAlWp9aMvNxeHCpFoZ4D4vqdkeO6IIToV3xs4aJsCg== --cluster ceph --account-id RGW99999999999999999 --account-root 2026-04-01T15:39:22.950 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:22.950 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:22.964 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.963+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.964 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.964 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 realm 2026-04-01T15:39:22.964 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.964+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.965+0000 7f2a00dcb900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:22.965 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.965+0000 7f2a00dcb900 20 searching for the correct realm 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.970 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 zone default found 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 4 Realm: () 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 zonegroup default 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.970+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.971+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.971+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.971+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.971+0000 7f2a00dcb900 20 started sync module instance, tier type = 2026-04-01T15:39:22.971 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.971+0000 7f2a00dcb900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:22.975 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=1 2026-04-01T15:39:22.975 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=4 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=0 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=6 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=2 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=7 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=3 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 20 add_watcher() i=5 2026-04-01T15:39:22.976 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.975+0000 7f2a00dcb900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:22.978 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.977+0000 7f2a00dcb900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:22.978 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.977+0000 7f2a00dcb900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:22.978 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.977+0000 7f2a00dcb900 5 note: GC not initialized 2026-04-01T15:39:22.978 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:22.978+0000 7f29a8fe9640 20 reqs_thread_entry: start 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.010+0000 7f2a00dcb900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.010+0000 7f2a00dcb900 20 Filter name: none 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.010+0000 7f29a27fc640 20 reqs_thread_entry: start 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.010+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+users.uid+root2.client.0 : miss 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.010+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 adding default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+users.email+root2+test@test.test : miss 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x0 2026-04-01T15:39:23.011 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 adding default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP : miss 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP info.flags=0x0 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 adding default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP to cache LRU end 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+accounts+account.RGW99999999999999999 : miss 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 20 rados_obj.operate() r=0 bl.length=184 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 adding default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP : hit (negative entry) 2026-04-01T15:39:23.012 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.011+0000 7f2a00dcb900 10 cache get: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP : hit (negative entry) 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f2a00dcb900 10 moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f2a00dcb900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root2.client.0, ofs0, ns] 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f29d67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018892 cookie 94079559170848 notifier 4359 bl.length()=681 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f29d67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2026-04-01T15:39:23.013 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.013+0000 7f29d67fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.014+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x7 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.014+0000 7f2a00dcb900 10 moving default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.014+0000 7f2a00dcb900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:root2+test@test.test, ofs0, ns] 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.015+0000 7f29d67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018890 cookie 94079559182912 notifier 4359 bl.length()=189 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.015+0000 7f29d67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x7 2026-04-01T15:39:23.015 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.015+0000 7f29d67fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2026-04-01T15:39:23.016 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f2a00dcb900 10 cache put: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP info.flags=0x7 2026-04-01T15:39:23.016 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f2a00dcb900 10 moving default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP to cache LRU end 2026-04-01T15:39:23.016 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f2a00dcb900 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:DDGCSEXNYDLFPZMJDUQP, ofs0, ns] 2026-04-01T15:39:23.017 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f29d6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94079559188944 notifier 4359 bl.length()=188 2026-04-01T15:39:23.017 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f29d6ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP info.flags=0x7 2026-04-01T15:39:23.017 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.016+0000 7f29d6ffd640 10 rgw watcher librados: moving default.rgw.meta+users.keys+DDGCSEXNYDLFPZMJDUQP to cache LRU end 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.017+0000 7f2a00dcb900 20 linked user root2.client.0 to account RGW99999999999999999 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout:{ 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "user_id": "root2.client.0", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "display_name": "Mr.RGW99999999999999999", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "email": "root2+test@test.test", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "suspended": 0, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_buckets": 1000, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "subusers": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "keys": [ 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: { 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "user": "root2.client.0", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "access_key": "DDGCSEXNYDLFPZMJDUQP", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "secret_key": "0x4kfomXVK7BCAlWp9aMvNxeHCpFoZ4D4vqdkeO6IIToV3xs4aJsCg==", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "active": true, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:23.012591Z" 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: } 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: ], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "swift_keys": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "caps": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "op_mask": "read, write, delete", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "default_placement": "", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "default_storage_class": "", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "placement_tags": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "bucket_quota": { 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "user_quota": { 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "enabled": false, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "check_on_raw": false, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_size": -1, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_size_kb": 0, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "max_objects": -1 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: }, 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "temp_url_keys": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "type": "root", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "mfa_ids": [], 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "account_id": "RGW99999999999999999", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "path": "/", 2026-04-01T15:39:23.018 INFO:teuthology.orchestra.run.vm07.stdout: "create_date": "2026-04-01T15:39:23.012586Z", 2026-04-01T15:39:23.019 INFO:teuthology.orchestra.run.vm07.stdout: "tags": [], 2026-04-01T15:39:23.019 INFO:teuthology.orchestra.run.vm07.stdout: "group_ids": [] 2026-04-01T15:39:23.019 INFO:teuthology.orchestra.run.vm07.stdout:} 2026-04-01T15:39:23.019 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:23.021 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=3 2026-04-01T15:39:23.021 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 2 removed watcher, disabling cache 2026-04-01T15:39:23.021 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=2 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=4 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=7 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=6 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=1 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=5 2026-04-01T15:39:23.022 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.021+0000 7f2a00dcb900 20 remove_watcher() i=0 2026-04-01T15:39:23.027 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid root2.client.0 --totp-serial 3292604076 --totp-seed RYGLMXNR4BOJXYRXHVZ3GYMGBSFETNFVYAN34GEBCPSPMBGZPFLDDVK3GMCMP7QG --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2026-04-01T15:39:23.110 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:39:23.110 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:39:23.125 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.125+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.125+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.125+0000 7f05905a3900 20 realm 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.125+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:23.126 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.127 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:39:23.127 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.126+0000 7f05905a3900 20 searching for the correct realm 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.132+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 zone default found 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 4 Realm: () 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 zonegroup default 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.133 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.134 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.134 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.134 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:39:23.134 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 started sync module instance, tier type = 2026-04-01T15:39:23.134 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.133+0000 7f05905a3900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:39:23.138 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.137+0000 7f05905a3900 20 add_watcher() i=4 2026-04-01T15:39:23.138 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.138+0000 7f05905a3900 20 add_watcher() i=1 2026-04-01T15:39:23.138 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.138+0000 7f05905a3900 20 add_watcher() i=0 2026-04-01T15:39:23.138 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.138+0000 7f05905a3900 20 add_watcher() i=3 2026-04-01T15:39:23.139 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.138+0000 7f05905a3900 20 add_watcher() i=2 2026-04-01T15:39:23.139 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.138+0000 7f05905a3900 20 add_watcher() i=6 2026-04-01T15:39:23.139 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.139+0000 7f05905a3900 20 add_watcher() i=5 2026-04-01T15:39:23.139 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.139+0000 7f05905a3900 20 add_watcher() i=7 2026-04-01T15:39:23.139 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.139+0000 7f05905a3900 2 all 8 watchers are set, enabling cache 2026-04-01T15:39:23.141 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.140+0000 7f05905a3900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:39:23.141 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.140+0000 7f05905a3900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:39:23.141 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.141+0000 7f05905a3900 5 note: GC not initialized 2026-04-01T15:39:23.141 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.141+0000 7f0539feb640 20 reqs_thread_entry: start 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.204+0000 7f05905a3900 20 init_complete bucket index max shards: 11 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.204+0000 7f05905a3900 20 Filter name: none 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.204+0000 7f0532ffd640 20 reqs_thread_entry: start 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 10 cache get: name=default.rgw.meta+users.uid+root2.client.0 : miss 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 20 rados->read ofs=0 len=0 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 20 rados_obj.operate() r=0 bl.length=494 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x13 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 10 adding default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.205 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.205+0000 7f05905a3900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05905a3900 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05905a3900 10 moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05905a3900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root2.client.0, ofs0, ns] 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05677fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018893 cookie 94110192286848 notifier 4366 bl.length()=695 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05677fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2026-04-01T15:39:23.208 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.207+0000 7f05677fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.211+0000 7f05905a3900 20 remove_watcher() i=1 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.211+0000 7f05905a3900 2 removed watcher, disabling cache 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.211+0000 7f05905a3900 20 remove_watcher() i=5 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=4 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=0 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=7 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=3 2026-04-01T15:39:23.212 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=2 2026-04-01T15:39:23.213 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:39:23.212+0000 7f05905a3900 20 remove_watcher() i=6 2026-04-01T15:39:23.218 INFO:tasks.s3tests:Configuring s3-tests... 2026-04-01T15:39:23.218 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:39:23.218 DEBUG:teuthology.orchestra.run.vm07:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2026-04-01T15:39:23.235 INFO:tasks.s3tests:Configuring boto... 2026-04-01T15:39:23.235 DEBUG:teuthology.orchestra.run.vm07:> set -ex 2026-04-01T15:39:23.235 DEBUG:teuthology.orchestra.run.vm07:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2026-04-01T15:39:23.293 DEBUG:teuthology.orchestra.run.vm07:s3 tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not bucket_logging' 2026-04-01T15:39:23.632 INFO:teuthology.orchestra.run.vm07.stdout:py: install_deps> python -I -m pip install -r requirements.txt 2026-04-01T15:39:29.033 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' 2026-04-01T15:39:29.438 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: _optional_hooks> python /home/ubuntu/cephtest/tox-venv/lib64/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ 2026-04-01T15:39:29.598 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: get_requires_for_build_sdist> python /home/ubuntu/cephtest/tox-venv/lib64/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ 2026-04-01T15:39:29.725 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: get_requires_for_build_wheel> python /home/ubuntu/cephtest/tox-venv/lib64/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ 2026-04-01T15:39:29.837 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: prepare_metadata_for_build_wheel> python /home/ubuntu/cephtest/tox-venv/lib64/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ 2026-04-01T15:39:29.946 INFO:teuthology.orchestra.run.vm07.stdout:.pkg: build_sdist> python /home/ubuntu/cephtest/tox-venv/lib64/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ 2026-04-01T15:39:30.101 INFO:teuthology.orchestra.run.vm07.stdout:py: install_package_deps> python -I -m pip install PyYAML 'boto3>=1.0.0' 'gevent>=1.0' 'isodate>=0.4.4' 'munch>=2.0.0' 2026-04-01T15:39:30.298 INFO:teuthology.orchestra.run.vm07.stdout:py: install_package> python -I -m pip install --force-reinstall --no-deps /home/ubuntu/cephtest/s3-tests-client.0/.tox/.tmp/package/1/s3tests-0.0.1.tar.gz 2026-04-01T15:39:31.449 INFO:teuthology.orchestra.run.vm07.stdout:py: commands[0]> pytest -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not bucket_logging' 2026-04-01T15:39:31.533 INFO:teuthology.orchestra.run.vm07.stdout:============================= test session starts ============================== 2026-04-01T15:39:31.533 INFO:teuthology.orchestra.run.vm07.stdout:platform linux -- Python 3.9.23, pytest-8.4.2, pluggy-1.6.0 -- /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/bin/python 2026-04-01T15:39:31.533 INFO:teuthology.orchestra.run.vm07.stdout:cachedir: .tox/py/.pytest_cache 2026-04-01T15:39:31.533 INFO:teuthology.orchestra.run.vm07.stdout:rootdir: /home/ubuntu/cephtest/s3-tests-client.0 2026-04-01T15:39:31.533 INFO:teuthology.orchestra.run.vm07.stdout:configfile: pytest.ini 2026-04-01T15:39:32.794 INFO:teuthology.orchestra.run.vm07.stdout:collecting ... collected 893 items / 211 deselected / 682 selected 2026-04-01T15:39:32.794 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:39:32.993 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_short PASSED [ 0%] 2026-04-01T15:39:35.852 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_bad PASSED [ 0%] 2026-04-01T15:39:36.023 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_empty PASSED [ 0%] 2026-04-01T15:39:36.204 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_none PASSED [ 0%] 2026-04-01T15:39:36.385 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_expect_mismatch PASSED [ 0%] 2026-04-01T15:39:36.561 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_expect_empty PASSED [ 0%] 2026-04-01T15:39:36.769 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_expect_none PASSED [ 1%] 2026-04-01T15:39:36.897 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contentlength_negative PASSED [ 1%] 2026-04-01T15:39:37.081 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contenttype_invalid PASSED [ 1%] 2026-04-01T15:39:37.287 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contenttype_empty PASSED [ 1%] 2026-04-01T15:39:37.456 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contenttype_none PASSED [ 1%] 2026-04-01T15:39:37.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_put_bad_canned_acl PASSED [ 1%] 2026-04-01T15:39:37.738 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_expect_mismatch PASSED [ 1%] 2026-04-01T15:39:37.853 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_expect_empty PASSED [ 2%] 2026-04-01T15:39:37.900 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_contentlength_negative PASSED [ 2%] 2026-04-01T15:39:38.027 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws2 PASSED [ 2%] 2026-04-01T15:39:38.243 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws2 PASSED [ 2%] 2026-04-01T15:39:38.424 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws2 PASSED [ 2%] 2026-04-01T15:39:38.547 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws2 PASSED [ 2%] 2026-04-01T15:39:38.668 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws2 PASSED [ 2%] 2026-04-01T15:39:38.829 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws2 PASSED [ 3%] 2026-04-01T15:39:38.957 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws2 PASSED [ 3%] 2026-04-01T15:39:39.072 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws2 PASSED [ 3%] 2026-04-01T15:39:39.153 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws2 PASSED [ 3%] 2026-04-01T15:39:39.201 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws2 PASSED [ 3%] 2026-04-01T15:39:39.247 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws2 PASSED [ 3%] 2026-04-01T15:39:39.289 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws2 PASSED [ 3%] 2026-04-01T15:39:39.373 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws2 PASSED [ 4%] 2026-04-01T15:39:39.471 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_put_user_policy PASSED [ 4%] 2026-04-01T15:39:39.519 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_put_user_policy_invalid_user PASSED [ 4%] 2026-04-01T15:39:39.568 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_put_user_policy_parameter_limit PASSED [ 4%] 2026-04-01T15:39:39.751 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_put_existing_user_policy PASSED [ 4%] 2026-04-01T15:39:39.889 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_list_user_policy PASSED [ 4%] 2026-04-01T15:39:39.937 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_list_user_policy_invalid_user PASSED [ 4%] 2026-04-01T15:39:40.079 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_get_user_policy PASSED [ 5%] 2026-04-01T15:39:40.245 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_get_user_policy_invalid_user PASSED [ 5%] 2026-04-01T15:39:40.565 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_get_user_policy_from_multiple_policies PASSED [ 5%] 2026-04-01T15:39:40.663 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_delete_user_policy PASSED [ 5%] 2026-04-01T15:39:40.801 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_delete_user_policy_invalid_user PASSED [ 5%] 2026-04-01T15:39:40.975 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_delete_user_policy_invalid_policy_name PASSED [ 5%] 2026-04-01T15:39:41.298 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_delete_user_policy_from_multiple_policies PASSED [ 6%] 2026-04-01T15:39:41.446 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_allow_bucket_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:41.618 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_deny_bucket_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:41.796 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_allow_object_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:42.116 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_deny_object_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:43.615 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_allow_multipart_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:43.868 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_deny_multipart_actions_in_user_policy PASSED [ 6%] 2026-04-01T15:39:44.056 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_allow_tagging_actions_in_user_policy PASSED [ 7%] 2026-04-01T15:39:44.321 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_deny_tagging_actions_in_user_policy PASSED [ 7%] 2026-04-01T15:39:44.549 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_conflicting_user_policy_statements PASSED [ 7%] 2026-04-01T15:39:44.821 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_conflicting_user_policies PASSED [ 7%] 2026-04-01T15:39:44.972 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_allow_iam_actions PASSED [ 7%] 2026-04-01T15:39:45.681 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_create PASSED [ 7%] 2026-04-01T15:39:46.145 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_case_insensitive_name PASSED [ 7%] 2026-04-01T15:39:46.728 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_delete PASSED [ 8%] 2026-04-01T15:39:48.110 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_list PASSED [ 8%] 2026-04-01T15:39:50.051 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_list_path_prefix PASSED [ 8%] 2026-04-01T15:39:51.055 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_update_name PASSED [ 8%] 2026-04-01T15:39:51.920 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_update_path PASSED [ 8%] 2026-04-01T15:39:52.490 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_access_key_create PASSED [ 8%] 2026-04-01T15:39:52.834 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_current_user_access_key_create PASSED [ 8%] 2026-04-01T15:39:53.574 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_access_key_update PASSED [ 9%] 2026-04-01T15:39:54.055 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_current_user_access_key_update PASSED [ 9%] 2026-04-01T15:39:54.751 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_access_key_delete PASSED [ 9%] 2026-04-01T15:39:55.210 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_current_user_access_key_delete PASSED [ 9%] 2026-04-01T15:39:56.169 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_access_key_list PASSED [ 9%] 2026-04-01T15:39:56.742 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_bucket_policy_allow PASSED [ 9%] 2026-04-01T15:39:57.844 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_policy PASSED [ 9%] 2026-04-01T15:39:58.919 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_policy_managed PASSED [ 10%] 2026-04-01T15:39:59.589 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_user_policy_allow PASSED [ 10%] 2026-04-01T15:40:00.153 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_group_create PASSED [ 10%] 2026-04-01T15:40:00.643 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_group_case_insensitive_name PASSED [ 10%] 2026-04-01T15:40:02.009 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_group_list PASSED [ 10%] 2026-04-01T15:40:02.898 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_group_update PASSED [ 10%] 2026-04-01T15:40:04.022 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_inline_group_policy PASSED [ 10%] 2026-04-01T15:40:05.110 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_managed_group_policy PASSED [ 11%] 2026-04-01T15:40:06.020 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_inline_group_policy_allow PASSED [ 11%] 2026-04-01T15:40:06.932 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_managed_group_policy_allow PASSED [ 11%] 2026-04-01T15:40:07.595 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_create PASSED [ 11%] 2026-04-01T15:40:08.085 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_case_insensitive_name PASSED [ 11%] 2026-04-01T15:40:08.623 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_delete PASSED [ 11%] 2026-04-01T15:40:09.804 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_list PASSED [ 12%] 2026-04-01T15:40:11.484 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_list_path_prefix PASSED [ 12%] 2026-04-01T15:40:12.049 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_update PASSED [ 12%] 2026-04-01T15:40:13.044 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_policy PASSED [ 12%] 2026-04-01T15:40:14.057 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_policy_managed PASSED [ 12%] 2026-04-01T15:40:14.890 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_policy_allow PASSED [ 12%] 2026-04-01T15:40:15.695 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_same_account_role_policy_allow PASSED [ 12%] 2026-04-01T15:40:16.526 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_role_policy_allow PASSED [ 13%] 2026-04-01T15:40:17.381 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_policy_allow_create_bucket PASSED [ 13%] 2026-04-01T15:40:18.146 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_role_policy_allow_get_role PASSED [ 13%] 2026-04-01T15:40:18.668 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_account_oidc_provider PASSED [ 13%] 2026-04-01T15:40:19.140 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_add_new_client_id_to_oidc PASSED [ 13%] 2026-04-01T15:40:19.568 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_add_existing_client_id_to_oidc PASSED [ 13%] 2026-04-01T15:40:19.997 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_remove_client_id_from_oidc PASSED [ 13%] 2026-04-01T15:40:20.422 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_verify_update_thumbprintlist_of_oidc PASSED [ 14%] 2026-04-01T15:40:21.161 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_user_arn PASSED [ 14%] 2026-04-01T15:40:21.877 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_user_arn PASSED [ 14%] 2026-04-01T15:40:22.636 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_account_arn PASSED [ 14%] 2026-04-01T15:40:23.363 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_account_arn PASSED [ 14%] 2026-04-01T15:40:24.079 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_account_id PASSED [ 14%] 2026-04-01T15:40:24.805 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_account_id PASSED [ 14%] 2026-04-01T15:40:25.512 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_user_id PASSED [ 15%] 2026-04-01T15:40:26.207 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_user_id PASSED [ 15%] 2026-04-01T15:40:26.959 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_canonical_id PASSED [ 15%] 2026-04-01T15:40:27.651 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_canonical_id PASSED [ 15%] 2026-04-01T15:40:28.357 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_account_email PASSED [ 15%] 2026-04-01T15:40:29.152 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_account_email PASSED [ 15%] 2026-04-01T15:40:29.514 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_root_bucket_policy_allow_account_arn PASSED [ 15%] 2026-04-01T15:40:29.935 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_root_bucket_policy_allow_account_id PASSED [ 16%] 2026-04-01T15:40:30.269 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_root_bucket_acl_grant_canonical_id PASSED [ 16%] 2026-04-01T15:40:30.588 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py::test_cross_account_root_bucket_acl_grant_account_email PASSED [ 16%] 2026-04-01T15:40:30.818 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_empty PASSED [ 16%] 2026-04-01T15:40:31.112 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_distinct PASSED [ 16%] 2026-04-01T15:40:31.338 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_many PASSED [ 16%] 2026-04-01T15:40:31.603 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_many PASSED [ 17%] 2026-04-01T15:40:31.827 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_basic_key_count PASSED [ 17%] 2026-04-01T15:40:32.013 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_basic PASSED [ 17%] 2026-04-01T15:40:32.301 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_basic PASSED [ 17%] 2026-04-01T15:40:32.525 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_encoding_basic PASSED [ 17%] 2026-04-01T15:40:32.703 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_encoding_basic PASSED [ 17%] 2026-04-01T15:40:32.962 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_prefix PASSED [ 17%] 2026-04-01T15:40:33.182 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_prefix PASSED [ 18%] 2026-04-01T15:40:33.351 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_ends_with_delimiter PASSED [ 18%] 2026-04-01T15:40:33.557 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_prefix_ends_with_delimiter PASSED [ 18%] 2026-04-01T15:40:33.744 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_alt PASSED [ 18%] 2026-04-01T15:40:33.928 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_alt PASSED [ 18%] 2026-04-01T15:40:34.181 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_prefix_underscore PASSED [ 18%] 2026-04-01T15:40:34.444 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_underscore PASSED [ 18%] 2026-04-01T15:40:34.621 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_percentage PASSED [ 19%] 2026-04-01T15:40:34.803 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_percentage PASSED [ 19%] 2026-04-01T15:40:34.985 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_whitespace PASSED [ 19%] 2026-04-01T15:40:35.207 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_whitespace PASSED [ 19%] 2026-04-01T15:40:35.401 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_dot PASSED [ 19%] 2026-04-01T15:40:35.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_dot PASSED [ 19%] 2026-04-01T15:40:35.807 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_unreadable PASSED [ 19%] 2026-04-01T15:40:35.986 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_unreadable PASSED [ 20%] 2026-04-01T15:40:36.170 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_empty PASSED [ 20%] 2026-04-01T15:40:36.389 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_empty PASSED [ 20%] 2026-04-01T15:40:36.570 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_none PASSED [ 20%] 2026-04-01T15:40:36.751 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_none PASSED [ 20%] 2026-04-01T15:40:37.006 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_fetchowner_notempty PASSED [ 20%] 2026-04-01T15:40:37.189 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_fetchowner_defaultempty PASSED [ 20%] 2026-04-01T15:40:37.363 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_fetchowner_empty PASSED [ 21%] 2026-04-01T15:40:37.587 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_not_exist PASSED [ 21%] 2026-04-01T15:40:37.772 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_delimiter_not_exist PASSED [ 21%] 2026-04-01T15:40:43.392 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_delimiter_not_skip_special PASSED [ 21%] 2026-04-01T15:40:43.603 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_basic PASSED [ 21%] 2026-04-01T15:40:43.785 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_basic PASSED [ 21%] 2026-04-01T15:40:43.965 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_alt PASSED [ 21%] 2026-04-01T15:40:44.163 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_alt PASSED [ 22%] 2026-04-01T15:40:44.344 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_empty PASSED [ 22%] 2026-04-01T15:40:44.529 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_empty PASSED [ 22%] 2026-04-01T15:40:44.743 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_none PASSED [ 22%] 2026-04-01T15:40:44.934 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_none PASSED [ 22%] 2026-04-01T15:40:45.119 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_not_exist PASSED [ 22%] 2026-04-01T15:40:45.340 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_not_exist PASSED [ 23%] 2026-04-01T15:40:45.521 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_unreadable PASSED [ 23%] 2026-04-01T15:40:45.700 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_unreadable PASSED [ 23%] 2026-04-01T15:40:45.924 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_delimiter_basic PASSED [ 23%] 2026-04-01T15:40:46.109 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_basic PASSED [ 23%] 2026-04-01T15:40:46.292 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_delimiter_alt PASSED [ 23%] 2026-04-01T15:40:46.520 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_alt PASSED [ 23%] 2026-04-01T15:40:46.703 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_delimiter_prefix_not_exist PASSED [ 24%] 2026-04-01T15:40:46.898 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_not_exist PASSED [ 24%] 2026-04-01T15:40:47.129 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_delimiter_delimiter_not_exist PASSED [ 24%] 2026-04-01T15:40:48.320 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_delimiter_not_exist PASSED [ 24%] 2026-04-01T15:40:48.518 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist PASSED [ 24%] 2026-04-01T15:40:48.726 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist PASSED [ 24%] 2026-04-01T15:40:50.103 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_maxkeys_one PASSED [ 24%] 2026-04-01T15:40:50.328 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_maxkeys_one PASSED [ 25%] 2026-04-01T15:40:50.516 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_maxkeys_zero PASSED [ 25%] 2026-04-01T15:40:50.725 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_maxkeys_zero PASSED [ 25%] 2026-04-01T15:40:50.912 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_maxkeys_none PASSED [ 25%] 2026-04-01T15:40:51.098 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_maxkeys_none PASSED [ 25%] 2026-04-01T15:40:51.180 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_account_usage PASSED [ 25%] 2026-04-01T15:40:51.350 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_head_bucket_usage PASSED [ 25%] 2026-04-01T15:40:51.806 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_unordered PASSED [ 26%] 2026-04-01T15:40:52.287 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_unordered PASSED [ 26%] 2026-04-01T15:40:52.465 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_maxkeys_invalid PASSED [ 26%] 2026-04-01T15:40:52.674 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_marker_none PASSED [ 26%] 2026-04-01T15:40:52.860 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_marker_empty PASSED [ 26%] 2026-04-01T15:40:53.040 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_continuationtoken_empty PASSED [ 26%] 2026-04-01T15:40:53.297 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_continuationtoken PASSED [ 26%] 2026-04-01T15:40:53.525 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_both_continuationtoken_startafter PASSED [ 27%] 2026-04-01T15:40:53.704 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_marker_unreadable PASSED [ 27%] 2026-04-01T15:40:53.936 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_startafter_unreadable PASSED [ 27%] 2026-04-01T15:40:54.123 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_marker_not_in_list PASSED [ 27%] 2026-04-01T15:40:54.313 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_startafter_not_in_list PASSED [ 27%] 2026-04-01T15:40:54.500 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_marker_after_list PASSED [ 27%] 2026-04-01T15:40:54.679 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_startafter_after_list PASSED [ 28%] 2026-04-01T15:40:55.038 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_return_data PASSED [ 28%] 2026-04-01T15:40:55.505 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_return_data_versioning PASSED [ 28%] 2026-04-01T15:40:55.630 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_objects_anonymous PASSED [ 28%] 2026-04-01T15:40:55.766 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_objects_anonymous PASSED [ 28%] 2026-04-01T15:40:55.888 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_objects_anonymous_fail PASSED [ 28%] 2026-04-01T15:40:56.050 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_listv2_objects_anonymous_fail PASSED [ 28%] 2026-04-01T15:40:56.097 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_notexist PASSED [ 29%] 2026-04-01T15:40:56.142 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucketv2_notexist PASSED [ 29%] 2026-04-01T15:40:56.189 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_delete_notexist PASSED [ 29%] 2026-04-01T15:40:56.401 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_delete_nonempty PASSED [ 29%] 2026-04-01T15:40:56.613 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_concurrent_set_canned_acl PASSED [ 29%] 2026-04-01T15:40:56.653 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_to_nonexist_bucket PASSED [ 29%] 2026-04-01T15:40:56.852 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_with_chunked_transfer_encoding PASSED [ 29%] 2026-04-01T15:40:56.976 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_delete PASSED [ 30%] 2026-04-01T15:40:57.099 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_read_not_exist PASSED [ 30%] 2026-04-01T15:40:57.223 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_requestid_matches_header_on_error PASSED [ 30%] 2026-04-01T15:40:57.663 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_concurrent_multi_object_delete PASSED [ 30%] 2026-04-01T15:40:57.969 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multi_object_delete PASSED [ 30%] 2026-04-01T15:40:58.267 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_expected_bucket_owner PASSED [ 30%] 2026-04-01T15:40:58.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multi_objectv2_delete PASSED [ 30%] 2026-04-01T15:41:05.186 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multi_object_delete_key_limit PASSED [ 31%] 2026-04-01T15:41:11.234 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multi_objectv2_delete_key_limit PASSED [ 31%] 2026-04-01T15:41:11.403 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_head_zero_bytes PASSED [ 31%] 2026-04-01T15:41:11.569 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_check_etag PASSED [ 31%] 2026-04-01T15:41:11.777 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_cache_control PASSED [ 31%] 2026-04-01T15:41:11.951 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_expires PASSED [ 31%] 2026-04-01T15:41:12.180 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_read_update_read_delete PASSED [ 31%] 2026-04-01T15:41:12.359 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_set_get_metadata_none_to_good PASSED [ 32%] 2026-04-01T15:41:12.581 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_set_get_metadata_none_to_empty PASSED [ 32%] 2026-04-01T15:41:12.762 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_set_get_metadata_overwrite_to_empty PASSED [ 32%] 2026-04-01T15:41:12.929 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_metadata_replaced_on_put PASSED [ 32%] 2026-04-01T15:41:13.184 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_write_file PASSED [ 32%] 2026-04-01T15:41:13.391 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_anonymous_request PASSED [ 32%] 2026-04-01T15:41:13.561 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_authenticated_request PASSED [ 32%] 2026-04-01T15:41:13.769 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_authenticated_no_content_type PASSED [ 33%] 2026-04-01T15:41:13.923 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_authenticated_request_bad_access_key PASSED [ 33%] 2026-04-01T15:41:14.138 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_set_success_code PASSED [ 33%] 2026-04-01T15:41:14.308 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_set_invalid_success_code PASSED [ 33%] 2026-04-01T15:41:14.494 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_upload_larger_than_chunk PASSED [ 33%] 2026-04-01T15:41:14.695 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_set_key_from_filename PASSED [ 33%] 2026-04-01T15:41:14.861 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_ignored_header PASSED [ 34%] 2026-04-01T15:41:15.033 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_case_insensitive_condition_fields PASSED [ 34%] 2026-04-01T15:41:15.224 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_escaped_field_values PASSED [ 34%] 2026-04-01T15:41:15.435 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_success_redirect_action PASSED [ 34%] 2026-04-01T15:41:15.550 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_invalid_signature PASSED [ 34%] 2026-04-01T15:41:15.666 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_invalid_access_key PASSED [ 34%] 2026-04-01T15:41:15.803 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_invalid_date_format PASSED [ 34%] 2026-04-01T15:41:15.927 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_no_key_specified PASSED [ 35%] 2026-04-01T15:41:16.055 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_missing_signature PASSED [ 35%] 2026-04-01T15:41:16.217 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_missing_policy_condition PASSED [ 35%] 2026-04-01T15:41:16.383 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_user_specified_header PASSED [ 35%] 2026-04-01T15:41:16.498 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_request_missing_policy_specified_field PASSED [ 35%] 2026-04-01T15:41:16.616 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_condition_is_case_sensitive PASSED [ 35%] 2026-04-01T15:41:16.740 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_expires_is_case_sensitive PASSED [ 35%] 2026-04-01T15:41:16.875 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_expired_policy PASSED [ 36%] 2026-04-01T15:41:17.100 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_wrong_bucket PASSED [ 36%] 2026-04-01T15:41:17.223 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_invalid_request_field_value PASSED [ 36%] 2026-04-01T15:41:17.343 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_missing_expires_condition PASSED [ 36%] 2026-04-01T15:41:17.505 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_missing_conditions_list PASSED [ 36%] 2026-04-01T15:41:17.632 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_upload_size_limit_exceeded PASSED [ 36%] 2026-04-01T15:41:17.752 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_missing_content_length_argument PASSED [ 36%] 2026-04-01T15:41:17.873 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_invalid_content_length_argument PASSED [ 37%] 2026-04-01T15:41:18.035 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_upload_size_below_minimum PASSED [ 37%] 2026-04-01T15:41:18.243 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_upload_size_rgw_chunk_size_bug PASSED [ 37%] 2026-04-01T15:41:18.370 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_empty_conditions PASSED [ 37%] 2026-04-01T15:41:18.694 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifmatch_good PASSED [ 37%] 2026-04-01T15:41:19.088 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifmatch_failed PASSED [ 37%] 2026-04-01T15:41:19.302 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifnonematch_good PASSED [ 37%] 2026-04-01T15:41:19.544 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifnonematch_failed PASSED [ 38%] 2026-04-01T15:41:19.760 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifmodifiedsince_good PASSED [ 38%] 2026-04-01T15:41:20.953 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifmodifiedsince_failed PASSED [ 38%] 2026-04-01T15:41:21.173 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifunmodifiedsince_good PASSED [ 38%] 2026-04-01T15:41:21.472 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_ifunmodifiedsince_failed PASSED [ 38%] 2026-04-01T15:41:21.745 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifmatch_good PASSED [ 38%] 2026-04-01T15:41:22.056 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifmatch_failed PASSED [ 39%] 2026-04-01T15:41:22.365 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifmatch_overwrite_existed_good PASSED [ 39%] 2026-04-01T15:41:22.584 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifmatch_nonexisted_failed PASSED [ 39%] 2026-04-01T15:41:22.846 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifnonmatch_good PASSED [ 39%] 2026-04-01T15:41:23.157 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifnonmatch_failed PASSED [ 39%] 2026-04-01T15:41:23.411 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifnonmatch_nonexisted_good PASSED [ 39%] 2026-04-01T15:41:23.716 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_object_ifnonmatch_overwrite_existed_failed PASSED [ 39%] 2026-04-01T15:41:23.897 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get PASSED [ 40%] 2026-04-01T15:41:24.044 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_bucket_gone PASSED [ 40%] 2026-04-01T15:41:24.148 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_delete_key_bucket_gone PASSED [ 40%] 2026-04-01T15:41:24.297 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_object_gone PASSED [ 40%] 2026-04-01T15:41:24.470 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_head PASSED [ 40%] 2026-04-01T15:41:24.516 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_head_notexist PASSED [ 40%] 2026-04-01T15:41:24.723 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_head_extended PASSED [ 40%] 2026-04-01T15:41:24.924 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_bucket_acl PASSED [ 41%] 2026-04-01T15:41:25.151 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_object_acl PASSED [ 41%] 2026-04-01T15:41:27.563 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_put_acl_mtime PASSED [ 41%] 2026-04-01T15:41:27.749 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_authenticated PASSED [ 41%] 2026-04-01T15:41:27.971 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_response_headers PASSED [ 41%] 2026-04-01T15:41:28.145 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_authenticated_bucket_acl PASSED [ 41%] 2026-04-01T15:41:28.329 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_authenticated_object_acl PASSED [ 41%] 2026-04-01T15:41:28.460 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_authenticated_bucket_gone PASSED [ 42%] 2026-04-01T15:41:28.686 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_authenticated_object_gone PASSED [ 42%] 2026-04-01T15:41:28.857 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired PASSED [ 42%] 2026-04-01T15:41:29.034 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired_tenant PASSED [ 42%] 2026-04-01T15:41:29.266 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_range_zero PASSED [ 42%] 2026-04-01T15:41:29.445 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_max_range PASSED [ 42%] 2026-04-01T15:41:29.619 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_positive_range PASSED [ 42%] 2026-04-01T15:41:29.801 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_anon_put PASSED [ 43%] 2026-04-01T15:41:30.037 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_anon_put_write_access PASSED [ 43%] 2026-04-01T15:41:30.211 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_put_authenticated PASSED [ 43%] 2026-04-01T15:41:30.385 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_presigned_put_object_with_acl PASSED [ 43%] 2026-04-01T15:41:30.605 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_presigned_put_object_with_acl_tenant PASSED [ 43%] 2026-04-01T15:41:30.781 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_raw_put_authenticated_expired PASSED [ 43%] 2026-04-01T15:41:30.827 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_bad_starts_nonalpha PASSED [ 43%] 2026-04-01T15:41:30.871 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_bad_short_one PASSED [ 44%] 2026-04-01T15:41:30.966 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_bad_short_two PASSED [ 44%] 2026-04-01T15:41:31.087 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_long_60 PASSED [ 44%] 2026-04-01T15:41:31.215 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_long_61 PASSED [ 44%] 2026-04-01T15:41:31.337 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_long_62 PASSED [ 44%] 2026-04-01T15:41:31.508 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_long_63 PASSED [ 44%] 2026-04-01T15:41:31.677 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_long_name PASSED [ 45%] 2026-04-01T15:41:31.728 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_bad_ip PASSED [ 45%] 2026-04-01T15:41:31.778 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_underscore PASSED [ 45%] 2026-04-01T15:41:31.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_long PASSED [ 45%] 2026-04-01T15:41:31.993 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_dash_at_end PASSED [ 45%] 2026-04-01T15:41:32.041 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_dot_dot PASSED [ 45%] 2026-04-01T15:41:32.092 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_dot_dash PASSED [ 45%] 2026-04-01T15:41:32.184 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_dns_dash_dot PASSED [ 46%] 2026-04-01T15:41:32.317 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_exists PASSED [ 46%] 2026-04-01T15:41:32.485 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_get_location PASSED [ 46%] 2026-04-01T15:41:32.609 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_exists_nonowner PASSED [ 46%] 2026-04-01T15:41:32.818 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_recreate_overwrite_acl PASSED [ 46%] 2026-04-01T15:41:32.990 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_recreate_new_acl PASSED [ 46%] 2026-04-01T15:41:33.118 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_default PASSED [ 46%] 2026-04-01T15:41:33.338 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_canned_during_create PASSED [ 47%] 2026-04-01T15:41:33.596 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_canned PASSED [ 47%] 2026-04-01T15:41:33.769 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_canned_publicreadwrite PASSED [ 47%] 2026-04-01T15:41:33.941 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_canned_authenticatedread PASSED [ 47%] 2026-04-01T15:41:34.162 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_bucket_acl_grant_group_read PASSED [ 47%] 2026-04-01T15:41:34.387 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_default PASSED [ 47%] 2026-04-01T15:41:34.614 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned_during_create PASSED [ 47%] 2026-04-01T15:41:34.889 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned PASSED [ 48%] 2026-04-01T15:41:35.148 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned_publicreadwrite PASSED [ 48%] 2026-04-01T15:41:35.369 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned_authenticatedread PASSED [ 48%] 2026-04-01T15:41:35.627 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned_bucketownerread PASSED [ 48%] 2026-04-01T15:41:35.930 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_canned_bucketownerfullcontrol PASSED [ 48%] 2026-04-01T15:41:36.160 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_full_control_verify_owner PASSED [ 48%] 2026-04-01T15:41:36.353 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_full_control_verify_attributes PASSED [ 48%] 2026-04-01T15:41:36.526 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_canned_private_to_private PASSED [ 49%] 2026-04-01T15:41:36.789 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl PASSED [ 49%] 2026-04-01T15:41:37.084 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_write PASSED [ 49%] 2026-04-01T15:41:37.348 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_writeacp PASSED [ 49%] 2026-04-01T15:41:37.661 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_read PASSED [ 49%] 2026-04-01T15:41:37.918 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_acl_readacp PASSED [ 49%] 2026-04-01T15:41:38.166 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_userid_fullcontrol PASSED [ 50%] 2026-04-01T15:41:38.386 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_userid_read PASSED [ 50%] 2026-04-01T15:41:38.582 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_userid_readacp PASSED [ 50%] 2026-04-01T15:41:38.860 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_userid_write PASSED [ 50%] 2026-04-01T15:41:39.062 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_userid_writeacp PASSED [ 50%] 2026-04-01T15:41:39.229 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_nonexist_user PASSED [ 50%] 2026-04-01T15:41:39.444 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_header_acl_grants PASSED [ 50%] 2026-04-01T15:41:39.659 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_header_acl_grants PASSED [ 51%] 2026-04-01T15:41:39.834 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_email PASSED [ 51%] 2026-04-01T15:41:40.010 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_grant_email_not_exist PASSED [ 51%] 2026-04-01T15:41:40.278 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_acl_revoke_all PASSED [ 51%] 2026-04-01T15:41:40.632 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_object_private PASSED [ 51%] 2026-04-01T15:41:41.032 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_objectv2_private PASSED [ 51%] 2026-04-01T15:41:41.374 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_object_publicread PASSED [ 51%] 2026-04-01T15:41:41.741 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_objectv2_publicread PASSED [ 52%] 2026-04-01T15:41:42.071 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_object_publicreadwrite PASSED [ 52%] 2026-04-01T15:41:42.415 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_private_objectv2_publicreadwrite PASSED [ 52%] 2026-04-01T15:41:42.796 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicread_object_private PASSED [ 52%] 2026-04-01T15:41:43.134 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicread_object_publicread PASSED [ 52%] 2026-04-01T15:41:43.499 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicread_object_publicreadwrite PASSED [ 52%] 2026-04-01T15:41:43.791 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicreadwrite_object_private PASSED [ 52%] 2026-04-01T15:41:44.092 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicread PASSED [ 53%] 2026-04-01T15:41:44.426 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicreadwrite PASSED [ 53%] 2026-04-01T15:41:44.917 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_buckets_create_then_list PASSED [ 53%] 2026-04-01T15:41:45.412 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_buckets_list_ctime PASSED [ 53%] 2026-04-01T15:41:45.457 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_list_buckets_anonymous PASSED [ 53%] 2026-04-01T15:41:45.547 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_list_buckets_invalid_auth PASSED [ 53%] 2026-04-01T15:41:45.589 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_list_buckets_bad_auth PASSED [ 53%] 2026-04-01T15:41:45.717 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_starts_alpha PASSED [ 54%] 2026-04-01T15:41:45.900 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_starts_digit PASSED [ 54%] 2026-04-01T15:41:46.028 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_contains_period PASSED [ 54%] 2026-04-01T15:41:46.154 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_naming_good_contains_hyphen PASSED [ 54%] 2026-04-01T15:41:46.366 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_recreate_not_overriding PASSED [ 54%] 2026-04-01T15:41:47.168 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_create_special_key_names PASSED [ 54%] 2026-04-01T15:41:47.369 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_list_special_prefix PASSED [ 54%] 2026-04-01T15:41:47.643 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_zero_size PASSED [ 55%] 2026-04-01T15:41:47.940 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_16m PASSED [ 55%] 2026-04-01T15:41:48.202 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_same_bucket PASSED [ 55%] 2026-04-01T15:41:48.458 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_verify_contenttype PASSED [ 55%] 2026-04-01T15:41:48.730 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_to_itself PASSED [ 55%] 2026-04-01T15:41:48.960 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_to_itself_with_metadata PASSED [ 55%] 2026-04-01T15:41:49.353 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_diff_bucket PASSED [ 56%] 2026-04-01T15:41:49.636 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_not_owned_bucket PASSED [ 56%] 2026-04-01T15:41:49.885 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_not_owned_object_bucket PASSED [ 56%] 2026-04-01T15:41:50.170 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_canned_acl PASSED [ 56%] 2026-04-01T15:41:50.808 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_retaining_metadata PASSED [ 56%] 2026-04-01T15:41:51.221 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_replacing_metadata PASSED [ 56%] 2026-04-01T15:41:51.358 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_bucket_not_found PASSED [ 56%] 2026-04-01T15:41:51.538 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_key_not_found PASSED [ 57%] 2026-04-01T15:41:52.466 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_versioned_bucket PASSED [ 57%] 2026-04-01T15:41:52.752 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_versioned_url_encoding PASSED [ 57%] 2026-04-01T15:41:53.911 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_copy_versioning_multipart_upload PASSED [ 57%] 2026-04-01T15:41:54.061 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_empty PASSED [ 57%] 2026-04-01T15:41:54.325 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_small PASSED [ 57%] 2026-04-01T15:41:54.823 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_small PASSED [ 57%] 2026-04-01T15:41:55.073 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_invalid_range PASSED [ 58%] 2026-04-01T15:41:55.632 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_without_range PASSED [ 58%] 2026-04-01T15:41:56.671 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_special_names PASSED [ 58%] 2026-04-01T15:41:57.476 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload PASSED [ 58%] 2026-04-01T15:41:58.882 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_versioned PASSED [ 58%] 2026-04-01T15:42:04.114 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_resend_part PASSED [ 58%] 2026-04-01T15:42:04.832 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_multiple_sizes PASSED [ 58%] 2026-04-01T15:42:06.358 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_copy_multiple_sizes PASSED [ 59%] 2026-04-01T15:42:06.608 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_size_too_small PASSED [ 59%] 2026-04-01T15:42:07.052 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_contents PASSED [ 59%] 2026-04-01T15:42:07.539 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_overwrite_existing_object PASSED [ 59%] 2026-04-01T15:42:07.844 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_abort_multipart_upload PASSED [ 59%] 2026-04-01T15:42:08.068 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_abort_multipart_upload_not_found PASSED [ 59%] 2026-04-01T15:42:08.398 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_list_multipart_upload PASSED [ 59%] 2026-04-01T15:42:08.693 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_list_multipart_upload_owner PASSED [ 60%] 2026-04-01T15:42:08.890 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_missing_part PASSED [ 60%] 2026-04-01T15:42:09.085 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_incorrect_etag PASSED [ 60%] 2026-04-01T15:42:15.224 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_get_part PASSED [ 60%] 2026-04-01T15:42:21.893 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_sse_c_get_part PASSED [ 60%] 2026-04-01T15:42:22.506 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_single_get_part PASSED [ 60%] 2026-04-01T15:42:22.838 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_non_multipart_get_part PASSED [ 60%] 2026-04-01T15:42:23.107 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_non_multipart_sse_c_get_part PASSED [ 61%] 2026-04-01T15:42:23.241 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_100_continue PASSED [ 61%] 2026-04-01T15:42:23.468 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_set_cors PASSED [ 61%] 2026-04-01T15:42:26.711 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_origin_response PASSED [ 61%] 2026-04-01T15:42:29.851 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_origin_wildcard PASSED [ 61%] 2026-04-01T15:42:33.052 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_header_option PASSED [ 61%] 2026-04-01T15:42:33.242 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_get_object PASSED [ 62%] 2026-04-01T15:42:33.458 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_get_object_tenant PASSED [ 62%] 2026-04-01T15:42:33.650 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_put_object PASSED [ 62%] 2026-04-01T15:42:33.838 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_put_object_with_acl PASSED [ 62%] 2026-04-01T15:42:34.073 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_put_object_tenant PASSED [ 62%] 2026-04-01T15:42:34.261 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_cors_presigned_put_object_tenant_with_acl PASSED [ 62%] 2026-04-01T15:42:34.496 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_set_bucket_tagging PASSED [ 62%] 2026-04-01T15:42:34.729 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_read_1mb PASSED [ 63%] 2026-04-01T15:42:35.137 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_read_4mb PASSED [ 63%] 2026-04-01T15:42:35.473 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_read_8mb PASSED [ 63%] 2026-04-01T15:42:35.806 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_write_1mb PASSED [ 63%] 2026-04-01T15:42:36.132 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_write_4mb PASSED [ 63%] 2026-04-01T15:42:36.578 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_write_8mb PASSED [ 63%] 2026-04-01T15:42:36.832 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_dual_write_1mb PASSED [ 63%] 2026-04-01T15:42:37.182 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_dual_write_4mb PASSED [ 64%] 2026-04-01T15:42:37.667 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_dual_write_8mb PASSED [ 64%] 2026-04-01T15:42:37.979 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_conditional_write_1mb PASSED [ 64%] 2026-04-01T15:42:38.462 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_atomic_multipart_upload_write PASSED [ 64%] 2026-04-01T15:42:38.906 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_resend_first_finishes_last PASSED [ 64%] 2026-04-01T15:42:39.138 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_request_response_code PASSED [ 64%] 2026-04-01T15:42:41.493 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_big_request_response_code PASSED [ 64%] 2026-04-01T15:42:41.755 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_request_skip_leading_bytes_response_code PASSED [ 65%] 2026-04-01T15:42:41.983 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_request_return_trailing_bytes_response_code PASSED [ 65%] 2026-04-01T15:42:42.218 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_request_invalid_range PASSED [ 65%] 2026-04-01T15:42:42.447 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ranged_request_empty_object PASSED [ 65%] 2026-04-01T15:42:42.842 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_bucket_create_suspend PASSED [ 65%] 2026-04-01T15:42:48.915 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_create_read_remove PASSED [ 65%] 2026-04-01T15:42:49.265 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_create_read_remove_head PASSED [ 65%] 2026-04-01T15:42:49.620 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_stack_delete_merkers PASSED [ 66%] 2026-04-01T15:42:49.926 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_plain_null_version_removal PASSED [ 66%] 2026-04-01T15:42:50.325 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite PASSED [ 66%] 2026-04-01T15:42:50.819 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite_suspended PASSED [ 66%] 2026-04-01T15:42:53.287 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_suspend_versions PASSED [ 66%] 2026-04-01T15:42:53.665 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_suspended_copy PASSED [ 66%] 2026-04-01T15:42:56.882 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_create_versions_remove_all PASSED [ 67%] 2026-04-01T15:43:09.044 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_create_versions_remove_special_names PASSED [ 67%] 2026-04-01T15:43:10.501 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_create_overwrite_multipart PASSED [ 67%] 2026-04-01T15:43:11.307 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_obj_list_marker PASSED [ 67%] 2026-04-01T15:43:12.336 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_copy_obj_version PASSED [ 67%] 2026-04-01T15:43:12.760 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_multi_object_delete PASSED [ 67%] 2026-04-01T15:43:13.138 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_multi_object_delete_with_marker PASSED [ 67%] 2026-04-01T15:43:13.419 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_multi_object_delete_with_marker_create PASSED [ 68%] 2026-04-01T15:43:13.852 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioned_object_acl PASSED [ 68%] 2026-04-01T15:43:14.267 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioned_object_acl_no_version_specified PASSED [ 68%] 2026-04-01T15:43:15.449 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioned_concurrent_object_create_concurrent_remove PASSED [ 68%] 2026-04-01T15:43:15.956 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioned_concurrent_object_create_and_remove PASSED [ 68%] 2026-04-01T15:43:16.106 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set PASSED [ 68%] 2026-04-01T15:43:16.304 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_get PASSED [ 68%] 2026-04-01T15:43:16.583 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_delete PASSED [ 69%] 2026-04-01T15:43:16.813 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_get_no_id PASSED [ 69%] 2026-04-01T15:43:16.967 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_id_too_long PASSED [ 69%] 2026-04-01T15:43:17.152 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_same_id PASSED [ 69%] 2026-04-01T15:43:17.438 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_invalid_status PASSED [ 69%] 2026-04-01T15:43:17.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_date PASSED [ 69%] 2026-04-01T15:43:17.721 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_invalid_date PASSED [ 69%] 2026-04-01T15:43:17.956 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_noncurrent PASSED [ 70%] 2026-04-01T15:43:18.150 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_deletemarker PASSED [ 70%] 2026-04-01T15:43:18.327 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_filter PASSED [ 70%] 2026-04-01T15:43:18.479 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_empty_filter PASSED [ 70%] 2026-04-01T15:43:18.692 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_set_multipart PASSED [ 70%] 2026-04-01T15:43:18.853 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_lifecycle_transition_set_invalid_date PASSED [ 70%] 2026-04-01T15:43:18.899 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_restore_object_temporary SKIPPED [ 70%] 2026-04-01T15:43:18.948 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_restore_object_permanent SKIPPED [ 71%] 2026-04-01T15:43:18.990 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_read_through SKIPPED ([s3 cloud]...) [ 71%] 2026-04-01T15:43:19.074 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_restore_noncur_obj SKIPPED ([s3 ...) [ 71%] 2026-04-01T15:43:19.298 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encrypted_transfer_1b PASSED [ 71%] 2026-04-01T15:43:19.540 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encrypted_transfer_1kb PASSED [ 71%] 2026-04-01T15:43:19.799 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encrypted_transfer_1MB PASSED [ 71%] 2026-04-01T15:43:20.049 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encrypted_transfer_13b PASSED [ 71%] 2026-04-01T15:43:20.231 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_method_head PASSED [ 72%] 2026-04-01T15:43:20.457 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_present PASSED [ 72%] 2026-04-01T15:43:20.739 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_other_key PASSED [ 72%] 2026-04-01T15:43:20.878 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_invalid_md5 PASSED [ 72%] 2026-04-01T15:43:21.053 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_no_md5 PASSED [ 72%] 2026-04-01T15:43:21.233 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_no_key PASSED [ 72%] 2026-04-01T15:43:21.408 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_key_no_sse_c PASSED [ 73%] 2026-04-01T15:43:22.918 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_multipart_upload PASSED [ 73%] 2026-04-01T15:43:24.283 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_unaligned_multipart_upload PASSED [ 73%] 2026-04-01T15:43:24.978 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_multipart_bad_download PASSED [ 73%] 2026-04-01T15:43:25.154 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_post_object_authenticated_request PASSED [ 73%] 2026-04-01T15:44:25.718 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_enforced_with_bucket_policy PASSED [ 73%] 2026-04-01T15:45:26.121 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_deny_algo_with_bucket_policy PASSED [ 73%] 2026-04-01T15:45:26.308 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_method_head PASSED [ 74%] 2026-04-01T15:45:26.534 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_present PASSED [ 74%] 2026-04-01T15:45:26.658 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_no_key PASSED [ 74%] 2026-04-01T15:45:26.838 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_not_declared PASSED [ 74%] 2026-04-01T15:45:27.980 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_multipart_upload PASSED [ 74%] 2026-04-01T15:45:28.490 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_1 PASSED [ 74%] 2026-04-01T15:45:29.005 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_2 PASSED [ 74%] 2026-04-01T15:45:29.183 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_post_object_authenticated_request PASSED [ 75%] 2026-04-01T15:45:29.418 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_transfer_1b PASSED [ 75%] 2026-04-01T15:45:29.691 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_transfer_1kb PASSED [ 75%] 2026-04-01T15:45:29.881 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_transfer_1MB PASSED [ 75%] 2026-04-01T15:45:30.101 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_transfer_13b PASSED [ 75%] 2026-04-01T15:45:30.338 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_read_declare PASSED [ 75%] 2026-04-01T15:45:30.568 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy PASSED [ 75%] 2026-04-01T15:45:30.759 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucketv2_policy PASSED [ 76%] 2026-04-01T15:45:31.376 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_deny_self_denied_policy PASSED [ 76%] 2026-04-01T15:45:32.115 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_deny_self_denied_policy_confirm_header PASSED [ 76%] 2026-04-01T15:45:32.319 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_acl PASSED [ 76%] 2026-04-01T15:45:32.518 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucketv2_policy_acl PASSED [ 76%] 2026-04-01T15:45:32.817 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_different_tenant PASSED [ 76%] 2026-04-01T15:45:33.066 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_multipart PASSED [ 76%] 2026-04-01T15:45:33.258 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_tenanted_bucket PASSED [ 77%] 2026-04-01T15:45:33.678 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_another_bucket PASSED [ 77%] 2026-04-01T15:45:34.085 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucketv2_policy_another_bucket PASSED [ 77%] 2026-04-01T15:45:34.366 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_set_get_del_bucket_policy PASSED [ 77%] 2026-04-01T15:45:34.713 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_set_multipart_tagging PASSED [ 77%] 2026-04-01T15:45:34.992 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_obj_tagging PASSED [ 77%] 2026-04-01T15:45:35.256 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_obj_head_tagging PASSED [ 78%] 2026-04-01T15:45:35.575 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_max_tags PASSED [ 78%] 2026-04-01T15:45:35.868 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_excess_tags PASSED [ 78%] 2026-04-01T15:45:36.136 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_max_kvsize_tags PASSED [ 78%] 2026-04-01T15:45:36.445 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_excess_key_tags PASSED [ 78%] 2026-04-01T15:45:36.926 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_excess_val_tags PASSED [ 78%] 2026-04-01T15:45:37.252 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_modify_tags PASSED [ 78%] 2026-04-01T15:45:37.633 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_delete_tags PASSED [ 79%] 2026-04-01T15:45:37.894 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_tags_anonymous_request PASSED [ 79%] 2026-04-01T15:45:38.076 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_tags_authenticated_request PASSED [ 79%] 2026-04-01T15:45:38.346 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_obj_with_tags PASSED [ 79%] 2026-04-01T15:45:38.613 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_tags_acl_public PASSED [ 79%] 2026-04-01T15:45:38.910 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_tags_acl_public PASSED [ 79%] 2026-04-01T15:45:39.199 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_delete_tags_obj_public PASSED [ 79%] 2026-04-01T15:45:39.875 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_bucket_atomic_upload_return_version_id PASSED [ 80%] 2026-04-01T15:45:41.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_versioning_bucket_multipart_upload_return_version_id PASSED [ 80%] 2026-04-01T15:45:41.875 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_get_obj_existing_tag PASSED [ 80%] 2026-04-01T15:45:42.274 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_get_obj_tagging_existing_tag PASSED [ 80%] 2026-04-01T15:45:42.584 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_tagging_existing_tag PASSED [ 80%] 2026-04-01T15:45:43.487 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_upload_part_copy PASSED [ 80%] 2026-04-01T15:45:44.004 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_copy_source PASSED [ 80%] 2026-04-01T15:45:44.430 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_copy_source_meta PASSED [ 81%] 2026-04-01T15:45:44.715 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_acl PASSED [ 81%] 2026-04-01T15:45:45.132 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_grant PASSED [ 81%] 2026-04-01T15:45:45.267 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_obj_enc_conflict_c_s3 PASSED [ 81%] 2026-04-01T15:45:45.450 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_obj_enc_conflict_c_kms PASSED [ 81%] 2026-04-01T15:45:45.585 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_obj_enc_conflict_s3_kms PASSED [ 81%] 2026-04-01T15:45:45.724 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_obj_enc_conflict_bad_enc_kms PASSED [ 81%] 2026-04-01T15:45:45.946 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_kms_noenc PASSED [ 82%] 2026-04-01T15:45:46.178 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_kms_s3 PASSED [ 82%] 2026-04-01T15:45:46.625 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_get_obj_acl_existing_tag PASSED [ 82%] 2026-04-01T15:45:46.809 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock PASSED [ 82%] 2026-04-01T15:45:47.039 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_bucket PASSED [ 82%] 2026-04-01T15:45:47.412 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_enable_after_create PASSED [ 82%] 2026-04-01T15:45:47.601 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_with_days_and_years PASSED [ 82%] 2026-04-01T15:45:47.867 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_days PASSED [ 83%] 2026-04-01T15:45:48.047 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_years PASSED [ 83%] 2026-04-01T15:45:48.275 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_mode PASSED [ 83%] 2026-04-01T15:45:48.481 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_status PASSED [ 83%] 2026-04-01T15:45:48.738 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_suspend_versioning PASSED [ 83%] 2026-04-01T15:45:48.931 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_lock PASSED [ 83%] 2026-04-01T15:45:49.112 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_lock_invalid_bucket PASSED [ 84%] 2026-04-01T15:45:49.324 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention PASSED [ 84%] 2026-04-01T15:45:49.576 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_bucket PASSED [ 84%] 2026-04-01T15:45:49.846 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_mode PASSED [ 84%] 2026-04-01T15:45:50.055 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_retention PASSED [ 84%] 2026-04-01T15:45:50.251 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_retention_iso8601 PASSED [ 84%] 2026-04-01T15:45:50.512 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_retention_invalid_bucket PASSED [ 84%] 2026-04-01T15:45:50.758 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_versionid PASSED [ 85%] 2026-04-01T15:45:50.967 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_override_default_retention PASSED [ 85%] 2026-04-01T15:45:51.162 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_increase_period PASSED [ 85%] 2026-04-01T15:45:51.382 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period PASSED [ 85%] 2026-04-01T15:45:51.574 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period_bypass PASSED [ 85%] 2026-04-01T15:45:51.758 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_object_with_retention PASSED [ 85%] 2026-04-01T15:45:52.049 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_multipart_object_with_retention PASSED [ 85%] 2026-04-01T15:45:52.358 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_object_with_retention_and_marker PASSED [ 86%] 2026-04-01T15:45:52.613 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_multi_delete_object_with_retention PASSED [ 86%] 2026-04-01T15:45:52.818 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_legal_hold PASSED [ 86%] 2026-04-01T15:45:53.057 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_bucket PASSED [ 86%] 2026-04-01T15:45:53.336 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_status PASSED [ 86%] 2026-04-01T15:45:53.637 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_legal_hold PASSED [ 86%] 2026-04-01T15:45:53.883 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_legal_hold_invalid_bucket PASSED [ 86%] 2026-04-01T15:45:54.201 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_on PASSED [ 87%] 2026-04-01T15:45:54.538 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_multipart_object_with_legal_hold_on PASSED [ 87%] 2026-04-01T15:45:54.696 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_off PASSED [ 87%] 2026-04-01T15:45:54.868 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_get_obj_metadata PASSED [ 87%] 2026-04-01T15:45:55.102 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_uploading_obj PASSED [ 87%] 2026-04-01T15:46:05.243 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_changing_mode_from_governance_with_bypass PASSED [ 87%] 2026-04-01T15:46:05.473 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_changing_mode_from_governance_without_bypass PASSED [ 87%] 2026-04-01T15:46:15.101 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_lock_changing_mode_from_compliance PASSED [ 88%] 2026-04-01T15:46:15.435 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_copy_object_ifmatch_good PASSED [ 88%] 2026-04-01T15:46:15.726 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_copy_object_ifnonematch_failed PASSED [ 88%] 2026-04-01T15:46:15.876 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_bucket_policy_status PASSED [ 88%] 2026-04-01T15:46:16.128 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_public_acl_bucket_policy_status PASSED [ 88%] 2026-04-01T15:46:16.329 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_authpublic_acl_bucket_policy_status PASSED [ 88%] 2026-04-01T15:46:16.524 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_publicpolicy_acl_bucket_policy_status PASSED [ 89%] 2026-04-01T15:46:16.708 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_nonpublicpolicy_acl_bucket_policy_status PASSED [ 89%] 2026-04-01T15:46:16.975 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_nonpublicpolicy_principal_bucket_policy_status PASSED [ 89%] 2026-04-01T15:46:17.145 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_bucket_policy_allow_notprincipal PASSED [ 89%] 2026-04-01T15:46:17.346 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_undefined_public_block PASSED [ 89%] 2026-04-01T15:46:17.594 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_public_block_deny_bucket_policy PASSED [ 89%] 2026-04-01T15:46:17.839 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_public_block PASSED [ 89%] 2026-04-01T15:46:18.275 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_block_public_put_bucket_acls PASSED [ 90%] 2026-04-01T15:46:18.580 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_block_public_object_canned_acls PASSED [ 90%] 2026-04-01T15:46:18.843 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_block_public_policy PASSED [ 90%] 2026-04-01T15:46:19.006 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_block_public_policy_with_principal PASSED [ 90%] 2026-04-01T15:46:19.272 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_block_public_restrict_public_buckets PASSED [ 90%] 2026-04-01T15:46:19.679 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_ignore_public_acls PASSED [ 90%] 2026-04-01T15:46:19.931 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_get_delete_public_block PASSED [ 90%] 2026-04-01T15:46:20.696 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_upload_on_a_bucket_with_policy PASSED [ 91%] 2026-04-01T15:46:20.860 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_put_bucket_encryption_kms PASSED [ 91%] 2026-04-01T15:46:21.144 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_bucket_encryption_kms PASSED [ 91%] 2026-04-01T15:46:21.347 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_delete_bucket_encryption_kms PASSED [ 91%] 2026-04-01T15:46:21.577 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_sse_kms_default_post_object_authenticated_request PASSED [ 91%] 2026-04-01T15:46:21.970 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_torrent PASSED [ 91%] 2026-04-01T15:46:30.163 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_checksum_sha256 PASSED [ 91%] 2026-04-01T15:46:35.348 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_object_checksum_crc64nvme PASSED [ 92%] 2026-04-01T15:46:50.299 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_checksum_sha256 PASSED [ 92%] 2026-04-01T15:46:50.740 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_use_cksum_helper_sha256 PASSED [ 92%] 2026-04-01T15:46:51.125 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_use_cksum_helper_crc64nvme PASSED [ 92%] 2026-04-01T15:46:51.479 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_use_cksum_helper_crc32 PASSED [ 92%] 2026-04-01T15:46:51.846 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_use_cksum_helper_crc32c PASSED [ 92%] 2026-04-01T15:46:52.180 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_multipart_use_cksum_helper_sha1 PASSED [ 92%] 2026-04-01T15:46:52.370 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_post_object_upload_checksum PASSED [ 93%] 2026-04-01T15:46:53.083 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_multipart_checksum_object_attributes PASSED [ 93%] 2026-04-01T15:46:53.530 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_multipart_object_attributes PASSED [ 93%] 2026-04-01T15:46:54.019 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_paginated_multipart_object_attributes PASSED [ 93%] 2026-04-01T15:46:54.291 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_single_multipart_object_attributes PASSED [ 93%] 2026-04-01T15:46:54.546 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_checksum_object_attributes PASSED [ 93%] 2026-04-01T15:46:54.869 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_versioned_object_attributes PASSED [ 93%] 2026-04-01T15:46:55.128 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_sse_c_encrypted_object_attributes PASSED [ 94%] 2026-04-01T15:46:55.382 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py::test_get_object_attributes PASSED [ 94%] 2026-04-01T15:46:56.167 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_generate_where_clause PASSED [ 94%] 2026-04-01T15:46:56.918 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_generate_projection PASSED [ 94%] 2026-04-01T15:46:57.114 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_count_operation PASSED [ 94%] 2026-04-01T15:46:57.331 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_count_json_operation PASSED [ 94%] 2026-04-01T15:46:57.885 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_json_column_sum_min_max PASSED [ 95%] 2026-04-01T15:46:58.390 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_json_nullif_expressions PASSED [ 95%] 2026-04-01T15:46:58.753 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_column_sum_min_max PASSED [ 95%] 2026-04-01T15:46:59.025 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_nullif_expressions PASSED [ 95%] 2026-04-01T15:46:59.366 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_nulliftrue_expressions PASSED [ 95%] 2026-04-01T15:46:59.640 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_is_not_null_expressions PASSED [ 95%] 2026-04-01T15:46:59.864 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_lowerupper_expressions PASSED [ 95%] 2026-04-01T15:47:00.230 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_in_expressions PASSED [ 96%] 2026-04-01T15:47:00.694 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_true_false_in_expressions PASSED [ 96%] 2026-04-01T15:47:01.585 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_like_expressions PASSED [ 96%] 2026-04-01T15:47:01.864 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_truefalselike_expressions PASSED [ 96%] 2026-04-01T15:47:02.166 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_complex_expressions PASSED [ 96%] 2026-04-01T15:47:02.416 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_alias PASSED [ 96%] 2026-04-01T15:47:02.646 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_alias_cyclic_refernce PASSED [ 96%] 2026-04-01T15:47:02.932 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_datetime PASSED [ 97%] 2026-04-01T15:47:03.196 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_true_false_datetime PASSED [ 97%] 2026-04-01T15:47:03.450 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_csv_parser PASSED [ 97%] 2026-04-01T15:47:03.710 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_csv_definition PASSED [ 97%] 2026-04-01T15:47:04.005 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_schema_definition PASSED [ 97%] 2026-04-01T15:47:04.328 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_when_then_else_expressions PASSED [ 97%] 2026-04-01T15:47:04.601 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_coalesce_expressions PASSED [ 97%] 2026-04-01T15:47:04.923 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_cast_expressions PASSED [ 98%] 2026-04-01T15:47:04.961 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_version PASSED [ 98%] 2026-04-01T15:47:05.285 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_trim_expressions PASSED [ 98%] 2026-04-01T15:47:05.737 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_truefalse_trim_expressions PASSED [ 98%] 2026-04-01T15:47:06.011 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_escape_expressions PASSED [ 98%] 2026-04-01T15:47:06.308 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_case_value_expressions PASSED [ 98%] 2026-04-01T15:47:06.546 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_bool_cast_expressions PASSED [ 98%] 2026-04-01T15:47:14.143 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_progress_expressions PASSED [ 99%] 2026-04-01T15:47:14.183 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py::test_output_serial_expressions PASSED [ 99%] 2026-04-01T15:47:14.708 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_sns.py::test_account_topic PASSED [ 99%] 2026-04-01T15:47:15.086 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_sns.py::test_cross_account_topic PASSED [ 99%] 2026-04-01T15:47:15.404 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_sns.py::test_account_topic_publish PASSED [ 99%] 2026-04-01T15:47:15.821 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_sns.py::test_cross_account_topic_publish PASSED [ 99%] 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_utils.py::test_generate PASSED [100%] 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:=============================== warnings summary =============================== 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_headers.py: 28 warnings 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_iam.py: 84 warnings 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3.py: 531 warnings 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_s3select.py: 34 warnings 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:s3tests/functional/test_sns.py: 4 warnings 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/lib/python3.9/site-packages/boto3/compat.py:89: PythonDeprecationWarning: Boto3 will no longer support Python 3.9 starting April 29, 2026. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.10 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout: warnings.warn(warning, PythonDeprecationWarning) 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout: 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2026-04-01T15:47:15.945 INFO:teuthology.orchestra.run.vm07.stdout:=== 678 passed, 4 skipped, 211 deselected, 681 warnings in 464.30s (0:07:44) === 2026-04-01T15:47:16.032 INFO:teuthology.orchestra.run.vm07.stdout: py: OK (472.58=setup[8.00]+cmd[464.58] seconds) 2026-04-01T15:47:16.032 INFO:teuthology.orchestra.run.vm07.stdout: congratulations :) (472.59 seconds) 2026-04-01T15:47:16.052 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2026-04-01T15:47:16.052 DEBUG:teuthology.orchestra.run.vm07:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2026-04-01T15:47:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-04-01T15:47:16.114 INFO:tasks.s3tests:Cleaning up boto... 2026-04-01T15:47:16.114 DEBUG:teuthology.orchestra.run.vm07:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2026-04-01T15:47:16.131 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2026-04-01T15:47:16.210 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:16.210 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 realm 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:16.223 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.222+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:16.224 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.223+0000 7f82d9b4b900 20 searching for the correct realm 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:16.229 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.228+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 zone default found 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 4 Realm: () 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 zonegroup default 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.230 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.229+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.230+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.230+0000 7f82d9b4b900 20 started sync module instance, tier type = 2026-04-01T15:47:16.231 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.230+0000 7f82d9b4b900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:16.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.233+0000 7f82d9b4b900 20 add_watcher() i=3 2026-04-01T15:47:16.234 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.233+0000 7f82d9b4b900 20 add_watcher() i=2 2026-04-01T15:47:16.235 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.233+0000 7f82d9b4b900 20 add_watcher() i=1 2026-04-01T15:47:16.235 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.234+0000 7f82d9b4b900 20 add_watcher() i=6 2026-04-01T15:47:16.235 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.234+0000 7f82d9b4b900 20 add_watcher() i=5 2026-04-01T15:47:16.235 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.234+0000 7f82d9b4b900 20 add_watcher() i=0 2026-04-01T15:47:16.236 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.235+0000 7f82d9b4b900 20 add_watcher() i=4 2026-04-01T15:47:16.236 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.235+0000 7f82d9b4b900 20 add_watcher() i=7 2026-04-01T15:47:16.236 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.235+0000 7f82d9b4b900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:16.237 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.236+0000 7f82d9b4b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:16.237 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.236+0000 7f82d9b4b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:16.257 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.256+0000 7f8282fed640 20 reqs_thread_entry: start 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.282+0000 7f82d9b4b900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 20 Filter name: none 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f8280fe9640 20 reqs_thread_entry: start 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=474 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-04-01T15:47:16.284 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.283+0000 7f82d9b4b900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 : miss 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=421 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 info.flags=0x17 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 to cache LRU end 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 updating xattr: name=user.rgw.acl bl.length()=183 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 2026-04-01T15:47:16.285 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.284+0000 7f82d9b4b900 10 cls_bucket_list_unordered: got DF781C 2026-04-01T15:47:16.287 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 10 cls_bucket_list_unordered: got 947919 2026-04-01T15:47:16.287 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 20 get_obj_state: octx=0x5611c4b78380 obj=testbuck2:DF781C state=0x5611c4c299e8 s->prefetch_data=0 2026-04-01T15:47:16.287 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 10 manifest: total_size = 1028989 2026-04-01T15:47:16.287 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 20 get_obj_state: setting s->obj_tag to 26234e52-e0ad-4466-b829-a8bdf0048973.4266.7107242169410188286 2026-04-01T15:47:16.287 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 20 get_obj_state: octx=0x5611c4b78380 obj=testbuck2:DF781C state=0x5611c4c299e8 s->prefetch_data=0 2026-04-01T15:47:16.288 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 20 prepare_atomic_modification: state is not atomic. state=0x5611c4c299e8 2026-04-01T15:47:16.288 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.286+0000 7f82d9b4b900 20 bucket index object: :.dir.26234e52-e0ad-4466-b829-a8bdf0048973.4266.607.0 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 20 get_obj_state: octx=0x5611c4b78380 obj=testbuck2:947919 state=0x5611c4c299e8 s->prefetch_data=0 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 10 manifest: total_size = 306314 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 20 get_obj_state: setting s->obj_tag to 26234e52-e0ad-4466-b829-a8bdf0048973.4266.16159362772627835431 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 20 get_obj_state: octx=0x5611c4b78380 obj=testbuck2:947919 state=0x5611c4c299e8 s->prefetch_data=0 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 20 prepare_atomic_modification: state is not atomic. state=0x5611c4c299e8 2026-04-01T15:47:16.290 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82d9b4b900 20 bucket index object: :.dir.26234e52-e0ad-4466-b829-a8bdf0048973.4266.607.9 2026-04-01T15:47:16.291 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.289+0000 7f82b4ff9640 20 handle_completion(): completion ok for obj=DF781C 2026-04-01T15:47:16.293 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.292+0000 7f82d9b4b900 20 list_objects_ordered: starting attempt 1 2026-04-01T15:47:16.293 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.292+0000 7f82d9b4b900 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2026-04-01T15:47:16.293 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.292+0000 7f82b57fa640 20 handle_completion(): completion ok for obj=947919 2026-04-01T15:47:16.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.293+0000 7f82d9b4b900 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2026-04-01T15:47:16.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.293+0000 7f82d9b4b900 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2026-04-01T15:47:16.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.293+0000 7f82d9b4b900 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2026-04-01T15:47:16.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.293+0000 7f82d9b4b900 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2026-04-01T15:47:16.294 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.293+0000 7f82d9b4b900 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[26234e52-e0ad-4466-b829-a8bdf0048973.4266.607])"; uploads.size()=0; is_truncated=0 2026-04-01T15:47:16.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.295+0000 7f82d9b4b900 10 cache get: name=default.rgw.meta+root+testbuck2 : miss 2026-04-01T15:47:16.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.295+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.295+0000 7f82d9b4b900 20 rados_obj.operate() r=0 bl.length=191 2026-04-01T15:47:16.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.295+0000 7f82d9b4b900 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x11 2026-04-01T15:47:16.296 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.295+0000 7f82d9b4b900 10 adding default.rgw.meta+root+testbuck2 to cache LRU end 2026-04-01T15:47:16.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.296+0000 7f82d9b4b900 10 removing default.rgw.meta+root+testbuck2 from cache 2026-04-01T15:47:16.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.296+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2026-04-01T15:47:16.297 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.296+0000 7f82b57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758850 cookie 94634315548000 notifier 4373 bl.length()=153 2026-04-01T15:47:16.299 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.298+0000 7f82d9b4b900 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 from cache 2026-04-01T15:47:16.299 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.298+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:26234e52-e0ad-4466-b829-a8bdf0048973.4266.607, ofs0, ns] 2026-04-01T15:47:16.300 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.299+0000 7f82b4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757495 cookie 94634316366848 notifier 4373 bl.length()=212 2026-04-01T15:47:16.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.303+0000 7f82d9b4b900 10 cache get: name=default.rgw.log++pubsub..bucket.testbuck2/26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 : miss 2026-04-01T15:47:16.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.303+0000 7f82d9b4b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.303+0000 7f82d9b4b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.303+0000 7f82d9b4b900 10 cache put: name=default.rgw.log++pubsub..bucket.testbuck2/26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 info.flags=0x0 2026-04-01T15:47:16.304 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.303+0000 7f82d9b4b900 10 adding default.rgw.log++pubsub..bucket.testbuck2/26234e52-e0ad-4466-b829-a8bdf0048973.4266.607 to cache LRU end 2026-04-01T15:47:16.306 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.305+0000 7f82d9b4b900 10 removing key index: ZWFWRHXCMUDPESQFULHC 2026-04-01T15:47:16.307 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.306+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:ZWFWRHXCMUDPESQFULHC, ofs0, ns] 2026-04-01T15:47:16.307 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.306+0000 7f82b57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758851 cookie 94634316355040 notifier 4373 bl.length()=170 2026-04-01T15:47:16.307 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.306+0000 7f82d9b4b900 10 removing email index: foo+test@test.test 2026-04-01T15:47:16.308 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.307+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.email:foo+test@test.test, ofs0, ns] 2026-04-01T15:47:16.308 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.307+0000 7f82b4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758852 cookie 94634315548000 notifier 4373 bl.length()=169 2026-04-01T15:47:16.308 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.307+0000 7f82d9b4b900 10 removing user buckets index 2026-04-01T15:47:16.310 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.309+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2026-04-01T15:47:16.310 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.309+0000 7f82b57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758853 cookie 94634316355040 notifier 4373 bl.length()=169 2026-04-01T15:47:16.310 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.309+0000 7f82d9b4b900 10 removing user index: foo.client.0 2026-04-01T15:47:16.311 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.310+0000 7f82d9b4b900 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2026-04-01T15:47:16.311 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.310+0000 7f82d9b4b900 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2026-04-01T15:47:16.311 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.310+0000 7f82b4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758798 cookie 94634316364480 notifier 4373 bl.length()=161 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=3 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 2 removed watcher, disabling cache 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=5 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=1 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=0 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=6 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=7 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=2 2026-04-01T15:47:16.315 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.314+0000 7f82d9b4b900 20 remove_watcher() i=4 2026-04-01T15:47:16.320 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2026-04-01T15:47:16.394 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:16.394 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:16.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.406 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 realm 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.405+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:16.407 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.406+0000 7f7365b3a900 20 searching for the correct realm 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 zone default found 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 4 Realm: () 2026-04-01T15:47:16.412 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 zonegroup default 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.411+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.412+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.412+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.412+0000 7f7365b3a900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.412+0000 7f7365b3a900 20 started sync module instance, tier type = 2026-04-01T15:47:16.413 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.412+0000 7f7365b3a900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.414+0000 7f7365b3a900 20 add_watcher() i=1 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.414+0000 7f7365b3a900 20 add_watcher() i=4 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.414+0000 7f7365b3a900 20 add_watcher() i=6 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 20 add_watcher() i=5 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 20 add_watcher() i=0 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 20 add_watcher() i=2 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 20 add_watcher() i=3 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 20 add_watcher() i=7 2026-04-01T15:47:16.416 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.415+0000 7f7365b3a900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:16.418 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.417+0000 7f7365b3a900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:16.418 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.417+0000 7f7365b3a900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:16.437 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.436+0000 7f730efed640 20 reqs_thread_entry: start 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.462+0000 7f7365b3a900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.462+0000 7f7365b3a900 20 Filter name: none 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.462+0000 7f730cfe9640 20 reqs_thread_entry: start 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.462+0000 7f7365b3a900 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.462+0000 7f7365b3a900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 20 rados_obj.operate() r=0 bl.length=474 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 10 updating xattr: name=user.rgw.user-policy bl.length()=4 2026-04-01T15:47:16.464 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.463+0000 7f7365b3a900 10 removing key index: VTHCUXXAKWGNFWMWKYSP 2026-04-01T15:47:16.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.464+0000 7f7365b3a900 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:VTHCUXXAKWGNFWMWKYSP, ofs0, ns] 2026-04-01T15:47:16.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.464+0000 7f7340ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758854 cookie 94027571982384 notifier 4380 bl.length()=170 2026-04-01T15:47:16.465 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.464+0000 7f7365b3a900 10 removing email index: bar+test@test.test 2026-04-01T15:47:16.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.465+0000 7f7365b3a900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.email:bar+test@test.test, ofs0, ns] 2026-04-01T15:47:16.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.465+0000 7f73417fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758799 cookie 94027571974640 notifier 4380 bl.length()=169 2026-04-01T15:47:16.466 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.465+0000 7f7365b3a900 10 removing user buckets index 2026-04-01T15:47:16.467 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.466+0000 7f7365b3a900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2026-04-01T15:47:16.467 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.466+0000 7f73417fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757496 cookie 94027571978112 notifier 4380 bl.length()=169 2026-04-01T15:47:16.467 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.466+0000 7f7365b3a900 10 removing user index: bar.client.0 2026-04-01T15:47:16.468 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.467+0000 7f7365b3a900 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2026-04-01T15:47:16.468 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.467+0000 7f7365b3a900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2026-04-01T15:47:16.468 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.467+0000 7f73417fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758855 cookie 94027571955632 notifier 4380 bl.length()=161 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.470+0000 7f7365b3a900 20 remove_watcher() i=3 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.470+0000 7f7365b3a900 2 removed watcher, disabling cache 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.470+0000 7f7365b3a900 20 remove_watcher() i=2 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=0 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=1 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=4 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=5 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=6 2026-04-01T15:47:16.472 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.471+0000 7f7365b3a900 20 remove_watcher() i=7 2026-04-01T15:47:16.477 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2026-04-01T15:47:16.551 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:16.551 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:16.564 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.563+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.563+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 realm 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:16.565 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.564+0000 7ff2abc0f900 20 searching for the correct realm 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.570 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.569+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 zone default found 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 4 Realm: () 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 zonegroup default 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 started sync module instance, tier type = 2026-04-01T15:47:16.571 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.570+0000 7ff2abc0f900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=0 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=5 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=7 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=1 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=4 2026-04-01T15:47:16.575 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.574+0000 7ff2abc0f900 20 add_watcher() i=6 2026-04-01T15:47:16.576 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.575+0000 7ff2abc0f900 20 add_watcher() i=3 2026-04-01T15:47:16.576 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.575+0000 7ff2abc0f900 20 add_watcher() i=2 2026-04-01T15:47:16.576 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.575+0000 7ff2abc0f900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:16.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.576+0000 7ff2abc0f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:16.577 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.576+0000 7ff2abc0f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:16.597 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.596+0000 7ff254ff1640 20 reqs_thread_entry: start 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 20 Filter name: none 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff24dffb640 20 reqs_thread_entry: start 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 20 rados_obj.operate() r=0 bl.length=533 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2026-04-01T15:47:16.622 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.623 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.621+0000 7ff2abc0f900 10 removing key index: MZGZEHDDEGJMHVGQWJSM 2026-04-01T15:47:16.623 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.622+0000 7ff2abc0f900 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:MZGZEHDDEGJMHVGQWJSM, ofs0, ns] 2026-04-01T15:47:16.624 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.622+0000 7ff286ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757497 cookie 94505995109056 notifier 4387 bl.length()=170 2026-04-01T15:47:16.624 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.623+0000 7ff2abc0f900 10 removing email index: testx$tenanteduser+test@test.test 2026-04-01T15:47:16.625 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.623+0000 7ff2abc0f900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser+test@test.test, ofs0, ns] 2026-04-01T15:47:16.625 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.624+0000 7ff2867fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758800 cookie 94505995086848 notifier 4387 bl.length()=184 2026-04-01T15:47:16.625 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.624+0000 7ff2abc0f900 10 removing user buckets index 2026-04-01T15:47:16.626 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.625+0000 7ff2abc0f900 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2026-04-01T15:47:16.626 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.625+0000 7ff286ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758856 cookie 94505995105472 notifier 4387 bl.length()=184 2026-04-01T15:47:16.626 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.625+0000 7ff2abc0f900 10 removing user index: testx$tenanteduser.client.0 2026-04-01T15:47:16.627 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.626+0000 7ff2abc0f900 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2026-04-01T15:47:16.627 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.626+0000 7ff2abc0f900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2026-04-01T15:47:16.627 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.626+0000 7ff2867fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758801 cookie 94505995089824 notifier 4387 bl.length()=176 2026-04-01T15:47:16.630 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.629+0000 7ff2abc0f900 20 remove_watcher() i=5 2026-04-01T15:47:16.630 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.629+0000 7ff2abc0f900 2 removed watcher, disabling cache 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=0 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=7 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=2 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=3 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=1 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=6 2026-04-01T15:47:16.631 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.630+0000 7ff2abc0f900 20 remove_watcher() i=4 2026-04-01T15:47:16.636 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foobar.client.0 --purge-data --cluster ceph 2026-04-01T15:47:16.711 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:16.711 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.722+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 realm 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:16.724 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.723+0000 7fcf1b956900 20 searching for the correct realm 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:16.729 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.728+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 zone default found 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 4 Realm: () 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 zonegroup default 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 started sync module instance, tier type = 2026-04-01T15:47:16.730 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.729+0000 7fcf1b956900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=0 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=3 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=2 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=5 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=6 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=7 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=1 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 20 add_watcher() i=4 2026-04-01T15:47:16.734 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.733+0000 7fcf1b956900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:16.735 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.734+0000 7fcf1b956900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:16.735 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.734+0000 7fcf1b956900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:16.756 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.754+0000 7fcebffff640 20 reqs_thread_entry: start 2026-04-01T15:47:16.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:16.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 20 Filter name: none 2026-04-01T15:47:16.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcebdffb640 20 reqs_thread_entry: start 2026-04-01T15:47:16.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2026-04-01T15:47:16.782 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 20 rados_obj.operate() r=0 bl.length=539 2026-04-01T15:47:16.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2026-04-01T15:47:16.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.781+0000 7fcf1b956900 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2026-04-01T15:47:16.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.782+0000 7fcf1b956900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.783 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.782+0000 7fcf1b956900 10 removing key index: RZFKORFICEAKVFPVQFDI 2026-04-01T15:47:16.784 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.782+0000 7fcf1b956900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:RZFKORFICEAKVFPVQFDI, ofs0, ns] 2026-04-01T15:47:16.784 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.783+0000 7fcef6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757498 cookie 94481989123232 notifier 4393 bl.length()=170 2026-04-01T15:47:16.784 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.783+0000 7fcf1b956900 10 removing email index: foobar+test@test.test 2026-04-01T15:47:16.785 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.784+0000 7fcf1b956900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:foobar+test@test.test, ofs0, ns] 2026-04-01T15:47:16.785 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.784+0000 7fcef6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758857 cookie 94481989135792 notifier 4393 bl.length()=172 2026-04-01T15:47:16.785 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.784+0000 7fcf1b956900 10 removing user buckets index 2026-04-01T15:47:16.786 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.785+0000 7fcf1b956900 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foobar.client.0.buckets, ofs0, ns] 2026-04-01T15:47:16.786 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.785+0000 7fcef67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758802 cookie 94481989116320 notifier 4393 bl.length()=172 2026-04-01T15:47:16.786 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.785+0000 7fcf1b956900 10 removing user index: foobar.client.0 2026-04-01T15:47:16.787 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.786+0000 7fcf1b956900 10 removing default.rgw.meta+users.uid+foobar.client.0 from cache 2026-04-01T15:47:16.787 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.786+0000 7fcf1b956900 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2026-04-01T15:47:16.787 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.786+0000 7fcef6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758803 cookie 94481989110288 notifier 4393 bl.length()=164 2026-04-01T15:47:16.790 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=5 2026-04-01T15:47:16.790 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 2 removed watcher, disabling cache 2026-04-01T15:47:16.790 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=1 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=2 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=3 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=0 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.789+0000 7fcf1b956900 20 remove_watcher() i=7 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.790+0000 7fcf1b956900 20 remove_watcher() i=4 2026-04-01T15:47:16.791 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.790+0000 7fcf1b956900 20 remove_watcher() i=6 2026-04-01T15:47:16.796 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid root1.client.0 --purge-data --cluster ceph 2026-04-01T15:47:16.870 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:16.871 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:16.883 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.882+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.882+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.882+0000 7f023972b900 20 realm 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.882+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:16.884 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.883+0000 7f023972b900 20 searching for the correct realm 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.888+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 zone default found 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 4 Realm: () 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:16.890 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 zonegroup default 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.889+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.890+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.890+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.890+0000 7f023972b900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.890+0000 7f023972b900 20 started sync module instance, tier type = 2026-04-01T15:47:16.891 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.890+0000 7f023972b900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=3 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=2 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=4 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=0 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=7 2026-04-01T15:47:16.894 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.893+0000 7f023972b900 20 add_watcher() i=5 2026-04-01T15:47:16.895 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.894+0000 7f023972b900 20 add_watcher() i=1 2026-04-01T15:47:16.895 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.894+0000 7f023972b900 20 add_watcher() i=6 2026-04-01T15:47:16.895 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.894+0000 7f023972b900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:16.896 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.895+0000 7f023972b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:16.896 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.895+0000 7f023972b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:16.920 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.919+0000 7f01e27ec640 20 reqs_thread_entry: start 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.943+0000 7f023972b900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.943+0000 7f023972b900 20 Filter name: none 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f01dbfff640 20 reqs_thread_entry: start 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 20 rados->read ofs=0 len=0 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 20 rados_obj.operate() r=0 bl.length=508 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x13 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:16.945 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.944+0000 7f023972b900 10 removing key index: POMSSXZFMOTDNYCCGIME 2026-04-01T15:47:16.946 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.945+0000 7f023972b900 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:POMSSXZFMOTDNYCCGIME, ofs0, ns] 2026-04-01T15:47:16.946 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.945+0000 7f020ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757499 cookie 94389103235472 notifier 4400 bl.length()=170 2026-04-01T15:47:16.947 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.946+0000 7f023972b900 10 removing email index: root1+test@test.test 2026-04-01T15:47:16.947 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.946+0000 7f023972b900 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.email:root1+test@test.test, ofs0, ns] 2026-04-01T15:47:16.948 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.946+0000 7f020ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758804 cookie 94389103225376 notifier 4400 bl.length()=171 2026-04-01T15:47:16.948 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.947+0000 7f023972b900 10 removing user index: root1.client.0 2026-04-01T15:47:16.949 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.947+0000 7f023972b900 10 removing default.rgw.meta+users.uid+root1.client.0 from cache 2026-04-01T15:47:16.949 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.948+0000 7f023972b900 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:root1.client.0, ofs0, ns] 2026-04-01T15:47:16.949 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.948+0000 7f020ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758858 cookie 94389103205024 notifier 4400 bl.length()=163 2026-04-01T15:47:16.952 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.951+0000 7f023972b900 20 remove_watcher() i=1 2026-04-01T15:47:16.952 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.951+0000 7f023972b900 2 removed watcher, disabling cache 2026-04-01T15:47:16.952 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.951+0000 7f023972b900 20 remove_watcher() i=5 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.951+0000 7f023972b900 20 remove_watcher() i=3 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.952+0000 7f023972b900 20 remove_watcher() i=2 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.952+0000 7f023972b900 20 remove_watcher() i=6 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.952+0000 7f023972b900 20 remove_watcher() i=7 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.952+0000 7f023972b900 20 remove_watcher() i=0 2026-04-01T15:47:16.953 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:16.952+0000 7f023972b900 20 remove_watcher() i=4 2026-04-01T15:47:16.958 DEBUG:teuthology.orchestra.run.vm07:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph account rm --account-id RGW88888888888888888 --purge-data 2026-04-01T15:47:17.032 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setuser ceph since I am not root 2026-04-01T15:47:17.032 INFO:teuthology.orchestra.run.vm07.stderr:ignoring --setgroup ceph since I am not root 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 realm 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2026-04-01T15:47:17.045 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.044+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=1214 2026-04-01T15:47:17.046 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.045+0000 7f9c60fa3900 20 searching for the correct realm 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got default.zonegroup. 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got default.zone. 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got zonegroup_info.3cf91ed5-4b81-44e4-9181-00c11e15fb67 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got zone_info.26234e52-e0ad-4466-b829-a8bdf0048973 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got zone_names.default 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 RGWRados::pool_iterate: got zonegroups_names.default 2026-04-01T15:47:17.051 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.050+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=46 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=470 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 zone default found 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 4 Realm: () 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 4 ZoneGroup: default (3cf91ed5-4b81-44e4-9181-00c11e15fb67) 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 4 Zone: default (26234e52-e0ad-4466-b829-a8bdf0048973) 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 10 cannot find current period zonegroup using local zonegroup configuration 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 zonegroup default 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.052 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.051+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.053 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.052+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.053 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.052+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.053 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.052+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.053 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.052+0000 7f9c60fa3900 20 started sync module instance, tier type = 2026-04-01T15:47:17.053 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.052+0000 7f9c60fa3900 20 started zone id=26234e52-e0ad-4466-b829-a8bdf0048973 (name=default) with tier type = 2026-04-01T15:47:17.056 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.055+0000 7f9c60fa3900 20 add_watcher() i=4 2026-04-01T15:47:17.056 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.055+0000 7f9c60fa3900 20 add_watcher() i=1 2026-04-01T15:47:17.056 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.055+0000 7f9c60fa3900 20 add_watcher() i=6 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 20 add_watcher() i=3 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 20 add_watcher() i=2 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 20 add_watcher() i=0 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 20 add_watcher() i=5 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 20 add_watcher() i=7 2026-04-01T15:47:17.057 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.056+0000 7f9c60fa3900 2 all 8 watchers are set, enabling cache 2026-04-01T15:47:17.059 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.058+0000 7f9c60fa3900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-04-01T15:47:17.059 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.058+0000 7f9c60fa3900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-04-01T15:47:17.059 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.058+0000 7f9c60fa3900 5 note: GC not initialized 2026-04-01T15:47:17.059 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.058+0000 7f9c09feb640 20 reqs_thread_entry: start 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.087+0000 7f9c60fa3900 20 init_complete bucket index max shards: 11 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.087+0000 7f9c60fa3900 20 Filter name: none 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c037fe640 20 reqs_thread_entry: start 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 cache get: name=default.rgw.meta+accounts+account.RGW88888888888888888 : miss 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=184 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 adding default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 20 rados_obj.operate() r=-2 bl.length=0 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x0 2026-04-01T15:47:17.089 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.088+0000 7f9c60fa3900 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-s7o8y48xbt7ky2657lboagpe-479:26234e52-e0ad-4466-b829-a8bdf0048973.4266.467 : miss 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 20 rados->read ofs=0 len=0 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 20 rados_obj.operate() r=0 bl.length=427 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-s7o8y48xbt7ky2657lboagpe-479:26234e52-e0ad-4466-b829-a8bdf0048973.4266.467 info.flags=0x17 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 adding default.rgw.meta+root+.bucket.meta.test-s7o8y48xbt7ky2657lboagpe-479:26234e52-e0ad-4466-b829-a8bdf0048973.4266.467 to cache LRU end 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 updating xattr: name=user.rgw.acl bl.length()=231 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 updating xattr: name=user.rgw.iam-policy bl.length()=275 2026-04-01T15:47:17.090 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.089+0000 7f9c60fa3900 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-s7o8y48xbt7ky2657lboagpe-479:26234e52-e0ad-4466-b829-a8bdf0048973.4266.467 2026-04-01T15:47:17.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.091+0000 7f9c60fa3900 20 list_objects_ordered: starting attempt 1 2026-04-01T15:47:17.092 INFO:teuthology.orchestra.run.vm07.stderr:2026-04-01T15:47:17.091+0000 7f9c60fa3900 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2026-04-01T17:36:20.166 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2026-04-01T17:36:20.166 DEBUG:teuthology.exit:Finished running handlers